Training: 2021-06-06 19:02:28,376-rank_id: 0 Training: 2021-06-06 19:03:32,436-softmax weight init successfully! Training: 2021-06-06 19:03:32,436-softmax weight mom init successfully! Training: 2021-06-06 19:03:32,466-Total Step is: 249100 Training: 2021-06-06 19:04:44,527-Reducer buckets have been rebuilt in this iteration. Training: 2021-06-06 19:05:22,166-Speed 126.48 samples/sec Loss 46.6889 Epoch: 0 Global Step: 10 Fp16 Grad Scale: 64 Required: 321 hours Training: 2021-06-06 19:05:42,109-Speed 128.37 samples/sec Loss 46.6425 Epoch: 0 Global Step: 15 Fp16 Grad Scale: 64 Required: 307 hours Training: 2021-06-06 19:06:02,402-Speed 126.16 samples/sec Loss 46.5812 Epoch: 0 Global Step: 20 Fp16 Grad Scale: 64 Required: 300 hours Training: 2021-06-06 19:06:22,389-Speed 128.09 samples/sec Loss 46.7483 Epoch: 0 Global Step: 25 Fp16 Grad Scale: 64 Required: 296 hours Training: 2021-06-06 19:06:42,769-Speed 125.62 samples/sec Loss 46.7590 Epoch: 0 Global Step: 30 Fp16 Grad Scale: 64 Required: 294 hours Training: 2021-06-06 19:07:03,414-Speed 124.01 samples/sec Loss 46.7193 Epoch: 0 Global Step: 35 Fp16 Grad Scale: 64 Required: 293 hours Training: 2021-06-06 19:07:23,800-Speed 125.58 samples/sec Loss 46.7828 Epoch: 0 Global Step: 40 Fp16 Grad Scale: 64 Required: 291 hours Training: 2021-06-06 19:07:43,882-Speed 127.48 samples/sec Loss 46.8093 Epoch: 0 Global Step: 45 Fp16 Grad Scale: 64 Required: 290 hours Training: 2021-06-06 19:08:04,410-Speed 124.72 samples/sec Loss 46.9862 Epoch: 0 Global Step: 50 Fp16 Grad Scale: 64 Required: 289 hours Training: 2021-06-06 19:08:24,820-Speed 125.43 samples/sec Loss 47.0167 Epoch: 0 Global Step: 55 Fp16 Grad Scale: 64 Required: 289 hours Training: 2021-06-06 19:08:44,949-Speed 127.19 samples/sec Loss 47.2250 Epoch: 0 Global Step: 60 Fp16 Grad Scale: 64 Required: 288 hours Training: 2021-06-06 19:09:05,438-Speed 124.95 samples/sec Loss 47.1263 Epoch: 0 Global Step: 65 Fp16 Grad Scale: 64 Required: 287 hours Training: 2021-06-06 19:09:25,561-Speed 127.23 samples/sec Loss 47.1659 Epoch: 0 Global Step: 70 Fp16 Grad Scale: 64 Required: 287 hours Training: 2021-06-06 19:09:45,886-Speed 125.96 samples/sec Loss 47.1399 Epoch: 0 Global Step: 75 Fp16 Grad Scale: 64 Required: 286 hours Training: 2021-06-06 19:10:05,963-Speed 127.52 samples/sec Loss 47.3431 Epoch: 0 Global Step: 80 Fp16 Grad Scale: 64 Required: 286 hours Training: 2021-06-06 19:10:25,836-Speed 128.82 samples/sec Loss 47.2571 Epoch: 0 Global Step: 85 Fp16 Grad Scale: 64 Required: 285 hours Training: 2021-06-06 19:10:45,608-Speed 129.48 samples/sec Loss 47.5572 Epoch: 0 Global Step: 90 Fp16 Grad Scale: 64 Required: 285 hours Training: 2021-06-06 19:11:05,557-Speed 128.33 samples/sec Loss 47.6079 Epoch: 0 Global Step: 95 Fp16 Grad Scale: 64 Required: 284 hours Training: 2021-06-06 19:11:25,446-Speed 128.73 samples/sec Loss 47.4796 Epoch: 0 Global Step: 100 Fp16 Grad Scale: 128 Required: 284 hours Training: 2021-06-06 19:11:45,047-Speed 130.61 samples/sec Loss 47.5171 Epoch: 0 Global Step: 105 Fp16 Grad Scale: 128 Required: 283 hours Training: 2021-06-06 19:12:04,973-Speed 128.48 samples/sec Loss 47.4137 Epoch: 0 Global Step: 110 Fp16 Grad Scale: 128 Required: 283 hours Training: 2021-06-06 19:12:24,553-Speed 130.75 samples/sec Loss 47.7380 Epoch: 0 Global Step: 115 Fp16 Grad Scale: 128 Required: 282 hours Training: 2021-06-06 19:12:44,456-Speed 128.64 samples/sec Loss 47.7292 Epoch: 0 Global Step: 120 Fp16 Grad Scale: 128 Required: 282 hours Training: 2021-06-06 19:13:04,438-Speed 128.12 samples/sec Loss 47.7173 Epoch: 0 Global Step: 125 Fp16 Grad Scale: 128 Required: 282 hours Training: 2021-06-06 19:13:24,374-Speed 128.42 samples/sec Loss 47.6338 Epoch: 0 Global Step: 130 Fp16 Grad Scale: 128 Required: 281 hours Training: 2021-06-06 19:13:43,953-Speed 130.76 samples/sec Loss 47.6409 Epoch: 0 Global Step: 135 Fp16 Grad Scale: 128 Required: 281 hours Training: 2021-06-06 19:14:03,828-Speed 128.81 samples/sec Loss 47.6691 Epoch: 0 Global Step: 140 Fp16 Grad Scale: 128 Required: 281 hours Training: 2021-06-06 19:14:24,375-Speed 124.60 samples/sec Loss 47.5041 Epoch: 0 Global Step: 145 Fp16 Grad Scale: 128 Required: 281 hours Training: 2021-06-06 19:14:43,946-Speed 130.81 samples/sec Loss 47.3835 Epoch: 0 Global Step: 150 Fp16 Grad Scale: 128 Required: 281 hours Training: 2021-06-06 19:15:03,820-Speed 128.82 samples/sec Loss 47.5716 Epoch: 0 Global Step: 155 Fp16 Grad Scale: 128 Required: 280 hours Training: 2021-06-06 19:15:23,374-Speed 130.93 samples/sec Loss 47.5220 Epoch: 0 Global Step: 160 Fp16 Grad Scale: 128 Required: 280 hours Training: 2021-06-06 19:15:43,242-Speed 128.86 samples/sec Loss 47.6322 Epoch: 0 Global Step: 165 Fp16 Grad Scale: 128 Required: 280 hours Training: 2021-06-06 19:16:03,219-Speed 128.16 samples/sec Loss 47.4072 Epoch: 0 Global Step: 170 Fp16 Grad Scale: 128 Required: 280 hours Training: 2021-06-06 19:16:23,103-Speed 128.76 samples/sec Loss 47.5362 Epoch: 0 Global Step: 175 Fp16 Grad Scale: 128 Required: 280 hours Training: 2021-06-06 19:16:42,670-Speed 130.84 samples/sec Loss 47.6565 Epoch: 0 Global Step: 180 Fp16 Grad Scale: 128 Required: 279 hours Training: 2021-06-06 19:17:02,548-Speed 128.79 samples/sec Loss 47.5408 Epoch: 0 Global Step: 185 Fp16 Grad Scale: 128 Required: 279 hours Training: 2021-06-06 19:17:22,428-Speed 128.78 samples/sec Loss 47.4997 Epoch: 0 Global Step: 190 Fp16 Grad Scale: 128 Required: 279 hours Training: 2021-06-06 19:17:41,996-Speed 130.84 samples/sec Loss 47.3504 Epoch: 0 Global Step: 195 Fp16 Grad Scale: 128 Required: 279 hours Training: 2021-06-06 19:18:01,931-Speed 128.43 samples/sec Loss 47.3482 Epoch: 0 Global Step: 200 Fp16 Grad Scale: 256 Required: 279 hours Training: 2021-06-06 19:18:21,497-Speed 130.85 samples/sec Loss 47.4735 Epoch: 0 Global Step: 205 Fp16 Grad Scale: 256 Required: 279 hours Training: 2021-06-06 19:18:41,387-Speed 128.72 samples/sec Loss 47.4907 Epoch: 0 Global Step: 210 Fp16 Grad Scale: 256 Required: 279 hours Training: 2021-06-06 19:19:01,369-Speed 128.13 samples/sec Loss 47.5415 Epoch: 0 Global Step: 215 Fp16 Grad Scale: 256 Required: 279 hours Training: 2021-06-06 19:19:21,309-Speed 128.39 samples/sec Loss 47.4874 Epoch: 0 Global Step: 220 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:19:40,925-Speed 130.51 samples/sec Loss 47.4100 Epoch: 0 Global Step: 225 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:20:00,853-Speed 128.47 samples/sec Loss 47.3380 Epoch: 0 Global Step: 230 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:20:20,723-Speed 128.84 samples/sec Loss 47.1566 Epoch: 0 Global Step: 235 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:20:40,990-Speed 126.33 samples/sec Loss 47.2465 Epoch: 0 Global Step: 240 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:21:00,862-Speed 128.83 samples/sec Loss 47.1234 Epoch: 0 Global Step: 245 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:21:20,430-Speed 130.83 samples/sec Loss 47.1155 Epoch: 0 Global Step: 250 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:21:40,331-Speed 128.65 samples/sec Loss 46.9391 Epoch: 0 Global Step: 255 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:22:00,534-Speed 126.72 samples/sec Loss 47.0133 Epoch: 0 Global Step: 260 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:22:20,440-Speed 128.61 samples/sec Loss 47.1952 Epoch: 0 Global Step: 265 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:22:40,053-Speed 130.53 samples/sec Loss 47.0146 Epoch: 0 Global Step: 270 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:23:00,011-Speed 128.28 samples/sec Loss 46.8721 Epoch: 0 Global Step: 275 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:23:20,018-Speed 127.96 samples/sec Loss 46.8414 Epoch: 0 Global Step: 280 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:23:39,602-Speed 130.73 samples/sec Loss 46.8426 Epoch: 0 Global Step: 285 Fp16 Grad Scale: 256 Required: 278 hours Training: 2021-06-06 19:23:59,504-Speed 128.65 samples/sec Loss 46.6806 Epoch: 0 Global Step: 290 Fp16 Grad Scale: 256 Required: 277 hours Training: 2021-06-06 19:24:19,079-Speed 130.78 samples/sec Loss 46.6612 Epoch: 0 Global Step: 295 Fp16 Grad Scale: 256 Required: 277 hours Training: 2021-06-06 19:24:38,967-Speed 128.73 samples/sec Loss 46.7249 Epoch: 0 Global Step: 300 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:24:58,955-Speed 128.08 samples/sec Loss 46.8616 Epoch: 0 Global Step: 305 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:25:18,978-Speed 127.86 samples/sec Loss 46.6664 Epoch: 0 Global Step: 310 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:25:38,565-Speed 130.71 samples/sec Loss 46.6318 Epoch: 0 Global Step: 315 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:25:58,459-Speed 128.69 samples/sec Loss 46.5939 Epoch: 0 Global Step: 320 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:26:18,373-Speed 128.56 samples/sec Loss 46.6190 Epoch: 0 Global Step: 325 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:26:37,950-Speed 130.77 samples/sec Loss 46.4942 Epoch: 0 Global Step: 330 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:26:57,873-Speed 128.51 samples/sec Loss 46.4961 Epoch: 0 Global Step: 335 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:27:17,442-Speed 130.83 samples/sec Loss 46.5220 Epoch: 0 Global Step: 340 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:27:37,573-Speed 127.18 samples/sec Loss 46.3150 Epoch: 0 Global Step: 345 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:27:57,599-Speed 127.85 samples/sec Loss 46.3201 Epoch: 0 Global Step: 350 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:28:17,509-Speed 128.58 samples/sec Loss 46.3582 Epoch: 0 Global Step: 355 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:28:37,103-Speed 130.66 samples/sec Loss 46.3802 Epoch: 0 Global Step: 360 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:28:57,022-Speed 128.53 samples/sec Loss 46.2774 Epoch: 0 Global Step: 365 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:29:16,923-Speed 128.64 samples/sec Loss 46.3638 Epoch: 0 Global Step: 370 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:29:36,529-Speed 130.58 samples/sec Loss 46.3518 Epoch: 0 Global Step: 375 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:29:56,419-Speed 128.72 samples/sec Loss 46.3016 Epoch: 0 Global Step: 380 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:30:16,087-Speed 130.17 samples/sec Loss 46.3134 Epoch: 0 Global Step: 385 Fp16 Grad Scale: 512 Required: 277 hours Training: 2021-06-06 19:30:35,962-Speed 128.81 samples/sec Loss 46.3585 Epoch: 0 Global Step: 390 Fp16 Grad Scale: 512 Required: 276 hours Training: 2021-06-06 19:30:55,974-Speed 127.94 samples/sec Loss 46.0575 Epoch: 0 Global Step: 395 Fp16 Grad Scale: 512 Required: 276 hours Training: 2021-06-06 19:31:15,865-Speed 128.71 samples/sec Loss 46.3820 Epoch: 0 Global Step: 400 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:31:35,451-Speed 130.71 samples/sec Loss 46.1172 Epoch: 0 Global Step: 405 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:31:55,335-Speed 128.75 samples/sec Loss 46.1280 Epoch: 0 Global Step: 410 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:32:15,574-Speed 126.50 samples/sec Loss 46.0399 Epoch: 0 Global Step: 415 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:32:35,202-Speed 130.43 samples/sec Loss 46.0442 Epoch: 0 Global Step: 420 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:32:55,094-Speed 128.70 samples/sec Loss 45.9032 Epoch: 0 Global Step: 425 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:33:14,760-Speed 130.18 samples/sec Loss 45.8653 Epoch: 0 Global Step: 430 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:33:34,799-Speed 127.77 samples/sec Loss 45.9199 Epoch: 0 Global Step: 435 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:33:54,776-Speed 128.16 samples/sec Loss 45.9012 Epoch: 0 Global Step: 440 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:34:14,712-Speed 128.41 samples/sec Loss 45.7617 Epoch: 0 Global Step: 445 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:34:34,289-Speed 130.77 samples/sec Loss 45.6643 Epoch: 0 Global Step: 450 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:34:54,315-Speed 127.84 samples/sec Loss 45.7437 Epoch: 0 Global Step: 455 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:35:14,243-Speed 128.47 samples/sec Loss 45.6028 Epoch: 0 Global Step: 460 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:35:33,838-Speed 130.65 samples/sec Loss 45.5207 Epoch: 0 Global Step: 465 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:35:53,706-Speed 128.86 samples/sec Loss 45.5209 Epoch: 0 Global Step: 470 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:36:13,319-Speed 130.54 samples/sec Loss 45.5454 Epoch: 0 Global Step: 475 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:36:33,208-Speed 128.73 samples/sec Loss 45.4047 Epoch: 0 Global Step: 480 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:36:53,213-Speed 127.98 samples/sec Loss 45.3605 Epoch: 0 Global Step: 485 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:37:13,107-Speed 128.69 samples/sec Loss 45.3262 Epoch: 0 Global Step: 490 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:37:32,684-Speed 130.77 samples/sec Loss 45.2213 Epoch: 0 Global Step: 495 Fp16 Grad Scale: 1024 Required: 276 hours Training: 2021-06-06 19:37:52,592-Speed 128.60 samples/sec Loss 45.2701 Epoch: 0 Global Step: 500 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:38:12,480-Speed 128.73 samples/sec Loss 45.2851 Epoch: 0 Global Step: 505 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:38:32,067-Speed 130.71 samples/sec Loss 45.3620 Epoch: 0 Global Step: 510 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:38:51,953-Speed 128.74 samples/sec Loss 45.2499 Epoch: 0 Global Step: 515 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:39:11,575-Speed 130.47 samples/sec Loss 45.1841 Epoch: 0 Global Step: 520 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:39:31,507-Speed 128.45 samples/sec Loss 45.1967 Epoch: 0 Global Step: 525 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:39:51,484-Speed 128.16 samples/sec Loss 45.0501 Epoch: 0 Global Step: 530 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:40:11,295-Speed 129.23 samples/sec Loss 45.1457 Epoch: 0 Global Step: 535 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:40:30,775-Speed 131.43 samples/sec Loss 45.1744 Epoch: 0 Global Step: 540 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:40:50,685-Speed 128.59 samples/sec Loss 45.1519 Epoch: 0 Global Step: 545 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:41:10,405-Speed 129.83 samples/sec Loss 45.1455 Epoch: 0 Global Step: 550 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:41:29,919-Speed 131.20 samples/sec Loss 44.8079 Epoch: 0 Global Step: 555 Fp16 Grad Scale: 2048 Required: 276 hours Training: 2021-06-06 19:41:49,857-Speed 128.40 samples/sec Loss 44.8337 Epoch: 0 Global Step: 560 Fp16 Grad Scale: 2048 Required: 275 hours Training: 2021-06-06 19:42:09,449-Speed 130.68 samples/sec Loss 44.8480 Epoch: 0 Global Step: 565 Fp16 Grad Scale: 2048 Required: 275 hours Training: 2021-06-06 19:42:29,405-Speed 128.29 samples/sec Loss 44.8027 Epoch: 0 Global Step: 570 Fp16 Grad Scale: 2048 Required: 275 hours Training: 2021-06-06 19:42:49,342-Speed 128.41 samples/sec Loss 44.7510 Epoch: 0 Global Step: 575 Fp16 Grad Scale: 2048 Required: 275 hours Training: 2021-06-06 19:43:09,238-Speed 128.68 samples/sec Loss 44.7862 Epoch: 0 Global Step: 580 Fp16 Grad Scale: 2048 Required: 275 hours Training: 2021-06-06 19:43:28,942-Speed 129.93 samples/sec Loss 44.6383 Epoch: 0 Global Step: 585 Fp16 Grad Scale: 2048 Required: 275 hours Training: 2021-06-06 19:43:48,917-Speed 128.16 samples/sec Loss 44.7572 Epoch: 0 Global Step: 590 Fp16 Grad Scale: 2048 Required: 275 hours Training: 2021-06-06 19:44:09,106-Speed 126.81 samples/sec Loss 44.7456 Epoch: 0 Global Step: 595 Fp16 Grad Scale: 2048 Required: 275 hours Training: 2021-06-06 19:44:28,732-Speed 130.45 samples/sec Loss 44.6311 Epoch: 0 Global Step: 600 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:44:48,466-Speed 129.73 samples/sec Loss 44.5367 Epoch: 0 Global Step: 605 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:45:07,975-Speed 131.23 samples/sec Loss 44.4856 Epoch: 0 Global Step: 610 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:45:27,880-Speed 128.62 samples/sec Loss 44.4883 Epoch: 0 Global Step: 615 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:45:47,620-Speed 129.69 samples/sec Loss 44.2581 Epoch: 0 Global Step: 620 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:46:08,092-Speed 125.05 samples/sec Loss 44.3513 Epoch: 0 Global Step: 625 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:46:27,478-Speed 132.07 samples/sec Loss 44.3034 Epoch: 0 Global Step: 630 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:46:47,363-Speed 128.75 samples/sec Loss 44.3894 Epoch: 0 Global Step: 635 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:47:07,288-Speed 128.49 samples/sec Loss 44.3945 Epoch: 0 Global Step: 640 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:47:26,873-Speed 130.72 samples/sec Loss 44.2745 Epoch: 0 Global Step: 645 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:47:46,716-Speed 129.03 samples/sec Loss 44.2634 Epoch: 0 Global Step: 650 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:48:06,257-Speed 131.02 samples/sec Loss 44.2101 Epoch: 0 Global Step: 655 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:48:25,925-Speed 130.17 samples/sec Loss 44.1273 Epoch: 0 Global Step: 660 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:48:45,761-Speed 129.06 samples/sec Loss 44.1697 Epoch: 0 Global Step: 665 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:49:05,447-Speed 130.05 samples/sec Loss 44.1503 Epoch: 0 Global Step: 670 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:49:25,109-Speed 130.21 samples/sec Loss 44.0604 Epoch: 0 Global Step: 675 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:49:44,949-Speed 129.04 samples/sec Loss 44.0098 Epoch: 0 Global Step: 680 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:50:04,858-Speed 128.59 samples/sec Loss 43.9636 Epoch: 0 Global Step: 685 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:50:24,428-Speed 130.82 samples/sec Loss 43.8765 Epoch: 0 Global Step: 690 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:50:44,308-Speed 128.79 samples/sec Loss 44.0270 Epoch: 0 Global Step: 695 Fp16 Grad Scale: 4096 Required: 275 hours Training: 2021-06-06 19:51:03,879-Speed 130.81 samples/sec Loss 43.8627 Epoch: 0 Global Step: 700 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:51:23,820-Speed 128.39 samples/sec Loss 43.7935 Epoch: 0 Global Step: 705 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:51:43,661-Speed 129.04 samples/sec Loss 43.6572 Epoch: 0 Global Step: 710 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:52:03,346-Speed 130.05 samples/sec Loss 43.6774 Epoch: 0 Global Step: 715 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:52:23,045-Speed 129.97 samples/sec Loss 43.6785 Epoch: 0 Global Step: 720 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:52:42,948-Speed 128.63 samples/sec Loss 43.6056 Epoch: 0 Global Step: 725 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:53:02,853-Speed 128.62 samples/sec Loss 43.5651 Epoch: 0 Global Step: 730 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:53:22,535-Speed 130.08 samples/sec Loss 43.5284 Epoch: 0 Global Step: 735 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:53:42,458-Speed 128.50 samples/sec Loss 43.4923 Epoch: 0 Global Step: 740 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:54:02,040-Speed 130.74 samples/sec Loss 43.6812 Epoch: 0 Global Step: 745 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:54:21,938-Speed 128.67 samples/sec Loss 43.4139 Epoch: 0 Global Step: 750 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:54:41,894-Speed 128.29 samples/sec Loss 43.5167 Epoch: 0 Global Step: 755 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:55:01,833-Speed 128.40 samples/sec Loss 43.4361 Epoch: 0 Global Step: 760 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:55:21,444-Speed 130.54 samples/sec Loss 43.1350 Epoch: 0 Global Step: 765 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:55:41,341-Speed 128.67 samples/sec Loss 43.3164 Epoch: 0 Global Step: 770 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:56:01,419-Speed 127.51 samples/sec Loss 43.2925 Epoch: 0 Global Step: 775 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:56:21,032-Speed 130.54 samples/sec Loss 43.2543 Epoch: 0 Global Step: 780 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:56:40,946-Speed 128.56 samples/sec Loss 43.0909 Epoch: 0 Global Step: 785 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:57:00,501-Speed 130.92 samples/sec Loss 43.0975 Epoch: 0 Global Step: 790 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:57:20,388-Speed 128.74 samples/sec Loss 43.1342 Epoch: 0 Global Step: 795 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:57:40,397-Speed 127.95 samples/sec Loss 42.9614 Epoch: 0 Global Step: 800 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:58:00,290-Speed 128.70 samples/sec Loss 43.1339 Epoch: 0 Global Step: 805 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:58:19,876-Speed 130.71 samples/sec Loss 43.0773 Epoch: 0 Global Step: 810 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:58:39,808-Speed 128.45 samples/sec Loss 43.0573 Epoch: 0 Global Step: 815 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:58:59,693-Speed 128.75 samples/sec Loss 43.0477 Epoch: 0 Global Step: 820 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:59:19,243-Speed 130.96 samples/sec Loss 42.9372 Epoch: 0 Global Step: 825 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:59:39,013-Speed 129.50 samples/sec Loss 42.8680 Epoch: 0 Global Step: 830 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-06 19:59:58,604-Speed 130.68 samples/sec Loss 42.9514 Epoch: 0 Global Step: 835 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:00:18,567-Speed 128.24 samples/sec Loss 42.7063 Epoch: 0 Global Step: 840 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:00:38,550-Speed 128.12 samples/sec Loss 42.8634 Epoch: 0 Global Step: 845 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:00:58,441-Speed 128.71 samples/sec Loss 42.9058 Epoch: 0 Global Step: 850 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:01:17,812-Speed 132.17 samples/sec Loss 42.7135 Epoch: 0 Global Step: 855 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:01:37,713-Speed 128.64 samples/sec Loss 42.8181 Epoch: 0 Global Step: 860 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:01:57,601-Speed 128.73 samples/sec Loss 42.5770 Epoch: 0 Global Step: 865 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:02:17,205-Speed 130.60 samples/sec Loss 42.6082 Epoch: 0 Global Step: 870 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:02:37,094-Speed 128.72 samples/sec Loss 42.3871 Epoch: 0 Global Step: 875 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:02:56,644-Speed 130.96 samples/sec Loss 42.4871 Epoch: 0 Global Step: 880 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:03:16,540-Speed 128.68 samples/sec Loss 42.4151 Epoch: 0 Global Step: 885 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:03:36,373-Speed 129.08 samples/sec Loss 42.5534 Epoch: 0 Global Step: 890 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:03:56,263-Speed 128.71 samples/sec Loss 42.7093 Epoch: 0 Global Step: 895 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:04:15,843-Speed 130.76 samples/sec Loss 42.4016 Epoch: 0 Global Step: 900 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:04:35,712-Speed 128.85 samples/sec Loss 42.3317 Epoch: 0 Global Step: 905 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:04:55,610-Speed 128.67 samples/sec Loss 42.5148 Epoch: 0 Global Step: 910 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:05:15,181-Speed 130.82 samples/sec Loss 42.3737 Epoch: 0 Global Step: 915 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:05:35,061-Speed 128.78 samples/sec Loss 42.2589 Epoch: 0 Global Step: 920 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:05:54,636-Speed 130.79 samples/sec Loss 42.4191 Epoch: 0 Global Step: 925 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:06:14,643-Speed 127.97 samples/sec Loss 42.0714 Epoch: 0 Global Step: 930 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:06:34,624-Speed 128.13 samples/sec Loss 42.0436 Epoch: 0 Global Step: 935 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:06:54,519-Speed 128.68 samples/sec Loss 42.2640 Epoch: 0 Global Step: 940 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:07:14,085-Speed 130.85 samples/sec Loss 42.0557 Epoch: 0 Global Step: 945 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:07:33,981-Speed 128.67 samples/sec Loss 42.0563 Epoch: 0 Global Step: 950 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:07:53,860-Speed 128.80 samples/sec Loss 41.9913 Epoch: 0 Global Step: 955 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:08:13,429-Speed 130.83 samples/sec Loss 42.0878 Epoch: 0 Global Step: 960 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:08:33,329-Speed 128.65 samples/sec Loss 42.0519 Epoch: 0 Global Step: 965 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:08:52,881-Speed 130.95 samples/sec Loss 41.8772 Epoch: 0 Global Step: 970 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:09:12,563-Speed 130.07 samples/sec Loss 41.8799 Epoch: 0 Global Step: 975 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:09:32,527-Speed 128.24 samples/sec Loss 41.7726 Epoch: 0 Global Step: 980 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:09:52,428-Speed 128.65 samples/sec Loss 41.8208 Epoch: 0 Global Step: 985 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:10:12,007-Speed 130.76 samples/sec Loss 41.4547 Epoch: 0 Global Step: 990 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:10:31,853-Speed 129.00 samples/sec Loss 41.7312 Epoch: 0 Global Step: 995 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:10:51,727-Speed 128.82 samples/sec Loss 41.6799 Epoch: 0 Global Step: 1000 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:11:11,301-Speed 130.80 samples/sec Loss 41.8338 Epoch: 0 Global Step: 1005 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:11:31,035-Speed 129.74 samples/sec Loss 41.7279 Epoch: 0 Global Step: 1010 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:11:50,601-Speed 130.84 samples/sec Loss 41.7287 Epoch: 0 Global Step: 1015 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:12:10,440-Speed 129.05 samples/sec Loss 41.5043 Epoch: 0 Global Step: 1020 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:12:30,445-Speed 127.98 samples/sec Loss 41.6912 Epoch: 0 Global Step: 1025 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:12:50,952-Speed 124.84 samples/sec Loss 41.5573 Epoch: 0 Global Step: 1030 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:13:10,304-Speed 132.29 samples/sec Loss 41.5069 Epoch: 0 Global Step: 1035 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:13:30,277-Speed 128.18 samples/sec Loss 41.3875 Epoch: 0 Global Step: 1040 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:13:49,946-Speed 130.16 samples/sec Loss 41.3891 Epoch: 0 Global Step: 1045 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:14:09,517-Speed 130.81 samples/sec Loss 41.3075 Epoch: 0 Global Step: 1050 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:14:29,203-Speed 130.05 samples/sec Loss 41.4114 Epoch: 0 Global Step: 1055 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:14:48,766-Speed 130.87 samples/sec Loss 41.0158 Epoch: 0 Global Step: 1060 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:15:08,632-Speed 128.87 samples/sec Loss 41.1428 Epoch: 0 Global Step: 1065 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:15:28,602-Speed 128.20 samples/sec Loss 41.3469 Epoch: 0 Global Step: 1070 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:15:48,500-Speed 128.66 samples/sec Loss 41.2363 Epoch: 0 Global Step: 1075 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:16:08,023-Speed 131.14 samples/sec Loss 41.3123 Epoch: 0 Global Step: 1080 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:16:27,887-Speed 128.88 samples/sec Loss 41.0683 Epoch: 0 Global Step: 1085 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:16:47,792-Speed 128.63 samples/sec Loss 41.0910 Epoch: 0 Global Step: 1090 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:17:07,277-Speed 131.39 samples/sec Loss 40.9822 Epoch: 0 Global Step: 1095 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:17:27,196-Speed 128.53 samples/sec Loss 40.7960 Epoch: 0 Global Step: 1100 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:17:46,789-Speed 130.67 samples/sec Loss 41.0849 Epoch: 0 Global Step: 1105 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:18:06,659-Speed 128.85 samples/sec Loss 41.0250 Epoch: 0 Global Step: 1110 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:18:26,593-Speed 128.44 samples/sec Loss 40.8873 Epoch: 0 Global Step: 1115 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:18:46,480-Speed 128.73 samples/sec Loss 40.5026 Epoch: 0 Global Step: 1120 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:19:05,854-Speed 132.15 samples/sec Loss 41.0409 Epoch: 0 Global Step: 1125 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:19:25,839-Speed 128.11 samples/sec Loss 40.8930 Epoch: 0 Global Step: 1130 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:19:45,670-Speed 129.09 samples/sec Loss 40.7366 Epoch: 0 Global Step: 1135 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:20:05,135-Speed 131.53 samples/sec Loss 40.4504 Epoch: 0 Global Step: 1140 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:20:25,006-Speed 128.84 samples/sec Loss 40.7247 Epoch: 0 Global Step: 1145 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:20:44,448-Speed 131.68 samples/sec Loss 40.5339 Epoch: 0 Global Step: 1150 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:21:04,171-Speed 129.81 samples/sec Loss 40.4791 Epoch: 0 Global Step: 1155 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:21:24,101-Speed 128.45 samples/sec Loss 40.2979 Epoch: 0 Global Step: 1160 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:21:43,995-Speed 128.69 samples/sec Loss 40.5782 Epoch: 0 Global Step: 1165 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:22:03,545-Speed 130.96 samples/sec Loss 40.5373 Epoch: 0 Global Step: 1170 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:22:23,431-Speed 128.74 samples/sec Loss 40.5198 Epoch: 0 Global Step: 1175 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:22:43,474-Speed 127.73 samples/sec Loss 40.3625 Epoch: 0 Global Step: 1180 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:23:03,036-Speed 130.88 samples/sec Loss 40.4501 Epoch: 0 Global Step: 1185 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:23:22,935-Speed 128.66 samples/sec Loss 40.1412 Epoch: 0 Global Step: 1190 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:23:42,417-Speed 131.42 samples/sec Loss 40.1909 Epoch: 0 Global Step: 1195 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:24:02,257-Speed 129.04 samples/sec Loss 40.2333 Epoch: 0 Global Step: 1200 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:24:22,123-Speed 128.88 samples/sec Loss 40.2937 Epoch: 0 Global Step: 1205 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:24:42,041-Speed 128.53 samples/sec Loss 40.5019 Epoch: 0 Global Step: 1210 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:25:01,596-Speed 130.92 samples/sec Loss 40.2632 Epoch: 0 Global Step: 1215 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:25:21,393-Speed 129.32 samples/sec Loss 40.1467 Epoch: 0 Global Step: 1220 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:25:41,255-Speed 128.90 samples/sec Loss 40.1184 Epoch: 0 Global Step: 1225 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-06 20:26:00,873-Speed 130.50 samples/sec Loss 39.9864 Epoch: 0 Global Step: 1230 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:26:20,715-Speed 129.03 samples/sec Loss 39.7998 Epoch: 0 Global Step: 1235 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:26:40,311-Speed 130.65 samples/sec Loss 40.1326 Epoch: 0 Global Step: 1240 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:27:00,196-Speed 128.75 samples/sec Loss 39.8174 Epoch: 0 Global Step: 1245 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:27:20,344-Speed 127.07 samples/sec Loss 39.7920 Epoch: 0 Global Step: 1250 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:27:40,219-Speed 128.81 samples/sec Loss 39.8097 Epoch: 0 Global Step: 1255 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:27:59,762-Speed 131.00 samples/sec Loss 39.7759 Epoch: 0 Global Step: 1260 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:28:19,550-Speed 129.38 samples/sec Loss 39.9362 Epoch: 0 Global Step: 1265 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:28:39,395-Speed 129.01 samples/sec Loss 39.8812 Epoch: 0 Global Step: 1270 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:28:59,001-Speed 130.58 samples/sec Loss 39.7618 Epoch: 0 Global Step: 1275 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:29:18,929-Speed 128.47 samples/sec Loss 39.9745 Epoch: 0 Global Step: 1280 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:29:38,490-Speed 130.88 samples/sec Loss 39.8372 Epoch: 0 Global Step: 1285 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:29:58,896-Speed 125.47 samples/sec Loss 39.7168 Epoch: 0 Global Step: 1290 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:30:18,882-Speed 128.09 samples/sec Loss 39.4551 Epoch: 0 Global Step: 1295 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:30:38,789-Speed 128.60 samples/sec Loss 39.4621 Epoch: 0 Global Step: 1300 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:30:58,397-Speed 130.57 samples/sec Loss 39.5746 Epoch: 0 Global Step: 1305 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:31:18,305-Speed 128.60 samples/sec Loss 39.5327 Epoch: 0 Global Step: 1310 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:31:38,184-Speed 128.79 samples/sec Loss 39.3347 Epoch: 0 Global Step: 1315 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:31:57,645-Speed 131.55 samples/sec Loss 39.2317 Epoch: 0 Global Step: 1320 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:32:17,519-Speed 128.82 samples/sec Loss 39.7021 Epoch: 0 Global Step: 1325 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:32:37,079-Speed 130.89 samples/sec Loss 39.2772 Epoch: 0 Global Step: 1330 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:32:56,752-Speed 130.14 samples/sec Loss 38.8853 Epoch: 0 Global Step: 1335 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:33:16,630-Speed 128.79 samples/sec Loss 39.0483 Epoch: 0 Global Step: 1340 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:33:36,535-Speed 128.61 samples/sec Loss 39.2837 Epoch: 0 Global Step: 1345 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:33:56,141-Speed 130.58 samples/sec Loss 39.1236 Epoch: 0 Global Step: 1350 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:34:16,043-Speed 128.64 samples/sec Loss 39.1659 Epoch: 0 Global Step: 1355 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:34:35,911-Speed 128.86 samples/sec Loss 39.0825 Epoch: 0 Global Step: 1360 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:34:55,469-Speed 130.90 samples/sec Loss 39.0753 Epoch: 0 Global Step: 1365 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:35:15,151-Speed 130.08 samples/sec Loss 39.3669 Epoch: 0 Global Step: 1370 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:35:34,787-Speed 130.38 samples/sec Loss 39.0437 Epoch: 0 Global Step: 1375 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:35:54,455-Speed 130.17 samples/sec Loss 38.8446 Epoch: 0 Global Step: 1380 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:36:14,411-Speed 128.29 samples/sec Loss 39.0128 Epoch: 0 Global Step: 1385 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:36:34,092-Speed 130.09 samples/sec Loss 39.0697 Epoch: 0 Global Step: 1390 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:36:53,814-Speed 129.81 samples/sec Loss 38.8431 Epoch: 0 Global Step: 1395 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:37:13,681-Speed 128.86 samples/sec Loss 38.9738 Epoch: 0 Global Step: 1400 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:37:33,549-Speed 128.86 samples/sec Loss 38.8565 Epoch: 0 Global Step: 1405 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:37:53,121-Speed 130.81 samples/sec Loss 38.6708 Epoch: 0 Global Step: 1410 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:38:12,901-Speed 129.43 samples/sec Loss 38.5673 Epoch: 0 Global Step: 1415 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:38:32,466-Speed 130.86 samples/sec Loss 38.5533 Epoch: 0 Global Step: 1420 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:38:52,349-Speed 128.76 samples/sec Loss 38.5066 Epoch: 0 Global Step: 1425 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:39:12,329-Speed 128.14 samples/sec Loss 38.6855 Epoch: 0 Global Step: 1430 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:39:32,153-Speed 129.14 samples/sec Loss 38.5762 Epoch: 0 Global Step: 1435 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:39:51,820-Speed 130.18 samples/sec Loss 38.5980 Epoch: 0 Global Step: 1440 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:40:11,711-Speed 128.71 samples/sec Loss 38.7585 Epoch: 0 Global Step: 1445 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:40:31,612-Speed 128.65 samples/sec Loss 38.4425 Epoch: 0 Global Step: 1450 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:40:51,382-Speed 129.50 samples/sec Loss 38.2353 Epoch: 0 Global Step: 1455 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:41:11,274-Speed 128.70 samples/sec Loss 38.2497 Epoch: 0 Global Step: 1460 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:41:30,780-Speed 131.25 samples/sec Loss 38.3267 Epoch: 0 Global Step: 1465 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:41:50,687-Speed 128.61 samples/sec Loss 38.2533 Epoch: 0 Global Step: 1470 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:42:10,677-Speed 128.07 samples/sec Loss 38.2718 Epoch: 0 Global Step: 1475 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:42:30,612-Speed 128.42 samples/sec Loss 38.1371 Epoch: 0 Global Step: 1480 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:42:50,178-Speed 130.85 samples/sec Loss 38.0651 Epoch: 0 Global Step: 1485 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:43:10,051-Speed 128.83 samples/sec Loss 38.4470 Epoch: 0 Global Step: 1490 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:43:29,936-Speed 128.75 samples/sec Loss 38.0213 Epoch: 0 Global Step: 1495 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:43:49,417-Speed 131.42 samples/sec Loss 37.8078 Epoch: 0 Global Step: 1500 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:44:09,290-Speed 128.83 samples/sec Loss 37.8363 Epoch: 0 Global Step: 1505 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:44:28,844-Speed 130.93 samples/sec Loss 37.9089 Epoch: 0 Global Step: 1510 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:44:48,706-Speed 128.90 samples/sec Loss 37.7416 Epoch: 0 Global Step: 1515 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:45:08,610-Speed 128.63 samples/sec Loss 37.9067 Epoch: 0 Global Step: 1520 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:45:28,504-Speed 128.69 samples/sec Loss 37.9692 Epoch: 0 Global Step: 1525 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:45:48,103-Speed 130.62 samples/sec Loss 37.8728 Epoch: 0 Global Step: 1530 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:46:08,002-Speed 128.66 samples/sec Loss 37.6226 Epoch: 0 Global Step: 1535 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:46:27,878-Speed 128.81 samples/sec Loss 37.5790 Epoch: 0 Global Step: 1540 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:46:47,499-Speed 130.48 samples/sec Loss 37.6483 Epoch: 0 Global Step: 1545 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:47:07,377-Speed 128.79 samples/sec Loss 37.6350 Epoch: 0 Global Step: 1550 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:47:26,967-Speed 130.69 samples/sec Loss 37.9018 Epoch: 0 Global Step: 1555 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:47:46,860-Speed 128.70 samples/sec Loss 37.7376 Epoch: 0 Global Step: 1560 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:48:06,869-Speed 127.95 samples/sec Loss 37.4443 Epoch: 0 Global Step: 1565 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:48:26,794-Speed 128.49 samples/sec Loss 37.4803 Epoch: 0 Global Step: 1570 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:48:46,335-Speed 131.02 samples/sec Loss 37.4773 Epoch: 0 Global Step: 1575 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:49:06,402-Speed 127.58 samples/sec Loss 37.2944 Epoch: 0 Global Step: 1580 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:49:26,115-Speed 129.87 samples/sec Loss 37.4709 Epoch: 0 Global Step: 1585 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:49:45,706-Speed 130.68 samples/sec Loss 37.2657 Epoch: 0 Global Step: 1590 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:50:05,662-Speed 128.29 samples/sec Loss 37.2238 Epoch: 0 Global Step: 1595 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:50:25,220-Speed 130.91 samples/sec Loss 37.2912 Epoch: 0 Global Step: 1600 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:50:45,078-Speed 128.92 samples/sec Loss 37.0015 Epoch: 0 Global Step: 1605 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:51:04,937-Speed 128.92 samples/sec Loss 37.0527 Epoch: 0 Global Step: 1610 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:51:24,843-Speed 128.61 samples/sec Loss 37.1639 Epoch: 0 Global Step: 1615 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:51:44,403-Speed 130.89 samples/sec Loss 37.2783 Epoch: 0 Global Step: 1620 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:52:04,303-Speed 128.66 samples/sec Loss 37.3613 Epoch: 0 Global Step: 1625 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:52:24,228-Speed 128.49 samples/sec Loss 37.0888 Epoch: 0 Global Step: 1630 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:52:43,800-Speed 130.81 samples/sec Loss 37.0732 Epoch: 0 Global Step: 1635 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:53:03,686-Speed 128.74 samples/sec Loss 37.0282 Epoch: 0 Global Step: 1640 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:53:23,291-Speed 130.59 samples/sec Loss 37.0059 Epoch: 0 Global Step: 1645 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:53:43,148-Speed 128.93 samples/sec Loss 36.9279 Epoch: 0 Global Step: 1650 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:54:03,079-Speed 128.45 samples/sec Loss 36.7699 Epoch: 0 Global Step: 1655 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:54:22,955-Speed 128.81 samples/sec Loss 36.9383 Epoch: 0 Global Step: 1660 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:54:42,323-Speed 132.19 samples/sec Loss 36.6141 Epoch: 0 Global Step: 1665 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:55:02,239-Speed 128.54 samples/sec Loss 36.7724 Epoch: 0 Global Step: 1670 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:55:22,181-Speed 128.38 samples/sec Loss 36.5805 Epoch: 0 Global Step: 1675 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:55:41,742-Speed 130.88 samples/sec Loss 36.4536 Epoch: 0 Global Step: 1680 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:56:01,648-Speed 128.62 samples/sec Loss 36.4484 Epoch: 0 Global Step: 1685 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:56:21,192-Speed 131.00 samples/sec Loss 36.7209 Epoch: 0 Global Step: 1690 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:56:40,918-Speed 129.79 samples/sec Loss 36.4110 Epoch: 0 Global Step: 1695 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:57:00,641-Speed 129.81 samples/sec Loss 36.3801 Epoch: 0 Global Step: 1700 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:57:20,616-Speed 128.16 samples/sec Loss 36.0923 Epoch: 0 Global Step: 1705 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:57:40,032-Speed 131.86 samples/sec Loss 36.3562 Epoch: 0 Global Step: 1710 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:57:59,834-Speed 129.29 samples/sec Loss 36.3793 Epoch: 0 Global Step: 1715 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:58:19,694-Speed 128.91 samples/sec Loss 36.4837 Epoch: 0 Global Step: 1720 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:58:39,159-Speed 131.53 samples/sec Loss 36.1506 Epoch: 0 Global Step: 1725 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:58:58,990-Speed 129.10 samples/sec Loss 36.0221 Epoch: 0 Global Step: 1730 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:59:18,545-Speed 130.92 samples/sec Loss 35.7336 Epoch: 0 Global Step: 1735 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:59:38,574-Speed 127.83 samples/sec Loss 35.9738 Epoch: 0 Global Step: 1740 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 20:59:58,556-Speed 128.13 samples/sec Loss 36.2902 Epoch: 0 Global Step: 1745 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:00:18,489-Speed 128.43 samples/sec Loss 35.8284 Epoch: 0 Global Step: 1750 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:00:38,077-Speed 130.70 samples/sec Loss 36.1161 Epoch: 0 Global Step: 1755 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:00:57,951-Speed 128.82 samples/sec Loss 35.9436 Epoch: 0 Global Step: 1760 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:01:17,836-Speed 128.75 samples/sec Loss 35.8700 Epoch: 0 Global Step: 1765 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:01:37,402-Speed 130.84 samples/sec Loss 35.5724 Epoch: 0 Global Step: 1770 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:01:57,088-Speed 130.05 samples/sec Loss 35.9143 Epoch: 0 Global Step: 1775 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:02:16,674-Speed 130.72 samples/sec Loss 35.6439 Epoch: 0 Global Step: 1780 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:02:36,551-Speed 128.81 samples/sec Loss 35.7118 Epoch: 0 Global Step: 1785 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:02:56,510-Speed 128.27 samples/sec Loss 35.7685 Epoch: 0 Global Step: 1790 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:03:16,191-Speed 130.08 samples/sec Loss 35.4998 Epoch: 0 Global Step: 1795 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:03:35,900-Speed 129.90 samples/sec Loss 35.4587 Epoch: 0 Global Step: 1800 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:03:55,780-Speed 128.78 samples/sec Loss 35.7247 Epoch: 0 Global Step: 1805 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:04:15,647-Speed 128.87 samples/sec Loss 35.2700 Epoch: 0 Global Step: 1810 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-06 21:04:35,239-Speed 130.67 samples/sec Loss 35.4521 Epoch: 0 Global Step: 1815 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:04:55,142-Speed 128.63 samples/sec Loss 35.2446 Epoch: 0 Global Step: 1820 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:05:14,706-Speed 130.86 samples/sec Loss 35.0684 Epoch: 0 Global Step: 1825 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:05:34,689-Speed 128.12 samples/sec Loss 35.3697 Epoch: 0 Global Step: 1830 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:05:54,673-Speed 128.11 samples/sec Loss 35.1501 Epoch: 0 Global Step: 1835 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:06:14,670-Speed 128.02 samples/sec Loss 35.1682 Epoch: 0 Global Step: 1840 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:06:34,236-Speed 130.85 samples/sec Loss 35.0344 Epoch: 0 Global Step: 1845 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:06:54,155-Speed 128.53 samples/sec Loss 34.9929 Epoch: 0 Global Step: 1850 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:07:14,057-Speed 128.64 samples/sec Loss 34.7234 Epoch: 0 Global Step: 1855 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:07:33,629-Speed 130.81 samples/sec Loss 35.3740 Epoch: 0 Global Step: 1860 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:07:53,466-Speed 129.06 samples/sec Loss 35.3344 Epoch: 0 Global Step: 1865 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:08:13,018-Speed 130.94 samples/sec Loss 34.7688 Epoch: 0 Global Step: 1870 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:08:32,689-Speed 130.15 samples/sec Loss 35.1279 Epoch: 0 Global Step: 1875 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:08:52,605-Speed 128.55 samples/sec Loss 35.0332 Epoch: 0 Global Step: 1880 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:09:12,614-Speed 127.95 samples/sec Loss 34.6230 Epoch: 0 Global Step: 1885 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:09:32,213-Speed 130.63 samples/sec Loss 34.8286 Epoch: 0 Global Step: 1890 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:09:52,066-Speed 128.96 samples/sec Loss 34.8814 Epoch: 0 Global Step: 1895 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:10:11,941-Speed 128.81 samples/sec Loss 34.7998 Epoch: 0 Global Step: 1900 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:10:31,528-Speed 130.71 samples/sec Loss 34.2584 Epoch: 0 Global Step: 1905 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:10:51,424-Speed 128.68 samples/sec Loss 34.7655 Epoch: 0 Global Step: 1910 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:11:11,223-Speed 129.31 samples/sec Loss 34.4026 Epoch: 0 Global Step: 1915 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:11:31,405-Speed 126.85 samples/sec Loss 34.3282 Epoch: 0 Global Step: 1920 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:11:51,359-Speed 128.31 samples/sec Loss 34.7166 Epoch: 0 Global Step: 1925 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:12:11,074-Speed 129.85 samples/sec Loss 34.7777 Epoch: 0 Global Step: 1930 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:12:30,699-Speed 130.46 samples/sec Loss 34.3971 Epoch: 0 Global Step: 1935 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:12:50,504-Speed 129.27 samples/sec Loss 34.2041 Epoch: 0 Global Step: 1940 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:13:10,406-Speed 128.64 samples/sec Loss 34.2326 Epoch: 0 Global Step: 1945 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:13:30,001-Speed 130.65 samples/sec Loss 34.1888 Epoch: 0 Global Step: 1950 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:13:49,933-Speed 128.45 samples/sec Loss 34.1133 Epoch: 0 Global Step: 1955 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:14:09,545-Speed 130.54 samples/sec Loss 34.6498 Epoch: 0 Global Step: 1960 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:14:29,451-Speed 128.62 samples/sec Loss 34.0270 Epoch: 0 Global Step: 1965 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:14:49,722-Speed 126.29 samples/sec Loss 34.1486 Epoch: 0 Global Step: 1970 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:15:09,561-Speed 129.05 samples/sec Loss 33.8312 Epoch: 0 Global Step: 1975 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:15:29,483-Speed 128.51 samples/sec Loss 33.9350 Epoch: 0 Global Step: 1980 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:15:49,276-Speed 129.35 samples/sec Loss 34.0078 Epoch: 0 Global Step: 1985 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:16:09,194-Speed 128.53 samples/sec Loss 33.9606 Epoch: 0 Global Step: 1990 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:16:28,754-Speed 130.89 samples/sec Loss 33.9312 Epoch: 0 Global Step: 1995 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:16:48,486-Speed 129.75 samples/sec Loss 33.7523 Epoch: 0 Global Step: 2000 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:17:07,998-Speed 131.21 samples/sec Loss 33.9490 Epoch: 0 Global Step: 2005 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:17:27,673-Speed 130.12 samples/sec Loss 33.8519 Epoch: 0 Global Step: 2010 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:17:47,430-Speed 129.59 samples/sec Loss 33.5267 Epoch: 0 Global Step: 2015 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:18:07,348-Speed 128.53 samples/sec Loss 33.5627 Epoch: 0 Global Step: 2020 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:18:26,919-Speed 130.82 samples/sec Loss 33.6089 Epoch: 0 Global Step: 2025 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:18:46,845-Speed 128.48 samples/sec Loss 33.3217 Epoch: 0 Global Step: 2030 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:19:06,597-Speed 129.62 samples/sec Loss 33.2747 Epoch: 0 Global Step: 2035 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:19:26,123-Speed 131.12 samples/sec Loss 33.6312 Epoch: 0 Global Step: 2040 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:19:46,002-Speed 128.79 samples/sec Loss 33.4506 Epoch: 0 Global Step: 2045 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:20:05,478-Speed 131.46 samples/sec Loss 33.3884 Epoch: 0 Global Step: 2050 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:20:25,385-Speed 128.61 samples/sec Loss 33.2665 Epoch: 0 Global Step: 2055 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:20:45,325-Speed 128.40 samples/sec Loss 33.4999 Epoch: 0 Global Step: 2060 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:21:05,214-Speed 128.72 samples/sec Loss 33.3289 Epoch: 0 Global Step: 2065 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:21:24,690-Speed 131.46 samples/sec Loss 33.4753 Epoch: 0 Global Step: 2070 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:21:44,481-Speed 129.36 samples/sec Loss 33.1172 Epoch: 0 Global Step: 2075 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:22:04,376-Speed 128.69 samples/sec Loss 33.0952 Epoch: 0 Global Step: 2080 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:22:23,886-Speed 131.22 samples/sec Loss 33.1633 Epoch: 0 Global Step: 2085 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:22:43,701-Speed 129.20 samples/sec Loss 33.2484 Epoch: 0 Global Step: 2090 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:23:03,367-Speed 130.18 samples/sec Loss 33.2397 Epoch: 0 Global Step: 2095 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:23:23,043-Speed 130.12 samples/sec Loss 32.6239 Epoch: 0 Global Step: 2100 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:23:42,860-Speed 129.19 samples/sec Loss 33.0252 Epoch: 0 Global Step: 2105 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:24:02,759-Speed 128.65 samples/sec Loss 33.0054 Epoch: 0 Global Step: 2110 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:24:22,149-Speed 132.03 samples/sec Loss 33.0591 Epoch: 0 Global Step: 2115 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:24:42,057-Speed 128.60 samples/sec Loss 32.6072 Epoch: 0 Global Step: 2120 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:25:01,949-Speed 128.70 samples/sec Loss 32.5393 Epoch: 0 Global Step: 2125 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:25:21,530-Speed 130.75 samples/sec Loss 32.4607 Epoch: 0 Global Step: 2130 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:25:41,448-Speed 128.54 samples/sec Loss 32.8401 Epoch: 0 Global Step: 2135 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:26:01,023-Speed 130.79 samples/sec Loss 32.4525 Epoch: 0 Global Step: 2140 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:26:20,864-Speed 129.04 samples/sec Loss 32.4108 Epoch: 0 Global Step: 2145 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:26:41,241-Speed 125.64 samples/sec Loss 32.3859 Epoch: 0 Global Step: 2150 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:27:00,951-Speed 129.89 samples/sec Loss 32.2780 Epoch: 0 Global Step: 2155 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:27:20,601-Speed 130.29 samples/sec Loss 32.2587 Epoch: 0 Global Step: 2160 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:27:40,424-Speed 129.16 samples/sec Loss 32.2888 Epoch: 0 Global Step: 2165 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:28:00,064-Speed 130.35 samples/sec Loss 32.2811 Epoch: 0 Global Step: 2170 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:28:19,586-Speed 131.15 samples/sec Loss 32.0806 Epoch: 0 Global Step: 2175 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:28:39,254-Speed 130.17 samples/sec Loss 32.1970 Epoch: 0 Global Step: 2180 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:28:58,841-Speed 130.71 samples/sec Loss 32.1036 Epoch: 0 Global Step: 2185 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:29:18,536-Speed 129.99 samples/sec Loss 32.3401 Epoch: 0 Global Step: 2190 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:29:38,303-Speed 129.52 samples/sec Loss 32.4214 Epoch: 0 Global Step: 2195 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:29:58,009-Speed 129.92 samples/sec Loss 31.9968 Epoch: 0 Global Step: 2200 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:30:17,598-Speed 130.69 samples/sec Loss 32.1452 Epoch: 0 Global Step: 2205 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:30:37,367-Speed 129.51 samples/sec Loss 32.0823 Epoch: 0 Global Step: 2210 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:30:57,261-Speed 128.69 samples/sec Loss 32.0704 Epoch: 0 Global Step: 2215 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:31:16,745-Speed 131.40 samples/sec Loss 31.9019 Epoch: 0 Global Step: 2220 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:31:36,533-Speed 129.38 samples/sec Loss 32.0166 Epoch: 0 Global Step: 2225 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:31:56,006-Speed 131.48 samples/sec Loss 31.9112 Epoch: 0 Global Step: 2230 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:32:15,535-Speed 131.09 samples/sec Loss 31.7365 Epoch: 0 Global Step: 2235 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:32:35,319-Speed 129.41 samples/sec Loss 31.6841 Epoch: 0 Global Step: 2240 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:32:55,119-Speed 129.30 samples/sec Loss 31.9138 Epoch: 0 Global Step: 2245 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:33:14,573-Speed 131.60 samples/sec Loss 31.6905 Epoch: 0 Global Step: 2250 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:33:34,424-Speed 128.97 samples/sec Loss 31.3066 Epoch: 0 Global Step: 2255 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:33:54,230-Speed 129.26 samples/sec Loss 31.6322 Epoch: 0 Global Step: 2260 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:34:13,702-Speed 131.48 samples/sec Loss 31.3220 Epoch: 0 Global Step: 2265 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:34:33,553-Speed 128.97 samples/sec Loss 31.6017 Epoch: 0 Global Step: 2270 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:34:53,058-Speed 131.26 samples/sec Loss 31.4507 Epoch: 0 Global Step: 2275 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:35:12,768-Speed 129.89 samples/sec Loss 30.8880 Epoch: 0 Global Step: 2280 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:35:32,488-Speed 129.83 samples/sec Loss 31.0851 Epoch: 0 Global Step: 2285 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:35:52,550-Speed 127.61 samples/sec Loss 31.0490 Epoch: 0 Global Step: 2290 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:36:11,907-Speed 132.26 samples/sec Loss 31.0178 Epoch: 0 Global Step: 2295 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:36:31,563-Speed 130.25 samples/sec Loss 30.9951 Epoch: 0 Global Step: 2300 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:36:51,347-Speed 129.41 samples/sec Loss 31.1517 Epoch: 0 Global Step: 2305 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:37:10,811-Speed 131.53 samples/sec Loss 31.1601 Epoch: 0 Global Step: 2310 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:37:30,673-Speed 128.90 samples/sec Loss 30.4872 Epoch: 0 Global Step: 2315 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:37:50,029-Speed 132.26 samples/sec Loss 30.9372 Epoch: 0 Global Step: 2320 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:38:09,768-Speed 129.70 samples/sec Loss 31.4334 Epoch: 0 Global Step: 2325 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:38:29,506-Speed 129.71 samples/sec Loss 31.4777 Epoch: 0 Global Step: 2330 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:38:49,225-Speed 129.83 samples/sec Loss 30.9457 Epoch: 0 Global Step: 2335 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:39:08,605-Speed 132.11 samples/sec Loss 31.2542 Epoch: 0 Global Step: 2340 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:39:28,505-Speed 128.65 samples/sec Loss 31.0076 Epoch: 0 Global Step: 2345 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:39:48,179-Speed 130.13 samples/sec Loss 30.6483 Epoch: 0 Global Step: 2350 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:40:07,587-Speed 131.91 samples/sec Loss 30.3011 Epoch: 0 Global Step: 2355 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:40:27,443-Speed 128.94 samples/sec Loss 30.8564 Epoch: 0 Global Step: 2360 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:40:46,836-Speed 132.02 samples/sec Loss 30.8792 Epoch: 0 Global Step: 2365 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:41:06,615-Speed 129.44 samples/sec Loss 30.4358 Epoch: 0 Global Step: 2370 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:41:26,302-Speed 130.05 samples/sec Loss 30.3507 Epoch: 0 Global Step: 2375 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:41:46,138-Speed 129.06 samples/sec Loss 30.5386 Epoch: 0 Global Step: 2380 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:42:05,504-Speed 132.20 samples/sec Loss 30.6524 Epoch: 0 Global Step: 2385 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:42:25,381-Speed 128.80 samples/sec Loss 30.3347 Epoch: 0 Global Step: 2390 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:42:45,092-Speed 129.89 samples/sec Loss 30.4101 Epoch: 0 Global Step: 2395 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:43:04,609-Speed 131.18 samples/sec Loss 30.7920 Epoch: 0 Global Step: 2400 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:43:24,426-Speed 129.19 samples/sec Loss 30.2569 Epoch: 0 Global Step: 2405 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:43:43,928-Speed 131.28 samples/sec Loss 30.1336 Epoch: 0 Global Step: 2410 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-06 21:44:03,619-Speed 130.02 samples/sec Loss 30.2732 Epoch: 0 Global Step: 2415 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:44:23,343-Speed 129.80 samples/sec Loss 30.0639 Epoch: 0 Global Step: 2420 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:44:43,133-Speed 129.36 samples/sec Loss 30.2979 Epoch: 0 Global Step: 2425 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:45:02,496-Speed 132.22 samples/sec Loss 29.6635 Epoch: 0 Global Step: 2430 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:45:22,230-Speed 129.74 samples/sec Loss 30.0190 Epoch: 0 Global Step: 2435 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:45:42,034-Speed 129.28 samples/sec Loss 29.9163 Epoch: 0 Global Step: 2440 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:46:01,588-Speed 130.93 samples/sec Loss 30.1342 Epoch: 0 Global Step: 2445 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:46:21,455-Speed 128.87 samples/sec Loss 29.8534 Epoch: 0 Global Step: 2450 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:46:40,897-Speed 131.68 samples/sec Loss 29.7834 Epoch: 0 Global Step: 2455 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:47:00,439-Speed 131.01 samples/sec Loss 29.8669 Epoch: 0 Global Step: 2460 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:47:20,172-Speed 129.74 samples/sec Loss 29.0428 Epoch: 0 Global Step: 2465 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:47:40,051-Speed 128.78 samples/sec Loss 29.3882 Epoch: 0 Global Step: 2470 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:47:59,515-Speed 131.54 samples/sec Loss 29.6293 Epoch: 0 Global Step: 2475 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:48:19,330-Speed 129.20 samples/sec Loss 29.6615 Epoch: 0 Global Step: 2480 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:48:39,158-Speed 129.12 samples/sec Loss 29.6596 Epoch: 0 Global Step: 2485 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:48:58,541-Speed 132.08 samples/sec Loss 29.4439 Epoch: 0 Global Step: 2490 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:49:18,433-Speed 128.71 samples/sec Loss 29.4246 Epoch: 0 Global Step: 2495 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:49:38,002-Speed 130.83 samples/sec Loss 28.9192 Epoch: 0 Global Step: 2500 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:49:57,679-Speed 130.11 samples/sec Loss 29.6504 Epoch: 0 Global Step: 2505 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:50:17,395-Speed 129.85 samples/sec Loss 29.4523 Epoch: 0 Global Step: 2510 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:50:37,175-Speed 129.43 samples/sec Loss 29.5203 Epoch: 0 Global Step: 2515 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:50:56,542-Speed 132.19 samples/sec Loss 29.3818 Epoch: 0 Global Step: 2520 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:51:16,338-Speed 129.33 samples/sec Loss 29.0418 Epoch: 0 Global Step: 2525 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:51:36,101-Speed 129.55 samples/sec Loss 29.4135 Epoch: 0 Global Step: 2530 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:51:55,561-Speed 131.56 samples/sec Loss 28.9446 Epoch: 0 Global Step: 2535 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:52:15,353-Speed 129.36 samples/sec Loss 29.0122 Epoch: 0 Global Step: 2540 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:52:34,885-Speed 131.07 samples/sec Loss 28.8863 Epoch: 0 Global Step: 2545 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:52:54,594-Speed 129.90 samples/sec Loss 29.0639 Epoch: 0 Global Step: 2550 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:53:14,322-Speed 129.77 samples/sec Loss 29.2194 Epoch: 0 Global Step: 2555 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:53:34,220-Speed 128.67 samples/sec Loss 28.8563 Epoch: 0 Global Step: 2560 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:53:53,573-Speed 132.28 samples/sec Loss 28.9460 Epoch: 0 Global Step: 2565 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:54:13,384-Speed 129.23 samples/sec Loss 28.5411 Epoch: 0 Global Step: 2570 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:54:33,150-Speed 129.53 samples/sec Loss 28.7918 Epoch: 0 Global Step: 2575 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:54:52,631-Speed 131.42 samples/sec Loss 29.0290 Epoch: 0 Global Step: 2580 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:55:12,462-Speed 129.10 samples/sec Loss 28.9697 Epoch: 0 Global Step: 2585 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:55:31,933-Speed 131.49 samples/sec Loss 28.4505 Epoch: 0 Global Step: 2590 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:55:51,796-Speed 128.89 samples/sec Loss 28.6060 Epoch: 0 Global Step: 2595 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:56:11,521-Speed 129.80 samples/sec Loss 28.9880 Epoch: 0 Global Step: 2600 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:56:31,387-Speed 128.87 samples/sec Loss 28.3364 Epoch: 0 Global Step: 2605 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:56:50,744-Speed 132.26 samples/sec Loss 28.6037 Epoch: 0 Global Step: 2610 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:57:10,635-Speed 128.71 samples/sec Loss 28.6609 Epoch: 0 Global Step: 2615 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:57:30,254-Speed 130.49 samples/sec Loss 28.5082 Epoch: 0 Global Step: 2620 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:57:49,806-Speed 130.94 samples/sec Loss 28.2843 Epoch: 0 Global Step: 2625 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:58:09,404-Speed 130.64 samples/sec Loss 28.5665 Epoch: 0 Global Step: 2630 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:58:28,852-Speed 131.64 samples/sec Loss 28.1977 Epoch: 0 Global Step: 2635 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:58:48,675-Speed 129.15 samples/sec Loss 28.4251 Epoch: 0 Global Step: 2640 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:59:08,546-Speed 128.84 samples/sec Loss 28.3419 Epoch: 0 Global Step: 2645 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:59:28,153-Speed 130.57 samples/sec Loss 28.1080 Epoch: 0 Global Step: 2650 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 21:59:47,517-Speed 132.22 samples/sec Loss 28.5029 Epoch: 0 Global Step: 2655 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:00:07,368-Speed 128.97 samples/sec Loss 28.0115 Epoch: 0 Global Step: 2660 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:00:27,060-Speed 130.01 samples/sec Loss 28.2770 Epoch: 0 Global Step: 2665 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:00:46,720-Speed 130.23 samples/sec Loss 28.1859 Epoch: 0 Global Step: 2670 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:01:06,883-Speed 126.97 samples/sec Loss 28.3880 Epoch: 0 Global Step: 2675 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:01:26,367-Speed 131.40 samples/sec Loss 27.7703 Epoch: 0 Global Step: 2680 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:01:46,192-Speed 129.14 samples/sec Loss 27.9739 Epoch: 0 Global Step: 2685 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:02:06,106-Speed 128.56 samples/sec Loss 28.2308 Epoch: 0 Global Step: 2690 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:02:25,649-Speed 131.00 samples/sec Loss 28.4539 Epoch: 0 Global Step: 2695 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:02:45,121-Speed 131.48 samples/sec Loss 27.9357 Epoch: 0 Global Step: 2700 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:03:04,954-Speed 129.08 samples/sec Loss 27.7356 Epoch: 0 Global Step: 2705 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:03:24,555-Speed 130.61 samples/sec Loss 27.9663 Epoch: 0 Global Step: 2710 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:03:44,117-Speed 130.88 samples/sec Loss 27.6513 Epoch: 0 Global Step: 2715 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:04:03,749-Speed 130.41 samples/sec Loss 27.9705 Epoch: 0 Global Step: 2720 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:04:23,217-Speed 131.51 samples/sec Loss 27.8238 Epoch: 0 Global Step: 2725 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:04:43,069-Speed 128.97 samples/sec Loss 27.8725 Epoch: 0 Global Step: 2730 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:05:02,885-Speed 129.20 samples/sec Loss 27.4407 Epoch: 0 Global Step: 2735 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:05:22,413-Speed 131.10 samples/sec Loss 27.5520 Epoch: 0 Global Step: 2740 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:05:41,957-Speed 130.99 samples/sec Loss 27.7766 Epoch: 0 Global Step: 2745 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:06:01,918-Speed 128.26 samples/sec Loss 27.5084 Epoch: 0 Global Step: 2750 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:06:21,429-Speed 131.22 samples/sec Loss 27.1958 Epoch: 0 Global Step: 2755 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:06:40,976-Speed 130.97 samples/sec Loss 27.4023 Epoch: 0 Global Step: 2760 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:07:00,644-Speed 130.17 samples/sec Loss 27.4665 Epoch: 0 Global Step: 2765 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:07:20,164-Speed 131.15 samples/sec Loss 27.0987 Epoch: 0 Global Step: 2770 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:07:39,839-Speed 130.13 samples/sec Loss 27.2341 Epoch: 0 Global Step: 2775 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:07:59,733-Speed 128.69 samples/sec Loss 27.7150 Epoch: 0 Global Step: 2780 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:08:19,378-Speed 130.32 samples/sec Loss 27.3417 Epoch: 0 Global Step: 2785 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:08:39,038-Speed 130.22 samples/sec Loss 27.0911 Epoch: 0 Global Step: 2790 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:08:58,726-Speed 130.04 samples/sec Loss 27.1219 Epoch: 0 Global Step: 2795 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:09:18,292-Speed 130.85 samples/sec Loss 27.2828 Epoch: 0 Global Step: 2800 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:09:37,810-Speed 131.18 samples/sec Loss 26.9936 Epoch: 0 Global Step: 2805 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:09:57,485-Speed 130.12 samples/sec Loss 27.2720 Epoch: 0 Global Step: 2810 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:10:17,035-Speed 130.96 samples/sec Loss 27.0859 Epoch: 0 Global Step: 2815 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:10:36,562-Speed 131.11 samples/sec Loss 27.0474 Epoch: 0 Global Step: 2820 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:10:56,327-Speed 129.53 samples/sec Loss 26.2849 Epoch: 0 Global Step: 2825 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:11:16,009-Speed 130.08 samples/sec Loss 27.1210 Epoch: 0 Global Step: 2830 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:11:35,687-Speed 130.10 samples/sec Loss 26.8028 Epoch: 0 Global Step: 2835 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:11:55,385-Speed 129.97 samples/sec Loss 27.1564 Epoch: 0 Global Step: 2840 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:12:14,984-Speed 130.63 samples/sec Loss 26.8970 Epoch: 0 Global Step: 2845 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:12:34,495-Speed 131.21 samples/sec Loss 26.5723 Epoch: 0 Global Step: 2850 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:12:54,174-Speed 130.10 samples/sec Loss 26.6762 Epoch: 0 Global Step: 2855 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:13:13,736-Speed 130.88 samples/sec Loss 27.0440 Epoch: 0 Global Step: 2860 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:13:33,249-Speed 131.20 samples/sec Loss 27.0627 Epoch: 0 Global Step: 2865 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:13:53,017-Speed 129.51 samples/sec Loss 26.5252 Epoch: 0 Global Step: 2870 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:14:12,748-Speed 129.76 samples/sec Loss 26.6728 Epoch: 0 Global Step: 2875 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:14:32,321-Speed 130.80 samples/sec Loss 26.7898 Epoch: 0 Global Step: 2880 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:14:51,931-Speed 130.55 samples/sec Loss 26.6902 Epoch: 0 Global Step: 2885 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:15:11,790-Speed 128.92 samples/sec Loss 26.2821 Epoch: 0 Global Step: 2890 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:15:31,241-Speed 131.62 samples/sec Loss 25.8830 Epoch: 0 Global Step: 2895 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:15:50,980-Speed 129.70 samples/sec Loss 26.0719 Epoch: 0 Global Step: 2900 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:16:10,496-Speed 131.18 samples/sec Loss 25.9948 Epoch: 0 Global Step: 2905 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:16:30,037-Speed 131.02 samples/sec Loss 26.2739 Epoch: 0 Global Step: 2910 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:16:49,794-Speed 129.58 samples/sec Loss 26.4546 Epoch: 0 Global Step: 2915 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:17:09,572-Speed 129.44 samples/sec Loss 26.5207 Epoch: 0 Global Step: 2920 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:17:29,066-Speed 131.33 samples/sec Loss 26.5872 Epoch: 0 Global Step: 2925 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:17:48,603-Speed 131.03 samples/sec Loss 26.4779 Epoch: 0 Global Step: 2930 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:18:08,425-Speed 129.16 samples/sec Loss 26.2478 Epoch: 0 Global Step: 2935 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:18:27,871-Speed 131.66 samples/sec Loss 26.0020 Epoch: 0 Global Step: 2940 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:18:47,965-Speed 127.41 samples/sec Loss 26.1404 Epoch: 0 Global Step: 2945 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:19:07,465-Speed 131.29 samples/sec Loss 26.0673 Epoch: 0 Global Step: 2950 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:19:27,184-Speed 129.83 samples/sec Loss 26.4639 Epoch: 0 Global Step: 2955 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:19:46,931-Speed 129.65 samples/sec Loss 26.1769 Epoch: 0 Global Step: 2960 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:20:06,805-Speed 128.82 samples/sec Loss 26.2136 Epoch: 0 Global Step: 2965 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:20:26,167-Speed 132.23 samples/sec Loss 26.2250 Epoch: 0 Global Step: 2970 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-06 22:20:45,782-Speed 130.52 samples/sec Loss 25.8074 Epoch: 0 Global Step: 2975 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:21:05,595-Speed 129.22 samples/sec Loss 26.0440 Epoch: 0 Global Step: 2980 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:21:24,977-Speed 132.09 samples/sec Loss 25.5589 Epoch: 0 Global Step: 2985 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:21:44,826-Speed 128.99 samples/sec Loss 25.3450 Epoch: 0 Global Step: 2990 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:22:04,275-Speed 131.63 samples/sec Loss 25.6822 Epoch: 0 Global Step: 2995 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:22:23,945-Speed 130.16 samples/sec Loss 25.7979 Epoch: 0 Global Step: 3000 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:22:43,733-Speed 129.38 samples/sec Loss 25.7224 Epoch: 0 Global Step: 3005 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:23:03,624-Speed 128.71 samples/sec Loss 25.4864 Epoch: 0 Global Step: 3010 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:23:23,116-Speed 131.35 samples/sec Loss 25.9284 Epoch: 0 Global Step: 3015 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:23:42,851-Speed 129.72 samples/sec Loss 25.3214 Epoch: 0 Global Step: 3020 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:24:02,660-Speed 129.24 samples/sec Loss 25.5613 Epoch: 0 Global Step: 3025 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:24:22,155-Speed 131.32 samples/sec Loss 25.5359 Epoch: 0 Global Step: 3030 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:24:42,008-Speed 128.96 samples/sec Loss 25.5060 Epoch: 0 Global Step: 3035 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:25:01,361-Speed 132.29 samples/sec Loss 25.7345 Epoch: 0 Global Step: 3040 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:25:21,195-Speed 129.08 samples/sec Loss 25.5898 Epoch: 0 Global Step: 3045 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:25:40,931-Speed 129.72 samples/sec Loss 25.3738 Epoch: 0 Global Step: 3050 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:26:00,800-Speed 128.85 samples/sec Loss 25.3572 Epoch: 0 Global Step: 3055 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:26:20,158-Speed 132.25 samples/sec Loss 25.4715 Epoch: 0 Global Step: 3060 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:26:39,784-Speed 130.45 samples/sec Loss 25.1558 Epoch: 0 Global Step: 3065 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:26:59,598-Speed 129.21 samples/sec Loss 25.3385 Epoch: 0 Global Step: 3070 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:27:18,999-Speed 131.96 samples/sec Loss 25.3185 Epoch: 0 Global Step: 3075 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:27:38,858-Speed 128.92 samples/sec Loss 25.3603 Epoch: 0 Global Step: 3080 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:27:58,284-Speed 131.79 samples/sec Loss 25.3293 Epoch: 0 Global Step: 3085 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:28:17,953-Speed 130.16 samples/sec Loss 25.5298 Epoch: 0 Global Step: 3090 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:28:37,680-Speed 129.78 samples/sec Loss 25.3059 Epoch: 0 Global Step: 3095 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:28:57,530-Speed 128.97 samples/sec Loss 25.0565 Epoch: 0 Global Step: 3100 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:29:16,889-Speed 132.25 samples/sec Loss 25.0186 Epoch: 0 Global Step: 3105 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:29:36,736-Speed 128.99 samples/sec Loss 25.4314 Epoch: 0 Global Step: 3110 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:29:56,421-Speed 130.05 samples/sec Loss 25.2098 Epoch: 0 Global Step: 3115 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:30:15,897-Speed 131.46 samples/sec Loss 24.6634 Epoch: 0 Global Step: 3120 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:30:35,655-Speed 129.58 samples/sec Loss 25.4259 Epoch: 0 Global Step: 3125 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:30:55,108-Speed 131.60 samples/sec Loss 24.6225 Epoch: 0 Global Step: 3130 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:31:14,815-Speed 129.91 samples/sec Loss 24.9662 Epoch: 0 Global Step: 3135 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:31:34,561-Speed 129.66 samples/sec Loss 24.9526 Epoch: 0 Global Step: 3140 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:31:54,444-Speed 128.76 samples/sec Loss 24.9562 Epoch: 0 Global Step: 3145 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:32:13,805-Speed 132.23 samples/sec Loss 24.9641 Epoch: 0 Global Step: 3150 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:32:33,592-Speed 129.38 samples/sec Loss 24.4995 Epoch: 0 Global Step: 3155 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:32:53,226-Speed 130.40 samples/sec Loss 24.8766 Epoch: 0 Global Step: 3160 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:33:12,711-Speed 131.39 samples/sec Loss 24.5802 Epoch: 0 Global Step: 3165 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:33:32,476-Speed 129.53 samples/sec Loss 24.9513 Epoch: 0 Global Step: 3170 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:33:51,859-Speed 132.09 samples/sec Loss 24.4512 Epoch: 0 Global Step: 3175 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:34:11,746-Speed 128.74 samples/sec Loss 24.8337 Epoch: 0 Global Step: 3180 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:34:31,460-Speed 129.87 samples/sec Loss 24.3862 Epoch: 0 Global Step: 3185 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:34:51,279-Speed 129.17 samples/sec Loss 24.6263 Epoch: 0 Global Step: 3190 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:35:10,654-Speed 132.14 samples/sec Loss 24.5621 Epoch: 0 Global Step: 3195 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:35:30,393-Speed 129.70 samples/sec Loss 24.4385 Epoch: 0 Global Step: 3200 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:35:50,068-Speed 130.12 samples/sec Loss 24.5000 Epoch: 0 Global Step: 3205 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:36:09,479-Speed 131.90 samples/sec Loss 24.2545 Epoch: 0 Global Step: 3210 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:36:29,343-Speed 128.89 samples/sec Loss 24.3669 Epoch: 0 Global Step: 3215 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:36:48,694-Speed 132.30 samples/sec Loss 24.6039 Epoch: 0 Global Step: 3220 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:37:08,565-Speed 128.84 samples/sec Loss 24.5500 Epoch: 0 Global Step: 3225 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:37:28,277-Speed 129.88 samples/sec Loss 24.8572 Epoch: 0 Global Step: 3230 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:37:48,116-Speed 129.05 samples/sec Loss 24.4131 Epoch: 0 Global Step: 3235 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:38:07,475-Speed 132.24 samples/sec Loss 24.3856 Epoch: 0 Global Step: 3240 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:38:27,272-Speed 129.32 samples/sec Loss 24.3138 Epoch: 0 Global Step: 3245 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:38:46,875-Speed 130.60 samples/sec Loss 24.3731 Epoch: 0 Global Step: 3250 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:39:06,336-Speed 131.55 samples/sec Loss 24.1384 Epoch: 0 Global Step: 3255 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:39:26,060-Speed 129.80 samples/sec Loss 23.8457 Epoch: 0 Global Step: 3260 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:39:45,563-Speed 131.27 samples/sec Loss 23.9992 Epoch: 0 Global Step: 3265 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:40:07,430-Speed 117.08 samples/sec Loss 24.1239 Epoch: 0 Global Step: 3270 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:40:27,761-Speed 125.92 samples/sec Loss 23.9984 Epoch: 0 Global Step: 3275 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:40:47,656-Speed 128.68 samples/sec Loss 23.9507 Epoch: 0 Global Step: 3280 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:41:07,017-Speed 132.23 samples/sec Loss 24.0078 Epoch: 0 Global Step: 3285 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:41:26,794-Speed 129.45 samples/sec Loss 23.6197 Epoch: 0 Global Step: 3290 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:41:46,606-Speed 129.23 samples/sec Loss 24.1935 Epoch: 0 Global Step: 3295 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:42:06,023-Speed 131.85 samples/sec Loss 24.4753 Epoch: 0 Global Step: 3300 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:42:25,879-Speed 128.93 samples/sec Loss 24.3145 Epoch: 0 Global Step: 3305 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:42:45,301-Speed 131.82 samples/sec Loss 23.6974 Epoch: 0 Global Step: 3310 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:43:04,854-Speed 130.94 samples/sec Loss 24.0819 Epoch: 0 Global Step: 3315 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:43:24,577-Speed 129.81 samples/sec Loss 23.7117 Epoch: 0 Global Step: 3320 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:43:44,399-Speed 129.15 samples/sec Loss 24.0934 Epoch: 0 Global Step: 3325 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:44:03,927-Speed 131.11 samples/sec Loss 23.7045 Epoch: 0 Global Step: 3330 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:44:23,764-Speed 129.06 samples/sec Loss 24.1925 Epoch: 0 Global Step: 3335 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:44:43,423-Speed 130.23 samples/sec Loss 23.7358 Epoch: 0 Global Step: 3340 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:45:02,875-Speed 131.61 samples/sec Loss 23.7111 Epoch: 0 Global Step: 3345 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:45:22,731-Speed 128.94 samples/sec Loss 23.6442 Epoch: 0 Global Step: 3350 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:45:42,107-Speed 132.13 samples/sec Loss 23.5264 Epoch: 0 Global Step: 3355 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:46:01,957-Speed 128.97 samples/sec Loss 23.5986 Epoch: 0 Global Step: 3360 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:46:21,657-Speed 129.96 samples/sec Loss 23.8534 Epoch: 0 Global Step: 3365 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:46:41,504-Speed 129.00 samples/sec Loss 23.7496 Epoch: 0 Global Step: 3370 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:47:00,852-Speed 132.32 samples/sec Loss 23.8364 Epoch: 0 Global Step: 3375 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:47:20,593-Speed 129.68 samples/sec Loss 23.5222 Epoch: 0 Global Step: 3380 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:47:40,280-Speed 130.04 samples/sec Loss 23.8872 Epoch: 0 Global Step: 3385 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:47:59,724-Speed 131.67 samples/sec Loss 23.8862 Epoch: 0 Global Step: 3390 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:48:19,548-Speed 129.15 samples/sec Loss 23.6524 Epoch: 0 Global Step: 3395 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:48:38,913-Speed 132.21 samples/sec Loss 23.6539 Epoch: 0 Global Step: 3400 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:48:58,655-Speed 129.68 samples/sec Loss 23.4656 Epoch: 0 Global Step: 3405 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:49:18,351-Speed 129.98 samples/sec Loss 23.7661 Epoch: 0 Global Step: 3410 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:49:38,255-Speed 128.63 samples/sec Loss 23.6914 Epoch: 0 Global Step: 3415 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:49:57,601-Speed 132.33 samples/sec Loss 23.3059 Epoch: 0 Global Step: 3420 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:50:17,364-Speed 129.54 samples/sec Loss 23.1792 Epoch: 0 Global Step: 3425 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:50:37,046-Speed 130.08 samples/sec Loss 23.3060 Epoch: 0 Global Step: 3430 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:50:56,518-Speed 131.48 samples/sec Loss 23.4860 Epoch: 0 Global Step: 3435 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:51:16,309-Speed 129.36 samples/sec Loss 23.1875 Epoch: 0 Global Step: 3440 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:51:35,668-Speed 132.24 samples/sec Loss 23.3301 Epoch: 0 Global Step: 3445 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:51:55,448-Speed 129.43 samples/sec Loss 23.6470 Epoch: 0 Global Step: 3450 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:52:15,145-Speed 129.98 samples/sec Loss 22.9506 Epoch: 0 Global Step: 3455 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:52:35,026-Speed 128.77 samples/sec Loss 23.1527 Epoch: 0 Global Step: 3460 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:52:54,372-Speed 132.34 samples/sec Loss 23.5978 Epoch: 0 Global Step: 3465 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:53:14,193-Speed 129.16 samples/sec Loss 23.2616 Epoch: 0 Global Step: 3470 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:53:33,871-Speed 130.10 samples/sec Loss 23.0466 Epoch: 0 Global Step: 3475 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:53:53,329-Speed 131.57 samples/sec Loss 23.0748 Epoch: 0 Global Step: 3480 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:54:13,426-Speed 127.39 samples/sec Loss 23.0191 Epoch: 0 Global Step: 3485 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:54:32,866-Speed 131.71 samples/sec Loss 23.1202 Epoch: 0 Global Step: 3490 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:54:52,651-Speed 129.40 samples/sec Loss 23.3209 Epoch: 0 Global Step: 3495 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:55:12,334-Speed 130.07 samples/sec Loss 22.9317 Epoch: 0 Global Step: 3500 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:55:32,207-Speed 128.83 samples/sec Loss 22.5868 Epoch: 0 Global Step: 3505 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:55:51,575-Speed 132.18 samples/sec Loss 23.2531 Epoch: 0 Global Step: 3510 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:56:11,517-Speed 128.38 samples/sec Loss 23.3825 Epoch: 0 Global Step: 3515 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:56:31,211-Speed 130.00 samples/sec Loss 22.8371 Epoch: 0 Global Step: 3520 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:56:50,621-Speed 131.90 samples/sec Loss 23.5569 Epoch: 0 Global Step: 3525 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:57:10,457-Speed 129.07 samples/sec Loss 22.8274 Epoch: 0 Global Step: 3530 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:57:29,926-Speed 131.50 samples/sec Loss 22.7246 Epoch: 0 Global Step: 3535 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:57:49,615-Speed 130.03 samples/sec Loss 23.0204 Epoch: 0 Global Step: 3540 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:58:09,339-Speed 129.80 samples/sec Loss 22.9381 Epoch: 0 Global Step: 3545 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:58:29,223-Speed 128.76 samples/sec Loss 23.0186 Epoch: 0 Global Step: 3550 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:58:48,588-Speed 132.20 samples/sec Loss 22.8402 Epoch: 0 Global Step: 3555 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:59:08,571-Speed 128.12 samples/sec Loss 22.6675 Epoch: 0 Global Step: 3560 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:59:28,248-Speed 130.11 samples/sec Loss 22.6045 Epoch: 0 Global Step: 3565 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 22:59:47,647-Speed 131.98 samples/sec Loss 22.9317 Epoch: 0 Global Step: 3570 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:00:07,460-Speed 129.21 samples/sec Loss 22.8217 Epoch: 0 Global Step: 3575 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:00:26,899-Speed 131.70 samples/sec Loss 22.7842 Epoch: 0 Global Step: 3580 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:00:46,761-Speed 128.90 samples/sec Loss 22.9687 Epoch: 0 Global Step: 3585 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:01:06,469-Speed 129.91 samples/sec Loss 22.5787 Epoch: 0 Global Step: 3590 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:01:26,307-Speed 129.06 samples/sec Loss 22.8859 Epoch: 0 Global Step: 3595 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:01:45,662-Speed 132.27 samples/sec Loss 22.7022 Epoch: 0 Global Step: 3600 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:02:05,526-Speed 128.88 samples/sec Loss 22.7243 Epoch: 0 Global Step: 3605 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:02:25,184-Speed 130.23 samples/sec Loss 22.1877 Epoch: 0 Global Step: 3610 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:02:44,652-Speed 131.51 samples/sec Loss 22.2048 Epoch: 0 Global Step: 3615 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:03:04,372-Speed 129.83 samples/sec Loss 22.4179 Epoch: 0 Global Step: 3620 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:03:23,846-Speed 131.47 samples/sec Loss 22.6952 Epoch: 0 Global Step: 3625 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:03:43,721-Speed 128.81 samples/sec Loss 22.3798 Epoch: 0 Global Step: 3630 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:04:03,433-Speed 129.88 samples/sec Loss 22.1869 Epoch: 0 Global Step: 3635 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:04:23,253-Speed 129.16 samples/sec Loss 22.4329 Epoch: 0 Global Step: 3640 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-06 23:04:42,598-Speed 132.34 samples/sec Loss 22.2731 Epoch: 0 Global Step: 3645 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:05:02,463-Speed 128.88 samples/sec Loss 22.5740 Epoch: 0 Global Step: 3650 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:05:22,060-Speed 130.64 samples/sec Loss 22.4745 Epoch: 0 Global Step: 3655 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:05:41,601-Speed 131.02 samples/sec Loss 22.6608 Epoch: 0 Global Step: 3660 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:06:01,125-Speed 131.13 samples/sec Loss 22.0895 Epoch: 0 Global Step: 3665 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:06:20,611-Speed 131.39 samples/sec Loss 22.8183 Epoch: 0 Global Step: 3670 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:06:40,464-Speed 128.96 samples/sec Loss 22.3245 Epoch: 0 Global Step: 3675 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:07:00,159-Speed 129.99 samples/sec Loss 22.4887 Epoch: 0 Global Step: 3680 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:07:20,004-Speed 129.01 samples/sec Loss 22.0877 Epoch: 0 Global Step: 3685 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:07:39,377-Speed 132.15 samples/sec Loss 22.2376 Epoch: 0 Global Step: 3690 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:07:59,247-Speed 128.85 samples/sec Loss 21.9149 Epoch: 0 Global Step: 3695 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:08:18,908-Speed 130.21 samples/sec Loss 22.3657 Epoch: 0 Global Step: 3700 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:08:38,431-Speed 131.14 samples/sec Loss 22.3400 Epoch: 0 Global Step: 3705 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:08:58,130-Speed 129.96 samples/sec Loss 22.1625 Epoch: 0 Global Step: 3710 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:09:17,560-Speed 131.77 samples/sec Loss 22.0482 Epoch: 0 Global Step: 3715 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:09:37,414-Speed 128.95 samples/sec Loss 22.2377 Epoch: 0 Global Step: 3720 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:09:57,139-Speed 129.79 samples/sec Loss 22.4413 Epoch: 0 Global Step: 3725 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:10:16,932-Speed 129.35 samples/sec Loss 22.0266 Epoch: 0 Global Step: 3730 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:10:36,302-Speed 132.17 samples/sec Loss 22.0972 Epoch: 0 Global Step: 3735 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:10:56,018-Speed 129.85 samples/sec Loss 22.5084 Epoch: 0 Global Step: 3740 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:11:15,678-Speed 130.22 samples/sec Loss 22.7772 Epoch: 0 Global Step: 3745 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:11:35,241-Speed 130.87 samples/sec Loss 21.9429 Epoch: 0 Global Step: 3750 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:11:54,952-Speed 129.89 samples/sec Loss 22.0653 Epoch: 0 Global Step: 3755 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:12:14,358-Speed 131.93 samples/sec Loss 22.2502 Epoch: 0 Global Step: 3760 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:12:34,191-Speed 129.09 samples/sec Loss 22.3614 Epoch: 0 Global Step: 3765 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:12:53,953-Speed 129.56 samples/sec Loss 21.6950 Epoch: 0 Global Step: 3770 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:13:13,646-Speed 130.00 samples/sec Loss 21.7432 Epoch: 0 Global Step: 3775 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:13:33,124-Speed 131.44 samples/sec Loss 21.7245 Epoch: 0 Global Step: 3780 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:13:52,946-Speed 129.16 samples/sec Loss 21.8542 Epoch: 0 Global Step: 3785 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:14:12,474-Speed 131.10 samples/sec Loss 22.0483 Epoch: 0 Global Step: 3790 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:14:32,136-Speed 130.21 samples/sec Loss 21.8817 Epoch: 0 Global Step: 3795 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:14:51,780-Speed 130.33 samples/sec Loss 21.7425 Epoch: 0 Global Step: 3800 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:15:11,381-Speed 130.61 samples/sec Loss 21.9052 Epoch: 0 Global Step: 3805 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:15:31,212-Speed 129.10 samples/sec Loss 21.7636 Epoch: 0 Global Step: 3810 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:15:50,999-Speed 129.39 samples/sec Loss 21.9478 Epoch: 0 Global Step: 3815 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:16:10,680-Speed 130.08 samples/sec Loss 21.7767 Epoch: 0 Global Step: 3820 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:16:30,167-Speed 131.38 samples/sec Loss 22.0264 Epoch: 0 Global Step: 3825 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:16:49,930-Speed 129.54 samples/sec Loss 22.0241 Epoch: 0 Global Step: 3830 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:17:09,593-Speed 130.20 samples/sec Loss 21.6589 Epoch: 0 Global Step: 3835 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:17:29,160-Speed 130.84 samples/sec Loss 21.6892 Epoch: 0 Global Step: 3840 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:17:48,831-Speed 130.15 samples/sec Loss 21.6373 Epoch: 0 Global Step: 3845 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:18:08,277-Speed 131.65 samples/sec Loss 22.2580 Epoch: 0 Global Step: 3850 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:18:28,028-Speed 129.62 samples/sec Loss 21.3352 Epoch: 0 Global Step: 3855 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:18:48,248-Speed 126.61 samples/sec Loss 21.5161 Epoch: 0 Global Step: 3860 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:19:07,928-Speed 130.09 samples/sec Loss 21.7350 Epoch: 0 Global Step: 3865 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:19:27,451-Speed 131.14 samples/sec Loss 21.5979 Epoch: 0 Global Step: 3870 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:19:47,270-Speed 129.18 samples/sec Loss 21.1954 Epoch: 0 Global Step: 3875 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:20:06,973-Speed 129.94 samples/sec Loss 21.4660 Epoch: 0 Global Step: 3880 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:20:26,515-Speed 131.00 samples/sec Loss 21.5531 Epoch: 0 Global Step: 3885 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:20:46,156-Speed 130.35 samples/sec Loss 21.7685 Epoch: 0 Global Step: 3890 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:21:05,750-Speed 130.66 samples/sec Loss 21.5433 Epoch: 0 Global Step: 3895 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:21:25,532-Speed 129.42 samples/sec Loss 21.2705 Epoch: 0 Global Step: 3900 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:21:45,648-Speed 127.27 samples/sec Loss 21.6346 Epoch: 0 Global Step: 3905 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:22:05,359-Speed 129.88 samples/sec Loss 21.6417 Epoch: 0 Global Step: 3910 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:22:24,947-Speed 130.70 samples/sec Loss 21.0793 Epoch: 0 Global Step: 3915 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:22:44,587-Speed 130.36 samples/sec Loss 21.4560 Epoch: 0 Global Step: 3920 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:23:04,477-Speed 128.72 samples/sec Loss 21.1447 Epoch: 0 Global Step: 3925 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:23:23,874-Speed 131.99 samples/sec Loss 21.1854 Epoch: 0 Global Step: 3930 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:23:43,583-Speed 129.90 samples/sec Loss 21.3961 Epoch: 0 Global Step: 3935 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:24:03,078-Speed 131.32 samples/sec Loss 21.4164 Epoch: 0 Global Step: 3940 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:24:22,751-Speed 130.14 samples/sec Loss 20.6898 Epoch: 0 Global Step: 3945 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:24:42,513-Speed 129.55 samples/sec Loss 21.1689 Epoch: 0 Global Step: 3950 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:25:02,315-Speed 129.29 samples/sec Loss 21.4017 Epoch: 0 Global Step: 3955 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:25:21,865-Speed 130.95 samples/sec Loss 21.4235 Epoch: 0 Global Step: 3960 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:25:41,517-Speed 130.27 samples/sec Loss 21.4607 Epoch: 0 Global Step: 3965 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:26:01,378-Speed 128.90 samples/sec Loss 21.5897 Epoch: 0 Global Step: 3970 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:26:20,811-Speed 131.75 samples/sec Loss 21.2080 Epoch: 0 Global Step: 3975 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:26:40,504-Speed 130.00 samples/sec Loss 21.2247 Epoch: 0 Global Step: 3980 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:27:00,004-Speed 131.29 samples/sec Loss 21.0763 Epoch: 0 Global Step: 3985 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:27:19,681-Speed 130.11 samples/sec Loss 21.1814 Epoch: 0 Global Step: 3990 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:27:39,426-Speed 129.66 samples/sec Loss 21.0329 Epoch: 0 Global Step: 3995 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:27:59,117-Speed 130.01 samples/sec Loss 21.0434 Epoch: 0 Global Step: 4000 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:28:18,656-Speed 131.03 samples/sec Loss 21.1989 Epoch: 0 Global Step: 4005 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:28:38,288-Speed 130.40 samples/sec Loss 20.6253 Epoch: 0 Global Step: 4010 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:28:58,151-Speed 128.89 samples/sec Loss 20.9863 Epoch: 0 Global Step: 4015 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:29:17,593-Speed 131.68 samples/sec Loss 21.0779 Epoch: 0 Global Step: 4020 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:29:37,272-Speed 130.09 samples/sec Loss 21.0342 Epoch: 0 Global Step: 4025 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:29:56,784-Speed 131.21 samples/sec Loss 20.9449 Epoch: 0 Global Step: 4030 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:30:16,537-Speed 129.61 samples/sec Loss 21.2559 Epoch: 0 Global Step: 4035 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:30:36,292-Speed 129.60 samples/sec Loss 21.0394 Epoch: 0 Global Step: 4040 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:30:56,252-Speed 128.26 samples/sec Loss 20.9243 Epoch: 0 Global Step: 4045 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:31:15,633-Speed 132.10 samples/sec Loss 20.8155 Epoch: 0 Global Step: 4050 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:31:35,402-Speed 129.50 samples/sec Loss 21.4847 Epoch: 0 Global Step: 4055 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:31:55,226-Speed 129.14 samples/sec Loss 20.9036 Epoch: 0 Global Step: 4060 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:32:14,598-Speed 132.16 samples/sec Loss 20.7046 Epoch: 0 Global Step: 4065 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:32:34,318-Speed 129.82 samples/sec Loss 20.8945 Epoch: 0 Global Step: 4070 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:32:53,781-Speed 131.54 samples/sec Loss 21.4742 Epoch: 0 Global Step: 4075 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:33:13,456-Speed 130.13 samples/sec Loss 20.9793 Epoch: 0 Global Step: 4080 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:33:33,209-Speed 129.61 samples/sec Loss 20.8173 Epoch: 0 Global Step: 4085 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:33:53,050-Speed 129.03 samples/sec Loss 21.0362 Epoch: 0 Global Step: 4090 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:34:12,405-Speed 132.28 samples/sec Loss 20.9794 Epoch: 0 Global Step: 4095 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:34:32,132-Speed 129.78 samples/sec Loss 20.8402 Epoch: 0 Global Step: 4100 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:34:51,950-Speed 129.18 samples/sec Loss 20.6376 Epoch: 0 Global Step: 4105 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:35:11,299-Speed 132.32 samples/sec Loss 20.7697 Epoch: 0 Global Step: 4110 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:35:31,170-Speed 128.84 samples/sec Loss 20.8064 Epoch: 0 Global Step: 4115 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:35:50,622-Speed 131.62 samples/sec Loss 20.9412 Epoch: 0 Global Step: 4120 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:36:10,311-Speed 130.03 samples/sec Loss 20.8873 Epoch: 0 Global Step: 4125 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:36:30,035-Speed 129.79 samples/sec Loss 20.8641 Epoch: 0 Global Step: 4130 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:36:50,027-Speed 128.06 samples/sec Loss 20.7693 Epoch: 0 Global Step: 4135 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:37:09,388-Speed 132.23 samples/sec Loss 20.8749 Epoch: 0 Global Step: 4140 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:37:29,186-Speed 129.32 samples/sec Loss 20.7297 Epoch: 0 Global Step: 4145 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:37:48,897-Speed 129.88 samples/sec Loss 21.0709 Epoch: 0 Global Step: 4150 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:38:08,394-Speed 131.32 samples/sec Loss 20.8461 Epoch: 0 Global Step: 4155 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:38:28,240-Speed 129.00 samples/sec Loss 20.2775 Epoch: 0 Global Step: 4160 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:38:47,631-Speed 132.03 samples/sec Loss 20.6064 Epoch: 0 Global Step: 4165 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:39:07,334-Speed 129.94 samples/sec Loss 20.2206 Epoch: 0 Global Step: 4170 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:39:27,067-Speed 129.74 samples/sec Loss 20.2175 Epoch: 0 Global Step: 4175 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:39:46,962-Speed 128.69 samples/sec Loss 20.2291 Epoch: 0 Global Step: 4180 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:40:06,328-Speed 132.19 samples/sec Loss 20.0697 Epoch: 0 Global Step: 4185 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:40:26,124-Speed 129.33 samples/sec Loss 20.4733 Epoch: 0 Global Step: 4190 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:40:45,743-Speed 130.49 samples/sec Loss 20.2010 Epoch: 0 Global Step: 4195 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:41:05,303-Speed 130.89 samples/sec Loss 20.7592 Epoch: 0 Global Step: 4200 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:41:25,160-Speed 128.93 samples/sec Loss 20.4899 Epoch: 0 Global Step: 4205 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:41:44,616-Speed 131.59 samples/sec Loss 20.3576 Epoch: 0 Global Step: 4210 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:42:04,382-Speed 129.53 samples/sec Loss 20.5538 Epoch: 0 Global Step: 4215 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:42:24,096-Speed 129.87 samples/sec Loss 20.9003 Epoch: 0 Global Step: 4220 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:42:43,911-Speed 129.20 samples/sec Loss 20.4581 Epoch: 0 Global Step: 4225 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:43:03,295-Speed 132.08 samples/sec Loss 20.6518 Epoch: 0 Global Step: 4230 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:43:23,139-Speed 129.01 samples/sec Loss 20.5058 Epoch: 0 Global Step: 4235 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:43:42,820-Speed 130.08 samples/sec Loss 20.5294 Epoch: 0 Global Step: 4240 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:44:02,267-Speed 131.65 samples/sec Loss 20.4626 Epoch: 0 Global Step: 4245 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:44:22,011-Speed 129.67 samples/sec Loss 20.5483 Epoch: 0 Global Step: 4250 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:44:41,474-Speed 131.54 samples/sec Loss 20.3799 Epoch: 0 Global Step: 4255 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:45:01,199-Speed 129.80 samples/sec Loss 20.2908 Epoch: 0 Global Step: 4260 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:45:20,914-Speed 129.86 samples/sec Loss 20.1827 Epoch: 0 Global Step: 4265 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:45:40,850-Speed 128.42 samples/sec Loss 20.3938 Epoch: 0 Global Step: 4270 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:46:00,218-Speed 132.19 samples/sec Loss 20.3686 Epoch: 0 Global Step: 4275 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:46:20,080-Speed 128.90 samples/sec Loss 20.6722 Epoch: 0 Global Step: 4280 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:46:39,771-Speed 130.01 samples/sec Loss 20.7328 Epoch: 0 Global Step: 4285 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:46:59,250-Speed 131.44 samples/sec Loss 20.4265 Epoch: 0 Global Step: 4290 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:47:18,977-Speed 129.78 samples/sec Loss 20.2088 Epoch: 0 Global Step: 4295 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:47:38,469-Speed 131.34 samples/sec Loss 20.6485 Epoch: 0 Global Step: 4300 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:47:58,210-Speed 129.69 samples/sec Loss 20.3892 Epoch: 0 Global Step: 4305 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:48:17,997-Speed 129.38 samples/sec Loss 20.0758 Epoch: 0 Global Step: 4310 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:48:37,758-Speed 129.56 samples/sec Loss 20.2479 Epoch: 0 Global Step: 4315 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:48:57,186-Speed 131.78 samples/sec Loss 20.4145 Epoch: 0 Global Step: 4320 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:49:16,937-Speed 129.62 samples/sec Loss 20.3968 Epoch: 0 Global Step: 4325 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:49:36,588-Speed 130.29 samples/sec Loss 20.1619 Epoch: 0 Global Step: 4330 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:49:57,216-Speed 124.11 samples/sec Loss 19.7528 Epoch: 0 Global Step: 4335 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:50:16,864-Speed 130.30 samples/sec Loss 20.1790 Epoch: 0 Global Step: 4340 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:50:36,263-Speed 131.98 samples/sec Loss 20.0449 Epoch: 0 Global Step: 4345 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:50:56,091-Speed 129.12 samples/sec Loss 19.8954 Epoch: 0 Global Step: 4350 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:51:16,003-Speed 128.57 samples/sec Loss 19.7351 Epoch: 0 Global Step: 4355 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:51:35,668-Speed 130.19 samples/sec Loss 20.3113 Epoch: 0 Global Step: 4360 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:51:55,169-Speed 131.29 samples/sec Loss 19.9517 Epoch: 0 Global Step: 4365 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-06 23:52:14,832-Speed 130.20 samples/sec Loss 20.0422 Epoch: 0 Global Step: 4370 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:52:34,495-Speed 130.20 samples/sec Loss 20.0856 Epoch: 0 Global Step: 4375 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:52:54,061-Speed 130.85 samples/sec Loss 20.1805 Epoch: 0 Global Step: 4380 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:53:13,696-Speed 130.39 samples/sec Loss 20.3037 Epoch: 0 Global Step: 4385 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:53:33,173-Speed 131.44 samples/sec Loss 20.4452 Epoch: 0 Global Step: 4390 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:53:52,925-Speed 129.62 samples/sec Loss 20.1352 Epoch: 0 Global Step: 4395 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:54:12,806-Speed 128.77 samples/sec Loss 19.9283 Epoch: 0 Global Step: 4400 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:54:32,710-Speed 128.62 samples/sec Loss 20.4855 Epoch: 0 Global Step: 4405 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:54:52,152-Speed 131.69 samples/sec Loss 20.0285 Epoch: 0 Global Step: 4410 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:55:11,899-Speed 129.65 samples/sec Loss 19.9755 Epoch: 0 Global Step: 4415 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:55:31,573-Speed 130.13 samples/sec Loss 20.1170 Epoch: 0 Global Step: 4420 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:55:51,123-Speed 130.96 samples/sec Loss 19.8614 Epoch: 0 Global Step: 4425 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:56:10,650-Speed 131.12 samples/sec Loss 20.1897 Epoch: 0 Global Step: 4430 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:56:30,145-Speed 131.32 samples/sec Loss 19.8723 Epoch: 0 Global Step: 4435 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:56:49,895-Speed 129.63 samples/sec Loss 19.9723 Epoch: 0 Global Step: 4440 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:57:09,730-Speed 129.07 samples/sec Loss 19.7594 Epoch: 0 Global Step: 4445 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:57:29,415-Speed 130.05 samples/sec Loss 19.2924 Epoch: 0 Global Step: 4450 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:57:48,806-Speed 132.03 samples/sec Loss 20.1576 Epoch: 0 Global Step: 4455 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:58:08,735-Speed 128.46 samples/sec Loss 19.9226 Epoch: 0 Global Step: 4460 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:58:28,389-Speed 130.26 samples/sec Loss 19.7917 Epoch: 0 Global Step: 4465 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:58:47,962-Speed 130.80 samples/sec Loss 20.0084 Epoch: 0 Global Step: 4470 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:59:07,469-Speed 131.24 samples/sec Loss 19.8624 Epoch: 0 Global Step: 4475 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:59:27,015-Speed 130.98 samples/sec Loss 20.1447 Epoch: 0 Global Step: 4480 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-06 23:59:46,663-Speed 130.31 samples/sec Loss 19.6535 Epoch: 0 Global Step: 4485 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:00:06,479-Speed 129.20 samples/sec Loss 19.5519 Epoch: 0 Global Step: 4490 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:00:26,062-Speed 130.73 samples/sec Loss 19.7612 Epoch: 0 Global Step: 4495 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:00:45,449-Speed 132.06 samples/sec Loss 20.0126 Epoch: 0 Global Step: 4500 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:01:05,272-Speed 129.15 samples/sec Loss 19.6491 Epoch: 0 Global Step: 4505 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:01:24,913-Speed 130.35 samples/sec Loss 19.9173 Epoch: 0 Global Step: 4510 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:01:44,458-Speed 130.99 samples/sec Loss 19.8835 Epoch: 0 Global Step: 4515 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:02:04,122-Speed 130.19 samples/sec Loss 19.7077 Epoch: 0 Global Step: 4520 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:02:23,655-Speed 131.07 samples/sec Loss 19.6939 Epoch: 0 Global Step: 4525 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:02:43,244-Speed 130.70 samples/sec Loss 19.3203 Epoch: 0 Global Step: 4530 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:03:03,061-Speed 129.19 samples/sec Loss 19.8153 Epoch: 0 Global Step: 4535 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:03:22,689-Speed 130.44 samples/sec Loss 19.7497 Epoch: 0 Global Step: 4540 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:03:42,369-Speed 130.09 samples/sec Loss 19.8722 Epoch: 0 Global Step: 4545 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:04:02,136-Speed 129.51 samples/sec Loss 19.7323 Epoch: 0 Global Step: 4550 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:04:21,807-Speed 130.16 samples/sec Loss 19.3049 Epoch: 0 Global Step: 4555 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:04:41,362-Speed 130.92 samples/sec Loss 19.3499 Epoch: 0 Global Step: 4560 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:05:01,037-Speed 130.12 samples/sec Loss 19.7646 Epoch: 0 Global Step: 4565 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:05:20,589-Speed 130.94 samples/sec Loss 19.4788 Epoch: 0 Global Step: 4570 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:05:40,227-Speed 130.37 samples/sec Loss 19.3818 Epoch: 0 Global Step: 4575 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:06:00,063-Speed 129.07 samples/sec Loss 20.1323 Epoch: 0 Global Step: 4580 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:06:19,700-Speed 130.38 samples/sec Loss 19.5111 Epoch: 0 Global Step: 4585 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:06:39,368-Speed 130.16 samples/sec Loss 19.5586 Epoch: 0 Global Step: 4590 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:06:59,097-Speed 129.77 samples/sec Loss 19.6980 Epoch: 0 Global Step: 4595 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:07:18,795-Speed 129.97 samples/sec Loss 19.2768 Epoch: 0 Global Step: 4600 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:07:38,307-Speed 131.21 samples/sec Loss 19.6351 Epoch: 0 Global Step: 4605 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:07:57,998-Speed 130.02 samples/sec Loss 19.4780 Epoch: 0 Global Step: 4610 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:08:17,527-Speed 131.10 samples/sec Loss 19.4599 Epoch: 0 Global Step: 4615 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:08:37,087-Speed 130.88 samples/sec Loss 19.6432 Epoch: 0 Global Step: 4620 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:08:56,865-Speed 129.45 samples/sec Loss 19.3809 Epoch: 0 Global Step: 4625 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:09:16,526-Speed 130.21 samples/sec Loss 19.6618 Epoch: 0 Global Step: 4630 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:09:36,146-Speed 130.49 samples/sec Loss 19.1301 Epoch: 0 Global Step: 4635 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:09:55,795-Speed 130.29 samples/sec Loss 19.1003 Epoch: 0 Global Step: 4640 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:10:15,534-Speed 129.70 samples/sec Loss 19.3987 Epoch: 0 Global Step: 4645 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:10:35,509-Speed 128.17 samples/sec Loss 19.5234 Epoch: 0 Global Step: 4650 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:10:55,150-Speed 130.35 samples/sec Loss 19.3796 Epoch: 0 Global Step: 4655 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:11:14,701-Speed 130.95 samples/sec Loss 19.1167 Epoch: 0 Global Step: 4660 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:11:34,277-Speed 130.78 samples/sec Loss 19.3679 Epoch: 0 Global Step: 4665 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:11:54,019-Speed 129.68 samples/sec Loss 19.3091 Epoch: 0 Global Step: 4670 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:12:13,750-Speed 129.76 samples/sec Loss 19.4496 Epoch: 0 Global Step: 4675 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:12:33,287-Speed 131.04 samples/sec Loss 19.1567 Epoch: 0 Global Step: 4680 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:12:53,029-Speed 129.68 samples/sec Loss 19.2933 Epoch: 0 Global Step: 4685 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:13:12,888-Speed 128.92 samples/sec Loss 19.0832 Epoch: 0 Global Step: 4690 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:13:32,362-Speed 131.47 samples/sec Loss 19.3634 Epoch: 0 Global Step: 4695 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:13:51,914-Speed 130.95 samples/sec Loss 19.6322 Epoch: 0 Global Step: 4700 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:14:11,421-Speed 131.24 samples/sec Loss 18.8326 Epoch: 0 Global Step: 4705 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:14:31,096-Speed 130.13 samples/sec Loss 19.3673 Epoch: 0 Global Step: 4710 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:14:50,853-Speed 129.58 samples/sec Loss 19.2446 Epoch: 0 Global Step: 4715 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:15:10,738-Speed 128.75 samples/sec Loss 19.5531 Epoch: 0 Global Step: 4720 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:15:30,132-Speed 132.00 samples/sec Loss 19.3451 Epoch: 0 Global Step: 4725 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:15:49,715-Speed 130.74 samples/sec Loss 19.2629 Epoch: 0 Global Step: 4730 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:16:09,580-Speed 128.87 samples/sec Loss 19.4778 Epoch: 0 Global Step: 4735 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:16:29,004-Speed 131.80 samples/sec Loss 19.4307 Epoch: 0 Global Step: 4740 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:16:48,743-Speed 129.71 samples/sec Loss 19.5222 Epoch: 0 Global Step: 4745 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:17:08,338-Speed 130.66 samples/sec Loss 19.3104 Epoch: 0 Global Step: 4750 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:17:27,870-Speed 131.08 samples/sec Loss 19.3933 Epoch: 0 Global Step: 4755 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:17:47,622-Speed 129.62 samples/sec Loss 18.9762 Epoch: 0 Global Step: 4760 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:18:07,399-Speed 129.45 samples/sec Loss 19.1728 Epoch: 0 Global Step: 4765 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:18:26,842-Speed 131.68 samples/sec Loss 19.1970 Epoch: 0 Global Step: 4770 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:18:46,512-Speed 130.15 samples/sec Loss 19.5778 Epoch: 0 Global Step: 4775 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:19:06,382-Speed 128.84 samples/sec Loss 19.3056 Epoch: 0 Global Step: 4780 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:19:25,766-Speed 132.08 samples/sec Loss 19.2790 Epoch: 0 Global Step: 4785 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:19:45,530-Speed 129.54 samples/sec Loss 19.1839 Epoch: 0 Global Step: 4790 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:20:04,957-Speed 131.78 samples/sec Loss 18.6452 Epoch: 0 Global Step: 4795 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:20:24,542-Speed 130.72 samples/sec Loss 19.5374 Epoch: 0 Global Step: 4800 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:20:44,287-Speed 129.66 samples/sec Loss 19.3626 Epoch: 0 Global Step: 4805 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:21:04,154-Speed 128.86 samples/sec Loss 19.3579 Epoch: 0 Global Step: 4810 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:21:23,515-Speed 132.23 samples/sec Loss 18.7687 Epoch: 0 Global Step: 4815 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:21:43,084-Speed 130.83 samples/sec Loss 18.8642 Epoch: 0 Global Step: 4820 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:22:02,908-Speed 129.14 samples/sec Loss 19.1292 Epoch: 0 Global Step: 4825 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:22:22,664-Speed 129.59 samples/sec Loss 18.9071 Epoch: 0 Global Step: 4830 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:22:42,419-Speed 129.59 samples/sec Loss 19.0727 Epoch: 0 Global Step: 4835 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:23:01,784-Speed 132.21 samples/sec Loss 19.2349 Epoch: 0 Global Step: 4840 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:23:21,559-Speed 129.47 samples/sec Loss 19.0084 Epoch: 0 Global Step: 4845 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:23:41,261-Speed 129.95 samples/sec Loss 18.9259 Epoch: 0 Global Step: 4850 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:24:01,127-Speed 128.87 samples/sec Loss 18.9162 Epoch: 0 Global Step: 4855 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:24:20,579-Speed 131.61 samples/sec Loss 19.3860 Epoch: 0 Global Step: 4860 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:24:40,222-Speed 130.33 samples/sec Loss 18.6145 Epoch: 0 Global Step: 4865 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:25:00,001-Speed 129.44 samples/sec Loss 19.0304 Epoch: 0 Global Step: 4870 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:25:19,449-Speed 131.64 samples/sec Loss 18.9492 Epoch: 0 Global Step: 4875 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:25:39,126-Speed 130.11 samples/sec Loss 19.0934 Epoch: 0 Global Step: 4880 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:25:58,582-Speed 131.59 samples/sec Loss 19.1149 Epoch: 0 Global Step: 4885 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:26:18,318-Speed 129.73 samples/sec Loss 18.6994 Epoch: 0 Global Step: 4890 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:26:38,053-Speed 129.73 samples/sec Loss 18.7232 Epoch: 0 Global Step: 4895 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:26:57,909-Speed 128.93 samples/sec Loss 19.2081 Epoch: 0 Global Step: 4900 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:27:17,267-Speed 132.25 samples/sec Loss 19.1234 Epoch: 0 Global Step: 4905 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:27:37,135-Speed 128.86 samples/sec Loss 19.1973 Epoch: 0 Global Step: 4910 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:27:56,795-Speed 130.23 samples/sec Loss 18.8837 Epoch: 0 Global Step: 4915 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:28:16,186-Speed 132.03 samples/sec Loss 18.6636 Epoch: 0 Global Step: 4920 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:28:35,941-Speed 129.60 samples/sec Loss 19.0473 Epoch: 0 Global Step: 4925 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:28:55,417-Speed 131.46 samples/sec Loss 18.9648 Epoch: 0 Global Step: 4930 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:29:15,124-Speed 129.91 samples/sec Loss 19.1100 Epoch: 0 Global Step: 4935 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:29:34,803-Speed 130.10 samples/sec Loss 19.0423 Epoch: 0 Global Step: 4940 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:29:54,591-Speed 129.37 samples/sec Loss 18.7657 Epoch: 0 Global Step: 4945 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:30:13,958-Speed 132.19 samples/sec Loss 19.0274 Epoch: 0 Global Step: 4950 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:30:33,837-Speed 128.79 samples/sec Loss 18.9932 Epoch: 0 Global Step: 4955 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:30:53,498-Speed 130.21 samples/sec Loss 19.0111 Epoch: 0 Global Step: 4960 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:31:12,902-Speed 131.95 samples/sec Loss 18.7642 Epoch: 0 Global Step: 4965 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:31:32,697-Speed 129.33 samples/sec Loss 18.6030 Epoch: 0 Global Step: 4970 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:31:52,231-Speed 131.06 samples/sec Loss 18.4248 Epoch: 0 Global Step: 4975 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:32:11,989-Speed 129.58 samples/sec Loss 18.7714 Epoch: 0 Global Step: 4980 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:32:31,710-Speed 129.82 samples/sec Loss 18.9288 Epoch: 0 Global Step: 4985 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:32:51,406-Speed 129.98 samples/sec Loss 18.6260 Epoch: 0 Global Step: 4990 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:33:10,766-Speed 132.24 samples/sec Loss 18.8029 Epoch: 0 Global Step: 4995 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:33:30,589-Speed 129.15 samples/sec Loss 18.9756 Epoch: 0 Global Step: 5000 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:33:50,258-Speed 130.17 samples/sec Loss 18.4588 Epoch: 0 Global Step: 5005 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:34:09,742-Speed 131.40 samples/sec Loss 19.0306 Epoch: 0 Global Step: 5010 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:34:29,598-Speed 128.94 samples/sec Loss 18.6566 Epoch: 0 Global Step: 5015 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:34:48,961-Speed 132.22 samples/sec Loss 19.0510 Epoch: 0 Global Step: 5020 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:35:08,823-Speed 128.89 samples/sec Loss 18.8789 Epoch: 0 Global Step: 5025 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:35:28,448-Speed 130.45 samples/sec Loss 18.6214 Epoch: 0 Global Step: 5030 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:35:48,356-Speed 128.60 samples/sec Loss 18.9254 Epoch: 0 Global Step: 5035 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:36:07,737-Speed 132.10 samples/sec Loss 19.0445 Epoch: 0 Global Step: 5040 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:36:27,487-Speed 129.62 samples/sec Loss 18.5969 Epoch: 0 Global Step: 5045 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:36:47,161-Speed 130.13 samples/sec Loss 18.6035 Epoch: 0 Global Step: 5050 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:37:06,718-Speed 130.91 samples/sec Loss 18.7437 Epoch: 0 Global Step: 5055 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:37:26,452-Speed 129.73 samples/sec Loss 18.6341 Epoch: 0 Global Step: 5060 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:37:45,907-Speed 131.60 samples/sec Loss 18.5291 Epoch: 0 Global Step: 5065 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:38:05,729-Speed 129.16 samples/sec Loss 18.3758 Epoch: 0 Global Step: 5070 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:38:25,461-Speed 129.75 samples/sec Loss 19.2665 Epoch: 0 Global Step: 5075 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:38:45,308-Speed 128.99 samples/sec Loss 18.7736 Epoch: 0 Global Step: 5080 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:39:04,674-Speed 132.20 samples/sec Loss 19.0643 Epoch: 0 Global Step: 5085 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:39:24,535-Speed 128.90 samples/sec Loss 18.3771 Epoch: 0 Global Step: 5090 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:39:44,198-Speed 130.20 samples/sec Loss 18.6909 Epoch: 0 Global Step: 5095 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:40:03,760-Speed 130.88 samples/sec Loss 18.3910 Epoch: 0 Global Step: 5100 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 00:40:23,541-Speed 129.42 samples/sec Loss 18.7009 Epoch: 0 Global Step: 5105 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:40:42,925-Speed 132.08 samples/sec Loss 18.4426 Epoch: 0 Global Step: 5110 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:41:02,742-Speed 129.19 samples/sec Loss 18.4533 Epoch: 0 Global Step: 5115 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:41:22,600-Speed 128.92 samples/sec Loss 18.4988 Epoch: 0 Global Step: 5120 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:41:42,217-Speed 130.51 samples/sec Loss 18.2014 Epoch: 0 Global Step: 5125 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:42:01,857-Speed 130.36 samples/sec Loss 18.6856 Epoch: 0 Global Step: 5130 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:42:21,645-Speed 129.38 samples/sec Loss 18.4673 Epoch: 0 Global Step: 5135 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:42:41,335-Speed 130.02 samples/sec Loss 18.4360 Epoch: 0 Global Step: 5140 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:43:00,893-Speed 130.90 samples/sec Loss 18.4407 Epoch: 0 Global Step: 5145 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:43:20,555-Speed 130.21 samples/sec Loss 18.4048 Epoch: 0 Global Step: 5150 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:43:40,625-Speed 127.57 samples/sec Loss 18.4365 Epoch: 0 Global Step: 5155 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:44:00,442-Speed 129.19 samples/sec Loss 18.4758 Epoch: 0 Global Step: 5160 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:44:20,325-Speed 128.76 samples/sec Loss 18.5082 Epoch: 0 Global Step: 5165 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:44:39,839-Speed 131.20 samples/sec Loss 18.6033 Epoch: 0 Global Step: 5170 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:44:59,340-Speed 131.28 samples/sec Loss 18.3768 Epoch: 0 Global Step: 5175 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:45:19,115-Speed 129.47 samples/sec Loss 18.4105 Epoch: 0 Global Step: 5180 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:45:38,791-Speed 130.12 samples/sec Loss 18.1155 Epoch: 0 Global Step: 5185 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:45:58,359-Speed 130.83 samples/sec Loss 18.2976 Epoch: 0 Global Step: 5190 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:46:18,014-Speed 130.26 samples/sec Loss 18.3828 Epoch: 0 Global Step: 5195 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:46:37,474-Speed 131.56 samples/sec Loss 18.2333 Epoch: 0 Global Step: 5200 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:46:57,314-Speed 129.04 samples/sec Loss 18.3868 Epoch: 0 Global Step: 5205 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:47:17,173-Speed 128.92 samples/sec Loss 18.0377 Epoch: 0 Global Step: 5210 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:47:36,831-Speed 130.23 samples/sec Loss 18.4646 Epoch: 0 Global Step: 5215 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:47:56,218-Speed 132.06 samples/sec Loss 18.0691 Epoch: 0 Global Step: 5220 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:48:16,034-Speed 129.19 samples/sec Loss 18.1082 Epoch: 0 Global Step: 5225 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:48:35,705-Speed 130.15 samples/sec Loss 18.4628 Epoch: 0 Global Step: 5230 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:48:55,254-Speed 130.96 samples/sec Loss 18.1234 Epoch: 0 Global Step: 5235 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:49:15,005-Speed 129.62 samples/sec Loss 18.1436 Epoch: 0 Global Step: 5240 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:49:34,548-Speed 131.00 samples/sec Loss 18.0814 Epoch: 0 Global Step: 5245 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:49:54,239-Speed 130.02 samples/sec Loss 18.3587 Epoch: 0 Global Step: 5250 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:50:14,071-Speed 129.09 samples/sec Loss 18.3139 Epoch: 0 Global Step: 5255 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:50:33,717-Speed 130.31 samples/sec Loss 18.3736 Epoch: 0 Global Step: 5260 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:50:53,310-Speed 130.67 samples/sec Loss 18.0903 Epoch: 0 Global Step: 5265 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:51:13,062-Speed 129.62 samples/sec Loss 18.1793 Epoch: 0 Global Step: 5270 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:51:32,684-Speed 130.47 samples/sec Loss 18.3124 Epoch: 0 Global Step: 5275 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:51:52,196-Speed 131.22 samples/sec Loss 18.1513 Epoch: 0 Global Step: 5280 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:52:11,831-Speed 130.38 samples/sec Loss 18.2610 Epoch: 0 Global Step: 5285 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:52:31,367-Speed 131.05 samples/sec Loss 18.7113 Epoch: 0 Global Step: 5290 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:52:51,002-Speed 130.39 samples/sec Loss 18.4347 Epoch: 0 Global Step: 5295 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:53:10,755-Speed 129.61 samples/sec Loss 18.0669 Epoch: 0 Global Step: 5300 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:53:30,448-Speed 130.00 samples/sec Loss 18.0800 Epoch: 0 Global Step: 5305 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:53:50,104-Speed 130.25 samples/sec Loss 18.1553 Epoch: 0 Global Step: 5310 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:54:09,772-Speed 130.17 samples/sec Loss 17.6495 Epoch: 0 Global Step: 5315 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:54:29,573-Speed 129.30 samples/sec Loss 17.9191 Epoch: 0 Global Step: 5320 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:54:49,024-Speed 131.62 samples/sec Loss 18.2383 Epoch: 0 Global Step: 5325 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:55:08,659-Speed 130.39 samples/sec Loss 18.4124 Epoch: 0 Global Step: 5330 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:55:28,193-Speed 131.06 samples/sec Loss 18.1346 Epoch: 0 Global Step: 5335 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:55:47,730-Speed 131.04 samples/sec Loss 18.3160 Epoch: 0 Global Step: 5340 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:56:07,492-Speed 129.55 samples/sec Loss 18.2178 Epoch: 0 Global Step: 5345 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:56:27,133-Speed 130.35 samples/sec Loss 17.9476 Epoch: 0 Global Step: 5350 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:56:46,831-Speed 129.97 samples/sec Loss 18.5202 Epoch: 0 Global Step: 5355 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:57:06,502-Speed 130.15 samples/sec Loss 18.2688 Epoch: 0 Global Step: 5360 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:57:26,230-Speed 129.78 samples/sec Loss 18.1366 Epoch: 0 Global Step: 5365 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:57:45,736-Speed 131.25 samples/sec Loss 18.2229 Epoch: 0 Global Step: 5370 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:58:05,417-Speed 130.09 samples/sec Loss 18.2430 Epoch: 0 Global Step: 5375 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:58:24,965-Speed 130.97 samples/sec Loss 18.0575 Epoch: 0 Global Step: 5380 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:58:44,623-Speed 130.24 samples/sec Loss 17.8323 Epoch: 0 Global Step: 5385 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:59:04,367-Speed 129.67 samples/sec Loss 18.1226 Epoch: 0 Global Step: 5390 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:59:24,041-Speed 130.13 samples/sec Loss 18.0382 Epoch: 0 Global Step: 5395 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 00:59:43,618-Speed 130.77 samples/sec Loss 18.3036 Epoch: 0 Global Step: 5400 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:00:03,218-Speed 130.62 samples/sec Loss 17.6888 Epoch: 0 Global Step: 5405 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:00:22,916-Speed 129.97 samples/sec Loss 17.9797 Epoch: 0 Global Step: 5410 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:00:42,348-Speed 131.75 samples/sec Loss 18.1867 Epoch: 0 Global Step: 5415 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:01:02,024-Speed 130.12 samples/sec Loss 18.1133 Epoch: 0 Global Step: 5420 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:01:21,565-Speed 131.02 samples/sec Loss 18.1875 Epoch: 0 Global Step: 5425 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:01:41,242-Speed 130.11 samples/sec Loss 17.9567 Epoch: 0 Global Step: 5430 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:02:00,990-Speed 129.64 samples/sec Loss 18.1456 Epoch: 0 Global Step: 5435 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:02:20,660-Speed 130.15 samples/sec Loss 18.0151 Epoch: 0 Global Step: 5440 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:02:40,305-Speed 130.32 samples/sec Loss 17.5060 Epoch: 0 Global Step: 5445 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:02:59,895-Speed 130.68 samples/sec Loss 17.7529 Epoch: 0 Global Step: 5450 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:03:19,546-Speed 130.29 samples/sec Loss 17.8969 Epoch: 0 Global Step: 5455 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:03:39,081-Speed 131.06 samples/sec Loss 17.8451 Epoch: 0 Global Step: 5460 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:03:58,612-Speed 131.08 samples/sec Loss 17.6097 Epoch: 0 Global Step: 5465 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:04:18,163-Speed 130.95 samples/sec Loss 18.1774 Epoch: 0 Global Step: 5470 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:04:37,847-Speed 130.07 samples/sec Loss 18.1269 Epoch: 0 Global Step: 5475 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:04:57,613-Speed 129.52 samples/sec Loss 17.5854 Epoch: 0 Global Step: 5480 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:05:17,354-Speed 129.69 samples/sec Loss 18.1157 Epoch: 0 Global Step: 5485 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:05:36,880-Speed 131.11 samples/sec Loss 18.5397 Epoch: 0 Global Step: 5490 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:05:56,646-Speed 129.53 samples/sec Loss 18.0078 Epoch: 0 Global Step: 5495 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:06:16,498-Speed 128.96 samples/sec Loss 17.8486 Epoch: 0 Global Step: 5500 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:06:35,978-Speed 131.43 samples/sec Loss 17.7668 Epoch: 0 Global Step: 5505 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:06:55,635-Speed 130.25 samples/sec Loss 18.1788 Epoch: 0 Global Step: 5510 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:07:15,081-Speed 131.66 samples/sec Loss 17.9806 Epoch: 0 Global Step: 5515 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:07:34,732-Speed 130.28 samples/sec Loss 17.7252 Epoch: 0 Global Step: 5520 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:07:54,504-Speed 129.49 samples/sec Loss 17.9426 Epoch: 0 Global Step: 5525 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:08:14,359-Speed 128.94 samples/sec Loss 18.1309 Epoch: 0 Global Step: 5530 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:08:33,757-Speed 131.98 samples/sec Loss 17.8828 Epoch: 0 Global Step: 5535 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:08:53,318-Speed 130.88 samples/sec Loss 18.0853 Epoch: 0 Global Step: 5540 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:09:13,268-Speed 128.33 samples/sec Loss 18.1526 Epoch: 0 Global Step: 5545 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:09:32,703-Speed 131.73 samples/sec Loss 18.0513 Epoch: 0 Global Step: 5550 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:09:52,388-Speed 130.06 samples/sec Loss 17.9538 Epoch: 0 Global Step: 5555 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:10:11,943-Speed 130.92 samples/sec Loss 17.7628 Epoch: 0 Global Step: 5560 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:10:31,620-Speed 130.11 samples/sec Loss 17.9407 Epoch: 0 Global Step: 5565 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:10:51,373-Speed 129.61 samples/sec Loss 17.9103 Epoch: 0 Global Step: 5570 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:11:11,231-Speed 128.92 samples/sec Loss 17.6023 Epoch: 0 Global Step: 5575 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:11:30,576-Speed 132.34 samples/sec Loss 17.4895 Epoch: 0 Global Step: 5580 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:11:50,200-Speed 130.46 samples/sec Loss 17.9353 Epoch: 0 Global Step: 5585 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:12:10,034-Speed 129.08 samples/sec Loss 17.8560 Epoch: 0 Global Step: 5590 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:12:29,481-Speed 131.65 samples/sec Loss 18.0621 Epoch: 0 Global Step: 5595 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:12:49,166-Speed 130.05 samples/sec Loss 17.5731 Epoch: 0 Global Step: 5600 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:13:08,528-Speed 132.22 samples/sec Loss 17.9569 Epoch: 0 Global Step: 5605 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:13:28,234-Speed 129.92 samples/sec Loss 18.0898 Epoch: 0 Global Step: 5610 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:13:47,943-Speed 129.90 samples/sec Loss 18.0486 Epoch: 0 Global Step: 5615 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:14:07,797-Speed 128.95 samples/sec Loss 18.1147 Epoch: 0 Global Step: 5620 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:14:27,160-Speed 132.22 samples/sec Loss 17.7307 Epoch: 0 Global Step: 5625 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:14:46,913-Speed 129.60 samples/sec Loss 17.9084 Epoch: 0 Global Step: 5630 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:15:06,592-Speed 130.10 samples/sec Loss 17.7725 Epoch: 0 Global Step: 5635 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:15:25,991-Speed 131.97 samples/sec Loss 18.1047 Epoch: 0 Global Step: 5640 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:15:45,639-Speed 130.31 samples/sec Loss 17.5956 Epoch: 0 Global Step: 5645 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:16:05,083-Speed 131.67 samples/sec Loss 17.5917 Epoch: 0 Global Step: 5650 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:16:24,888-Speed 129.27 samples/sec Loss 18.0389 Epoch: 0 Global Step: 5655 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:16:44,537-Speed 130.30 samples/sec Loss 17.6189 Epoch: 0 Global Step: 5660 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:17:04,332-Speed 129.33 samples/sec Loss 17.6399 Epoch: 0 Global Step: 5665 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:17:23,690-Speed 132.25 samples/sec Loss 17.5816 Epoch: 0 Global Step: 5670 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:17:43,533-Speed 129.02 samples/sec Loss 17.5497 Epoch: 0 Global Step: 5675 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:18:03,212-Speed 130.10 samples/sec Loss 17.4503 Epoch: 0 Global Step: 5680 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:18:22,615-Speed 131.94 samples/sec Loss 17.9577 Epoch: 0 Global Step: 5685 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:18:42,415-Speed 129.31 samples/sec Loss 17.4455 Epoch: 0 Global Step: 5690 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:19:01,901-Speed 131.39 samples/sec Loss 18.0020 Epoch: 0 Global Step: 5695 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:19:21,765-Speed 128.88 samples/sec Loss 17.7787 Epoch: 0 Global Step: 5700 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:19:41,491-Speed 129.79 samples/sec Loss 17.5672 Epoch: 0 Global Step: 5705 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:20:01,279-Speed 129.38 samples/sec Loss 17.5658 Epoch: 0 Global Step: 5710 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:20:20,804-Speed 131.12 samples/sec Loss 17.9222 Epoch: 0 Global Step: 5715 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:20:40,623-Speed 129.17 samples/sec Loss 17.8211 Epoch: 0 Global Step: 5720 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:21:00,277-Speed 130.26 samples/sec Loss 17.5045 Epoch: 0 Global Step: 5725 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:21:19,821-Speed 131.00 samples/sec Loss 17.5178 Epoch: 0 Global Step: 5730 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:21:39,472-Speed 130.28 samples/sec Loss 17.7725 Epoch: 0 Global Step: 5735 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:21:58,914-Speed 131.68 samples/sec Loss 17.5891 Epoch: 0 Global Step: 5740 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:22:18,753-Speed 129.04 samples/sec Loss 17.4629 Epoch: 0 Global Step: 5745 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:22:38,720-Speed 128.22 samples/sec Loss 17.7100 Epoch: 0 Global Step: 5750 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:22:58,368-Speed 130.30 samples/sec Loss 17.4253 Epoch: 0 Global Step: 5755 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:23:17,772-Speed 131.94 samples/sec Loss 17.6532 Epoch: 0 Global Step: 5760 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:23:37,615-Speed 129.02 samples/sec Loss 17.8683 Epoch: 0 Global Step: 5765 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:23:57,270-Speed 130.25 samples/sec Loss 17.6509 Epoch: 0 Global Step: 5770 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:24:16,821-Speed 130.95 samples/sec Loss 17.5400 Epoch: 0 Global Step: 5775 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:24:36,416-Speed 130.65 samples/sec Loss 17.4746 Epoch: 0 Global Step: 5780 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:24:55,864-Speed 131.64 samples/sec Loss 17.6908 Epoch: 0 Global Step: 5785 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:25:15,656-Speed 129.36 samples/sec Loss 17.2460 Epoch: 0 Global Step: 5790 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:25:35,850-Speed 126.78 samples/sec Loss 17.4893 Epoch: 0 Global Step: 5795 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:25:55,573-Speed 129.80 samples/sec Loss 17.3430 Epoch: 0 Global Step: 5800 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:26:15,054-Speed 131.42 samples/sec Loss 17.3629 Epoch: 0 Global Step: 5805 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:26:34,873-Speed 129.18 samples/sec Loss 17.4009 Epoch: 0 Global Step: 5810 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:26:54,544-Speed 130.15 samples/sec Loss 17.6536 Epoch: 0 Global Step: 5815 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:27:14,081-Speed 131.04 samples/sec Loss 17.5021 Epoch: 0 Global Step: 5820 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:27:33,784-Speed 129.94 samples/sec Loss 17.3062 Epoch: 0 Global Step: 5825 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:27:53,381-Speed 130.64 samples/sec Loss 17.5646 Epoch: 0 Global Step: 5830 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:28:12,957-Speed 130.78 samples/sec Loss 17.4860 Epoch: 0 Global Step: 5835 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:28:32,767-Speed 129.24 samples/sec Loss 17.6592 Epoch: 0 Global Step: 5840 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:28:52,275-Speed 131.23 samples/sec Loss 17.3848 Epoch: 0 Global Step: 5845 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:29:11,936-Speed 130.22 samples/sec Loss 17.3272 Epoch: 0 Global Step: 5850 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:29:31,592-Speed 130.24 samples/sec Loss 17.3397 Epoch: 0 Global Step: 5855 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:29:51,289-Speed 129.98 samples/sec Loss 17.1078 Epoch: 0 Global Step: 5860 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:30:10,784-Speed 131.32 samples/sec Loss 17.3958 Epoch: 0 Global Step: 5865 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:30:30,588-Speed 129.28 samples/sec Loss 17.0854 Epoch: 0 Global Step: 5870 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:30:50,131-Speed 131.01 samples/sec Loss 17.4940 Epoch: 0 Global Step: 5875 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:31:09,737-Speed 130.58 samples/sec Loss 17.2184 Epoch: 0 Global Step: 5880 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 01:31:29,496-Speed 129.57 samples/sec Loss 17.6562 Epoch: 0 Global Step: 5885 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:31:49,205-Speed 129.89 samples/sec Loss 17.5907 Epoch: 0 Global Step: 5890 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:32:08,754-Speed 130.96 samples/sec Loss 17.5616 Epoch: 0 Global Step: 5895 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:32:28,454-Speed 129.96 samples/sec Loss 17.4998 Epoch: 0 Global Step: 5900 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:32:48,303-Speed 128.98 samples/sec Loss 17.2830 Epoch: 0 Global Step: 5905 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:33:07,870-Speed 130.84 samples/sec Loss 17.7591 Epoch: 0 Global Step: 5910 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:33:27,548-Speed 130.11 samples/sec Loss 17.6591 Epoch: 0 Global Step: 5915 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:33:47,111-Speed 130.87 samples/sec Loss 17.2938 Epoch: 0 Global Step: 5920 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:34:06,797-Speed 130.05 samples/sec Loss 17.5699 Epoch: 0 Global Step: 5925 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:34:26,583-Speed 129.40 samples/sec Loss 17.1926 Epoch: 0 Global Step: 5930 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:34:46,303-Speed 129.82 samples/sec Loss 17.1478 Epoch: 0 Global Step: 5935 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:35:05,842-Speed 131.03 samples/sec Loss 17.4602 Epoch: 0 Global Step: 5940 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:35:25,489-Speed 130.30 samples/sec Loss 17.1235 Epoch: 0 Global Step: 5945 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:35:45,377-Speed 128.73 samples/sec Loss 17.3585 Epoch: 0 Global Step: 5950 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:36:04,820-Speed 131.68 samples/sec Loss 17.4091 Epoch: 0 Global Step: 5955 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:36:24,537-Speed 129.85 samples/sec Loss 17.3158 Epoch: 0 Global Step: 5960 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:36:44,039-Speed 131.28 samples/sec Loss 17.4256 Epoch: 0 Global Step: 5965 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:37:03,657-Speed 130.50 samples/sec Loss 17.2507 Epoch: 0 Global Step: 5970 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:37:23,401-Speed 129.67 samples/sec Loss 17.3025 Epoch: 0 Global Step: 5975 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:37:43,265-Speed 128.88 samples/sec Loss 17.3207 Epoch: 0 Global Step: 5980 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:38:02,627-Speed 132.22 samples/sec Loss 17.4692 Epoch: 0 Global Step: 5985 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:38:22,267-Speed 130.35 samples/sec Loss 17.2311 Epoch: 0 Global Step: 5990 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:38:42,134-Speed 128.87 samples/sec Loss 17.0533 Epoch: 0 Global Step: 5995 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:39:01,596-Speed 131.55 samples/sec Loss 17.8138 Epoch: 0 Global Step: 6000 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:39:21,333-Speed 129.72 samples/sec Loss 17.3233 Epoch: 0 Global Step: 6005 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:39:40,793-Speed 131.56 samples/sec Loss 17.6358 Epoch: 0 Global Step: 6010 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:40:00,402-Speed 130.56 samples/sec Loss 17.4659 Epoch: 0 Global Step: 6015 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:40:20,164-Speed 129.55 samples/sec Loss 17.3361 Epoch: 0 Global Step: 6020 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:40:40,052-Speed 128.72 samples/sec Loss 17.1175 Epoch: 0 Global Step: 6025 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:40:59,397-Speed 132.34 samples/sec Loss 17.3234 Epoch: 0 Global Step: 6030 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:41:19,116-Speed 129.83 samples/sec Loss 16.9320 Epoch: 0 Global Step: 6035 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:41:38,919-Speed 129.28 samples/sec Loss 17.2337 Epoch: 0 Global Step: 6040 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:41:58,352-Speed 131.75 samples/sec Loss 17.2642 Epoch: 0 Global Step: 6045 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:42:18,095-Speed 129.67 samples/sec Loss 16.9859 Epoch: 0 Global Step: 6050 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:42:37,564-Speed 131.51 samples/sec Loss 17.0344 Epoch: 0 Global Step: 6055 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:42:57,369-Speed 129.27 samples/sec Loss 17.0567 Epoch: 0 Global Step: 6060 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:43:17,115-Speed 129.65 samples/sec Loss 17.1639 Epoch: 0 Global Step: 6065 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:43:36,964-Speed 128.98 samples/sec Loss 17.3348 Epoch: 0 Global Step: 6070 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:43:56,302-Speed 132.39 samples/sec Loss 17.6951 Epoch: 0 Global Step: 6075 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:44:15,960-Speed 130.23 samples/sec Loss 17.1811 Epoch: 0 Global Step: 6080 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:44:35,776-Speed 129.20 samples/sec Loss 17.1163 Epoch: 0 Global Step: 6085 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:44:55,114-Speed 132.39 samples/sec Loss 17.6820 Epoch: 0 Global Step: 6090 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:45:14,806-Speed 130.01 samples/sec Loss 17.2751 Epoch: 0 Global Step: 6095 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:45:34,312-Speed 131.25 samples/sec Loss 17.3201 Epoch: 0 Global Step: 6100 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:45:53,962-Speed 130.29 samples/sec Loss 17.2756 Epoch: 0 Global Step: 6105 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:46:13,731-Speed 129.50 samples/sec Loss 17.2238 Epoch: 0 Global Step: 6110 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:46:33,580-Speed 128.98 samples/sec Loss 17.2334 Epoch: 0 Global Step: 6115 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:46:52,919-Speed 132.38 samples/sec Loss 17.2669 Epoch: 0 Global Step: 6120 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:47:12,918-Speed 128.01 samples/sec Loss 17.4934 Epoch: 0 Global Step: 6125 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:47:32,728-Speed 129.24 samples/sec Loss 17.3439 Epoch: 0 Global Step: 6130 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:47:52,172-Speed 131.67 samples/sec Loss 17.2623 Epoch: 0 Global Step: 6135 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:48:12,110-Speed 128.41 samples/sec Loss 17.0223 Epoch: 0 Global Step: 6140 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:48:31,521-Speed 131.89 samples/sec Loss 17.0037 Epoch: 0 Global Step: 6145 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:48:51,114-Speed 130.67 samples/sec Loss 17.3097 Epoch: 0 Global Step: 6150 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:49:10,861-Speed 129.65 samples/sec Loss 16.9812 Epoch: 0 Global Step: 6155 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:49:30,731-Speed 128.84 samples/sec Loss 17.2600 Epoch: 0 Global Step: 6160 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:49:50,063-Speed 132.43 samples/sec Loss 17.3577 Epoch: 0 Global Step: 6165 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:50:09,811-Speed 129.64 samples/sec Loss 17.3615 Epoch: 0 Global Step: 6170 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:50:29,653-Speed 129.03 samples/sec Loss 17.5061 Epoch: 0 Global Step: 6175 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:50:48,993-Speed 132.38 samples/sec Loss 17.3312 Epoch: 0 Global Step: 6180 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:51:08,864-Speed 128.84 samples/sec Loss 16.9521 Epoch: 0 Global Step: 6185 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:51:28,315-Speed 131.62 samples/sec Loss 17.1662 Epoch: 0 Global Step: 6190 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:51:48,072-Speed 129.58 samples/sec Loss 17.4853 Epoch: 0 Global Step: 6195 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:52:07,778-Speed 129.92 samples/sec Loss 16.7723 Epoch: 0 Global Step: 6200 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:52:27,663-Speed 128.74 samples/sec Loss 17.2939 Epoch: 0 Global Step: 6205 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:52:47,016-Speed 132.29 samples/sec Loss 16.7737 Epoch: 0 Global Step: 6210 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:53:06,862-Speed 129.00 samples/sec Loss 17.0223 Epoch: 0 Global Step: 6215 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:53:26,521-Speed 130.23 samples/sec Loss 16.8253 Epoch: 0 Global Step: 6220 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:53:45,956-Speed 131.73 samples/sec Loss 16.7504 Epoch: 0 Global Step: 6225 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:54:05,668-Speed 129.88 samples/sec Loss 16.9034 Epoch: 0 Global Step: 6230 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:54:25,218-Speed 130.95 samples/sec Loss 17.1470 Epoch: 0 Global Step: 6235 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:54:44,931-Speed 129.88 samples/sec Loss 16.7435 Epoch: 0 Global Step: 6240 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:55:04,695-Speed 129.54 samples/sec Loss 16.9708 Epoch: 0 Global Step: 6245 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:55:24,628-Speed 128.44 samples/sec Loss 16.8154 Epoch: 0 Global Step: 6250 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:55:43,966-Speed 132.39 samples/sec Loss 17.2137 Epoch: 0 Global Step: 6255 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:56:03,675-Speed 129.90 samples/sec Loss 16.7568 Epoch: 0 Global Step: 6260 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:56:23,303-Speed 130.44 samples/sec Loss 17.0779 Epoch: 0 Global Step: 6265 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:56:42,658-Speed 132.27 samples/sec Loss 16.4311 Epoch: 0 Global Step: 6270 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:57:02,501-Speed 129.03 samples/sec Loss 16.9390 Epoch: 0 Global Step: 6275 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:57:21,899-Speed 131.98 samples/sec Loss 16.9750 Epoch: 0 Global Step: 6280 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:57:41,721-Speed 129.15 samples/sec Loss 16.9454 Epoch: 0 Global Step: 6285 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:58:01,349-Speed 130.44 samples/sec Loss 16.6406 Epoch: 0 Global Step: 6290 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:58:21,050-Speed 129.95 samples/sec Loss 17.0125 Epoch: 0 Global Step: 6295 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:58:40,407-Speed 132.26 samples/sec Loss 16.8370 Epoch: 0 Global Step: 6300 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:59:00,274-Speed 128.86 samples/sec Loss 16.6730 Epoch: 0 Global Step: 6305 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:59:19,939-Speed 130.19 samples/sec Loss 16.6306 Epoch: 0 Global Step: 6310 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:59:39,343-Speed 131.94 samples/sec Loss 16.9586 Epoch: 0 Global Step: 6315 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 01:59:59,143-Speed 129.30 samples/sec Loss 17.0735 Epoch: 0 Global Step: 6320 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:00:18,631-Speed 131.37 samples/sec Loss 16.7835 Epoch: 0 Global Step: 6325 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:00:38,494-Speed 128.89 samples/sec Loss 17.0297 Epoch: 0 Global Step: 6330 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:00:58,105-Speed 130.55 samples/sec Loss 17.0222 Epoch: 0 Global Step: 6335 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:01:17,890-Speed 129.40 samples/sec Loss 17.0762 Epoch: 0 Global Step: 6340 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:01:37,223-Speed 132.43 samples/sec Loss 16.8836 Epoch: 0 Global Step: 6345 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:01:57,079-Speed 128.94 samples/sec Loss 16.8935 Epoch: 0 Global Step: 6350 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:02:16,574-Speed 131.32 samples/sec Loss 16.9597 Epoch: 0 Global Step: 6355 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:02:36,065-Speed 131.35 samples/sec Loss 17.0097 Epoch: 0 Global Step: 6360 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:02:55,903-Speed 129.05 samples/sec Loss 16.9839 Epoch: 0 Global Step: 6365 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:03:15,514-Speed 130.55 samples/sec Loss 16.5918 Epoch: 0 Global Step: 6370 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:03:35,352-Speed 129.05 samples/sec Loss 16.8445 Epoch: 0 Global Step: 6375 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:03:54,970-Speed 130.50 samples/sec Loss 16.9538 Epoch: 0 Global Step: 6380 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:04:15,353-Speed 125.60 samples/sec Loss 17.4414 Epoch: 0 Global Step: 6385 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:04:34,704-Speed 132.30 samples/sec Loss 16.8088 Epoch: 0 Global Step: 6390 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:04:54,559-Speed 128.94 samples/sec Loss 17.2265 Epoch: 0 Global Step: 6395 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:05:14,273-Speed 129.87 samples/sec Loss 16.9417 Epoch: 0 Global Step: 6400 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:05:33,815-Speed 131.01 samples/sec Loss 16.8545 Epoch: 0 Global Step: 6405 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:05:53,456-Speed 130.35 samples/sec Loss 16.7521 Epoch: 0 Global Step: 6410 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:06:12,910-Speed 131.61 samples/sec Loss 17.0284 Epoch: 0 Global Step: 6415 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:06:32,701-Speed 129.36 samples/sec Loss 16.8207 Epoch: 0 Global Step: 6420 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:06:52,605-Speed 128.63 samples/sec Loss 17.0115 Epoch: 0 Global Step: 6425 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:07:12,293-Speed 130.03 samples/sec Loss 17.1124 Epoch: 0 Global Step: 6430 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:07:31,649-Speed 132.27 samples/sec Loss 16.4709 Epoch: 0 Global Step: 6435 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:07:51,472-Speed 129.14 samples/sec Loss 16.6949 Epoch: 0 Global Step: 6440 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:08:11,108-Speed 130.39 samples/sec Loss 16.9226 Epoch: 0 Global Step: 6445 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:08:30,667-Speed 130.90 samples/sec Loss 16.7818 Epoch: 0 Global Step: 6450 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:08:50,297-Speed 130.42 samples/sec Loss 16.8472 Epoch: 0 Global Step: 6455 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:09:09,714-Speed 131.85 samples/sec Loss 16.4882 Epoch: 0 Global Step: 6460 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:09:29,523-Speed 129.25 samples/sec Loss 16.8149 Epoch: 0 Global Step: 6465 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:09:49,351-Speed 129.11 samples/sec Loss 17.1872 Epoch: 0 Global Step: 6470 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:10:08,971-Speed 130.49 samples/sec Loss 16.6586 Epoch: 0 Global Step: 6475 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:10:28,463-Speed 131.35 samples/sec Loss 16.1945 Epoch: 0 Global Step: 6480 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:10:48,148-Speed 130.05 samples/sec Loss 16.8274 Epoch: 0 Global Step: 6485 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:11:07,822-Speed 130.13 samples/sec Loss 16.8066 Epoch: 0 Global Step: 6490 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:11:27,352-Speed 131.09 samples/sec Loss 16.4786 Epoch: 0 Global Step: 6495 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:11:47,010-Speed 130.23 samples/sec Loss 16.8318 Epoch: 0 Global Step: 6500 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:12:06,883-Speed 128.83 samples/sec Loss 16.6861 Epoch: 0 Global Step: 6505 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:12:27,113-Speed 126.55 samples/sec Loss 16.7866 Epoch: 0 Global Step: 6510 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:12:46,950-Speed 129.06 samples/sec Loss 16.5100 Epoch: 0 Global Step: 6515 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:13:06,446-Speed 131.32 samples/sec Loss 17.0257 Epoch: 0 Global Step: 6520 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:13:26,081-Speed 130.39 samples/sec Loss 16.6247 Epoch: 0 Global Step: 6525 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:13:45,821-Speed 129.71 samples/sec Loss 16.5141 Epoch: 0 Global Step: 6530 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:14:05,592-Speed 129.49 samples/sec Loss 16.6462 Epoch: 0 Global Step: 6535 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:14:25,079-Speed 131.38 samples/sec Loss 16.7469 Epoch: 0 Global Step: 6540 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:14:44,724-Speed 130.32 samples/sec Loss 16.9780 Epoch: 0 Global Step: 6545 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:15:04,267-Speed 131.00 samples/sec Loss 16.5696 Epoch: 0 Global Step: 6550 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:15:23,927-Speed 130.22 samples/sec Loss 16.7143 Epoch: 0 Global Step: 6555 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:15:43,744-Speed 129.19 samples/sec Loss 16.6956 Epoch: 0 Global Step: 6560 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:16:03,427-Speed 130.07 samples/sec Loss 16.9411 Epoch: 0 Global Step: 6565 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:16:22,965-Speed 131.04 samples/sec Loss 16.4146 Epoch: 0 Global Step: 6570 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:16:42,593-Speed 130.43 samples/sec Loss 16.5970 Epoch: 0 Global Step: 6575 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:17:02,372-Speed 129.44 samples/sec Loss 16.8773 Epoch: 0 Global Step: 6580 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:17:21,820-Speed 131.64 samples/sec Loss 16.8740 Epoch: 0 Global Step: 6585 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:17:41,520-Speed 129.96 samples/sec Loss 16.6800 Epoch: 0 Global Step: 6590 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:18:01,023-Speed 131.27 samples/sec Loss 16.3636 Epoch: 0 Global Step: 6595 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:18:20,725-Speed 129.95 samples/sec Loss 16.7320 Epoch: 0 Global Step: 6600 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:18:40,463-Speed 129.71 samples/sec Loss 16.8808 Epoch: 0 Global Step: 6605 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:19:00,074-Speed 130.55 samples/sec Loss 16.1915 Epoch: 0 Global Step: 6610 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:19:19,645-Speed 130.81 samples/sec Loss 17.0681 Epoch: 0 Global Step: 6615 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:19:39,564-Speed 128.53 samples/sec Loss 16.3646 Epoch: 0 Global Step: 6620 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:19:59,396-Speed 129.09 samples/sec Loss 16.9402 Epoch: 0 Global Step: 6625 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:20:18,827-Speed 131.75 samples/sec Loss 16.5828 Epoch: 0 Global Step: 6630 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:20:38,411-Speed 130.73 samples/sec Loss 16.6288 Epoch: 0 Global Step: 6635 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:20:57,926-Speed 131.19 samples/sec Loss 16.5969 Epoch: 0 Global Step: 6640 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:21:17,692-Speed 129.52 samples/sec Loss 16.6421 Epoch: 0 Global Step: 6645 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:21:37,440-Speed 129.64 samples/sec Loss 16.5644 Epoch: 0 Global Step: 6650 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:21:57,405-Speed 128.24 samples/sec Loss 16.8503 Epoch: 0 Global Step: 6655 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:22:16,899-Speed 131.33 samples/sec Loss 16.7747 Epoch: 0 Global Step: 6660 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:22:36,677-Speed 129.45 samples/sec Loss 16.5438 Epoch: 0 Global Step: 6665 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:22:56,467-Speed 129.37 samples/sec Loss 16.3857 Epoch: 0 Global Step: 6670 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:23:16,951-Speed 124.98 samples/sec Loss 16.8322 Epoch: 0 Global Step: 6675 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:23:36,682-Speed 129.76 samples/sec Loss 16.5212 Epoch: 0 Global Step: 6680 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:23:56,164-Speed 131.41 samples/sec Loss 16.6752 Epoch: 0 Global Step: 6685 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:24:15,831-Speed 130.18 samples/sec Loss 16.7040 Epoch: 0 Global Step: 6690 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:24:35,585-Speed 129.60 samples/sec Loss 16.8236 Epoch: 0 Global Step: 6695 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:24:55,292-Speed 129.91 samples/sec Loss 16.5259 Epoch: 0 Global Step: 6700 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:25:14,639-Speed 132.33 samples/sec Loss 16.0379 Epoch: 0 Global Step: 6705 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:25:34,350-Speed 129.88 samples/sec Loss 16.4059 Epoch: 0 Global Step: 6710 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 02:25:54,160-Speed 129.24 samples/sec Loss 16.3218 Epoch: 0 Global Step: 6715 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:26:13,522-Speed 132.23 samples/sec Loss 16.6236 Epoch: 0 Global Step: 6720 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:26:33,396-Speed 128.82 samples/sec Loss 16.4922 Epoch: 0 Global Step: 6725 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:26:52,768-Speed 132.16 samples/sec Loss 16.5433 Epoch: 0 Global Step: 6730 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:27:12,458-Speed 130.03 samples/sec Loss 16.2002 Epoch: 0 Global Step: 6735 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:27:32,174-Speed 129.85 samples/sec Loss 16.4015 Epoch: 0 Global Step: 6740 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:27:51,994-Speed 129.17 samples/sec Loss 16.8606 Epoch: 0 Global Step: 6745 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:28:11,345-Speed 132.30 samples/sec Loss 16.5709 Epoch: 0 Global Step: 6750 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:28:31,032-Speed 130.04 samples/sec Loss 16.5337 Epoch: 0 Global Step: 6755 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:28:50,756-Speed 129.80 samples/sec Loss 16.5175 Epoch: 0 Global Step: 6760 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:29:10,138-Speed 132.09 samples/sec Loss 16.7295 Epoch: 0 Global Step: 6765 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:29:30,008-Speed 128.85 samples/sec Loss 16.7318 Epoch: 0 Global Step: 6770 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:29:49,427-Speed 131.84 samples/sec Loss 16.4108 Epoch: 0 Global Step: 6775 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:30:09,116-Speed 130.04 samples/sec Loss 16.6324 Epoch: 0 Global Step: 6780 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:30:28,840-Speed 129.80 samples/sec Loss 16.2820 Epoch: 0 Global Step: 6785 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:30:48,705-Speed 128.87 samples/sec Loss 16.4735 Epoch: 0 Global Step: 6790 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:31:08,062-Speed 132.26 samples/sec Loss 16.3366 Epoch: 0 Global Step: 6795 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:31:27,758-Speed 129.99 samples/sec Loss 16.2201 Epoch: 0 Global Step: 6800 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:31:47,413-Speed 130.25 samples/sec Loss 16.7499 Epoch: 0 Global Step: 6805 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:32:06,895-Speed 131.42 samples/sec Loss 16.7136 Epoch: 0 Global Step: 6810 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:32:26,600-Speed 129.93 samples/sec Loss 16.6492 Epoch: 0 Global Step: 6815 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:32:46,055-Speed 131.60 samples/sec Loss 16.4867 Epoch: 0 Global Step: 6820 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:33:05,703-Speed 130.30 samples/sec Loss 16.1579 Epoch: 0 Global Step: 6825 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:33:25,370-Speed 130.18 samples/sec Loss 16.3072 Epoch: 0 Global Step: 6830 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:33:45,255-Speed 128.75 samples/sec Loss 16.4716 Epoch: 0 Global Step: 6835 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:34:04,594-Speed 132.38 samples/sec Loss 16.3045 Epoch: 0 Global Step: 6840 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:34:24,414-Speed 129.17 samples/sec Loss 16.2214 Epoch: 0 Global Step: 6845 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:34:44,053-Speed 130.37 samples/sec Loss 16.3113 Epoch: 0 Global Step: 6850 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:35:03,491-Speed 131.71 samples/sec Loss 16.2972 Epoch: 0 Global Step: 6855 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:35:23,252-Speed 129.56 samples/sec Loss 16.4753 Epoch: 0 Global Step: 6860 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:35:42,721-Speed 131.50 samples/sec Loss 16.4605 Epoch: 0 Global Step: 6865 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:36:02,517-Speed 129.33 samples/sec Loss 16.0960 Epoch: 0 Global Step: 6870 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:36:22,197-Speed 130.09 samples/sec Loss 16.4646 Epoch: 0 Global Step: 6875 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:36:41,990-Speed 129.35 samples/sec Loss 16.7166 Epoch: 0 Global Step: 6880 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:37:01,330-Speed 132.37 samples/sec Loss 16.6403 Epoch: 0 Global Step: 6885 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:37:21,175-Speed 129.01 samples/sec Loss 15.9196 Epoch: 0 Global Step: 6890 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:37:40,837-Speed 130.20 samples/sec Loss 15.9554 Epoch: 0 Global Step: 6895 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:38:00,206-Speed 132.18 samples/sec Loss 16.5169 Epoch: 0 Global Step: 6900 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:38:20,027-Speed 129.16 samples/sec Loss 16.2364 Epoch: 0 Global Step: 6905 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:38:39,480-Speed 131.61 samples/sec Loss 16.1918 Epoch: 0 Global Step: 6910 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:38:59,277-Speed 129.32 samples/sec Loss 16.5261 Epoch: 0 Global Step: 6915 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:39:19,048-Speed 129.49 samples/sec Loss 16.2203 Epoch: 0 Global Step: 6920 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:39:38,778-Speed 129.75 samples/sec Loss 16.6795 Epoch: 0 Global Step: 6925 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:39:58,107-Speed 132.46 samples/sec Loss 16.3076 Epoch: 0 Global Step: 6930 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:40:17,960-Speed 128.95 samples/sec Loss 16.6039 Epoch: 0 Global Step: 6935 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:40:37,634-Speed 130.13 samples/sec Loss 16.4521 Epoch: 0 Global Step: 6940 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:40:57,083-Speed 131.63 samples/sec Loss 16.9854 Epoch: 0 Global Step: 6945 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:41:16,798-Speed 129.86 samples/sec Loss 16.2520 Epoch: 0 Global Step: 6950 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:41:36,236-Speed 131.71 samples/sec Loss 16.2836 Epoch: 0 Global Step: 6955 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:41:56,100-Speed 128.88 samples/sec Loss 16.3567 Epoch: 0 Global Step: 6960 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:42:15,706-Speed 130.58 samples/sec Loss 16.7451 Epoch: 0 Global Step: 6965 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:42:35,512-Speed 129.26 samples/sec Loss 16.1202 Epoch: 0 Global Step: 6970 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:42:54,895-Speed 132.08 samples/sec Loss 15.8069 Epoch: 0 Global Step: 6975 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:43:14,739-Speed 129.01 samples/sec Loss 16.3647 Epoch: 0 Global Step: 6980 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:43:34,364-Speed 130.46 samples/sec Loss 16.0592 Epoch: 0 Global Step: 6985 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:43:53,795-Speed 131.76 samples/sec Loss 16.4269 Epoch: 0 Global Step: 6990 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:44:13,618-Speed 129.15 samples/sec Loss 16.5081 Epoch: 0 Global Step: 6995 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:44:33,063-Speed 131.67 samples/sec Loss 16.4223 Epoch: 0 Global Step: 7000 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:44:52,923-Speed 128.91 samples/sec Loss 16.6678 Epoch: 0 Global Step: 7005 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:45:12,701-Speed 129.45 samples/sec Loss 16.5252 Epoch: 0 Global Step: 7010 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:45:32,426-Speed 129.79 samples/sec Loss 16.6082 Epoch: 0 Global Step: 7015 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:45:51,876-Speed 131.62 samples/sec Loss 16.2026 Epoch: 0 Global Step: 7020 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:46:11,745-Speed 128.85 samples/sec Loss 16.2983 Epoch: 0 Global Step: 7025 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:46:31,345-Speed 130.62 samples/sec Loss 16.1470 Epoch: 0 Global Step: 7030 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:46:50,855-Speed 131.23 samples/sec Loss 16.4451 Epoch: 0 Global Step: 7035 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:47:10,514-Speed 130.23 samples/sec Loss 16.0758 Epoch: 0 Global Step: 7040 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:47:29,978-Speed 131.53 samples/sec Loss 16.1013 Epoch: 0 Global Step: 7045 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:47:49,771-Speed 129.35 samples/sec Loss 16.3250 Epoch: 0 Global Step: 7050 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:48:09,590-Speed 129.18 samples/sec Loss 16.4080 Epoch: 0 Global Step: 7055 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:48:29,219-Speed 130.42 samples/sec Loss 16.0517 Epoch: 0 Global Step: 7060 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:48:48,687-Speed 131.51 samples/sec Loss 16.0600 Epoch: 0 Global Step: 7065 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:49:08,510-Speed 129.15 samples/sec Loss 15.9946 Epoch: 0 Global Step: 7070 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:49:28,059-Speed 130.96 samples/sec Loss 16.3093 Epoch: 0 Global Step: 7075 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:49:47,597-Speed 131.04 samples/sec Loss 16.0487 Epoch: 0 Global Step: 7080 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:50:07,246-Speed 130.29 samples/sec Loss 16.2902 Epoch: 0 Global Step: 7085 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:50:26,788-Speed 131.01 samples/sec Loss 16.1001 Epoch: 0 Global Step: 7090 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:50:46,411-Speed 130.47 samples/sec Loss 16.0932 Epoch: 0 Global Step: 7095 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:51:06,160-Speed 129.63 samples/sec Loss 16.3731 Epoch: 0 Global Step: 7100 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:51:25,738-Speed 130.77 samples/sec Loss 16.5697 Epoch: 0 Global Step: 7105 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:51:45,350-Speed 130.54 samples/sec Loss 16.2775 Epoch: 0 Global Step: 7110 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:52:05,083-Speed 129.74 samples/sec Loss 16.4970 Epoch: 0 Global Step: 7115 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:52:24,737-Speed 130.27 samples/sec Loss 16.1509 Epoch: 0 Global Step: 7120 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:52:44,230-Speed 131.34 samples/sec Loss 16.2208 Epoch: 0 Global Step: 7125 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:53:03,898-Speed 130.17 samples/sec Loss 16.1771 Epoch: 0 Global Step: 7130 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:53:23,418-Speed 131.16 samples/sec Loss 16.0125 Epoch: 0 Global Step: 7135 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:53:43,047-Speed 130.43 samples/sec Loss 16.3028 Epoch: 0 Global Step: 7140 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:54:02,818-Speed 129.49 samples/sec Loss 15.8869 Epoch: 0 Global Step: 7145 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:54:22,457-Speed 130.37 samples/sec Loss 16.5125 Epoch: 0 Global Step: 7150 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:54:42,123-Speed 130.19 samples/sec Loss 16.1026 Epoch: 0 Global Step: 7155 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:55:01,814-Speed 130.01 samples/sec Loss 16.1486 Epoch: 0 Global Step: 7160 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:55:21,426-Speed 130.55 samples/sec Loss 16.2749 Epoch: 0 Global Step: 7165 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:55:40,961-Speed 131.05 samples/sec Loss 16.1478 Epoch: 0 Global Step: 7170 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:56:00,679-Speed 129.84 samples/sec Loss 16.0307 Epoch: 0 Global Step: 7175 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:56:20,194-Speed 131.19 samples/sec Loss 16.3502 Epoch: 0 Global Step: 7180 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:56:39,812-Speed 130.50 samples/sec Loss 15.9732 Epoch: 0 Global Step: 7185 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:56:59,570-Speed 129.57 samples/sec Loss 15.9753 Epoch: 0 Global Step: 7190 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:57:19,169-Speed 130.63 samples/sec Loss 16.2078 Epoch: 0 Global Step: 7195 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:57:38,813-Speed 130.33 samples/sec Loss 16.0972 Epoch: 0 Global Step: 7200 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:57:58,499-Speed 130.04 samples/sec Loss 16.3437 Epoch: 0 Global Step: 7205 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:58:18,254-Speed 129.60 samples/sec Loss 15.8859 Epoch: 0 Global Step: 7210 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:58:37,714-Speed 131.56 samples/sec Loss 15.9673 Epoch: 0 Global Step: 7215 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:58:57,410-Speed 129.99 samples/sec Loss 16.0680 Epoch: 0 Global Step: 7220 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:59:16,910-Speed 131.31 samples/sec Loss 16.3224 Epoch: 0 Global Step: 7225 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:59:36,457-Speed 130.97 samples/sec Loss 16.0249 Epoch: 0 Global Step: 7230 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 02:59:56,188-Speed 129.76 samples/sec Loss 16.2026 Epoch: 0 Global Step: 7235 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:00:15,905-Speed 129.84 samples/sec Loss 15.9109 Epoch: 0 Global Step: 7240 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:00:35,466-Speed 130.89 samples/sec Loss 15.9026 Epoch: 0 Global Step: 7245 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:00:55,055-Speed 130.69 samples/sec Loss 16.0245 Epoch: 0 Global Step: 7250 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:01:14,890-Speed 129.07 samples/sec Loss 15.8912 Epoch: 0 Global Step: 7255 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:01:34,312-Speed 131.82 samples/sec Loss 16.1212 Epoch: 0 Global Step: 7260 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:01:54,005-Speed 130.01 samples/sec Loss 16.1236 Epoch: 0 Global Step: 7265 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:02:13,485-Speed 131.42 samples/sec Loss 16.0286 Epoch: 0 Global Step: 7270 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:02:33,122-Speed 130.38 samples/sec Loss 16.1067 Epoch: 0 Global Step: 7275 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:02:52,868-Speed 129.66 samples/sec Loss 16.0533 Epoch: 0 Global Step: 7280 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:03:12,563-Speed 129.99 samples/sec Loss 16.0981 Epoch: 0 Global Step: 7285 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:03:31,976-Speed 131.88 samples/sec Loss 16.0250 Epoch: 0 Global Step: 7290 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:03:51,551-Speed 130.78 samples/sec Loss 16.3489 Epoch: 0 Global Step: 7295 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:04:11,410-Speed 128.92 samples/sec Loss 15.8934 Epoch: 0 Global Step: 7300 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:04:30,745-Speed 132.41 samples/sec Loss 15.7262 Epoch: 0 Global Step: 7305 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:04:50,481-Speed 129.73 samples/sec Loss 15.9081 Epoch: 0 Global Step: 7310 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:05:09,917-Speed 131.72 samples/sec Loss 16.0344 Epoch: 0 Global Step: 7315 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:05:29,557-Speed 130.36 samples/sec Loss 15.6424 Epoch: 0 Global Step: 7320 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:05:49,335-Speed 129.44 samples/sec Loss 16.0808 Epoch: 0 Global Step: 7325 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:06:09,050-Speed 129.85 samples/sec Loss 15.9313 Epoch: 0 Global Step: 7330 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:06:28,557-Speed 131.25 samples/sec Loss 15.9685 Epoch: 0 Global Step: 7335 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:06:48,278-Speed 129.81 samples/sec Loss 15.8174 Epoch: 0 Global Step: 7340 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:07:08,112-Speed 129.08 samples/sec Loss 15.9377 Epoch: 0 Global Step: 7345 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:07:27,464-Speed 132.29 samples/sec Loss 16.1371 Epoch: 0 Global Step: 7350 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:07:47,303-Speed 129.05 samples/sec Loss 16.0930 Epoch: 0 Global Step: 7355 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:08:06,679-Speed 132.13 samples/sec Loss 15.9570 Epoch: 0 Global Step: 7360 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:08:26,388-Speed 129.90 samples/sec Loss 16.2936 Epoch: 0 Global Step: 7365 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:08:46,074-Speed 130.05 samples/sec Loss 16.2075 Epoch: 0 Global Step: 7370 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:09:06,960-Speed 122.57 samples/sec Loss 16.0481 Epoch: 0 Global Step: 7375 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:09:26,318-Speed 132.25 samples/sec Loss 16.4210 Epoch: 0 Global Step: 7380 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:09:46,069-Speed 129.62 samples/sec Loss 16.0112 Epoch: 0 Global Step: 7385 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:10:05,734-Speed 130.19 samples/sec Loss 16.0053 Epoch: 0 Global Step: 7390 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:10:25,087-Speed 132.29 samples/sec Loss 15.8947 Epoch: 0 Global Step: 7395 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:10:44,935-Speed 128.99 samples/sec Loss 15.8747 Epoch: 0 Global Step: 7400 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:11:04,326-Speed 132.03 samples/sec Loss 16.1070 Epoch: 0 Global Step: 7405 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:11:24,037-Speed 129.88 samples/sec Loss 16.0704 Epoch: 0 Global Step: 7410 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:11:43,735-Speed 129.97 samples/sec Loss 16.0188 Epoch: 0 Global Step: 7415 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:12:03,433-Speed 129.97 samples/sec Loss 16.1566 Epoch: 0 Global Step: 7420 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:12:22,827-Speed 132.01 samples/sec Loss 15.9569 Epoch: 0 Global Step: 7425 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:12:42,605-Speed 129.44 samples/sec Loss 16.0313 Epoch: 0 Global Step: 7430 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:13:02,252-Speed 130.31 samples/sec Loss 16.3095 Epoch: 0 Global Step: 7435 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:13:21,696-Speed 131.67 samples/sec Loss 16.2490 Epoch: 0 Global Step: 7440 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:13:41,548-Speed 128.96 samples/sec Loss 16.1011 Epoch: 0 Global Step: 7445 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:14:00,993-Speed 131.67 samples/sec Loss 16.2976 Epoch: 0 Global Step: 7450 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:14:20,648-Speed 130.25 samples/sec Loss 16.0269 Epoch: 0 Global Step: 7455 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:14:40,401-Speed 129.61 samples/sec Loss 15.8681 Epoch: 0 Global Step: 7460 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:15:00,101-Speed 129.96 samples/sec Loss 15.8201 Epoch: 0 Global Step: 7465 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:15:19,462-Speed 132.24 samples/sec Loss 16.2167 Epoch: 0 Global Step: 7470 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:15:39,276-Speed 129.21 samples/sec Loss 16.0252 Epoch: 0 Global Step: 7475 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:15:58,853-Speed 130.77 samples/sec Loss 16.0032 Epoch: 0 Global Step: 7480 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:16:18,258-Speed 131.94 samples/sec Loss 15.9324 Epoch: 0 Global Step: 7485 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:16:38,086-Speed 129.12 samples/sec Loss 15.8334 Epoch: 0 Global Step: 7490 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:16:57,476-Speed 132.04 samples/sec Loss 15.4638 Epoch: 0 Global Step: 7495 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:17:17,112-Speed 130.39 samples/sec Loss 15.9281 Epoch: 0 Global Step: 7500 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:17:36,873-Speed 129.56 samples/sec Loss 15.9214 Epoch: 0 Global Step: 7505 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:17:56,920-Speed 127.70 samples/sec Loss 15.9330 Epoch: 0 Global Step: 7510 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:18:16,270-Speed 132.31 samples/sec Loss 16.0402 Epoch: 0 Global Step: 7515 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:18:36,035-Speed 129.52 samples/sec Loss 15.9352 Epoch: 0 Global Step: 7520 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 03:18:55,535-Speed 131.29 samples/sec Loss 16.3470 Epoch: 0 Global Step: 7525 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:19:14,975-Speed 131.70 samples/sec Loss 15.8519 Epoch: 0 Global Step: 7530 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:19:34,827-Speed 128.97 samples/sec Loss 15.7434 Epoch: 0 Global Step: 7535 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:19:54,179-Speed 132.29 samples/sec Loss 16.3376 Epoch: 0 Global Step: 7540 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:20:13,900-Speed 129.83 samples/sec Loss 15.9043 Epoch: 0 Global Step: 7545 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:20:33,566-Speed 130.18 samples/sec Loss 15.8553 Epoch: 0 Global Step: 7550 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:20:53,347-Speed 129.43 samples/sec Loss 15.9800 Epoch: 0 Global Step: 7555 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:21:12,682-Speed 132.41 samples/sec Loss 15.9751 Epoch: 0 Global Step: 7560 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:21:32,624-Speed 128.38 samples/sec Loss 15.6364 Epoch: 0 Global Step: 7565 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:21:52,279-Speed 130.26 samples/sec Loss 15.7739 Epoch: 0 Global Step: 7570 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:22:11,824-Speed 130.99 samples/sec Loss 15.4198 Epoch: 0 Global Step: 7575 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:22:31,477-Speed 130.26 samples/sec Loss 15.7187 Epoch: 0 Global Step: 7580 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:22:50,885-Speed 131.92 samples/sec Loss 15.7944 Epoch: 0 Global Step: 7585 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:23:10,533-Speed 130.30 samples/sec Loss 15.5179 Epoch: 0 Global Step: 7590 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:23:30,257-Speed 129.80 samples/sec Loss 15.8143 Epoch: 0 Global Step: 7595 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:23:50,044-Speed 129.38 samples/sec Loss 15.7942 Epoch: 0 Global Step: 7600 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:24:09,400-Speed 132.27 samples/sec Loss 15.8951 Epoch: 0 Global Step: 7605 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:24:29,111-Speed 129.88 samples/sec Loss 15.8167 Epoch: 0 Global Step: 7610 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:24:48,760-Speed 130.30 samples/sec Loss 15.7718 Epoch: 0 Global Step: 7615 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:25:08,308-Speed 130.97 samples/sec Loss 15.6125 Epoch: 0 Global Step: 7620 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:25:28,008-Speed 129.96 samples/sec Loss 15.9330 Epoch: 0 Global Step: 7625 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:25:47,379-Speed 132.16 samples/sec Loss 15.5214 Epoch: 0 Global Step: 7630 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:26:07,174-Speed 129.34 samples/sec Loss 16.0379 Epoch: 0 Global Step: 7635 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:26:26,907-Speed 129.74 samples/sec Loss 15.8437 Epoch: 0 Global Step: 7640 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:26:46,692-Speed 129.39 samples/sec Loss 15.5848 Epoch: 0 Global Step: 7645 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:27:06,083-Speed 132.03 samples/sec Loss 15.8861 Epoch: 0 Global Step: 7650 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:27:25,730-Speed 130.30 samples/sec Loss 15.5378 Epoch: 0 Global Step: 7655 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:27:45,437-Speed 129.91 samples/sec Loss 15.7095 Epoch: 0 Global Step: 7660 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:28:05,002-Speed 130.85 samples/sec Loss 15.6027 Epoch: 0 Global Step: 7665 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:28:24,498-Speed 131.31 samples/sec Loss 16.1844 Epoch: 0 Global Step: 7670 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:28:43,905-Speed 131.93 samples/sec Loss 16.1259 Epoch: 0 Global Step: 7675 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:29:03,659-Speed 129.60 samples/sec Loss 15.6789 Epoch: 0 Global Step: 7680 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:29:23,542-Speed 128.76 samples/sec Loss 15.7548 Epoch: 0 Global Step: 7685 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:29:43,207-Speed 130.19 samples/sec Loss 15.7978 Epoch: 0 Global Step: 7690 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:30:02,602-Speed 132.00 samples/sec Loss 15.5951 Epoch: 0 Global Step: 7695 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:30:22,300-Speed 129.97 samples/sec Loss 15.5801 Epoch: 0 Global Step: 7700 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:30:41,978-Speed 130.10 samples/sec Loss 15.8100 Epoch: 0 Global Step: 7705 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:31:01,521-Speed 131.00 samples/sec Loss 15.8518 Epoch: 0 Global Step: 7710 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:31:21,017-Speed 131.32 samples/sec Loss 15.7256 Epoch: 0 Global Step: 7715 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:31:40,446-Speed 131.77 samples/sec Loss 15.8076 Epoch: 0 Global Step: 7720 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:32:00,238-Speed 129.36 samples/sec Loss 15.6145 Epoch: 0 Global Step: 7725 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:32:20,118-Speed 128.78 samples/sec Loss 16.1782 Epoch: 0 Global Step: 7730 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:32:39,741-Speed 130.47 samples/sec Loss 15.5591 Epoch: 0 Global Step: 7735 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:32:59,269-Speed 131.10 samples/sec Loss 15.7225 Epoch: 0 Global Step: 7740 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:33:19,058-Speed 129.37 samples/sec Loss 15.7073 Epoch: 0 Global Step: 7745 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:33:38,739-Speed 130.08 samples/sec Loss 15.5438 Epoch: 0 Global Step: 7750 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:33:58,262-Speed 131.14 samples/sec Loss 15.4154 Epoch: 0 Global Step: 7755 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:34:17,768-Speed 131.25 samples/sec Loss 15.7001 Epoch: 0 Global Step: 7760 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:34:37,174-Speed 131.93 samples/sec Loss 15.5994 Epoch: 0 Global Step: 7765 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:34:56,962-Speed 129.38 samples/sec Loss 16.0660 Epoch: 0 Global Step: 7770 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:35:16,818-Speed 128.94 samples/sec Loss 15.7515 Epoch: 0 Global Step: 7775 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:35:36,548-Speed 129.76 samples/sec Loss 15.4121 Epoch: 0 Global Step: 7780 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:35:55,926-Speed 132.12 samples/sec Loss 15.1953 Epoch: 0 Global Step: 7785 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:36:15,796-Speed 128.84 samples/sec Loss 15.7215 Epoch: 0 Global Step: 7790 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:36:35,455-Speed 130.23 samples/sec Loss 15.9686 Epoch: 0 Global Step: 7795 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:36:55,002-Speed 130.98 samples/sec Loss 15.6108 Epoch: 0 Global Step: 7800 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:37:14,657-Speed 130.25 samples/sec Loss 15.7341 Epoch: 0 Global Step: 7805 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:37:34,112-Speed 131.60 samples/sec Loss 15.4075 Epoch: 0 Global Step: 7810 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:37:53,887-Speed 129.47 samples/sec Loss 15.3412 Epoch: 0 Global Step: 7815 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:38:13,749-Speed 128.90 samples/sec Loss 15.6171 Epoch: 0 Global Step: 7820 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:38:33,270-Speed 131.15 samples/sec Loss 15.2561 Epoch: 0 Global Step: 7825 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:38:52,758-Speed 131.37 samples/sec Loss 15.1902 Epoch: 0 Global Step: 7830 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:39:12,431-Speed 130.14 samples/sec Loss 15.7467 Epoch: 0 Global Step: 7835 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:39:32,029-Speed 130.63 samples/sec Loss 15.5964 Epoch: 0 Global Step: 7840 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:39:51,556-Speed 131.11 samples/sec Loss 15.2988 Epoch: 0 Global Step: 7845 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:40:11,216-Speed 130.22 samples/sec Loss 15.6217 Epoch: 0 Global Step: 7850 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:40:30,666-Speed 131.63 samples/sec Loss 15.6918 Epoch: 0 Global Step: 7855 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:40:50,440-Speed 129.48 samples/sec Loss 15.7890 Epoch: 0 Global Step: 7860 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:41:10,311-Speed 128.84 samples/sec Loss 15.7119 Epoch: 0 Global Step: 7865 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:41:29,890-Speed 130.76 samples/sec Loss 15.8869 Epoch: 0 Global Step: 7870 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:41:49,340-Speed 131.63 samples/sec Loss 15.5892 Epoch: 0 Global Step: 7875 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:42:09,116-Speed 129.46 samples/sec Loss 15.6149 Epoch: 0 Global Step: 7880 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:42:28,783-Speed 130.17 samples/sec Loss 15.6582 Epoch: 0 Global Step: 7885 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:42:48,415-Speed 130.41 samples/sec Loss 15.9750 Epoch: 0 Global Step: 7890 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:43:08,000-Speed 130.72 samples/sec Loss 15.2751 Epoch: 0 Global Step: 7895 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:43:27,544-Speed 131.00 samples/sec Loss 15.6416 Epoch: 0 Global Step: 7900 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:43:47,049-Speed 131.26 samples/sec Loss 15.7592 Epoch: 0 Global Step: 7905 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:44:06,856-Speed 129.25 samples/sec Loss 15.3540 Epoch: 0 Global Step: 7910 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:44:26,577-Speed 129.82 samples/sec Loss 15.6766 Epoch: 0 Global Step: 7915 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:44:46,126-Speed 130.96 samples/sec Loss 15.4996 Epoch: 0 Global Step: 7920 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:45:05,830-Speed 129.93 samples/sec Loss 15.2351 Epoch: 0 Global Step: 7925 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:45:25,621-Speed 129.36 samples/sec Loss 15.4878 Epoch: 0 Global Step: 7930 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:45:45,124-Speed 131.27 samples/sec Loss 15.8999 Epoch: 0 Global Step: 7935 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:46:04,799-Speed 130.13 samples/sec Loss 15.6006 Epoch: 0 Global Step: 7940 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:46:24,327-Speed 131.10 samples/sec Loss 15.3260 Epoch: 0 Global Step: 7945 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:46:43,831-Speed 131.27 samples/sec Loss 15.5479 Epoch: 0 Global Step: 7950 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:47:03,626-Speed 129.34 samples/sec Loss 15.6466 Epoch: 0 Global Step: 7955 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:47:23,136-Speed 131.22 samples/sec Loss 15.5440 Epoch: 0 Global Step: 7960 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:47:42,806-Speed 130.16 samples/sec Loss 15.6019 Epoch: 0 Global Step: 7965 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:48:02,394-Speed 130.70 samples/sec Loss 15.3549 Epoch: 0 Global Step: 7970 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:48:22,084-Speed 130.02 samples/sec Loss 15.4367 Epoch: 0 Global Step: 7975 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:48:41,696-Speed 130.54 samples/sec Loss 15.7530 Epoch: 0 Global Step: 7980 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:49:01,359-Speed 130.21 samples/sec Loss 15.3708 Epoch: 0 Global Step: 7985 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:49:20,907-Speed 130.96 samples/sec Loss 15.1754 Epoch: 0 Global Step: 7990 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:49:40,717-Speed 129.24 samples/sec Loss 15.5422 Epoch: 0 Global Step: 7995 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:50:00,455-Speed 129.71 samples/sec Loss 15.6720 Epoch: 0 Global Step: 8000 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:50:20,525-Speed 127.56 samples/sec Loss 15.6246 Epoch: 0 Global Step: 8005 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:50:40,203-Speed 130.10 samples/sec Loss 15.3707 Epoch: 0 Global Step: 8010 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:50:59,798-Speed 130.65 samples/sec Loss 15.4025 Epoch: 0 Global Step: 8015 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:51:19,589-Speed 129.36 samples/sec Loss 15.5692 Epoch: 0 Global Step: 8020 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:51:39,015-Speed 131.80 samples/sec Loss 15.2986 Epoch: 0 Global Step: 8025 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:51:58,745-Speed 129.76 samples/sec Loss 15.3404 Epoch: 0 Global Step: 8030 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:52:18,248-Speed 131.27 samples/sec Loss 15.6041 Epoch: 0 Global Step: 8035 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:52:38,636-Speed 125.57 samples/sec Loss 15.6710 Epoch: 0 Global Step: 8040 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:52:58,392-Speed 129.60 samples/sec Loss 15.3969 Epoch: 0 Global Step: 8045 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:53:18,096-Speed 129.93 samples/sec Loss 15.6465 Epoch: 0 Global Step: 8050 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:53:37,634-Speed 131.03 samples/sec Loss 15.7872 Epoch: 0 Global Step: 8055 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:53:57,366-Speed 129.75 samples/sec Loss 15.3741 Epoch: 0 Global Step: 8060 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:54:17,213-Speed 128.99 samples/sec Loss 15.7174 Epoch: 0 Global Step: 8065 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:54:36,665-Speed 131.62 samples/sec Loss 15.4394 Epoch: 0 Global Step: 8070 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:54:56,244-Speed 130.76 samples/sec Loss 15.3591 Epoch: 0 Global Step: 8075 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:55:15,741-Speed 131.31 samples/sec Loss 15.6363 Epoch: 0 Global Step: 8080 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:55:35,433-Speed 130.01 samples/sec Loss 15.5680 Epoch: 0 Global Step: 8085 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:55:55,190-Speed 129.58 samples/sec Loss 15.6727 Epoch: 0 Global Step: 8090 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:56:14,896-Speed 129.91 samples/sec Loss 15.6541 Epoch: 0 Global Step: 8095 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:56:34,412-Speed 131.19 samples/sec Loss 15.6903 Epoch: 0 Global Step: 8100 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:56:54,129-Speed 129.84 samples/sec Loss 15.5774 Epoch: 0 Global Step: 8105 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:57:14,085-Speed 128.29 samples/sec Loss 15.2567 Epoch: 0 Global Step: 8110 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:57:33,541-Speed 131.59 samples/sec Loss 15.9305 Epoch: 0 Global Step: 8115 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:57:53,535-Speed 128.05 samples/sec Loss 16.0603 Epoch: 0 Global Step: 8120 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:58:12,974-Speed 131.70 samples/sec Loss 15.2351 Epoch: 0 Global Step: 8125 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:58:32,593-Speed 130.50 samples/sec Loss 15.5935 Epoch: 0 Global Step: 8130 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:58:52,330-Speed 129.72 samples/sec Loss 15.7694 Epoch: 0 Global Step: 8135 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:59:12,100-Speed 129.50 samples/sec Loss 15.7363 Epoch: 0 Global Step: 8140 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:59:31,543-Speed 131.68 samples/sec Loss 15.3059 Epoch: 0 Global Step: 8145 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 03:59:51,232-Speed 130.02 samples/sec Loss 15.4616 Epoch: 0 Global Step: 8150 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:00:11,058-Speed 129.13 samples/sec Loss 15.1778 Epoch: 0 Global Step: 8155 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:00:30,487-Speed 131.77 samples/sec Loss 15.3212 Epoch: 0 Global Step: 8160 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:00:50,167-Speed 130.09 samples/sec Loss 15.9676 Epoch: 0 Global Step: 8165 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:01:09,671-Speed 131.27 samples/sec Loss 15.4556 Epoch: 0 Global Step: 8170 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:01:29,514-Speed 129.02 samples/sec Loss 15.4700 Epoch: 0 Global Step: 8175 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:01:49,311-Speed 129.32 samples/sec Loss 15.3442 Epoch: 0 Global Step: 8180 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:02:09,138-Speed 129.12 samples/sec Loss 15.5033 Epoch: 0 Global Step: 8185 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:02:28,491-Speed 132.29 samples/sec Loss 15.5561 Epoch: 0 Global Step: 8190 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:02:48,221-Speed 129.76 samples/sec Loss 15.6924 Epoch: 0 Global Step: 8195 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:03:08,037-Speed 129.20 samples/sec Loss 15.0679 Epoch: 0 Global Step: 8200 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:03:27,479-Speed 131.68 samples/sec Loss 15.5104 Epoch: 0 Global Step: 8205 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:03:47,306-Speed 129.13 samples/sec Loss 15.5392 Epoch: 0 Global Step: 8210 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:04:06,641-Speed 132.41 samples/sec Loss 15.4149 Epoch: 0 Global Step: 8215 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:04:26,346-Speed 129.93 samples/sec Loss 15.3638 Epoch: 0 Global Step: 8220 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:04:46,062-Speed 129.85 samples/sec Loss 15.2122 Epoch: 0 Global Step: 8225 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:05:05,935-Speed 128.82 samples/sec Loss 15.7226 Epoch: 0 Global Step: 8230 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:05:25,271-Speed 132.41 samples/sec Loss 15.8690 Epoch: 0 Global Step: 8235 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:05:44,887-Speed 130.51 samples/sec Loss 15.4131 Epoch: 0 Global Step: 8240 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:06:04,648-Speed 129.56 samples/sec Loss 15.3453 Epoch: 0 Global Step: 8245 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:06:24,087-Speed 131.70 samples/sec Loss 15.8524 Epoch: 0 Global Step: 8250 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:06:43,917-Speed 129.11 samples/sec Loss 15.5319 Epoch: 0 Global Step: 8255 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:07:03,257-Speed 132.38 samples/sec Loss 15.3101 Epoch: 0 Global Step: 8260 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:07:22,953-Speed 129.99 samples/sec Loss 15.6752 Epoch: 0 Global Step: 8265 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:07:42,641-Speed 130.04 samples/sec Loss 15.6179 Epoch: 0 Global Step: 8270 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:08:02,480-Speed 129.05 samples/sec Loss 15.8009 Epoch: 0 Global Step: 8275 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:08:21,808-Speed 132.45 samples/sec Loss 15.5965 Epoch: 0 Global Step: 8280 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:08:41,607-Speed 129.31 samples/sec Loss 15.6518 Epoch: 0 Global Step: 8285 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:09:01,251-Speed 130.34 samples/sec Loss 15.5175 Epoch: 0 Global Step: 8290 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:09:20,641-Speed 132.03 samples/sec Loss 15.1109 Epoch: 0 Global Step: 8295 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:09:40,402-Speed 129.56 samples/sec Loss 15.1229 Epoch: 0 Global Step: 8300 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:10:00,304-Speed 128.64 samples/sec Loss 15.3571 Epoch: 0 Global Step: 8305 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:10:20,134-Speed 129.10 samples/sec Loss 15.4633 Epoch: 0 Global Step: 8310 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:10:39,723-Speed 130.69 samples/sec Loss 15.3024 Epoch: 0 Global Step: 8315 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:10:59,518-Speed 129.34 samples/sec Loss 15.6466 Epoch: 0 Global Step: 8320 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:11:18,843-Speed 132.47 samples/sec Loss 15.4832 Epoch: 0 Global Step: 8325 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:11:38,668-Speed 129.14 samples/sec Loss 15.4672 Epoch: 0 Global Step: 8330 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:11:58,263-Speed 130.66 samples/sec Loss 15.0084 Epoch: 0 Global Step: 8335 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:12:17,736-Speed 131.47 samples/sec Loss 15.3775 Epoch: 0 Global Step: 8340 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:12:37,359-Speed 130.47 samples/sec Loss 15.2025 Epoch: 0 Global Step: 8345 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:12:56,752-Speed 132.02 samples/sec Loss 15.1380 Epoch: 0 Global Step: 8350 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:13:16,626-Speed 128.82 samples/sec Loss 15.9314 Epoch: 0 Global Step: 8355 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 04:13:36,378-Speed 129.62 samples/sec Loss 15.9047 Epoch: 0 Global Step: 8360 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:13:56,016-Speed 130.36 samples/sec Loss 15.1189 Epoch: 0 Global Step: 8365 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:14:15,390-Speed 132.14 samples/sec Loss 15.6156 Epoch: 0 Global Step: 8370 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:14:35,374-Speed 128.11 samples/sec Loss 15.4613 Epoch: 0 Global Step: 8375 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:14:54,984-Speed 130.55 samples/sec Loss 15.5473 Epoch: 0 Global Step: 8380 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:15:14,530-Speed 130.98 samples/sec Loss 15.8420 Epoch: 0 Global Step: 8385 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:15:34,144-Speed 130.52 samples/sec Loss 15.2741 Epoch: 0 Global Step: 8390 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:15:53,615-Speed 131.49 samples/sec Loss 15.2042 Epoch: 0 Global Step: 8395 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:16:13,368-Speed 129.61 samples/sec Loss 15.3349 Epoch: 0 Global Step: 8400 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:16:33,214-Speed 129.01 samples/sec Loss 15.4261 Epoch: 0 Global Step: 8405 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:16:52,729-Speed 131.19 samples/sec Loss 15.7212 Epoch: 0 Global Step: 8410 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:17:12,206-Speed 131.45 samples/sec Loss 15.6139 Epoch: 0 Global Step: 8415 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:17:32,003-Speed 129.32 samples/sec Loss 15.4456 Epoch: 0 Global Step: 8420 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:17:51,623-Speed 130.49 samples/sec Loss 15.3196 Epoch: 0 Global Step: 8425 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:18:11,149-Speed 131.12 samples/sec Loss 15.3380 Epoch: 0 Global Step: 8430 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:18:30,812-Speed 130.20 samples/sec Loss 15.2696 Epoch: 0 Global Step: 8435 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:18:50,334-Speed 131.15 samples/sec Loss 15.7154 Epoch: 0 Global Step: 8440 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:19:09,980-Speed 130.32 samples/sec Loss 15.7514 Epoch: 0 Global Step: 8445 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:19:29,732-Speed 129.62 samples/sec Loss 15.0033 Epoch: 0 Global Step: 8450 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:19:49,371-Speed 130.36 samples/sec Loss 15.5542 Epoch: 0 Global Step: 8455 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:20:08,950-Speed 130.76 samples/sec Loss 15.3636 Epoch: 0 Global Step: 8460 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:20:28,525-Speed 130.79 samples/sec Loss 15.6225 Epoch: 0 Global Step: 8465 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:20:48,308-Speed 129.41 samples/sec Loss 15.4508 Epoch: 0 Global Step: 8470 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:21:07,787-Speed 131.43 samples/sec Loss 15.8858 Epoch: 0 Global Step: 8475 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:21:27,472-Speed 130.06 samples/sec Loss 15.6312 Epoch: 0 Global Step: 8480 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:21:47,014-Speed 131.01 samples/sec Loss 15.7096 Epoch: 0 Global Step: 8485 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:22:06,667-Speed 130.27 samples/sec Loss 15.4076 Epoch: 0 Global Step: 8490 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:22:26,421-Speed 129.61 samples/sec Loss 15.2940 Epoch: 0 Global Step: 8495 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:22:46,069-Speed 130.30 samples/sec Loss 15.6074 Epoch: 0 Global Step: 8500 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:23:05,721-Speed 130.28 samples/sec Loss 15.6544 Epoch: 0 Global Step: 8505 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:23:25,338-Speed 130.50 samples/sec Loss 15.3487 Epoch: 0 Global Step: 8510 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:23:45,134-Speed 129.33 samples/sec Loss 15.2646 Epoch: 0 Global Step: 8515 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:24:04,576-Speed 131.68 samples/sec Loss 15.0631 Epoch: 0 Global Step: 8520 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:24:24,233-Speed 130.24 samples/sec Loss 15.4442 Epoch: 0 Global Step: 8525 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:24:43,809-Speed 130.78 samples/sec Loss 15.4141 Epoch: 0 Global Step: 8530 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:25:03,465-Speed 130.25 samples/sec Loss 15.4439 Epoch: 0 Global Step: 8535 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:25:23,205-Speed 129.70 samples/sec Loss 15.4288 Epoch: 0 Global Step: 8540 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:25:42,968-Speed 129.53 samples/sec Loss 15.2009 Epoch: 0 Global Step: 8545 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:26:02,497-Speed 131.10 samples/sec Loss 15.2860 Epoch: 0 Global Step: 8550 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:26:22,182-Speed 130.06 samples/sec Loss 15.5319 Epoch: 0 Global Step: 8555 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:26:42,027-Speed 129.01 samples/sec Loss 15.4526 Epoch: 0 Global Step: 8560 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:27:01,464-Speed 131.71 samples/sec Loss 15.6789 Epoch: 0 Global Step: 8565 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:27:21,127-Speed 130.20 samples/sec Loss 15.3354 Epoch: 0 Global Step: 8570 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:27:40,628-Speed 131.29 samples/sec Loss 15.2365 Epoch: 0 Global Step: 8575 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:28:00,256-Speed 130.44 samples/sec Loss 15.4888 Epoch: 0 Global Step: 8580 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:28:19,984-Speed 129.78 samples/sec Loss 15.6246 Epoch: 0 Global Step: 8585 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:28:39,826-Speed 129.02 samples/sec Loss 15.1997 Epoch: 0 Global Step: 8590 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:28:59,243-Speed 131.85 samples/sec Loss 15.1909 Epoch: 0 Global Step: 8595 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:29:18,847-Speed 130.59 samples/sec Loss 15.1660 Epoch: 0 Global Step: 8600 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:29:38,668-Speed 129.17 samples/sec Loss 15.1134 Epoch: 0 Global Step: 8605 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:29:58,107-Speed 131.70 samples/sec Loss 15.1499 Epoch: 0 Global Step: 8610 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:30:17,907-Speed 129.31 samples/sec Loss 15.1719 Epoch: 0 Global Step: 8615 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:30:37,257-Speed 132.31 samples/sec Loss 15.5233 Epoch: 0 Global Step: 8620 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:30:56,929-Speed 130.14 samples/sec Loss 15.0451 Epoch: 0 Global Step: 8625 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:31:16,695-Speed 129.52 samples/sec Loss 15.1179 Epoch: 0 Global Step: 8630 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:31:36,581-Speed 128.74 samples/sec Loss 15.3506 Epoch: 0 Global Step: 8635 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:31:55,902-Speed 132.51 samples/sec Loss 15.2731 Epoch: 0 Global Step: 8640 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:32:15,635-Speed 129.74 samples/sec Loss 15.4299 Epoch: 0 Global Step: 8645 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:32:35,432-Speed 129.32 samples/sec Loss 14.8687 Epoch: 0 Global Step: 8650 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:32:54,870-Speed 131.72 samples/sec Loss 15.1755 Epoch: 0 Global Step: 8655 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:33:14,542-Speed 130.14 samples/sec Loss 15.2921 Epoch: 0 Global Step: 8660 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:33:33,977-Speed 131.73 samples/sec Loss 15.1459 Epoch: 0 Global Step: 8665 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:33:53,655-Speed 130.10 samples/sec Loss 15.1776 Epoch: 0 Global Step: 8670 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:34:13,353-Speed 129.97 samples/sec Loss 15.4112 Epoch: 0 Global Step: 8675 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:34:33,202-Speed 128.98 samples/sec Loss 15.3990 Epoch: 0 Global Step: 8680 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:34:52,801-Speed 130.62 samples/sec Loss 14.9848 Epoch: 0 Global Step: 8685 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:35:12,485-Speed 130.07 samples/sec Loss 15.1171 Epoch: 0 Global Step: 8690 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:35:32,302-Speed 129.19 samples/sec Loss 15.6466 Epoch: 0 Global Step: 8695 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:35:51,859-Speed 130.91 samples/sec Loss 15.0378 Epoch: 0 Global Step: 8700 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:36:11,648-Speed 129.37 samples/sec Loss 15.2228 Epoch: 0 Global Step: 8705 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:36:31,015-Speed 132.19 samples/sec Loss 15.3997 Epoch: 0 Global Step: 8710 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:36:50,700-Speed 130.06 samples/sec Loss 14.8370 Epoch: 0 Global Step: 8715 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:37:10,403-Speed 129.94 samples/sec Loss 15.4026 Epoch: 0 Global Step: 8720 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:37:30,207-Speed 129.27 samples/sec Loss 15.1349 Epoch: 0 Global Step: 8725 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:37:49,536-Speed 132.45 samples/sec Loss 15.2444 Epoch: 0 Global Step: 8730 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:38:09,373-Speed 129.06 samples/sec Loss 14.9756 Epoch: 0 Global Step: 8735 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:38:29,028-Speed 130.26 samples/sec Loss 15.2316 Epoch: 0 Global Step: 8740 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:38:48,408-Speed 132.11 samples/sec Loss 15.6884 Epoch: 0 Global Step: 8745 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:39:08,196-Speed 129.38 samples/sec Loss 15.2400 Epoch: 0 Global Step: 8750 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:39:27,591-Speed 132.00 samples/sec Loss 15.2542 Epoch: 0 Global Step: 8755 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:39:47,367-Speed 129.46 samples/sec Loss 15.4174 Epoch: 0 Global Step: 8760 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:40:07,027-Speed 130.23 samples/sec Loss 15.4542 Epoch: 0 Global Step: 8765 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:40:26,786-Speed 129.57 samples/sec Loss 15.2047 Epoch: 0 Global Step: 8770 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:40:46,147-Speed 132.23 samples/sec Loss 15.3306 Epoch: 0 Global Step: 8775 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:41:05,999-Speed 128.96 samples/sec Loss 15.0317 Epoch: 0 Global Step: 8780 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:41:25,638-Speed 130.36 samples/sec Loss 15.2098 Epoch: 0 Global Step: 8785 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:41:45,064-Speed 131.79 samples/sec Loss 14.9492 Epoch: 0 Global Step: 8790 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:42:04,871-Speed 129.26 samples/sec Loss 15.4210 Epoch: 0 Global Step: 8795 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:42:24,278-Speed 131.92 samples/sec Loss 15.3021 Epoch: 0 Global Step: 8800 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:42:44,067-Speed 129.37 samples/sec Loss 14.7229 Epoch: 0 Global Step: 8805 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:43:03,812-Speed 129.67 samples/sec Loss 15.4372 Epoch: 0 Global Step: 8810 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:43:23,565-Speed 129.60 samples/sec Loss 14.9823 Epoch: 0 Global Step: 8815 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:43:43,119-Speed 130.93 samples/sec Loss 15.0877 Epoch: 0 Global Step: 8820 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:44:02,885-Speed 129.52 samples/sec Loss 15.3413 Epoch: 0 Global Step: 8825 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:44:22,530-Speed 130.32 samples/sec Loss 15.0286 Epoch: 0 Global Step: 8830 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:44:42,068-Speed 131.04 samples/sec Loss 15.1664 Epoch: 0 Global Step: 8835 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:45:01,697-Speed 130.43 samples/sec Loss 15.0110 Epoch: 0 Global Step: 8840 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:45:21,121-Speed 131.80 samples/sec Loss 15.3418 Epoch: 0 Global Step: 8845 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:45:40,896-Speed 129.46 samples/sec Loss 15.1354 Epoch: 0 Global Step: 8850 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:46:00,681-Speed 129.40 samples/sec Loss 15.5336 Epoch: 0 Global Step: 8855 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:46:20,424-Speed 129.67 samples/sec Loss 15.3161 Epoch: 0 Global Step: 8860 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:46:39,786-Speed 132.23 samples/sec Loss 15.4079 Epoch: 0 Global Step: 8865 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:46:59,599-Speed 129.22 samples/sec Loss 15.2482 Epoch: 0 Global Step: 8870 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:47:19,224-Speed 130.46 samples/sec Loss 15.0604 Epoch: 0 Global Step: 8875 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:47:38,765-Speed 131.02 samples/sec Loss 15.1092 Epoch: 0 Global Step: 8880 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:47:58,402-Speed 130.37 samples/sec Loss 15.2945 Epoch: 0 Global Step: 8885 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:48:17,818-Speed 131.86 samples/sec Loss 14.8594 Epoch: 0 Global Step: 8890 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:48:37,568-Speed 129.63 samples/sec Loss 15.2332 Epoch: 0 Global Step: 8895 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:48:57,337-Speed 129.50 samples/sec Loss 15.1545 Epoch: 0 Global Step: 8900 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:49:17,132-Speed 129.33 samples/sec Loss 15.1956 Epoch: 0 Global Step: 8905 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:49:36,470-Speed 132.39 samples/sec Loss 15.0069 Epoch: 0 Global Step: 8910 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:49:56,298-Speed 129.12 samples/sec Loss 15.0242 Epoch: 0 Global Step: 8915 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:50:15,946-Speed 130.30 samples/sec Loss 15.6727 Epoch: 0 Global Step: 8920 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:50:35,497-Speed 130.95 samples/sec Loss 15.2200 Epoch: 0 Global Step: 8925 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:50:55,180-Speed 130.08 samples/sec Loss 15.3798 Epoch: 0 Global Step: 8930 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:51:14,552-Speed 132.16 samples/sec Loss 15.2973 Epoch: 0 Global Step: 8935 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:51:34,378-Speed 129.13 samples/sec Loss 15.1038 Epoch: 0 Global Step: 8940 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:51:54,073-Speed 129.99 samples/sec Loss 14.8747 Epoch: 0 Global Step: 8945 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:52:13,937-Speed 128.88 samples/sec Loss 15.2832 Epoch: 0 Global Step: 8950 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:52:33,482-Speed 130.99 samples/sec Loss 15.3518 Epoch: 0 Global Step: 8955 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:52:53,063-Speed 130.75 samples/sec Loss 15.0006 Epoch: 0 Global Step: 8960 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:53:12,971-Speed 128.61 samples/sec Loss 15.0499 Epoch: 0 Global Step: 8965 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:53:32,441-Speed 131.49 samples/sec Loss 15.2344 Epoch: 0 Global Step: 8970 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:53:52,232-Speed 129.36 samples/sec Loss 15.1070 Epoch: 0 Global Step: 8975 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:54:11,580-Speed 132.33 samples/sec Loss 15.3582 Epoch: 0 Global Step: 8980 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:54:31,426-Speed 129.00 samples/sec Loss 15.0576 Epoch: 0 Global Step: 8985 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:54:51,183-Speed 129.59 samples/sec Loss 15.1405 Epoch: 0 Global Step: 8990 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:55:10,995-Speed 129.22 samples/sec Loss 14.8215 Epoch: 0 Global Step: 8995 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:55:30,346-Speed 132.30 samples/sec Loss 14.9181 Epoch: 0 Global Step: 9000 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:55:50,173-Speed 129.12 samples/sec Loss 14.8731 Epoch: 0 Global Step: 9005 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:56:09,834-Speed 130.22 samples/sec Loss 14.8751 Epoch: 0 Global Step: 9010 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:56:29,376-Speed 131.01 samples/sec Loss 14.9616 Epoch: 0 Global Step: 9015 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:56:49,549-Speed 126.91 samples/sec Loss 14.9942 Epoch: 0 Global Step: 9020 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:57:09,013-Speed 131.53 samples/sec Loss 15.0354 Epoch: 0 Global Step: 9025 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:57:28,746-Speed 129.74 samples/sec Loss 15.3583 Epoch: 0 Global Step: 9030 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:57:48,589-Speed 129.02 samples/sec Loss 15.0827 Epoch: 0 Global Step: 9035 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:58:08,250-Speed 130.21 samples/sec Loss 15.1533 Epoch: 0 Global Step: 9040 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:58:27,628-Speed 132.12 samples/sec Loss 15.4034 Epoch: 0 Global Step: 9045 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:58:47,432-Speed 129.27 samples/sec Loss 14.8419 Epoch: 0 Global Step: 9050 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:59:07,191-Speed 129.57 samples/sec Loss 14.9828 Epoch: 0 Global Step: 9055 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:59:26,725-Speed 131.06 samples/sec Loss 15.2499 Epoch: 0 Global Step: 9060 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 04:59:46,490-Speed 129.53 samples/sec Loss 15.0914 Epoch: 0 Global Step: 9065 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:00:06,010-Speed 131.16 samples/sec Loss 14.7673 Epoch: 0 Global Step: 9070 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:00:25,666-Speed 130.25 samples/sec Loss 14.8106 Epoch: 0 Global Step: 9075 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:00:45,588-Speed 128.51 samples/sec Loss 15.1813 Epoch: 0 Global Step: 9080 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:01:05,147-Speed 130.89 samples/sec Loss 15.0988 Epoch: 0 Global Step: 9085 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:01:24,697-Speed 130.96 samples/sec Loss 15.5345 Epoch: 0 Global Step: 9090 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:01:44,362-Speed 130.18 samples/sec Loss 15.2595 Epoch: 0 Global Step: 9095 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:02:03,998-Speed 130.39 samples/sec Loss 14.7620 Epoch: 0 Global Step: 9100 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:02:23,533-Speed 131.05 samples/sec Loss 15.0087 Epoch: 0 Global Step: 9105 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:02:43,186-Speed 130.27 samples/sec Loss 15.0701 Epoch: 0 Global Step: 9110 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:03:02,713-Speed 131.11 samples/sec Loss 15.1589 Epoch: 0 Global Step: 9115 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:03:22,363-Speed 130.28 samples/sec Loss 15.1183 Epoch: 0 Global Step: 9120 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:03:42,163-Speed 129.31 samples/sec Loss 14.9168 Epoch: 0 Global Step: 9125 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:04:01,725-Speed 130.88 samples/sec Loss 14.6021 Epoch: 0 Global Step: 9130 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:04:21,290-Speed 130.85 samples/sec Loss 14.9151 Epoch: 0 Global Step: 9135 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:04:41,012-Speed 129.81 samples/sec Loss 15.1050 Epoch: 0 Global Step: 9140 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:05:00,697-Speed 130.06 samples/sec Loss 15.0119 Epoch: 0 Global Step: 9145 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:05:20,185-Speed 131.37 samples/sec Loss 14.8348 Epoch: 0 Global Step: 9150 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:05:39,695-Speed 131.22 samples/sec Loss 14.6023 Epoch: 0 Global Step: 9155 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:05:59,218-Speed 131.14 samples/sec Loss 14.9118 Epoch: 0 Global Step: 9160 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:06:18,904-Speed 130.05 samples/sec Loss 15.1953 Epoch: 0 Global Step: 9165 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:06:38,659-Speed 129.60 samples/sec Loss 14.9968 Epoch: 0 Global Step: 9170 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:06:58,341-Speed 130.07 samples/sec Loss 14.9825 Epoch: 0 Global Step: 9175 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:07:17,834-Speed 131.34 samples/sec Loss 15.0925 Epoch: 0 Global Step: 9180 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:07:37,613-Speed 129.44 samples/sec Loss 15.2188 Epoch: 0 Global Step: 9185 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:07:57,366-Speed 129.61 samples/sec Loss 15.2525 Epoch: 0 Global Step: 9190 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:08:16,688-Speed 132.50 samples/sec Loss 15.4318 Epoch: 0 Global Step: 9195 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:08:36,220-Speed 131.08 samples/sec Loss 15.1688 Epoch: 0 Global Step: 9200 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 05:08:55,709-Speed 131.36 samples/sec Loss 15.2689 Epoch: 0 Global Step: 9205 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:09:15,496-Speed 129.39 samples/sec Loss 14.8483 Epoch: 0 Global Step: 9210 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:09:35,237-Speed 129.68 samples/sec Loss 15.1142 Epoch: 0 Global Step: 9215 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:09:55,107-Speed 128.85 samples/sec Loss 14.7985 Epoch: 0 Global Step: 9220 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:10:14,466-Speed 132.24 samples/sec Loss 15.0643 Epoch: 0 Global Step: 9225 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:10:34,168-Speed 129.95 samples/sec Loss 15.1340 Epoch: 0 Global Step: 9230 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:10:54,011-Speed 129.02 samples/sec Loss 14.9878 Epoch: 0 Global Step: 9235 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:11:13,409-Speed 131.98 samples/sec Loss 14.4103 Epoch: 0 Global Step: 9240 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:11:33,118-Speed 129.90 samples/sec Loss 14.9149 Epoch: 0 Global Step: 9245 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:11:52,616-Speed 131.30 samples/sec Loss 14.9177 Epoch: 0 Global Step: 9250 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:12:12,106-Speed 131.36 samples/sec Loss 14.8646 Epoch: 0 Global Step: 9255 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:12:31,851-Speed 129.66 samples/sec Loss 14.9787 Epoch: 0 Global Step: 9260 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:12:51,552-Speed 129.95 samples/sec Loss 15.0177 Epoch: 0 Global Step: 9265 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:13:10,907-Speed 132.27 samples/sec Loss 14.8740 Epoch: 0 Global Step: 9270 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:13:30,604-Speed 129.98 samples/sec Loss 14.9214 Epoch: 0 Global Step: 9275 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:13:50,392-Speed 129.38 samples/sec Loss 15.1154 Epoch: 0 Global Step: 9280 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:14:09,734-Speed 132.37 samples/sec Loss 14.7556 Epoch: 0 Global Step: 9285 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:14:29,417-Speed 130.07 samples/sec Loss 14.8465 Epoch: 0 Global Step: 9290 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:14:48,990-Speed 130.80 samples/sec Loss 15.1653 Epoch: 0 Global Step: 9295 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:15:08,617-Speed 130.44 samples/sec Loss 14.5461 Epoch: 0 Global Step: 9300 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:15:28,346-Speed 129.77 samples/sec Loss 14.8880 Epoch: 0 Global Step: 9305 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:15:47,888-Speed 131.01 samples/sec Loss 15.0790 Epoch: 0 Global Step: 9310 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:16:07,455-Speed 130.84 samples/sec Loss 14.9674 Epoch: 0 Global Step: 9315 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:16:27,175-Speed 129.83 samples/sec Loss 15.3325 Epoch: 0 Global Step: 9320 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:16:46,956-Speed 129.42 samples/sec Loss 15.3403 Epoch: 0 Global Step: 9325 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:17:06,403-Speed 131.66 samples/sec Loss 15.0208 Epoch: 0 Global Step: 9330 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:17:26,152-Speed 129.63 samples/sec Loss 14.9570 Epoch: 0 Global Step: 9335 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:17:45,657-Speed 131.26 samples/sec Loss 14.9950 Epoch: 0 Global Step: 9340 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:18:05,315-Speed 130.24 samples/sec Loss 15.1032 Epoch: 0 Global Step: 9345 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:18:25,059-Speed 129.67 samples/sec Loss 14.8799 Epoch: 0 Global Step: 9350 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:18:44,827-Speed 129.51 samples/sec Loss 14.7421 Epoch: 0 Global Step: 9355 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:19:04,274-Speed 131.65 samples/sec Loss 15.1059 Epoch: 0 Global Step: 9360 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:19:24,022-Speed 129.64 samples/sec Loss 14.9432 Epoch: 0 Global Step: 9365 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:19:43,667-Speed 130.32 samples/sec Loss 14.9869 Epoch: 0 Global Step: 9370 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:20:03,192-Speed 131.13 samples/sec Loss 14.9919 Epoch: 0 Global Step: 9375 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:20:22,958-Speed 129.52 samples/sec Loss 14.6739 Epoch: 0 Global Step: 9380 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:20:42,394-Speed 131.73 samples/sec Loss 14.8995 Epoch: 0 Global Step: 9385 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:21:01,999-Speed 130.59 samples/sec Loss 15.1129 Epoch: 0 Global Step: 9390 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:21:21,726-Speed 129.78 samples/sec Loss 15.2161 Epoch: 0 Global Step: 9395 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:21:41,601-Speed 128.81 samples/sec Loss 14.4250 Epoch: 0 Global Step: 9400 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:22:00,939-Speed 132.39 samples/sec Loss 14.5357 Epoch: 0 Global Step: 9405 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:22:20,682-Speed 129.67 samples/sec Loss 14.8756 Epoch: 0 Global Step: 9410 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:22:40,355-Speed 130.13 samples/sec Loss 15.2529 Epoch: 0 Global Step: 9415 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:22:59,788-Speed 131.74 samples/sec Loss 14.8244 Epoch: 0 Global Step: 9420 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:23:19,611-Speed 129.15 samples/sec Loss 14.8368 Epoch: 0 Global Step: 9425 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:23:38,970-Speed 132.25 samples/sec Loss 14.7808 Epoch: 0 Global Step: 9430 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:23:58,554-Speed 130.73 samples/sec Loss 14.8252 Epoch: 0 Global Step: 9435 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:24:18,259-Speed 129.93 samples/sec Loss 15.0307 Epoch: 0 Global Step: 9440 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:24:38,087-Speed 129.12 samples/sec Loss 14.9010 Epoch: 0 Global Step: 9445 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:24:57,428-Speed 132.36 samples/sec Loss 15.0604 Epoch: 0 Global Step: 9450 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:25:17,172-Speed 129.67 samples/sec Loss 14.9030 Epoch: 0 Global Step: 9455 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:25:36,933-Speed 129.56 samples/sec Loss 14.7694 Epoch: 0 Global Step: 9460 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:25:56,357-Speed 131.81 samples/sec Loss 15.0474 Epoch: 0 Global Step: 9465 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:26:16,386-Speed 127.82 samples/sec Loss 15.0191 Epoch: 0 Global Step: 9470 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:26:35,823-Speed 131.71 samples/sec Loss 15.0723 Epoch: 0 Global Step: 9475 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:26:55,524-Speed 129.95 samples/sec Loss 14.4365 Epoch: 0 Global Step: 9480 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:27:15,174-Speed 130.29 samples/sec Loss 14.8815 Epoch: 0 Global Step: 9485 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:27:35,004-Speed 129.11 samples/sec Loss 14.6239 Epoch: 0 Global Step: 9490 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:27:54,338-Speed 132.41 samples/sec Loss 14.9721 Epoch: 0 Global Step: 9495 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:28:14,265-Speed 128.48 samples/sec Loss 15.2101 Epoch: 0 Global Step: 9500 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:28:33,928-Speed 130.20 samples/sec Loss 14.8628 Epoch: 0 Global Step: 9505 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:28:53,349-Speed 131.83 samples/sec Loss 14.9723 Epoch: 0 Global Step: 9510 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:29:13,126-Speed 129.46 samples/sec Loss 14.9906 Epoch: 0 Global Step: 9515 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:29:32,571-Speed 131.66 samples/sec Loss 14.7737 Epoch: 0 Global Step: 9520 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:29:52,378-Speed 129.26 samples/sec Loss 14.6886 Epoch: 0 Global Step: 9525 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:30:12,119-Speed 129.68 samples/sec Loss 14.8732 Epoch: 0 Global Step: 9530 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:30:31,940-Speed 129.17 samples/sec Loss 14.8888 Epoch: 0 Global Step: 9535 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:30:51,262-Speed 132.50 samples/sec Loss 15.0469 Epoch: 0 Global Step: 9540 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:31:11,071-Speed 129.24 samples/sec Loss 14.8865 Epoch: 0 Global Step: 9545 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:31:30,701-Speed 130.42 samples/sec Loss 14.7327 Epoch: 0 Global Step: 9550 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:31:50,130-Speed 131.77 samples/sec Loss 14.8154 Epoch: 0 Global Step: 9555 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:32:09,812-Speed 130.08 samples/sec Loss 14.8395 Epoch: 0 Global Step: 9560 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:32:29,225-Speed 131.88 samples/sec Loss 14.9641 Epoch: 0 Global Step: 9565 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:32:49,163-Speed 128.41 samples/sec Loss 14.9473 Epoch: 0 Global Step: 9570 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:33:08,742-Speed 130.76 samples/sec Loss 14.7896 Epoch: 0 Global Step: 9575 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:33:28,559-Speed 129.19 samples/sec Loss 15.2094 Epoch: 0 Global Step: 9580 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:33:47,877-Speed 132.52 samples/sec Loss 14.4726 Epoch: 0 Global Step: 9585 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:34:07,700-Speed 129.15 samples/sec Loss 15.0995 Epoch: 0 Global Step: 9590 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:34:27,335-Speed 130.39 samples/sec Loss 14.7213 Epoch: 0 Global Step: 9595 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:34:46,774-Speed 131.71 samples/sec Loss 14.5479 Epoch: 0 Global Step: 9600 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:35:06,538-Speed 129.54 samples/sec Loss 14.6467 Epoch: 0 Global Step: 9605 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:35:25,866-Speed 132.46 samples/sec Loss 14.6042 Epoch: 0 Global Step: 9610 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:35:45,714-Speed 128.99 samples/sec Loss 14.7381 Epoch: 0 Global Step: 9615 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:36:05,366-Speed 130.28 samples/sec Loss 14.5183 Epoch: 0 Global Step: 9620 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:36:25,171-Speed 129.27 samples/sec Loss 15.1070 Epoch: 0 Global Step: 9625 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:36:44,543-Speed 132.16 samples/sec Loss 14.8498 Epoch: 0 Global Step: 9630 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:37:04,386-Speed 129.02 samples/sec Loss 15.0228 Epoch: 0 Global Step: 9635 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:37:23,876-Speed 131.36 samples/sec Loss 14.7537 Epoch: 0 Global Step: 9640 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:37:43,420-Speed 131.00 samples/sec Loss 15.3843 Epoch: 0 Global Step: 9645 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:38:03,258-Speed 129.05 samples/sec Loss 15.3148 Epoch: 0 Global Step: 9650 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:38:22,702-Speed 131.67 samples/sec Loss 14.9355 Epoch: 0 Global Step: 9655 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:38:42,511-Speed 129.24 samples/sec Loss 14.9675 Epoch: 0 Global Step: 9660 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:39:02,254-Speed 129.67 samples/sec Loss 14.8544 Epoch: 0 Global Step: 9665 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:39:21,899-Speed 130.32 samples/sec Loss 15.0239 Epoch: 0 Global Step: 9670 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:39:41,276-Speed 132.12 samples/sec Loss 14.6115 Epoch: 0 Global Step: 9675 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:40:01,093-Speed 129.20 samples/sec Loss 14.9485 Epoch: 0 Global Step: 9680 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:40:20,684-Speed 130.68 samples/sec Loss 14.6550 Epoch: 0 Global Step: 9685 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:40:40,429-Speed 129.66 samples/sec Loss 14.9575 Epoch: 0 Global Step: 9690 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:40:59,973-Speed 130.99 samples/sec Loss 14.5220 Epoch: 0 Global Step: 9695 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:41:19,595-Speed 130.48 samples/sec Loss 14.8887 Epoch: 0 Global Step: 9700 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:41:39,230-Speed 130.39 samples/sec Loss 14.9086 Epoch: 0 Global Step: 9705 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:41:59,006-Speed 129.46 samples/sec Loss 15.2666 Epoch: 0 Global Step: 9710 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:42:18,665-Speed 130.22 samples/sec Loss 14.6071 Epoch: 0 Global Step: 9715 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:42:38,222-Speed 130.91 samples/sec Loss 14.7760 Epoch: 0 Global Step: 9720 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:42:57,973-Speed 129.62 samples/sec Loss 15.1949 Epoch: 0 Global Step: 9725 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:43:17,615-Speed 130.34 samples/sec Loss 14.8108 Epoch: 0 Global Step: 9730 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:43:37,097-Speed 131.41 samples/sec Loss 14.9470 Epoch: 0 Global Step: 9735 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:43:56,760-Speed 130.21 samples/sec Loss 14.7058 Epoch: 0 Global Step: 9740 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:44:16,282-Speed 131.14 samples/sec Loss 14.6583 Epoch: 0 Global Step: 9745 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:44:35,932-Speed 130.29 samples/sec Loss 14.7056 Epoch: 0 Global Step: 9750 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:44:55,665-Speed 129.74 samples/sec Loss 14.6179 Epoch: 0 Global Step: 9755 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:45:15,354-Speed 130.03 samples/sec Loss 14.7624 Epoch: 0 Global Step: 9760 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:45:34,926-Speed 130.81 samples/sec Loss 15.1021 Epoch: 0 Global Step: 9765 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:45:54,577-Speed 130.28 samples/sec Loss 14.8054 Epoch: 0 Global Step: 9770 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:46:14,360-Speed 129.41 samples/sec Loss 14.9597 Epoch: 0 Global Step: 9775 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:46:33,798-Speed 131.71 samples/sec Loss 14.8650 Epoch: 0 Global Step: 9780 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:46:53,500-Speed 129.95 samples/sec Loss 14.6900 Epoch: 0 Global Step: 9785 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:47:12,976-Speed 131.46 samples/sec Loss 14.9194 Epoch: 0 Global Step: 9790 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:47:32,592-Speed 130.51 samples/sec Loss 14.6369 Epoch: 0 Global Step: 9795 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:47:52,330-Speed 129.71 samples/sec Loss 14.9302 Epoch: 0 Global Step: 9800 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:48:12,212-Speed 128.77 samples/sec Loss 14.6809 Epoch: 0 Global Step: 9805 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:48:31,555-Speed 132.35 samples/sec Loss 14.7375 Epoch: 0 Global Step: 9810 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:48:51,178-Speed 130.46 samples/sec Loss 14.6391 Epoch: 0 Global Step: 9815 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:49:11,119-Speed 128.39 samples/sec Loss 14.7444 Epoch: 0 Global Step: 9820 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:49:30,493-Speed 132.15 samples/sec Loss 14.7627 Epoch: 0 Global Step: 9825 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:49:50,300-Speed 129.25 samples/sec Loss 14.8490 Epoch: 0 Global Step: 9830 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:50:09,730-Speed 131.76 samples/sec Loss 14.5247 Epoch: 0 Global Step: 9835 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:50:29,384-Speed 130.27 samples/sec Loss 15.0006 Epoch: 0 Global Step: 9840 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:50:49,080-Speed 129.98 samples/sec Loss 14.9236 Epoch: 0 Global Step: 9845 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:51:08,949-Speed 128.85 samples/sec Loss 14.8029 Epoch: 0 Global Step: 9850 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:51:28,266-Speed 132.53 samples/sec Loss 15.0463 Epoch: 0 Global Step: 9855 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:51:48,000-Speed 129.73 samples/sec Loss 14.7902 Epoch: 0 Global Step: 9860 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:52:07,875-Speed 128.81 samples/sec Loss 14.4971 Epoch: 0 Global Step: 9865 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:52:27,419-Speed 131.00 samples/sec Loss 14.8689 Epoch: 0 Global Step: 9870 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:52:47,273-Speed 128.95 samples/sec Loss 14.5305 Epoch: 0 Global Step: 9875 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:53:06,705-Speed 131.75 samples/sec Loss 14.9714 Epoch: 0 Global Step: 9880 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:53:26,510-Speed 129.27 samples/sec Loss 14.5782 Epoch: 0 Global Step: 9885 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:53:46,066-Speed 130.91 samples/sec Loss 14.7120 Epoch: 0 Global Step: 9890 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:54:05,926-Speed 128.91 samples/sec Loss 15.1456 Epoch: 0 Global Step: 9895 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:54:25,259-Speed 132.42 samples/sec Loss 14.8716 Epoch: 0 Global Step: 9900 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:54:45,116-Speed 128.93 samples/sec Loss 15.3174 Epoch: 0 Global Step: 9905 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:55:04,714-Speed 130.64 samples/sec Loss 14.9081 Epoch: 0 Global Step: 9910 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:55:24,153-Speed 131.70 samples/sec Loss 14.6862 Epoch: 0 Global Step: 9915 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:55:43,936-Speed 129.42 samples/sec Loss 14.8742 Epoch: 0 Global Step: 9920 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:56:03,374-Speed 131.70 samples/sec Loss 14.7188 Epoch: 0 Global Step: 9925 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:56:23,413-Speed 127.76 samples/sec Loss 14.6994 Epoch: 0 Global Step: 9930 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:56:43,044-Speed 130.42 samples/sec Loss 14.9608 Epoch: 0 Global Step: 9935 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:57:02,834-Speed 129.36 samples/sec Loss 15.0450 Epoch: 0 Global Step: 9940 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:57:22,173-Speed 132.38 samples/sec Loss 14.6628 Epoch: 0 Global Step: 9945 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:57:41,991-Speed 129.19 samples/sec Loss 14.7752 Epoch: 0 Global Step: 9950 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:58:01,574-Speed 130.73 samples/sec Loss 14.6544 Epoch: 0 Global Step: 9955 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:58:20,889-Speed 132.54 samples/sec Loss 14.6455 Epoch: 0 Global Step: 9960 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:58:43,786-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-07 05:58:55,054-Speed 74.93 samples/sec Loss 14.5183 Epoch: 1 Global Step: 9965 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:59:14,948-Speed 128.69 samples/sec Loss 14.0945 Epoch: 1 Global Step: 9970 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:59:34,753-Speed 129.27 samples/sec Loss 13.7952 Epoch: 1 Global Step: 9975 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 05:59:54,476-Speed 129.80 samples/sec Loss 14.2011 Epoch: 1 Global Step: 9980 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 06:00:13,952-Speed 131.45 samples/sec Loss 13.9245 Epoch: 1 Global Step: 9985 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 06:00:33,958-Speed 127.97 samples/sec Loss 14.0029 Epoch: 1 Global Step: 9990 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 06:00:53,923-Speed 128.23 samples/sec Loss 13.5498 Epoch: 1 Global Step: 9995 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 06:01:13,239-Speed 132.54 samples/sec Loss 13.6066 Epoch: 1 Global Step: 10000 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 06:16:10,735-[lfw][10000]XNorm: 22.242673 Training: 2021-06-07 06:16:10,736-[lfw][10000]Accuracy-Flip: 0.99467+-0.00379 Training: 2021-06-07 06:16:10,736-[lfw][10000]Accuracy-Highest: 0.99467 Training: 2021-06-07 06:33:46,170-[cfp_fp][10000]XNorm: 20.080025 Training: 2021-06-07 06:33:46,170-[cfp_fp][10000]Accuracy-Flip: 0.94686+-0.01008 Training: 2021-06-07 06:33:46,170-[cfp_fp][10000]Accuracy-Highest: 0.94686 Training: 2021-06-07 06:48:53,741-[agedb_30][10000]XNorm: 22.044511 Training: 2021-06-07 06:48:53,741-[agedb_30][10000]Accuracy-Flip: 0.95000+-0.01304 Training: 2021-06-07 06:48:53,741-[agedb_30][10000]Accuracy-Highest: 0.95000 Training: 2021-06-07 06:49:14,008-Speed 0.89 samples/sec Loss 14.1383 Epoch: 1 Global Step: 10005 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:49:34,282-Speed 126.28 samples/sec Loss 14.1525 Epoch: 1 Global Step: 10010 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:49:54,319-Speed 127.77 samples/sec Loss 14.0806 Epoch: 1 Global Step: 10015 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:50:14,221-Speed 128.64 samples/sec Loss 13.8696 Epoch: 1 Global Step: 10020 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:50:34,129-Speed 128.60 samples/sec Loss 14.1519 Epoch: 1 Global Step: 10025 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:50:54,030-Speed 128.64 samples/sec Loss 13.9501 Epoch: 1 Global Step: 10030 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:51:14,507-Speed 125.03 samples/sec Loss 13.8666 Epoch: 1 Global Step: 10035 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:51:35,068-Speed 124.52 samples/sec Loss 14.0864 Epoch: 1 Global Step: 10040 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:51:55,091-Speed 127.86 samples/sec Loss 14.2123 Epoch: 1 Global Step: 10045 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:52:15,030-Speed 128.40 samples/sec Loss 14.0064 Epoch: 1 Global Step: 10050 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:52:34,982-Speed 128.32 samples/sec Loss 13.8341 Epoch: 1 Global Step: 10055 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:52:54,994-Speed 127.93 samples/sec Loss 14.1318 Epoch: 1 Global Step: 10060 Fp16 Grad Scale: 8192 Required: 281 hours Training: 2021-06-07 06:53:15,011-Speed 127.90 samples/sec Loss 14.2057 Epoch: 1 Global Step: 10065 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:53:34,926-Speed 128.55 samples/sec Loss 13.9233 Epoch: 1 Global Step: 10070 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:53:54,885-Speed 128.27 samples/sec Loss 14.1793 Epoch: 1 Global Step: 10075 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:54:14,923-Speed 127.76 samples/sec Loss 14.4171 Epoch: 1 Global Step: 10080 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:54:35,720-Speed 123.10 samples/sec Loss 14.0281 Epoch: 1 Global Step: 10085 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:54:55,607-Speed 128.73 samples/sec Loss 13.8317 Epoch: 1 Global Step: 10090 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:55:15,498-Speed 128.71 samples/sec Loss 14.2526 Epoch: 1 Global Step: 10095 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:55:35,442-Speed 128.36 samples/sec Loss 14.2432 Epoch: 1 Global Step: 10100 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:55:55,327-Speed 128.75 samples/sec Loss 14.3732 Epoch: 1 Global Step: 10105 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:56:15,261-Speed 128.43 samples/sec Loss 14.1381 Epoch: 1 Global Step: 10110 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:56:35,234-Speed 128.18 samples/sec Loss 14.3445 Epoch: 1 Global Step: 10115 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:56:55,798-Speed 124.50 samples/sec Loss 13.8873 Epoch: 1 Global Step: 10120 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:57:15,570-Speed 129.49 samples/sec Loss 13.6730 Epoch: 1 Global Step: 10125 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:57:36,391-Speed 122.96 samples/sec Loss 14.0773 Epoch: 1 Global Step: 10130 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:57:56,252-Speed 128.90 samples/sec Loss 13.8317 Epoch: 1 Global Step: 10135 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:58:16,130-Speed 128.79 samples/sec Loss 14.0541 Epoch: 1 Global Step: 10140 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:58:36,006-Speed 128.81 samples/sec Loss 14.3038 Epoch: 1 Global Step: 10145 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:58:55,860-Speed 128.94 samples/sec Loss 14.2073 Epoch: 1 Global Step: 10150 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:59:15,735-Speed 128.82 samples/sec Loss 14.0176 Epoch: 1 Global Step: 10155 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:59:35,567-Speed 129.09 samples/sec Loss 14.3065 Epoch: 1 Global Step: 10160 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 06:59:55,533-Speed 128.22 samples/sec Loss 14.1034 Epoch: 1 Global Step: 10165 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:00:15,517-Speed 128.11 samples/sec Loss 14.0505 Epoch: 1 Global Step: 10170 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:00:36,122-Speed 124.24 samples/sec Loss 14.4001 Epoch: 1 Global Step: 10175 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:00:56,026-Speed 128.62 samples/sec Loss 13.9768 Epoch: 1 Global Step: 10180 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:01:15,915-Speed 128.72 samples/sec Loss 14.1638 Epoch: 1 Global Step: 10185 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:01:35,847-Speed 128.44 samples/sec Loss 14.0309 Epoch: 1 Global Step: 10190 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:01:55,722-Speed 128.82 samples/sec Loss 13.9136 Epoch: 1 Global Step: 10195 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:02:15,656-Speed 128.43 samples/sec Loss 13.8131 Epoch: 1 Global Step: 10200 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:02:36,045-Speed 125.57 samples/sec Loss 14.0925 Epoch: 1 Global Step: 10205 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:02:55,898-Speed 128.95 samples/sec Loss 13.7895 Epoch: 1 Global Step: 10210 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:03:15,948-Speed 127.69 samples/sec Loss 14.3472 Epoch: 1 Global Step: 10215 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:03:36,639-Speed 123.73 samples/sec Loss 14.0826 Epoch: 1 Global Step: 10220 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:03:56,523-Speed 128.76 samples/sec Loss 14.0551 Epoch: 1 Global Step: 10225 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:04:16,388-Speed 128.87 samples/sec Loss 14.3124 Epoch: 1 Global Step: 10230 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:04:36,227-Speed 129.05 samples/sec Loss 13.7543 Epoch: 1 Global Step: 10235 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:04:55,924-Speed 129.97 samples/sec Loss 13.9896 Epoch: 1 Global Step: 10240 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:05:15,731-Speed 129.26 samples/sec Loss 14.2418 Epoch: 1 Global Step: 10245 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:05:35,495-Speed 129.53 samples/sec Loss 14.0197 Epoch: 1 Global Step: 10250 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:05:55,403-Speed 128.61 samples/sec Loss 14.2852 Epoch: 1 Global Step: 10255 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:06:15,366-Speed 128.24 samples/sec Loss 14.2720 Epoch: 1 Global Step: 10260 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:06:36,003-Speed 124.06 samples/sec Loss 14.0480 Epoch: 1 Global Step: 10265 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:06:55,821-Speed 129.19 samples/sec Loss 14.1631 Epoch: 1 Global Step: 10270 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:07:15,738-Speed 128.54 samples/sec Loss 14.2221 Epoch: 1 Global Step: 10275 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:07:35,641-Speed 128.63 samples/sec Loss 14.2076 Epoch: 1 Global Step: 10280 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:07:55,494-Speed 128.96 samples/sec Loss 14.5087 Epoch: 1 Global Step: 10285 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:08:15,382-Speed 128.73 samples/sec Loss 14.2259 Epoch: 1 Global Step: 10290 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:08:35,294-Speed 128.57 samples/sec Loss 14.2884 Epoch: 1 Global Step: 10295 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:08:55,201-Speed 128.61 samples/sec Loss 14.2823 Epoch: 1 Global Step: 10300 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:09:15,018-Speed 129.19 samples/sec Loss 14.2329 Epoch: 1 Global Step: 10305 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:09:35,478-Speed 125.13 samples/sec Loss 14.0084 Epoch: 1 Global Step: 10310 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:09:55,368-Speed 128.75 samples/sec Loss 14.0949 Epoch: 1 Global Step: 10315 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:10:15,321-Speed 128.31 samples/sec Loss 13.8652 Epoch: 1 Global Step: 10320 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:10:35,201-Speed 128.78 samples/sec Loss 13.9622 Epoch: 1 Global Step: 10325 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:10:54,921-Speed 129.82 samples/sec Loss 14.0311 Epoch: 1 Global Step: 10330 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:11:14,757-Speed 129.07 samples/sec Loss 14.0546 Epoch: 1 Global Step: 10335 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:11:34,673-Speed 128.55 samples/sec Loss 14.0332 Epoch: 1 Global Step: 10340 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:11:54,689-Speed 127.91 samples/sec Loss 14.1240 Epoch: 1 Global Step: 10345 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:12:14,582-Speed 128.70 samples/sec Loss 14.1370 Epoch: 1 Global Step: 10350 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:12:35,165-Speed 124.38 samples/sec Loss 14.2746 Epoch: 1 Global Step: 10355 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:12:55,049-Speed 128.76 samples/sec Loss 14.0691 Epoch: 1 Global Step: 10360 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:13:14,924-Speed 128.81 samples/sec Loss 13.8802 Epoch: 1 Global Step: 10365 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:13:34,794-Speed 128.84 samples/sec Loss 13.9190 Epoch: 1 Global Step: 10370 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:13:54,686-Speed 128.70 samples/sec Loss 14.3210 Epoch: 1 Global Step: 10375 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:14:14,595-Speed 128.59 samples/sec Loss 14.4321 Epoch: 1 Global Step: 10380 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:14:34,459-Speed 128.89 samples/sec Loss 14.3188 Epoch: 1 Global Step: 10385 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:14:54,318-Speed 128.91 samples/sec Loss 14.0264 Epoch: 1 Global Step: 10390 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:15:14,237-Speed 128.53 samples/sec Loss 14.2555 Epoch: 1 Global Step: 10395 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:15:35,014-Speed 123.22 samples/sec Loss 14.2659 Epoch: 1 Global Step: 10400 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:15:54,892-Speed 128.79 samples/sec Loss 14.1199 Epoch: 1 Global Step: 10405 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:16:14,801-Speed 128.59 samples/sec Loss 14.0725 Epoch: 1 Global Step: 10410 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:16:34,540-Speed 129.70 samples/sec Loss 13.8516 Epoch: 1 Global Step: 10415 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:16:54,487-Speed 128.35 samples/sec Loss 13.9564 Epoch: 1 Global Step: 10420 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:17:14,340-Speed 128.96 samples/sec Loss 14.0552 Epoch: 1 Global Step: 10425 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:17:34,031-Speed 130.02 samples/sec Loss 14.4160 Epoch: 1 Global Step: 10430 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:17:53,998-Speed 128.22 samples/sec Loss 14.0381 Epoch: 1 Global Step: 10435 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:18:13,906-Speed 128.61 samples/sec Loss 14.3698 Epoch: 1 Global Step: 10440 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:18:34,588-Speed 123.79 samples/sec Loss 14.2532 Epoch: 1 Global Step: 10445 Fp16 Grad Scale: 8192 Required: 280 hours Training: 2021-06-07 07:18:54,592-Speed 127.98 samples/sec Loss 14.2995 Epoch: 1 Global Step: 10450 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:19:14,488-Speed 128.67 samples/sec Loss 14.1470 Epoch: 1 Global Step: 10455 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:19:34,461-Speed 128.18 samples/sec Loss 14.2209 Epoch: 1 Global Step: 10460 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:19:54,486-Speed 127.84 samples/sec Loss 14.2884 Epoch: 1 Global Step: 10465 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:20:14,448-Speed 128.25 samples/sec Loss 14.6497 Epoch: 1 Global Step: 10470 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:20:34,381-Speed 128.44 samples/sec Loss 14.2841 Epoch: 1 Global Step: 10475 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:20:54,065-Speed 130.06 samples/sec Loss 14.1995 Epoch: 1 Global Step: 10480 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:21:13,918-Speed 128.96 samples/sec Loss 14.2510 Epoch: 1 Global Step: 10485 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:21:34,369-Speed 125.19 samples/sec Loss 14.2937 Epoch: 1 Global Step: 10490 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:21:54,245-Speed 128.80 samples/sec Loss 14.7222 Epoch: 1 Global Step: 10495 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:22:14,106-Speed 128.90 samples/sec Loss 14.4178 Epoch: 1 Global Step: 10500 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:22:34,096-Speed 128.07 samples/sec Loss 13.7592 Epoch: 1 Global Step: 10505 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:22:53,977-Speed 128.78 samples/sec Loss 14.1769 Epoch: 1 Global Step: 10510 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:23:13,850-Speed 128.82 samples/sec Loss 14.0808 Epoch: 1 Global Step: 10515 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:23:33,748-Speed 128.67 samples/sec Loss 14.4644 Epoch: 1 Global Step: 10520 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:23:53,675-Speed 128.48 samples/sec Loss 14.6347 Epoch: 1 Global Step: 10525 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:24:13,569-Speed 128.69 samples/sec Loss 13.8295 Epoch: 1 Global Step: 10530 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:24:34,084-Speed 124.79 samples/sec Loss 14.2211 Epoch: 1 Global Step: 10535 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:24:53,946-Speed 128.89 samples/sec Loss 14.3951 Epoch: 1 Global Step: 10540 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:25:13,991-Speed 127.71 samples/sec Loss 14.3921 Epoch: 1 Global Step: 10545 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:25:33,962-Speed 128.19 samples/sec Loss 14.1963 Epoch: 1 Global Step: 10550 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:25:53,977-Speed 127.91 samples/sec Loss 14.6041 Epoch: 1 Global Step: 10555 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:26:13,848-Speed 128.84 samples/sec Loss 14.0655 Epoch: 1 Global Step: 10560 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:26:33,807-Speed 128.27 samples/sec Loss 14.2843 Epoch: 1 Global Step: 10565 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:26:53,686-Speed 128.79 samples/sec Loss 13.9922 Epoch: 1 Global Step: 10570 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:27:13,711-Speed 127.85 samples/sec Loss 14.4676 Epoch: 1 Global Step: 10575 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:27:34,282-Speed 124.46 samples/sec Loss 14.4734 Epoch: 1 Global Step: 10580 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:27:54,050-Speed 129.51 samples/sec Loss 14.0671 Epoch: 1 Global Step: 10585 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:28:13,743-Speed 130.00 samples/sec Loss 14.0538 Epoch: 1 Global Step: 10590 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:28:33,634-Speed 128.71 samples/sec Loss 14.3201 Epoch: 1 Global Step: 10595 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:28:53,641-Speed 127.97 samples/sec Loss 14.5057 Epoch: 1 Global Step: 10600 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:29:13,538-Speed 128.67 samples/sec Loss 14.0973 Epoch: 1 Global Step: 10605 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:29:33,436-Speed 128.67 samples/sec Loss 14.3030 Epoch: 1 Global Step: 10610 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:29:53,343-Speed 128.61 samples/sec Loss 14.4212 Epoch: 1 Global Step: 10615 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:30:13,286-Speed 128.37 samples/sec Loss 14.1600 Epoch: 1 Global Step: 10620 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:30:33,887-Speed 124.28 samples/sec Loss 14.4834 Epoch: 1 Global Step: 10625 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:30:53,760-Speed 128.82 samples/sec Loss 14.2146 Epoch: 1 Global Step: 10630 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:31:13,659-Speed 128.65 samples/sec Loss 14.0752 Epoch: 1 Global Step: 10635 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:31:33,558-Speed 128.66 samples/sec Loss 14.3501 Epoch: 1 Global Step: 10640 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:31:53,446-Speed 128.73 samples/sec Loss 14.5341 Epoch: 1 Global Step: 10645 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:32:13,334-Speed 128.73 samples/sec Loss 14.0706 Epoch: 1 Global Step: 10650 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:32:33,223-Speed 128.72 samples/sec Loss 14.0236 Epoch: 1 Global Step: 10655 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:32:53,174-Speed 128.32 samples/sec Loss 14.0429 Epoch: 1 Global Step: 10660 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:33:13,042-Speed 128.86 samples/sec Loss 14.3022 Epoch: 1 Global Step: 10665 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:33:33,508-Speed 125.09 samples/sec Loss 14.1742 Epoch: 1 Global Step: 10670 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:33:53,383-Speed 128.81 samples/sec Loss 14.0603 Epoch: 1 Global Step: 10675 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:34:13,316-Speed 128.43 samples/sec Loss 13.7397 Epoch: 1 Global Step: 10680 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:34:33,201-Speed 128.75 samples/sec Loss 13.9213 Epoch: 1 Global Step: 10685 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:34:53,175-Speed 128.17 samples/sec Loss 14.4117 Epoch: 1 Global Step: 10690 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:35:13,177-Speed 127.99 samples/sec Loss 14.5119 Epoch: 1 Global Step: 10695 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:35:33,151-Speed 128.18 samples/sec Loss 14.3183 Epoch: 1 Global Step: 10700 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:35:53,159-Speed 127.96 samples/sec Loss 13.9721 Epoch: 1 Global Step: 10705 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:36:13,028-Speed 128.85 samples/sec Loss 14.2929 Epoch: 1 Global Step: 10710 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:36:33,710-Speed 123.79 samples/sec Loss 14.6104 Epoch: 1 Global Step: 10715 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:36:53,651-Speed 128.38 samples/sec Loss 14.3540 Epoch: 1 Global Step: 10720 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:37:13,657-Speed 127.97 samples/sec Loss 14.3735 Epoch: 1 Global Step: 10725 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:37:33,621-Speed 128.24 samples/sec Loss 14.2267 Epoch: 1 Global Step: 10730 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:37:53,523-Speed 128.64 samples/sec Loss 14.2809 Epoch: 1 Global Step: 10735 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:38:13,474-Speed 128.32 samples/sec Loss 14.2730 Epoch: 1 Global Step: 10740 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:38:33,421-Speed 128.35 samples/sec Loss 14.2476 Epoch: 1 Global Step: 10745 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:38:54,270-Speed 122.79 samples/sec Loss 14.1810 Epoch: 1 Global Step: 10750 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:39:14,041-Speed 129.50 samples/sec Loss 14.5502 Epoch: 1 Global Step: 10755 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:39:34,616-Speed 124.43 samples/sec Loss 14.2246 Epoch: 1 Global Step: 10760 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:39:54,462-Speed 128.99 samples/sec Loss 14.2868 Epoch: 1 Global Step: 10765 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:40:14,426-Speed 128.24 samples/sec Loss 14.6037 Epoch: 1 Global Step: 10770 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:40:34,304-Speed 128.80 samples/sec Loss 14.4594 Epoch: 1 Global Step: 10775 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:40:54,279-Speed 128.17 samples/sec Loss 14.3536 Epoch: 1 Global Step: 10780 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:41:14,127-Speed 128.99 samples/sec Loss 14.3071 Epoch: 1 Global Step: 10785 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:41:34,105-Speed 128.15 samples/sec Loss 14.1794 Epoch: 1 Global Step: 10790 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:41:54,090-Speed 128.10 samples/sec Loss 14.4793 Epoch: 1 Global Step: 10795 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:42:13,945-Speed 128.95 samples/sec Loss 14.4118 Epoch: 1 Global Step: 10800 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:42:34,765-Speed 122.96 samples/sec Loss 14.4211 Epoch: 1 Global Step: 10805 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:42:54,603-Speed 129.05 samples/sec Loss 14.4841 Epoch: 1 Global Step: 10810 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:43:14,403-Speed 129.30 samples/sec Loss 14.0960 Epoch: 1 Global Step: 10815 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:43:34,178-Speed 129.46 samples/sec Loss 14.4120 Epoch: 1 Global Step: 10820 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:43:53,915-Speed 129.72 samples/sec Loss 14.2007 Epoch: 1 Global Step: 10825 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:44:13,613-Speed 129.97 samples/sec Loss 14.3316 Epoch: 1 Global Step: 10830 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:44:33,488-Speed 128.81 samples/sec Loss 14.1212 Epoch: 1 Global Step: 10835 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:44:53,215-Speed 129.78 samples/sec Loss 14.2369 Epoch: 1 Global Step: 10840 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:45:13,059-Speed 129.01 samples/sec Loss 14.5061 Epoch: 1 Global Step: 10845 Fp16 Grad Scale: 8192 Required: 279 hours Training: 2021-06-07 07:45:33,311-Speed 126.41 samples/sec Loss 14.4726 Epoch: 1 Global Step: 10850 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:45:53,090-Speed 129.43 samples/sec Loss 14.4383 Epoch: 1 Global Step: 10855 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:46:12,828-Speed 129.71 samples/sec Loss 14.2928 Epoch: 1 Global Step: 10860 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:46:32,714-Speed 128.74 samples/sec Loss 14.2327 Epoch: 1 Global Step: 10865 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:46:52,747-Speed 127.80 samples/sec Loss 14.6247 Epoch: 1 Global Step: 10870 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:47:13,007-Speed 126.37 samples/sec Loss 14.0790 Epoch: 1 Global Step: 10875 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:47:33,105-Speed 127.39 samples/sec Loss 14.3546 Epoch: 1 Global Step: 10880 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:47:53,103-Speed 128.02 samples/sec Loss 14.1834 Epoch: 1 Global Step: 10885 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:48:12,988-Speed 128.75 samples/sec Loss 14.7454 Epoch: 1 Global Step: 10890 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:48:33,539-Speed 124.58 samples/sec Loss 14.3338 Epoch: 1 Global Step: 10895 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:48:53,413-Speed 128.81 samples/sec Loss 14.1981 Epoch: 1 Global Step: 10900 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:49:13,160-Speed 129.65 samples/sec Loss 14.4446 Epoch: 1 Global Step: 10905 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:49:33,240-Speed 127.50 samples/sec Loss 14.3924 Epoch: 1 Global Step: 10910 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:49:53,083-Speed 129.02 samples/sec Loss 14.1266 Epoch: 1 Global Step: 10915 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:50:13,474-Speed 125.55 samples/sec Loss 14.3631 Epoch: 1 Global Step: 10920 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:50:33,311-Speed 129.07 samples/sec Loss 14.2027 Epoch: 1 Global Step: 10925 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:50:53,328-Speed 127.90 samples/sec Loss 13.9270 Epoch: 1 Global Step: 10930 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:51:13,046-Speed 129.84 samples/sec Loss 14.1283 Epoch: 1 Global Step: 10935 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:51:33,676-Speed 124.10 samples/sec Loss 14.1872 Epoch: 1 Global Step: 10940 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:51:53,553-Speed 128.80 samples/sec Loss 14.2561 Epoch: 1 Global Step: 10945 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:52:13,431-Speed 128.79 samples/sec Loss 14.5468 Epoch: 1 Global Step: 10950 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:52:33,327-Speed 128.67 samples/sec Loss 13.9624 Epoch: 1 Global Step: 10955 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:52:53,127-Speed 129.30 samples/sec Loss 14.2135 Epoch: 1 Global Step: 10960 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:53:13,145-Speed 127.89 samples/sec Loss 14.2681 Epoch: 1 Global Step: 10965 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:53:33,042-Speed 128.67 samples/sec Loss 14.0748 Epoch: 1 Global Step: 10970 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:53:52,873-Speed 129.10 samples/sec Loss 14.0537 Epoch: 1 Global Step: 10975 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:54:12,720-Speed 128.99 samples/sec Loss 14.2290 Epoch: 1 Global Step: 10980 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:54:33,172-Speed 125.18 samples/sec Loss 14.4627 Epoch: 1 Global Step: 10985 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:54:53,032-Speed 128.90 samples/sec Loss 14.3614 Epoch: 1 Global Step: 10990 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:55:13,206-Speed 126.90 samples/sec Loss 13.8711 Epoch: 1 Global Step: 10995 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:55:33,099-Speed 128.70 samples/sec Loss 14.6003 Epoch: 1 Global Step: 11000 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:55:52,959-Speed 128.91 samples/sec Loss 14.3099 Epoch: 1 Global Step: 11005 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:56:12,811-Speed 128.96 samples/sec Loss 14.1398 Epoch: 1 Global Step: 11010 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:56:32,643-Speed 129.10 samples/sec Loss 14.2580 Epoch: 1 Global Step: 11015 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:56:52,484-Speed 129.03 samples/sec Loss 14.4913 Epoch: 1 Global Step: 11020 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:57:12,330-Speed 129.00 samples/sec Loss 14.1526 Epoch: 1 Global Step: 11025 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:57:32,862-Speed 124.69 samples/sec Loss 14.2168 Epoch: 1 Global Step: 11030 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:57:52,549-Speed 130.04 samples/sec Loss 14.4325 Epoch: 1 Global Step: 11035 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:58:12,758-Speed 126.68 samples/sec Loss 14.2336 Epoch: 1 Global Step: 11040 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:58:32,874-Speed 127.27 samples/sec Loss 14.3365 Epoch: 1 Global Step: 11045 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:58:52,939-Speed 127.60 samples/sec Loss 14.5209 Epoch: 1 Global Step: 11050 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:59:12,864-Speed 128.49 samples/sec Loss 14.0909 Epoch: 1 Global Step: 11055 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:59:32,850-Speed 128.10 samples/sec Loss 13.9538 Epoch: 1 Global Step: 11060 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 07:59:52,749-Speed 128.66 samples/sec Loss 14.2617 Epoch: 1 Global Step: 11065 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:00:12,620-Speed 128.84 samples/sec Loss 14.1620 Epoch: 1 Global Step: 11070 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:00:33,181-Speed 124.52 samples/sec Loss 14.2409 Epoch: 1 Global Step: 11075 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:00:53,142-Speed 128.25 samples/sec Loss 14.0801 Epoch: 1 Global Step: 11080 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:01:13,211-Speed 127.57 samples/sec Loss 14.4784 Epoch: 1 Global Step: 11085 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:01:33,044-Speed 129.08 samples/sec Loss 14.3461 Epoch: 1 Global Step: 11090 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:01:53,033-Speed 128.08 samples/sec Loss 14.3108 Epoch: 1 Global Step: 11095 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:02:13,377-Speed 125.84 samples/sec Loss 14.5588 Epoch: 1 Global Step: 11100 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:02:33,331-Speed 128.31 samples/sec Loss 14.1550 Epoch: 1 Global Step: 11105 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:02:53,175-Speed 129.01 samples/sec Loss 13.8428 Epoch: 1 Global Step: 11110 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:03:13,101-Speed 128.48 samples/sec Loss 14.4688 Epoch: 1 Global Step: 11115 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:03:33,907-Speed 123.05 samples/sec Loss 14.3703 Epoch: 1 Global Step: 11120 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:03:54,044-Speed 127.14 samples/sec Loss 14.2667 Epoch: 1 Global Step: 11125 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:04:13,901-Speed 128.92 samples/sec Loss 14.2158 Epoch: 1 Global Step: 11130 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:04:33,749-Speed 128.99 samples/sec Loss 14.4399 Epoch: 1 Global Step: 11135 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:04:53,655-Speed 128.61 samples/sec Loss 14.0497 Epoch: 1 Global Step: 11140 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:05:13,529-Speed 128.82 samples/sec Loss 14.4027 Epoch: 1 Global Step: 11145 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:05:33,442-Speed 128.56 samples/sec Loss 14.3021 Epoch: 1 Global Step: 11150 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:05:53,454-Speed 127.93 samples/sec Loss 14.2100 Epoch: 1 Global Step: 11155 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:06:13,332-Speed 128.80 samples/sec Loss 14.2993 Epoch: 1 Global Step: 11160 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:06:33,905-Speed 124.45 samples/sec Loss 14.0117 Epoch: 1 Global Step: 11165 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:06:53,894-Speed 128.07 samples/sec Loss 14.2626 Epoch: 1 Global Step: 11170 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:07:13,719-Speed 129.14 samples/sec Loss 14.3041 Epoch: 1 Global Step: 11175 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:07:33,955-Speed 126.51 samples/sec Loss 14.2511 Epoch: 1 Global Step: 11180 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:07:53,905-Speed 128.33 samples/sec Loss 14.2238 Epoch: 1 Global Step: 11185 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:08:13,716-Speed 129.23 samples/sec Loss 14.3678 Epoch: 1 Global Step: 11190 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:08:33,569-Speed 128.96 samples/sec Loss 14.2552 Epoch: 1 Global Step: 11195 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:08:53,470-Speed 128.64 samples/sec Loss 14.4155 Epoch: 1 Global Step: 11200 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:09:13,643-Speed 126.91 samples/sec Loss 14.3544 Epoch: 1 Global Step: 11205 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:09:34,216-Speed 124.44 samples/sec Loss 14.2655 Epoch: 1 Global Step: 11210 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:09:54,129-Speed 128.57 samples/sec Loss 14.2079 Epoch: 1 Global Step: 11215 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:10:14,056-Speed 128.47 samples/sec Loss 14.3063 Epoch: 1 Global Step: 11220 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:10:34,041-Speed 128.11 samples/sec Loss 14.1040 Epoch: 1 Global Step: 11225 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:10:54,022-Speed 128.13 samples/sec Loss 14.3053 Epoch: 1 Global Step: 11230 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:11:14,144-Speed 127.23 samples/sec Loss 14.0890 Epoch: 1 Global Step: 11235 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:11:34,022-Speed 128.79 samples/sec Loss 14.0656 Epoch: 1 Global Step: 11240 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:11:53,780-Speed 129.57 samples/sec Loss 14.1345 Epoch: 1 Global Step: 11245 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:12:13,743-Speed 128.25 samples/sec Loss 14.0299 Epoch: 1 Global Step: 11250 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:12:34,431-Speed 123.75 samples/sec Loss 14.0847 Epoch: 1 Global Step: 11255 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:12:54,284-Speed 128.95 samples/sec Loss 14.1379 Epoch: 1 Global Step: 11260 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:13:14,319-Speed 127.78 samples/sec Loss 14.0133 Epoch: 1 Global Step: 11265 Fp16 Grad Scale: 8192 Required: 278 hours Training: 2021-06-07 08:13:34,131-Speed 129.22 samples/sec Loss 14.3976 Epoch: 1 Global Step: 11270 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:13:54,140-Speed 127.95 samples/sec Loss 14.2712 Epoch: 1 Global Step: 11275 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:14:14,019-Speed 128.79 samples/sec Loss 14.1180 Epoch: 1 Global Step: 11280 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:14:33,892-Speed 128.83 samples/sec Loss 14.2670 Epoch: 1 Global Step: 11285 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:14:53,860-Speed 128.22 samples/sec Loss 14.0756 Epoch: 1 Global Step: 11290 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:15:13,683-Speed 129.15 samples/sec Loss 14.3843 Epoch: 1 Global Step: 11295 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:15:34,244-Speed 124.51 samples/sec Loss 14.0500 Epoch: 1 Global Step: 11300 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:15:54,177-Speed 128.43 samples/sec Loss 14.1308 Epoch: 1 Global Step: 11305 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:16:13,891-Speed 129.87 samples/sec Loss 13.9895 Epoch: 1 Global Step: 11310 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:16:33,774-Speed 128.76 samples/sec Loss 14.0206 Epoch: 1 Global Step: 11315 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:16:53,658-Speed 128.75 samples/sec Loss 14.1242 Epoch: 1 Global Step: 11320 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:17:13,347-Speed 130.03 samples/sec Loss 14.0283 Epoch: 1 Global Step: 11325 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:17:33,208-Speed 128.90 samples/sec Loss 14.5782 Epoch: 1 Global Step: 11330 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:17:53,056-Speed 128.99 samples/sec Loss 14.1724 Epoch: 1 Global Step: 11335 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:18:12,894-Speed 129.05 samples/sec Loss 14.3726 Epoch: 1 Global Step: 11340 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:18:33,310-Speed 125.40 samples/sec Loss 14.4299 Epoch: 1 Global Step: 11345 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:18:53,305-Speed 128.04 samples/sec Loss 13.9953 Epoch: 1 Global Step: 11350 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:19:13,172-Speed 128.86 samples/sec Loss 13.9457 Epoch: 1 Global Step: 11355 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:19:33,012-Speed 129.04 samples/sec Loss 14.0843 Epoch: 1 Global Step: 11360 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:19:52,829-Speed 129.19 samples/sec Loss 14.3097 Epoch: 1 Global Step: 11365 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:20:12,546-Speed 129.84 samples/sec Loss 14.2236 Epoch: 1 Global Step: 11370 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:20:32,349-Speed 129.28 samples/sec Loss 14.0103 Epoch: 1 Global Step: 11375 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:20:52,091-Speed 129.68 samples/sec Loss 14.3128 Epoch: 1 Global Step: 11380 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:21:11,941-Speed 128.98 samples/sec Loss 14.0750 Epoch: 1 Global Step: 11385 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:21:32,714-Speed 123.24 samples/sec Loss 14.0199 Epoch: 1 Global Step: 11390 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:21:52,623-Speed 128.59 samples/sec Loss 14.1123 Epoch: 1 Global Step: 11395 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:22:12,456-Speed 129.08 samples/sec Loss 14.1079 Epoch: 1 Global Step: 11400 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:22:32,295-Speed 129.05 samples/sec Loss 13.8049 Epoch: 1 Global Step: 11405 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:22:52,174-Speed 128.79 samples/sec Loss 14.2852 Epoch: 1 Global Step: 11410 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:23:12,051-Speed 128.80 samples/sec Loss 14.2152 Epoch: 1 Global Step: 11415 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:23:32,057-Speed 127.97 samples/sec Loss 14.6916 Epoch: 1 Global Step: 11420 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:23:51,940-Speed 128.76 samples/sec Loss 14.2273 Epoch: 1 Global Step: 11425 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:24:11,634-Speed 130.00 samples/sec Loss 14.1161 Epoch: 1 Global Step: 11430 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:24:32,250-Speed 124.18 samples/sec Loss 14.6659 Epoch: 1 Global Step: 11435 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:24:52,229-Speed 128.14 samples/sec Loss 14.2757 Epoch: 1 Global Step: 11440 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:25:11,842-Speed 130.53 samples/sec Loss 13.9253 Epoch: 1 Global Step: 11445 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:25:31,628-Speed 129.39 samples/sec Loss 14.1332 Epoch: 1 Global Step: 11450 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:25:51,451-Speed 129.15 samples/sec Loss 13.9541 Epoch: 1 Global Step: 11455 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:26:11,279-Speed 129.12 samples/sec Loss 14.0350 Epoch: 1 Global Step: 11460 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:26:31,423-Speed 127.09 samples/sec Loss 14.2884 Epoch: 1 Global Step: 11465 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:26:51,412-Speed 128.08 samples/sec Loss 14.3261 Epoch: 1 Global Step: 11470 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:27:11,299-Speed 128.73 samples/sec Loss 14.0601 Epoch: 1 Global Step: 11475 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:27:31,966-Speed 123.88 samples/sec Loss 14.2248 Epoch: 1 Global Step: 11480 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:27:52,035-Speed 127.57 samples/sec Loss 14.1047 Epoch: 1 Global Step: 11485 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:28:11,873-Speed 129.05 samples/sec Loss 14.4857 Epoch: 1 Global Step: 11490 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:28:31,728-Speed 128.94 samples/sec Loss 14.0674 Epoch: 1 Global Step: 11495 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:28:51,562-Speed 129.08 samples/sec Loss 14.6591 Epoch: 1 Global Step: 11500 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:29:11,393-Speed 129.10 samples/sec Loss 13.8702 Epoch: 1 Global Step: 11505 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:29:31,241-Speed 128.99 samples/sec Loss 14.1069 Epoch: 1 Global Step: 11510 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:29:51,055-Speed 129.21 samples/sec Loss 13.9311 Epoch: 1 Global Step: 11515 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:30:10,890-Speed 129.07 samples/sec Loss 13.9487 Epoch: 1 Global Step: 11520 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:30:31,260-Speed 125.68 samples/sec Loss 14.1910 Epoch: 1 Global Step: 11525 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:30:51,098-Speed 129.05 samples/sec Loss 14.0921 Epoch: 1 Global Step: 11530 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:31:10,923-Speed 129.14 samples/sec Loss 14.2860 Epoch: 1 Global Step: 11535 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:31:30,747-Speed 129.14 samples/sec Loss 14.5496 Epoch: 1 Global Step: 11540 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:31:51,942-Speed 120.79 samples/sec Loss 14.2931 Epoch: 1 Global Step: 11545 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:32:12,080-Speed 127.13 samples/sec Loss 13.8194 Epoch: 1 Global Step: 11550 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:32:31,956-Speed 128.81 samples/sec Loss 13.9836 Epoch: 1 Global Step: 11555 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:32:51,815-Speed 128.91 samples/sec Loss 14.2525 Epoch: 1 Global Step: 11560 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:33:11,766-Speed 128.33 samples/sec Loss 14.1942 Epoch: 1 Global Step: 11565 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:33:32,156-Speed 125.56 samples/sec Loss 13.9385 Epoch: 1 Global Step: 11570 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:33:52,103-Speed 128.35 samples/sec Loss 14.0661 Epoch: 1 Global Step: 11575 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:34:11,780-Speed 130.11 samples/sec Loss 13.8256 Epoch: 1 Global Step: 11580 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:34:31,597-Speed 129.19 samples/sec Loss 13.7738 Epoch: 1 Global Step: 11585 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:34:51,368-Speed 129.49 samples/sec Loss 14.3250 Epoch: 1 Global Step: 11590 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:35:11,221-Speed 128.95 samples/sec Loss 14.1253 Epoch: 1 Global Step: 11595 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:35:31,495-Speed 126.28 samples/sec Loss 14.5953 Epoch: 1 Global Step: 11600 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:35:51,539-Speed 127.73 samples/sec Loss 14.1908 Epoch: 1 Global Step: 11605 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:36:11,529-Speed 128.07 samples/sec Loss 13.8694 Epoch: 1 Global Step: 11610 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:36:32,206-Speed 123.82 samples/sec Loss 13.8580 Epoch: 1 Global Step: 11615 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:36:52,186-Speed 128.13 samples/sec Loss 14.0739 Epoch: 1 Global Step: 11620 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:37:12,040-Speed 128.95 samples/sec Loss 14.1917 Epoch: 1 Global Step: 11625 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:37:31,882-Speed 129.03 samples/sec Loss 14.1812 Epoch: 1 Global Step: 11630 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:37:51,654-Speed 129.48 samples/sec Loss 13.9422 Epoch: 1 Global Step: 11635 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:38:11,412-Speed 129.57 samples/sec Loss 14.2922 Epoch: 1 Global Step: 11640 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:38:31,254-Speed 129.03 samples/sec Loss 14.0750 Epoch: 1 Global Step: 11645 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:38:51,127-Speed 128.82 samples/sec Loss 14.3833 Epoch: 1 Global Step: 11650 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:39:10,963-Speed 129.07 samples/sec Loss 14.1271 Epoch: 1 Global Step: 11655 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:39:31,712-Speed 123.38 samples/sec Loss 13.5423 Epoch: 1 Global Step: 11660 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:39:51,628-Speed 128.55 samples/sec Loss 14.1418 Epoch: 1 Global Step: 11665 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:40:11,638-Speed 127.94 samples/sec Loss 14.2586 Epoch: 1 Global Step: 11670 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:40:31,469-Speed 129.10 samples/sec Loss 14.0210 Epoch: 1 Global Step: 11675 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:40:51,097-Speed 130.43 samples/sec Loss 14.0111 Epoch: 1 Global Step: 11680 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:41:11,106-Speed 127.95 samples/sec Loss 14.0079 Epoch: 1 Global Step: 11685 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:41:30,795-Speed 130.03 samples/sec Loss 14.0524 Epoch: 1 Global Step: 11690 Fp16 Grad Scale: 8192 Required: 277 hours Training: 2021-06-07 08:41:50,698-Speed 128.63 samples/sec Loss 14.2336 Epoch: 1 Global Step: 11695 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:42:10,532-Speed 129.08 samples/sec Loss 14.0565 Epoch: 1 Global Step: 11700 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:42:30,938-Speed 125.46 samples/sec Loss 13.9212 Epoch: 1 Global Step: 11705 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:42:50,566-Speed 130.43 samples/sec Loss 14.2223 Epoch: 1 Global Step: 11710 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:43:10,358-Speed 129.35 samples/sec Loss 14.2935 Epoch: 1 Global Step: 11715 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:43:30,062-Speed 129.93 samples/sec Loss 14.1132 Epoch: 1 Global Step: 11720 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:43:49,921-Speed 128.92 samples/sec Loss 14.2951 Epoch: 1 Global Step: 11725 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:44:09,787-Speed 128.87 samples/sec Loss 14.0249 Epoch: 1 Global Step: 11730 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:44:29,605-Speed 129.18 samples/sec Loss 14.0202 Epoch: 1 Global Step: 11735 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:44:49,489-Speed 128.75 samples/sec Loss 14.0818 Epoch: 1 Global Step: 11740 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:45:09,348-Speed 128.92 samples/sec Loss 13.9806 Epoch: 1 Global Step: 11745 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:45:30,143-Speed 123.11 samples/sec Loss 14.2842 Epoch: 1 Global Step: 11750 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:45:50,196-Speed 127.67 samples/sec Loss 13.8522 Epoch: 1 Global Step: 11755 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:46:10,057-Speed 128.90 samples/sec Loss 14.2705 Epoch: 1 Global Step: 11760 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:46:30,607-Speed 124.58 samples/sec Loss 14.3583 Epoch: 1 Global Step: 11765 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:46:50,526-Speed 128.53 samples/sec Loss 13.9592 Epoch: 1 Global Step: 11770 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:47:10,898-Speed 125.67 samples/sec Loss 13.9890 Epoch: 1 Global Step: 11775 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:47:30,734-Speed 129.07 samples/sec Loss 14.2960 Epoch: 1 Global Step: 11780 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:47:50,386-Speed 130.28 samples/sec Loss 13.8590 Epoch: 1 Global Step: 11785 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:48:10,298-Speed 128.57 samples/sec Loss 14.0928 Epoch: 1 Global Step: 11790 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:48:30,840-Speed 124.63 samples/sec Loss 14.4277 Epoch: 1 Global Step: 11795 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:48:50,739-Speed 128.66 samples/sec Loss 14.3124 Epoch: 1 Global Step: 11800 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:49:10,630-Speed 128.71 samples/sec Loss 14.5249 Epoch: 1 Global Step: 11805 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:49:30,488-Speed 128.93 samples/sec Loss 14.0528 Epoch: 1 Global Step: 11810 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:49:50,553-Speed 127.59 samples/sec Loss 14.1302 Epoch: 1 Global Step: 11815 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:50:10,188-Speed 130.39 samples/sec Loss 14.0193 Epoch: 1 Global Step: 11820 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:50:30,004-Speed 129.19 samples/sec Loss 14.2100 Epoch: 1 Global Step: 11825 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:50:49,830-Speed 129.13 samples/sec Loss 14.2161 Epoch: 1 Global Step: 11830 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:51:09,547-Speed 129.85 samples/sec Loss 14.3266 Epoch: 1 Global Step: 11835 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:51:30,308-Speed 123.31 samples/sec Loss 14.0310 Epoch: 1 Global Step: 11840 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:51:50,198-Speed 128.71 samples/sec Loss 14.1585 Epoch: 1 Global Step: 11845 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:52:10,437-Speed 126.49 samples/sec Loss 13.9485 Epoch: 1 Global Step: 11850 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:52:30,305-Speed 128.86 samples/sec Loss 13.9012 Epoch: 1 Global Step: 11855 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:52:50,241-Speed 128.42 samples/sec Loss 13.9162 Epoch: 1 Global Step: 11860 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:53:10,066-Speed 129.14 samples/sec Loss 13.5886 Epoch: 1 Global Step: 11865 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:53:29,861-Speed 129.33 samples/sec Loss 13.5336 Epoch: 1 Global Step: 11870 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:53:50,200-Speed 125.87 samples/sec Loss 14.0088 Epoch: 1 Global Step: 11875 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:54:10,019-Speed 129.18 samples/sec Loss 13.8845 Epoch: 1 Global Step: 11880 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:54:30,410-Speed 125.55 samples/sec Loss 14.1735 Epoch: 1 Global Step: 11885 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:54:50,023-Speed 130.53 samples/sec Loss 13.8379 Epoch: 1 Global Step: 11890 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:55:09,831-Speed 129.25 samples/sec Loss 14.1418 Epoch: 1 Global Step: 11895 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:55:29,612-Speed 129.42 samples/sec Loss 13.8354 Epoch: 1 Global Step: 11900 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:55:50,209-Speed 124.30 samples/sec Loss 14.0678 Epoch: 1 Global Step: 11905 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:56:09,987-Speed 129.44 samples/sec Loss 14.4090 Epoch: 1 Global Step: 11910 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:56:29,958-Speed 128.20 samples/sec Loss 14.0755 Epoch: 1 Global Step: 11915 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:56:49,763-Speed 129.27 samples/sec Loss 14.1982 Epoch: 1 Global Step: 11920 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:57:09,610-Speed 128.99 samples/sec Loss 14.2897 Epoch: 1 Global Step: 11925 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:57:30,148-Speed 124.65 samples/sec Loss 14.2570 Epoch: 1 Global Step: 11930 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:57:49,973-Speed 129.14 samples/sec Loss 14.3003 Epoch: 1 Global Step: 11935 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:58:09,769-Speed 129.33 samples/sec Loss 14.2221 Epoch: 1 Global Step: 11940 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:58:29,740-Speed 128.19 samples/sec Loss 13.8470 Epoch: 1 Global Step: 11945 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:58:49,837-Speed 127.39 samples/sec Loss 14.1831 Epoch: 1 Global Step: 11950 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:59:09,786-Speed 128.34 samples/sec Loss 14.3228 Epoch: 1 Global Step: 11955 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:59:29,635-Speed 128.98 samples/sec Loss 14.0903 Epoch: 1 Global Step: 11960 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 08:59:49,756-Speed 127.24 samples/sec Loss 14.0615 Epoch: 1 Global Step: 11965 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:00:09,673-Speed 128.54 samples/sec Loss 14.3788 Epoch: 1 Global Step: 11970 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:00:30,194-Speed 124.75 samples/sec Loss 14.3370 Epoch: 1 Global Step: 11975 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:00:50,070-Speed 128.81 samples/sec Loss 14.0463 Epoch: 1 Global Step: 11980 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:01:09,954-Speed 128.75 samples/sec Loss 13.8846 Epoch: 1 Global Step: 11985 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:01:30,325-Speed 125.68 samples/sec Loss 14.3079 Epoch: 1 Global Step: 11990 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:01:50,298-Speed 128.18 samples/sec Loss 13.6390 Epoch: 1 Global Step: 11995 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:02:10,262-Speed 128.23 samples/sec Loss 14.4358 Epoch: 1 Global Step: 12000 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:02:30,162-Speed 128.65 samples/sec Loss 13.9787 Epoch: 1 Global Step: 12005 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:02:50,063-Speed 128.65 samples/sec Loss 14.0416 Epoch: 1 Global Step: 12010 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:03:09,971-Speed 128.60 samples/sec Loss 13.8871 Epoch: 1 Global Step: 12015 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:03:30,575-Speed 124.25 samples/sec Loss 14.1245 Epoch: 1 Global Step: 12020 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:03:50,438-Speed 128.89 samples/sec Loss 14.4368 Epoch: 1 Global Step: 12025 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:04:10,403-Speed 128.23 samples/sec Loss 14.2931 Epoch: 1 Global Step: 12030 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:04:30,241-Speed 129.06 samples/sec Loss 13.8407 Epoch: 1 Global Step: 12035 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:04:50,141-Speed 128.65 samples/sec Loss 14.2670 Epoch: 1 Global Step: 12040 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:05:10,010-Speed 128.85 samples/sec Loss 14.3893 Epoch: 1 Global Step: 12045 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:05:29,815-Speed 129.27 samples/sec Loss 14.0551 Epoch: 1 Global Step: 12050 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:05:49,624-Speed 129.24 samples/sec Loss 13.8609 Epoch: 1 Global Step: 12055 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:06:09,527-Speed 128.64 samples/sec Loss 14.1648 Epoch: 1 Global Step: 12060 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:06:30,239-Speed 123.61 samples/sec Loss 13.6481 Epoch: 1 Global Step: 12065 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:06:50,094-Speed 128.94 samples/sec Loss 13.7888 Epoch: 1 Global Step: 12070 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:07:09,966-Speed 128.83 samples/sec Loss 14.0787 Epoch: 1 Global Step: 12075 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:07:29,791-Speed 129.14 samples/sec Loss 13.8698 Epoch: 1 Global Step: 12080 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:07:49,768-Speed 128.15 samples/sec Loss 13.8959 Epoch: 1 Global Step: 12085 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:08:09,646-Speed 128.79 samples/sec Loss 13.8716 Epoch: 1 Global Step: 12090 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:08:29,465-Speed 129.17 samples/sec Loss 13.8916 Epoch: 1 Global Step: 12095 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:08:49,314-Speed 128.98 samples/sec Loss 14.2786 Epoch: 1 Global Step: 12100 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:09:09,248-Speed 128.43 samples/sec Loss 14.4092 Epoch: 1 Global Step: 12105 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:09:29,947-Speed 123.69 samples/sec Loss 13.9594 Epoch: 1 Global Step: 12110 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:09:49,771-Speed 129.14 samples/sec Loss 13.8731 Epoch: 1 Global Step: 12115 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:10:09,700-Speed 128.47 samples/sec Loss 13.8936 Epoch: 1 Global Step: 12120 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:10:29,589-Speed 128.72 samples/sec Loss 14.1371 Epoch: 1 Global Step: 12125 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:10:49,477-Speed 128.73 samples/sec Loss 14.2688 Epoch: 1 Global Step: 12130 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:11:09,422-Speed 128.36 samples/sec Loss 14.0608 Epoch: 1 Global Step: 12135 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:11:29,525-Speed 127.35 samples/sec Loss 13.8529 Epoch: 1 Global Step: 12140 Fp16 Grad Scale: 8192 Required: 276 hours Training: 2021-06-07 09:11:49,627-Speed 127.36 samples/sec Loss 13.9895 Epoch: 1 Global Step: 12145 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:12:09,624-Speed 128.03 samples/sec Loss 14.0431 Epoch: 1 Global Step: 12150 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:12:30,456-Speed 122.89 samples/sec Loss 14.1189 Epoch: 1 Global Step: 12155 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:12:51,197-Speed 123.43 samples/sec Loss 13.9689 Epoch: 1 Global Step: 12160 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:13:11,016-Speed 129.17 samples/sec Loss 14.1913 Epoch: 1 Global Step: 12165 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:13:30,853-Speed 129.06 samples/sec Loss 13.7771 Epoch: 1 Global Step: 12170 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:13:50,890-Speed 127.77 samples/sec Loss 13.7550 Epoch: 1 Global Step: 12175 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:14:10,814-Speed 128.50 samples/sec Loss 14.1605 Epoch: 1 Global Step: 12180 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:14:30,960-Speed 127.08 samples/sec Loss 14.0017 Epoch: 1 Global Step: 12185 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:14:50,917-Speed 128.28 samples/sec Loss 13.9811 Epoch: 1 Global Step: 12190 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:15:10,847-Speed 128.45 samples/sec Loss 14.0052 Epoch: 1 Global Step: 12195 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:15:31,528-Speed 123.79 samples/sec Loss 13.7966 Epoch: 1 Global Step: 12200 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:15:51,364-Speed 129.06 samples/sec Loss 13.9692 Epoch: 1 Global Step: 12205 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:16:11,225-Speed 128.90 samples/sec Loss 13.9845 Epoch: 1 Global Step: 12210 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:16:31,014-Speed 129.37 samples/sec Loss 14.2838 Epoch: 1 Global Step: 12215 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:16:51,029-Speed 127.91 samples/sec Loss 14.1994 Epoch: 1 Global Step: 12220 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:17:10,879-Speed 128.98 samples/sec Loss 14.2413 Epoch: 1 Global Step: 12225 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:17:30,694-Speed 129.20 samples/sec Loss 14.0442 Epoch: 1 Global Step: 12230 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:17:50,551-Speed 128.93 samples/sec Loss 13.8387 Epoch: 1 Global Step: 12235 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:18:10,527-Speed 128.16 samples/sec Loss 14.0550 Epoch: 1 Global Step: 12240 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:18:31,135-Speed 124.23 samples/sec Loss 13.9207 Epoch: 1 Global Step: 12245 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:18:50,924-Speed 129.37 samples/sec Loss 14.1820 Epoch: 1 Global Step: 12250 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:19:10,809-Speed 128.75 samples/sec Loss 14.2571 Epoch: 1 Global Step: 12255 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:19:30,700-Speed 128.71 samples/sec Loss 13.7608 Epoch: 1 Global Step: 12260 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:19:50,587-Speed 128.73 samples/sec Loss 13.5770 Epoch: 1 Global Step: 12265 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:20:10,423-Speed 129.07 samples/sec Loss 13.9112 Epoch: 1 Global Step: 12270 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:20:30,258-Speed 129.07 samples/sec Loss 13.9810 Epoch: 1 Global Step: 12275 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:20:50,136-Speed 128.79 samples/sec Loss 14.1265 Epoch: 1 Global Step: 12280 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:21:10,049-Speed 128.57 samples/sec Loss 13.9155 Epoch: 1 Global Step: 12285 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:21:30,870-Speed 122.96 samples/sec Loss 14.1554 Epoch: 1 Global Step: 12290 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:21:51,019-Speed 127.06 samples/sec Loss 14.1541 Epoch: 1 Global Step: 12295 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:22:10,901-Speed 128.76 samples/sec Loss 14.3400 Epoch: 1 Global Step: 12300 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:22:30,850-Speed 128.34 samples/sec Loss 13.8214 Epoch: 1 Global Step: 12305 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:22:50,892-Speed 127.74 samples/sec Loss 14.0467 Epoch: 1 Global Step: 12310 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:23:10,831-Speed 128.40 samples/sec Loss 13.7957 Epoch: 1 Global Step: 12315 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:23:30,796-Speed 128.23 samples/sec Loss 14.0894 Epoch: 1 Global Step: 12320 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:23:50,774-Speed 128.15 samples/sec Loss 14.2122 Epoch: 1 Global Step: 12325 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:24:10,963-Speed 126.81 samples/sec Loss 14.0213 Epoch: 1 Global Step: 12330 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:24:31,721-Speed 123.33 samples/sec Loss 13.8841 Epoch: 1 Global Step: 12335 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:24:51,955-Speed 126.53 samples/sec Loss 13.8206 Epoch: 1 Global Step: 12340 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:25:11,612-Speed 130.26 samples/sec Loss 14.3648 Epoch: 1 Global Step: 12345 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:25:31,486-Speed 128.81 samples/sec Loss 14.0337 Epoch: 1 Global Step: 12350 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:25:52,297-Speed 123.02 samples/sec Loss 14.1476 Epoch: 1 Global Step: 12355 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:26:12,219-Speed 128.51 samples/sec Loss 13.8850 Epoch: 1 Global Step: 12360 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:26:32,080-Speed 128.91 samples/sec Loss 13.8059 Epoch: 1 Global Step: 12365 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:26:51,934-Speed 128.95 samples/sec Loss 13.9862 Epoch: 1 Global Step: 12370 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:27:11,645-Speed 129.88 samples/sec Loss 13.9791 Epoch: 1 Global Step: 12375 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:27:32,315-Speed 123.86 samples/sec Loss 14.0608 Epoch: 1 Global Step: 12380 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:27:52,209-Speed 128.69 samples/sec Loss 14.0297 Epoch: 1 Global Step: 12385 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:28:12,078-Speed 128.85 samples/sec Loss 14.3349 Epoch: 1 Global Step: 12390 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:28:31,960-Speed 128.77 samples/sec Loss 13.7372 Epoch: 1 Global Step: 12395 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:28:51,704-Speed 129.67 samples/sec Loss 13.8976 Epoch: 1 Global Step: 12400 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:29:11,581-Speed 128.80 samples/sec Loss 13.9709 Epoch: 1 Global Step: 12405 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:29:31,556-Speed 128.17 samples/sec Loss 14.0103 Epoch: 1 Global Step: 12410 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:29:51,210-Speed 130.26 samples/sec Loss 13.9105 Epoch: 1 Global Step: 12415 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:30:11,061-Speed 128.97 samples/sec Loss 13.9882 Epoch: 1 Global Step: 12420 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:30:31,484-Speed 125.35 samples/sec Loss 13.8484 Epoch: 1 Global Step: 12425 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:30:51,347-Speed 128.89 samples/sec Loss 13.6141 Epoch: 1 Global Step: 12430 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:31:11,182-Speed 129.07 samples/sec Loss 13.4952 Epoch: 1 Global Step: 12435 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:31:31,013-Speed 129.10 samples/sec Loss 13.8500 Epoch: 1 Global Step: 12440 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:31:50,987-Speed 128.17 samples/sec Loss 13.5923 Epoch: 1 Global Step: 12445 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:32:10,973-Speed 128.10 samples/sec Loss 13.9062 Epoch: 1 Global Step: 12450 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:32:30,800-Speed 129.12 samples/sec Loss 13.7871 Epoch: 1 Global Step: 12455 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:32:50,643-Speed 129.02 samples/sec Loss 13.9163 Epoch: 1 Global Step: 12460 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:33:10,287-Speed 130.33 samples/sec Loss 13.9133 Epoch: 1 Global Step: 12465 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:33:30,821-Speed 124.68 samples/sec Loss 13.6925 Epoch: 1 Global Step: 12470 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:33:50,832-Speed 127.93 samples/sec Loss 14.1392 Epoch: 1 Global Step: 12475 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:34:10,706-Speed 128.82 samples/sec Loss 13.9670 Epoch: 1 Global Step: 12480 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:34:30,834-Speed 127.19 samples/sec Loss 13.8222 Epoch: 1 Global Step: 12485 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:34:50,715-Speed 128.77 samples/sec Loss 14.0651 Epoch: 1 Global Step: 12490 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:35:10,525-Speed 129.23 samples/sec Loss 13.8524 Epoch: 1 Global Step: 12495 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:35:30,372-Speed 128.99 samples/sec Loss 13.7677 Epoch: 1 Global Step: 12500 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:35:50,194-Speed 129.16 samples/sec Loss 13.8213 Epoch: 1 Global Step: 12505 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:36:10,020-Speed 129.13 samples/sec Loss 13.7303 Epoch: 1 Global Step: 12510 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:36:30,555-Speed 124.67 samples/sec Loss 13.6427 Epoch: 1 Global Step: 12515 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:36:50,406-Speed 128.97 samples/sec Loss 13.7706 Epoch: 1 Global Step: 12520 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:37:10,222-Speed 129.19 samples/sec Loss 13.8637 Epoch: 1 Global Step: 12525 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:37:30,219-Speed 128.03 samples/sec Loss 13.6617 Epoch: 1 Global Step: 12530 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:37:49,876-Speed 130.25 samples/sec Loss 14.0363 Epoch: 1 Global Step: 12535 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:38:09,818-Speed 128.37 samples/sec Loss 14.0767 Epoch: 1 Global Step: 12540 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:38:29,514-Speed 129.98 samples/sec Loss 13.9545 Epoch: 1 Global Step: 12545 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:38:49,370-Speed 128.94 samples/sec Loss 14.0661 Epoch: 1 Global Step: 12550 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:39:09,287-Speed 128.55 samples/sec Loss 13.4351 Epoch: 1 Global Step: 12555 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:39:29,808-Speed 124.76 samples/sec Loss 14.0122 Epoch: 1 Global Step: 12560 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:39:49,547-Speed 129.70 samples/sec Loss 14.0824 Epoch: 1 Global Step: 12565 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:40:09,386-Speed 129.04 samples/sec Loss 13.8018 Epoch: 1 Global Step: 12570 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:40:29,222-Speed 129.07 samples/sec Loss 14.0020 Epoch: 1 Global Step: 12575 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:40:49,034-Speed 129.22 samples/sec Loss 13.7778 Epoch: 1 Global Step: 12580 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:41:08,702-Speed 130.17 samples/sec Loss 14.1429 Epoch: 1 Global Step: 12585 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:41:28,687-Speed 128.10 samples/sec Loss 14.0830 Epoch: 1 Global Step: 12590 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:41:48,583-Speed 128.68 samples/sec Loss 13.8484 Epoch: 1 Global Step: 12595 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:42:08,466-Speed 128.76 samples/sec Loss 13.7114 Epoch: 1 Global Step: 12600 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:42:29,110-Speed 124.01 samples/sec Loss 14.0547 Epoch: 1 Global Step: 12605 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:42:49,028-Speed 128.53 samples/sec Loss 13.8203 Epoch: 1 Global Step: 12610 Fp16 Grad Scale: 8192 Required: 275 hours Training: 2021-06-07 09:43:08,667-Speed 130.36 samples/sec Loss 13.9622 Epoch: 1 Global Step: 12615 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:43:28,762-Speed 127.40 samples/sec Loss 14.1203 Epoch: 1 Global Step: 12620 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:43:48,652-Speed 128.72 samples/sec Loss 13.7974 Epoch: 1 Global Step: 12625 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:44:08,458-Speed 129.26 samples/sec Loss 13.7281 Epoch: 1 Global Step: 12630 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:44:28,257-Speed 129.31 samples/sec Loss 14.0842 Epoch: 1 Global Step: 12635 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:44:48,058-Speed 129.30 samples/sec Loss 13.6711 Epoch: 1 Global Step: 12640 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:45:07,784-Speed 129.79 samples/sec Loss 14.1835 Epoch: 1 Global Step: 12645 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:45:28,149-Speed 125.71 samples/sec Loss 13.4644 Epoch: 1 Global Step: 12650 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:45:47,816-Speed 130.17 samples/sec Loss 14.1662 Epoch: 1 Global Step: 12655 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:46:07,701-Speed 128.75 samples/sec Loss 13.9120 Epoch: 1 Global Step: 12660 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:46:27,606-Speed 128.62 samples/sec Loss 13.9752 Epoch: 1 Global Step: 12665 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:46:47,531-Speed 128.49 samples/sec Loss 13.7568 Epoch: 1 Global Step: 12670 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:47:07,385-Speed 128.95 samples/sec Loss 13.7967 Epoch: 1 Global Step: 12675 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:47:27,045-Speed 130.22 samples/sec Loss 13.9225 Epoch: 1 Global Step: 12680 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:47:46,881-Speed 129.07 samples/sec Loss 13.9327 Epoch: 1 Global Step: 12685 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:48:06,729-Speed 128.99 samples/sec Loss 13.7380 Epoch: 1 Global Step: 12690 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:48:27,352-Speed 124.14 samples/sec Loss 13.7422 Epoch: 1 Global Step: 12695 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:48:47,426-Speed 127.54 samples/sec Loss 13.8364 Epoch: 1 Global Step: 12700 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:49:07,301-Speed 128.81 samples/sec Loss 13.9225 Epoch: 1 Global Step: 12705 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:49:26,954-Speed 130.27 samples/sec Loss 14.0372 Epoch: 1 Global Step: 12710 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:49:46,743-Speed 129.37 samples/sec Loss 14.0812 Epoch: 1 Global Step: 12715 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:50:06,374-Speed 130.41 samples/sec Loss 14.0463 Epoch: 1 Global Step: 12720 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:50:26,224-Speed 128.97 samples/sec Loss 14.1191 Epoch: 1 Global Step: 12725 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:50:46,179-Speed 128.29 samples/sec Loss 13.7366 Epoch: 1 Global Step: 12730 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:51:06,031-Speed 128.96 samples/sec Loss 13.7382 Epoch: 1 Global Step: 12735 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:51:26,580-Speed 124.59 samples/sec Loss 13.5638 Epoch: 1 Global Step: 12740 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:51:46,522-Speed 128.38 samples/sec Loss 13.6889 Epoch: 1 Global Step: 12745 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:52:06,622-Speed 127.37 samples/sec Loss 13.7297 Epoch: 1 Global Step: 12750 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:52:26,956-Speed 125.90 samples/sec Loss 13.7681 Epoch: 1 Global Step: 12755 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:52:46,969-Speed 127.92 samples/sec Loss 13.9845 Epoch: 1 Global Step: 12760 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:53:07,046-Speed 127.52 samples/sec Loss 13.7709 Epoch: 1 Global Step: 12765 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:53:26,911-Speed 128.88 samples/sec Loss 13.7819 Epoch: 1 Global Step: 12770 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:53:46,688-Speed 129.45 samples/sec Loss 13.7354 Epoch: 1 Global Step: 12775 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:54:06,693-Speed 127.98 samples/sec Loss 13.6778 Epoch: 1 Global Step: 12780 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:54:27,138-Speed 125.22 samples/sec Loss 14.0565 Epoch: 1 Global Step: 12785 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:54:46,988-Speed 128.97 samples/sec Loss 13.9777 Epoch: 1 Global Step: 12790 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:55:06,846-Speed 128.93 samples/sec Loss 13.6081 Epoch: 1 Global Step: 12795 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:55:26,688-Speed 129.02 samples/sec Loss 13.6758 Epoch: 1 Global Step: 12800 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:55:46,854-Speed 126.96 samples/sec Loss 13.7857 Epoch: 1 Global Step: 12805 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:56:06,673-Speed 129.18 samples/sec Loss 13.8605 Epoch: 1 Global Step: 12810 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:56:26,503-Speed 129.11 samples/sec Loss 13.7109 Epoch: 1 Global Step: 12815 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:56:46,337-Speed 129.08 samples/sec Loss 14.0052 Epoch: 1 Global Step: 12820 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:57:06,085-Speed 129.64 samples/sec Loss 13.8026 Epoch: 1 Global Step: 12825 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:57:26,680-Speed 124.31 samples/sec Loss 13.8183 Epoch: 1 Global Step: 12830 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:57:46,490-Speed 129.23 samples/sec Loss 13.4499 Epoch: 1 Global Step: 12835 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:58:06,360-Speed 128.84 samples/sec Loss 14.0220 Epoch: 1 Global Step: 12840 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:58:26,185-Speed 129.14 samples/sec Loss 13.9507 Epoch: 1 Global Step: 12845 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:58:46,109-Speed 128.49 samples/sec Loss 14.1009 Epoch: 1 Global Step: 12850 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:59:05,845-Speed 129.72 samples/sec Loss 13.8370 Epoch: 1 Global Step: 12855 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:59:25,662-Speed 129.19 samples/sec Loss 13.9649 Epoch: 1 Global Step: 12860 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 09:59:45,438-Speed 129.45 samples/sec Loss 13.5366 Epoch: 1 Global Step: 12865 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:00:05,408-Speed 128.20 samples/sec Loss 13.6805 Epoch: 1 Global Step: 12870 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:00:25,986-Speed 124.41 samples/sec Loss 13.9937 Epoch: 1 Global Step: 12875 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:00:45,804-Speed 129.18 samples/sec Loss 13.9943 Epoch: 1 Global Step: 12880 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:01:05,580-Speed 129.46 samples/sec Loss 13.7049 Epoch: 1 Global Step: 12885 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:01:25,588-Speed 127.96 samples/sec Loss 13.9157 Epoch: 1 Global Step: 12890 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:01:45,779-Speed 126.81 samples/sec Loss 13.7181 Epoch: 1 Global Step: 12895 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:02:05,602-Speed 129.15 samples/sec Loss 13.7030 Epoch: 1 Global Step: 12900 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:02:25,466-Speed 128.89 samples/sec Loss 13.8785 Epoch: 1 Global Step: 12905 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:02:45,450-Speed 128.11 samples/sec Loss 13.8598 Epoch: 1 Global Step: 12910 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:03:05,315-Speed 128.88 samples/sec Loss 14.1673 Epoch: 1 Global Step: 12915 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:03:26,122-Speed 123.06 samples/sec Loss 14.1193 Epoch: 1 Global Step: 12920 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:03:45,948-Speed 129.13 samples/sec Loss 14.0657 Epoch: 1 Global Step: 12925 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:04:05,769-Speed 129.16 samples/sec Loss 13.6928 Epoch: 1 Global Step: 12930 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:04:25,585-Speed 129.19 samples/sec Loss 14.0215 Epoch: 1 Global Step: 12935 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:04:45,424-Speed 129.05 samples/sec Loss 13.8744 Epoch: 1 Global Step: 12940 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:05:05,409-Speed 128.10 samples/sec Loss 13.3333 Epoch: 1 Global Step: 12945 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:05:25,338-Speed 128.47 samples/sec Loss 13.8410 Epoch: 1 Global Step: 12950 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:05:45,311-Speed 128.18 samples/sec Loss 13.6462 Epoch: 1 Global Step: 12955 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:06:05,236-Speed 128.49 samples/sec Loss 14.0806 Epoch: 1 Global Step: 12960 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:06:25,647-Speed 125.43 samples/sec Loss 13.7276 Epoch: 1 Global Step: 12965 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:06:45,483-Speed 129.06 samples/sec Loss 13.7948 Epoch: 1 Global Step: 12970 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:07:05,242-Speed 129.57 samples/sec Loss 13.9552 Epoch: 1 Global Step: 12975 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:07:25,231-Speed 128.07 samples/sec Loss 13.8127 Epoch: 1 Global Step: 12980 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:07:45,140-Speed 128.60 samples/sec Loss 13.7543 Epoch: 1 Global Step: 12985 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:08:05,102-Speed 128.25 samples/sec Loss 13.8463 Epoch: 1 Global Step: 12990 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:08:25,128-Speed 127.84 samples/sec Loss 13.7447 Epoch: 1 Global Step: 12995 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:08:45,140-Speed 127.94 samples/sec Loss 13.8483 Epoch: 1 Global Step: 13000 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:09:05,083-Speed 128.37 samples/sec Loss 14.0693 Epoch: 1 Global Step: 13005 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:09:25,689-Speed 124.24 samples/sec Loss 13.9136 Epoch: 1 Global Step: 13010 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:09:45,613-Speed 128.50 samples/sec Loss 13.7205 Epoch: 1 Global Step: 13015 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:10:05,481-Speed 128.85 samples/sec Loss 13.6587 Epoch: 1 Global Step: 13020 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:10:25,460-Speed 128.15 samples/sec Loss 13.8524 Epoch: 1 Global Step: 13025 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:10:45,328-Speed 128.86 samples/sec Loss 13.5792 Epoch: 1 Global Step: 13030 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:11:05,131-Speed 129.28 samples/sec Loss 13.6273 Epoch: 1 Global Step: 13035 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:11:25,071-Speed 128.40 samples/sec Loss 14.0550 Epoch: 1 Global Step: 13040 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:11:44,988-Speed 128.54 samples/sec Loss 14.0515 Epoch: 1 Global Step: 13045 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:12:05,635-Speed 123.99 samples/sec Loss 13.7673 Epoch: 1 Global Step: 13050 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:12:26,268-Speed 124.08 samples/sec Loss 13.6833 Epoch: 1 Global Step: 13055 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:12:45,931-Speed 130.20 samples/sec Loss 13.6618 Epoch: 1 Global Step: 13060 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:13:05,999-Speed 127.57 samples/sec Loss 14.0756 Epoch: 1 Global Step: 13065 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:13:26,064-Speed 127.59 samples/sec Loss 14.0049 Epoch: 1 Global Step: 13070 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:13:46,049-Speed 128.10 samples/sec Loss 13.8715 Epoch: 1 Global Step: 13075 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:14:05,870-Speed 129.16 samples/sec Loss 13.8783 Epoch: 1 Global Step: 13080 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:14:25,717-Speed 128.99 samples/sec Loss 13.7266 Epoch: 1 Global Step: 13085 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:14:45,551-Speed 129.08 samples/sec Loss 13.5218 Epoch: 1 Global Step: 13090 Fp16 Grad Scale: 8192 Required: 274 hours Training: 2021-06-07 10:15:05,406-Speed 128.94 samples/sec Loss 13.9772 Epoch: 1 Global Step: 13095 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:15:25,902-Speed 124.91 samples/sec Loss 14.0723 Epoch: 1 Global Step: 13100 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:15:46,478-Speed 124.42 samples/sec Loss 13.7525 Epoch: 1 Global Step: 13105 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:16:06,547-Speed 127.56 samples/sec Loss 13.8437 Epoch: 1 Global Step: 13110 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:16:26,494-Speed 128.35 samples/sec Loss 13.5060 Epoch: 1 Global Step: 13115 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:16:46,338-Speed 129.01 samples/sec Loss 14.2166 Epoch: 1 Global Step: 13120 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:17:06,224-Speed 128.75 samples/sec Loss 13.9710 Epoch: 1 Global Step: 13125 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:17:25,938-Speed 129.86 samples/sec Loss 13.6097 Epoch: 1 Global Step: 13130 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:17:45,719-Speed 129.42 samples/sec Loss 13.7025 Epoch: 1 Global Step: 13135 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:18:05,507-Speed 129.38 samples/sec Loss 13.7323 Epoch: 1 Global Step: 13140 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:18:26,826-Speed 120.08 samples/sec Loss 13.7688 Epoch: 1 Global Step: 13145 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:18:46,558-Speed 129.75 samples/sec Loss 13.4422 Epoch: 1 Global Step: 13150 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:19:06,311-Speed 129.61 samples/sec Loss 13.7209 Epoch: 1 Global Step: 13155 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:19:26,117-Speed 129.26 samples/sec Loss 13.8782 Epoch: 1 Global Step: 13160 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:19:45,792-Speed 130.13 samples/sec Loss 13.7623 Epoch: 1 Global Step: 13165 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:20:05,633-Speed 129.03 samples/sec Loss 13.7154 Epoch: 1 Global Step: 13170 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:20:25,927-Speed 126.16 samples/sec Loss 13.7206 Epoch: 1 Global Step: 13175 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:20:45,890-Speed 128.25 samples/sec Loss 14.0308 Epoch: 1 Global Step: 13180 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:21:05,878-Speed 128.09 samples/sec Loss 13.5334 Epoch: 1 Global Step: 13185 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:21:26,529-Speed 123.97 samples/sec Loss 13.6680 Epoch: 1 Global Step: 13190 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:21:46,459-Speed 128.45 samples/sec Loss 13.8165 Epoch: 1 Global Step: 13195 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:22:06,115-Speed 130.25 samples/sec Loss 13.8133 Epoch: 1 Global Step: 13200 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:22:26,304-Speed 126.81 samples/sec Loss 13.9557 Epoch: 1 Global Step: 13205 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:22:46,152-Speed 128.98 samples/sec Loss 13.9355 Epoch: 1 Global Step: 13210 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:23:06,072-Speed 128.52 samples/sec Loss 13.7738 Epoch: 1 Global Step: 13215 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:23:25,811-Speed 129.70 samples/sec Loss 13.9317 Epoch: 1 Global Step: 13220 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:23:45,662-Speed 128.97 samples/sec Loss 13.8816 Epoch: 1 Global Step: 13225 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:24:05,550-Speed 128.73 samples/sec Loss 13.5472 Epoch: 1 Global Step: 13230 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:24:26,164-Speed 124.20 samples/sec Loss 13.6860 Epoch: 1 Global Step: 13235 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:24:46,016-Speed 128.96 samples/sec Loss 13.8151 Epoch: 1 Global Step: 13240 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:25:06,017-Speed 127.99 samples/sec Loss 13.7436 Epoch: 1 Global Step: 13245 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:25:26,124-Speed 127.33 samples/sec Loss 13.9671 Epoch: 1 Global Step: 13250 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:25:46,090-Speed 128.23 samples/sec Loss 13.9826 Epoch: 1 Global Step: 13255 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:26:05,959-Speed 128.85 samples/sec Loss 13.6651 Epoch: 1 Global Step: 13260 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:26:25,808-Speed 128.98 samples/sec Loss 13.5032 Epoch: 1 Global Step: 13265 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:26:45,697-Speed 128.72 samples/sec Loss 14.1289 Epoch: 1 Global Step: 13270 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:27:05,550-Speed 128.96 samples/sec Loss 13.7753 Epoch: 1 Global Step: 13275 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:27:26,099-Speed 124.58 samples/sec Loss 13.6040 Epoch: 1 Global Step: 13280 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:27:46,019-Speed 128.52 samples/sec Loss 13.9670 Epoch: 1 Global Step: 13285 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:28:05,973-Speed 128.30 samples/sec Loss 13.8946 Epoch: 1 Global Step: 13290 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:28:26,153-Speed 126.87 samples/sec Loss 13.9360 Epoch: 1 Global Step: 13295 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:28:46,190-Speed 127.77 samples/sec Loss 13.7576 Epoch: 1 Global Step: 13300 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:29:06,057-Speed 128.86 samples/sec Loss 13.4147 Epoch: 1 Global Step: 13305 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:29:25,906-Speed 128.98 samples/sec Loss 13.5788 Epoch: 1 Global Step: 13310 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:29:45,832-Speed 128.48 samples/sec Loss 13.7879 Epoch: 1 Global Step: 13315 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:30:05,613-Speed 129.43 samples/sec Loss 13.4473 Epoch: 1 Global Step: 13320 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:30:26,151-Speed 124.65 samples/sec Loss 13.8942 Epoch: 1 Global Step: 13325 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:30:45,981-Speed 129.11 samples/sec Loss 14.0910 Epoch: 1 Global Step: 13330 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:31:05,793-Speed 129.21 samples/sec Loss 13.4195 Epoch: 1 Global Step: 13335 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:31:25,775-Speed 128.13 samples/sec Loss 13.5074 Epoch: 1 Global Step: 13340 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:31:45,644-Speed 128.85 samples/sec Loss 13.6167 Epoch: 1 Global Step: 13345 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:32:05,740-Speed 127.39 samples/sec Loss 13.7154 Epoch: 1 Global Step: 13350 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:32:25,745-Speed 127.98 samples/sec Loss 13.7297 Epoch: 1 Global Step: 13355 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:32:45,594-Speed 128.98 samples/sec Loss 14.0488 Epoch: 1 Global Step: 13360 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:33:05,463-Speed 128.85 samples/sec Loss 13.7131 Epoch: 1 Global Step: 13365 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:33:25,984-Speed 124.76 samples/sec Loss 13.9463 Epoch: 1 Global Step: 13370 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:33:46,158-Speed 126.90 samples/sec Loss 14.2079 Epoch: 1 Global Step: 13375 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:34:06,143-Speed 128.10 samples/sec Loss 13.4308 Epoch: 1 Global Step: 13380 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:34:25,975-Speed 129.09 samples/sec Loss 14.1197 Epoch: 1 Global Step: 13385 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:34:45,832-Speed 128.93 samples/sec Loss 13.2154 Epoch: 1 Global Step: 13390 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:35:05,700-Speed 128.86 samples/sec Loss 13.8590 Epoch: 1 Global Step: 13395 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:35:25,794-Speed 127.41 samples/sec Loss 13.8395 Epoch: 1 Global Step: 13400 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:35:45,636-Speed 129.03 samples/sec Loss 13.6000 Epoch: 1 Global Step: 13405 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:36:05,384-Speed 129.64 samples/sec Loss 13.8232 Epoch: 1 Global Step: 13410 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:36:25,875-Speed 124.94 samples/sec Loss 13.6266 Epoch: 1 Global Step: 13415 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:36:45,606-Speed 129.75 samples/sec Loss 13.7000 Epoch: 1 Global Step: 13420 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:37:05,825-Speed 126.62 samples/sec Loss 13.7758 Epoch: 1 Global Step: 13425 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:37:25,753-Speed 128.47 samples/sec Loss 13.6924 Epoch: 1 Global Step: 13430 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:37:45,400-Speed 130.31 samples/sec Loss 13.7635 Epoch: 1 Global Step: 13435 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:38:05,504-Speed 127.34 samples/sec Loss 13.6777 Epoch: 1 Global Step: 13440 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:38:25,356-Speed 128.96 samples/sec Loss 13.8791 Epoch: 1 Global Step: 13445 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:38:45,235-Speed 128.79 samples/sec Loss 13.6074 Epoch: 1 Global Step: 13450 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:39:05,012-Speed 129.45 samples/sec Loss 14.1653 Epoch: 1 Global Step: 13455 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:39:25,851-Speed 122.85 samples/sec Loss 13.7355 Epoch: 1 Global Step: 13460 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:39:45,749-Speed 128.66 samples/sec Loss 13.8370 Epoch: 1 Global Step: 13465 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:40:05,587-Speed 129.05 samples/sec Loss 13.7451 Epoch: 1 Global Step: 13470 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:40:25,574-Speed 128.09 samples/sec Loss 13.7221 Epoch: 1 Global Step: 13475 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:40:45,567-Speed 128.05 samples/sec Loss 13.8651 Epoch: 1 Global Step: 13480 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:41:05,440-Speed 128.83 samples/sec Loss 13.4035 Epoch: 1 Global Step: 13485 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:41:25,327-Speed 128.73 samples/sec Loss 13.9083 Epoch: 1 Global Step: 13490 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:41:45,316-Speed 128.08 samples/sec Loss 13.9082 Epoch: 1 Global Step: 13495 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:42:05,217-Speed 128.64 samples/sec Loss 13.8380 Epoch: 1 Global Step: 13500 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:42:25,881-Speed 123.89 samples/sec Loss 13.8614 Epoch: 1 Global Step: 13505 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:42:45,716-Speed 129.07 samples/sec Loss 13.6072 Epoch: 1 Global Step: 13510 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:43:05,655-Speed 128.40 samples/sec Loss 13.9055 Epoch: 1 Global Step: 13515 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:43:25,480-Speed 129.13 samples/sec Loss 13.9994 Epoch: 1 Global Step: 13520 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:43:45,281-Speed 129.30 samples/sec Loss 13.6485 Epoch: 1 Global Step: 13525 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:44:05,460-Speed 126.87 samples/sec Loss 13.5573 Epoch: 1 Global Step: 13530 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:44:25,161-Speed 129.95 samples/sec Loss 13.5153 Epoch: 1 Global Step: 13535 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:44:44,971-Speed 129.24 samples/sec Loss 13.5516 Epoch: 1 Global Step: 13540 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:45:04,834-Speed 128.89 samples/sec Loss 13.3343 Epoch: 1 Global Step: 13545 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:45:25,399-Speed 124.49 samples/sec Loss 13.4995 Epoch: 1 Global Step: 13550 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:45:45,240-Speed 129.03 samples/sec Loss 13.7317 Epoch: 1 Global Step: 13555 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:46:05,106-Speed 128.87 samples/sec Loss 14.1303 Epoch: 1 Global Step: 13560 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:46:25,044-Speed 128.41 samples/sec Loss 13.9083 Epoch: 1 Global Step: 13565 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:46:45,116-Speed 127.55 samples/sec Loss 13.5532 Epoch: 1 Global Step: 13570 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:47:04,942-Speed 129.13 samples/sec Loss 13.6841 Epoch: 1 Global Step: 13575 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:47:24,954-Speed 127.93 samples/sec Loss 14.0904 Epoch: 1 Global Step: 13580 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:47:45,027-Speed 127.54 samples/sec Loss 13.9035 Epoch: 1 Global Step: 13585 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:48:05,144-Speed 127.26 samples/sec Loss 13.7248 Epoch: 1 Global Step: 13590 Fp16 Grad Scale: 8192 Required: 273 hours Training: 2021-06-07 10:48:25,786-Speed 124.03 samples/sec Loss 13.7395 Epoch: 1 Global Step: 13595 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:48:45,842-Speed 127.64 samples/sec Loss 13.6157 Epoch: 1 Global Step: 13600 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:49:05,603-Speed 129.55 samples/sec Loss 13.8082 Epoch: 1 Global Step: 13605 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:49:25,766-Speed 126.97 samples/sec Loss 13.4198 Epoch: 1 Global Step: 13610 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:49:45,588-Speed 129.16 samples/sec Loss 13.5493 Epoch: 1 Global Step: 13615 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:50:05,418-Speed 129.11 samples/sec Loss 13.5446 Epoch: 1 Global Step: 13620 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:50:25,228-Speed 129.23 samples/sec Loss 13.5075 Epoch: 1 Global Step: 13625 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:50:45,147-Speed 128.53 samples/sec Loss 13.6022 Epoch: 1 Global Step: 13630 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:51:05,015-Speed 128.86 samples/sec Loss 13.5879 Epoch: 1 Global Step: 13635 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:51:25,598-Speed 124.38 samples/sec Loss 13.7065 Epoch: 1 Global Step: 13640 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:51:45,471-Speed 128.82 samples/sec Loss 13.2999 Epoch: 1 Global Step: 13645 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:52:05,594-Speed 127.23 samples/sec Loss 13.6390 Epoch: 1 Global Step: 13650 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:52:25,681-Speed 127.46 samples/sec Loss 13.4763 Epoch: 1 Global Step: 13655 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:52:45,793-Speed 127.29 samples/sec Loss 13.7087 Epoch: 1 Global Step: 13660 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:53:05,761-Speed 128.21 samples/sec Loss 13.8187 Epoch: 1 Global Step: 13665 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:53:25,463-Speed 129.95 samples/sec Loss 13.9046 Epoch: 1 Global Step: 13670 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:53:45,378-Speed 128.55 samples/sec Loss 13.3560 Epoch: 1 Global Step: 13675 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:54:05,207-Speed 129.12 samples/sec Loss 13.5676 Epoch: 1 Global Step: 13680 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:54:25,729-Speed 124.75 samples/sec Loss 13.9851 Epoch: 1 Global Step: 13685 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:54:45,872-Speed 127.10 samples/sec Loss 13.3170 Epoch: 1 Global Step: 13690 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:55:05,709-Speed 129.06 samples/sec Loss 13.4228 Epoch: 1 Global Step: 13695 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:55:25,622-Speed 128.57 samples/sec Loss 13.1751 Epoch: 1 Global Step: 13700 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:55:45,572-Speed 128.33 samples/sec Loss 13.6283 Epoch: 1 Global Step: 13705 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:56:05,523-Speed 128.32 samples/sec Loss 13.6750 Epoch: 1 Global Step: 13710 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:56:25,370-Speed 129.00 samples/sec Loss 13.5814 Epoch: 1 Global Step: 13715 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:56:45,318-Speed 128.34 samples/sec Loss 13.4218 Epoch: 1 Global Step: 13720 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:57:04,989-Speed 130.15 samples/sec Loss 13.4003 Epoch: 1 Global Step: 13725 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:57:25,417-Speed 125.33 samples/sec Loss 13.5199 Epoch: 1 Global Step: 13730 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:57:45,243-Speed 129.12 samples/sec Loss 13.5285 Epoch: 1 Global Step: 13735 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:58:05,064-Speed 129.16 samples/sec Loss 13.5712 Epoch: 1 Global Step: 13740 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:58:24,911-Speed 128.99 samples/sec Loss 13.6142 Epoch: 1 Global Step: 13745 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:58:44,778-Speed 128.86 samples/sec Loss 13.6037 Epoch: 1 Global Step: 13750 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:59:04,616-Speed 129.06 samples/sec Loss 13.3275 Epoch: 1 Global Step: 13755 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:59:24,511-Speed 128.68 samples/sec Loss 13.5125 Epoch: 1 Global Step: 13760 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 10:59:44,366-Speed 128.94 samples/sec Loss 13.3637 Epoch: 1 Global Step: 13765 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:00:04,351-Speed 128.10 samples/sec Loss 13.5935 Epoch: 1 Global Step: 13770 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:00:25,033-Speed 123.79 samples/sec Loss 13.9639 Epoch: 1 Global Step: 13775 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:00:44,834-Speed 129.29 samples/sec Loss 13.3064 Epoch: 1 Global Step: 13780 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:01:04,769-Speed 128.42 samples/sec Loss 13.6555 Epoch: 1 Global Step: 13785 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:01:24,701-Speed 128.44 samples/sec Loss 14.1288 Epoch: 1 Global Step: 13790 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:01:44,383-Speed 130.08 samples/sec Loss 13.5655 Epoch: 1 Global Step: 13795 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:02:04,193-Speed 129.23 samples/sec Loss 13.5744 Epoch: 1 Global Step: 13800 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:02:24,069-Speed 128.81 samples/sec Loss 13.5985 Epoch: 1 Global Step: 13805 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:02:43,944-Speed 128.81 samples/sec Loss 13.8092 Epoch: 1 Global Step: 13810 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:03:04,225-Speed 126.23 samples/sec Loss 13.8932 Epoch: 1 Global Step: 13815 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:03:24,866-Speed 124.04 samples/sec Loss 13.3441 Epoch: 1 Global Step: 13820 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:03:45,049-Speed 126.84 samples/sec Loss 13.6189 Epoch: 1 Global Step: 13825 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:04:04,747-Speed 129.97 samples/sec Loss 13.6007 Epoch: 1 Global Step: 13830 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:04:24,693-Speed 128.35 samples/sec Loss 13.7066 Epoch: 1 Global Step: 13835 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:04:44,783-Speed 127.44 samples/sec Loss 13.7896 Epoch: 1 Global Step: 13840 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:05:04,753-Speed 128.20 samples/sec Loss 13.4307 Epoch: 1 Global Step: 13845 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:05:24,609-Speed 128.94 samples/sec Loss 13.3104 Epoch: 1 Global Step: 13850 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:05:44,570-Speed 128.26 samples/sec Loss 13.4063 Epoch: 1 Global Step: 13855 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:06:04,451-Speed 128.78 samples/sec Loss 13.8027 Epoch: 1 Global Step: 13860 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:06:25,134-Speed 123.78 samples/sec Loss 13.5217 Epoch: 1 Global Step: 13865 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:06:45,000-Speed 128.88 samples/sec Loss 13.4184 Epoch: 1 Global Step: 13870 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:07:04,866-Speed 128.87 samples/sec Loss 13.5013 Epoch: 1 Global Step: 13875 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:07:24,805-Speed 128.40 samples/sec Loss 13.8573 Epoch: 1 Global Step: 13880 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:07:44,810-Speed 127.97 samples/sec Loss 13.7560 Epoch: 1 Global Step: 13885 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:08:04,654-Speed 129.02 samples/sec Loss 13.5558 Epoch: 1 Global Step: 13890 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:08:24,740-Speed 127.46 samples/sec Loss 13.9897 Epoch: 1 Global Step: 13895 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:08:45,122-Speed 125.61 samples/sec Loss 13.8013 Epoch: 1 Global Step: 13900 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:09:05,158-Speed 127.78 samples/sec Loss 13.6636 Epoch: 1 Global Step: 13905 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:09:26,016-Speed 122.74 samples/sec Loss 13.8944 Epoch: 1 Global Step: 13910 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:09:45,870-Speed 128.94 samples/sec Loss 13.8687 Epoch: 1 Global Step: 13915 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:10:05,695-Speed 129.14 samples/sec Loss 13.4422 Epoch: 1 Global Step: 13920 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:10:25,543-Speed 128.99 samples/sec Loss 13.4333 Epoch: 1 Global Step: 13925 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:10:45,556-Speed 127.93 samples/sec Loss 13.8288 Epoch: 1 Global Step: 13930 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:11:05,457-Speed 128.64 samples/sec Loss 13.6934 Epoch: 1 Global Step: 13935 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:11:25,299-Speed 129.03 samples/sec Loss 13.4881 Epoch: 1 Global Step: 13940 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:11:45,161-Speed 128.90 samples/sec Loss 13.8854 Epoch: 1 Global Step: 13945 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:12:05,024-Speed 128.89 samples/sec Loss 13.4561 Epoch: 1 Global Step: 13950 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:12:25,644-Speed 124.15 samples/sec Loss 13.5850 Epoch: 1 Global Step: 13955 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:12:45,506-Speed 128.90 samples/sec Loss 13.3718 Epoch: 1 Global Step: 13960 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:13:05,405-Speed 128.65 samples/sec Loss 13.7277 Epoch: 1 Global Step: 13965 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:13:25,599-Speed 126.78 samples/sec Loss 13.3894 Epoch: 1 Global Step: 13970 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:13:45,676-Speed 127.52 samples/sec Loss 13.7882 Epoch: 1 Global Step: 13975 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:14:05,568-Speed 128.70 samples/sec Loss 13.3733 Epoch: 1 Global Step: 13980 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:14:25,611-Speed 127.73 samples/sec Loss 13.6606 Epoch: 1 Global Step: 13985 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:14:45,433-Speed 129.15 samples/sec Loss 13.2925 Epoch: 1 Global Step: 13990 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:15:05,180-Speed 129.65 samples/sec Loss 13.6442 Epoch: 1 Global Step: 13995 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:15:25,679-Speed 124.89 samples/sec Loss 13.3468 Epoch: 1 Global Step: 14000 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:15:45,515-Speed 129.07 samples/sec Loss 13.3390 Epoch: 1 Global Step: 14005 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:16:05,306-Speed 129.36 samples/sec Loss 13.7468 Epoch: 1 Global Step: 14010 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:16:25,239-Speed 128.43 samples/sec Loss 13.5669 Epoch: 1 Global Step: 14015 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:16:45,212-Speed 128.19 samples/sec Loss 13.5049 Epoch: 1 Global Step: 14020 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:17:04,912-Speed 129.95 samples/sec Loss 13.4012 Epoch: 1 Global Step: 14025 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:17:24,721-Speed 129.24 samples/sec Loss 13.4818 Epoch: 1 Global Step: 14030 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:17:44,366-Speed 130.32 samples/sec Loss 13.5347 Epoch: 1 Global Step: 14035 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:18:04,038-Speed 130.14 samples/sec Loss 13.5260 Epoch: 1 Global Step: 14040 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:18:24,464-Speed 125.34 samples/sec Loss 13.7324 Epoch: 1 Global Step: 14045 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:18:44,400-Speed 128.42 samples/sec Loss 13.7170 Epoch: 1 Global Step: 14050 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:19:04,410-Speed 127.95 samples/sec Loss 13.4860 Epoch: 1 Global Step: 14055 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:19:24,068-Speed 130.23 samples/sec Loss 13.2101 Epoch: 1 Global Step: 14060 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:19:43,888-Speed 129.17 samples/sec Loss 13.7344 Epoch: 1 Global Step: 14065 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:20:04,004-Speed 127.26 samples/sec Loss 13.8834 Epoch: 1 Global Step: 14070 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:20:23,992-Speed 128.09 samples/sec Loss 13.8618 Epoch: 1 Global Step: 14075 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:20:43,924-Speed 128.44 samples/sec Loss 13.7049 Epoch: 1 Global Step: 14080 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:21:03,782-Speed 128.92 samples/sec Loss 13.3537 Epoch: 1 Global Step: 14085 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:21:24,521-Speed 123.45 samples/sec Loss 13.8653 Epoch: 1 Global Step: 14090 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:21:44,349-Speed 129.12 samples/sec Loss 13.7673 Epoch: 1 Global Step: 14095 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:22:04,400-Speed 127.68 samples/sec Loss 13.7453 Epoch: 1 Global Step: 14100 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:22:25,235-Speed 122.88 samples/sec Loss 13.5072 Epoch: 1 Global Step: 14105 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:22:45,445-Speed 126.68 samples/sec Loss 13.9990 Epoch: 1 Global Step: 14110 Fp16 Grad Scale: 8192 Required: 272 hours Training: 2021-06-07 11:23:05,328-Speed 128.76 samples/sec Loss 13.4200 Epoch: 1 Global Step: 14115 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:23:25,204-Speed 128.81 samples/sec Loss 13.5398 Epoch: 1 Global Step: 14120 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:23:45,395-Speed 126.79 samples/sec Loss 13.6601 Epoch: 1 Global Step: 14125 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:24:05,362-Speed 128.22 samples/sec Loss 13.6714 Epoch: 1 Global Step: 14130 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:24:25,980-Speed 124.17 samples/sec Loss 13.7524 Epoch: 1 Global Step: 14135 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:24:45,774-Speed 129.34 samples/sec Loss 13.4626 Epoch: 1 Global Step: 14140 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:25:05,750-Speed 128.16 samples/sec Loss 13.5316 Epoch: 1 Global Step: 14145 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:25:26,081-Speed 125.92 samples/sec Loss 13.7179 Epoch: 1 Global Step: 14150 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:25:45,830-Speed 129.64 samples/sec Loss 13.7974 Epoch: 1 Global Step: 14155 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:26:05,762-Speed 128.44 samples/sec Loss 13.5015 Epoch: 1 Global Step: 14160 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:26:25,524-Speed 129.55 samples/sec Loss 13.8247 Epoch: 1 Global Step: 14165 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:26:45,513-Speed 128.08 samples/sec Loss 13.0856 Epoch: 1 Global Step: 14170 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:27:05,389-Speed 128.80 samples/sec Loss 13.6623 Epoch: 1 Global Step: 14175 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:27:25,774-Speed 125.59 samples/sec Loss 13.5419 Epoch: 1 Global Step: 14180 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:27:45,378-Speed 130.59 samples/sec Loss 13.4850 Epoch: 1 Global Step: 14185 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:28:05,294-Speed 128.55 samples/sec Loss 13.5539 Epoch: 1 Global Step: 14190 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:28:25,233-Speed 128.40 samples/sec Loss 13.8097 Epoch: 1 Global Step: 14195 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:28:45,085-Speed 128.96 samples/sec Loss 13.6464 Epoch: 1 Global Step: 14200 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:29:05,023-Speed 128.41 samples/sec Loss 13.3824 Epoch: 1 Global Step: 14205 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:29:25,584-Speed 124.51 samples/sec Loss 13.5738 Epoch: 1 Global Step: 14210 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:29:45,648-Speed 127.60 samples/sec Loss 13.5587 Epoch: 1 Global Step: 14215 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:30:05,827-Speed 126.87 samples/sec Loss 13.7893 Epoch: 1 Global Step: 14220 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:30:26,614-Speed 123.16 samples/sec Loss 13.6949 Epoch: 1 Global Step: 14225 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:30:46,613-Speed 128.01 samples/sec Loss 13.2964 Epoch: 1 Global Step: 14230 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:31:06,447-Speed 129.08 samples/sec Loss 13.3168 Epoch: 1 Global Step: 14235 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:31:26,513-Speed 127.58 samples/sec Loss 13.4794 Epoch: 1 Global Step: 14240 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:31:46,706-Speed 126.79 samples/sec Loss 13.1207 Epoch: 1 Global Step: 14245 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:32:06,898-Speed 126.79 samples/sec Loss 13.4705 Epoch: 1 Global Step: 14250 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:32:27,138-Speed 126.49 samples/sec Loss 13.4275 Epoch: 1 Global Step: 14255 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:32:47,184-Speed 127.71 samples/sec Loss 13.6841 Epoch: 1 Global Step: 14260 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:33:07,139-Speed 128.29 samples/sec Loss 13.4751 Epoch: 1 Global Step: 14265 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:33:27,753-Speed 124.20 samples/sec Loss 13.0783 Epoch: 1 Global Step: 14270 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:33:47,923-Speed 126.93 samples/sec Loss 13.7794 Epoch: 1 Global Step: 14275 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:34:07,765-Speed 129.03 samples/sec Loss 13.1082 Epoch: 1 Global Step: 14280 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:34:27,399-Speed 130.39 samples/sec Loss 13.6577 Epoch: 1 Global Step: 14285 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:34:47,206-Speed 129.25 samples/sec Loss 13.3494 Epoch: 1 Global Step: 14290 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:35:07,052-Speed 129.00 samples/sec Loss 13.2251 Epoch: 1 Global Step: 14295 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:35:26,893-Speed 129.04 samples/sec Loss 13.4815 Epoch: 1 Global Step: 14300 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:35:46,929-Speed 127.78 samples/sec Loss 13.8019 Epoch: 1 Global Step: 14305 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:36:06,816-Speed 128.73 samples/sec Loss 13.5179 Epoch: 1 Global Step: 14310 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:36:27,448-Speed 124.09 samples/sec Loss 13.5508 Epoch: 1 Global Step: 14315 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:36:47,258-Speed 129.23 samples/sec Loss 13.3516 Epoch: 1 Global Step: 14320 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:37:06,943-Speed 130.06 samples/sec Loss 13.4027 Epoch: 1 Global Step: 14325 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:37:26,753-Speed 129.23 samples/sec Loss 13.9524 Epoch: 1 Global Step: 14330 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:37:46,809-Speed 127.65 samples/sec Loss 13.8361 Epoch: 1 Global Step: 14335 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:38:06,951-Speed 127.10 samples/sec Loss 13.9540 Epoch: 1 Global Step: 14340 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:38:27,158-Speed 126.70 samples/sec Loss 13.4671 Epoch: 1 Global Step: 14345 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:38:46,986-Speed 129.12 samples/sec Loss 13.3933 Epoch: 1 Global Step: 14350 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:39:06,951-Speed 128.23 samples/sec Loss 13.8049 Epoch: 1 Global Step: 14355 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:39:27,711-Speed 123.32 samples/sec Loss 13.6672 Epoch: 1 Global Step: 14360 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:39:47,557-Speed 129.00 samples/sec Loss 13.6901 Epoch: 1 Global Step: 14365 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:40:07,405-Speed 128.98 samples/sec Loss 13.6673 Epoch: 1 Global Step: 14370 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:40:27,250-Speed 129.01 samples/sec Loss 13.3063 Epoch: 1 Global Step: 14375 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:40:47,211-Speed 128.25 samples/sec Loss 13.3076 Epoch: 1 Global Step: 14380 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:41:07,133-Speed 128.51 samples/sec Loss 13.6868 Epoch: 1 Global Step: 14385 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:41:26,959-Speed 129.13 samples/sec Loss 13.6092 Epoch: 1 Global Step: 14390 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:41:46,984-Speed 127.85 samples/sec Loss 13.4756 Epoch: 1 Global Step: 14395 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:42:06,832-Speed 128.99 samples/sec Loss 13.5959 Epoch: 1 Global Step: 14400 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:42:27,445-Speed 124.20 samples/sec Loss 13.5643 Epoch: 1 Global Step: 14405 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:42:47,483-Speed 127.76 samples/sec Loss 13.1778 Epoch: 1 Global Step: 14410 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:43:07,350-Speed 128.86 samples/sec Loss 13.6198 Epoch: 1 Global Step: 14415 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:43:27,250-Speed 128.65 samples/sec Loss 13.4492 Epoch: 1 Global Step: 14420 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:43:47,134-Speed 128.75 samples/sec Loss 13.4405 Epoch: 1 Global Step: 14425 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:44:06,949-Speed 129.21 samples/sec Loss 13.5604 Epoch: 1 Global Step: 14430 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:44:26,902-Speed 128.31 samples/sec Loss 13.4652 Epoch: 1 Global Step: 14435 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:44:46,853-Speed 128.33 samples/sec Loss 13.5122 Epoch: 1 Global Step: 14440 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:45:06,886-Speed 127.80 samples/sec Loss 13.8595 Epoch: 1 Global Step: 14445 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:45:27,721-Speed 122.87 samples/sec Loss 13.3168 Epoch: 1 Global Step: 14450 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:45:47,945-Speed 126.59 samples/sec Loss 13.4317 Epoch: 1 Global Step: 14455 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:46:07,852-Speed 128.61 samples/sec Loss 13.4298 Epoch: 1 Global Step: 14460 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:46:27,727-Speed 128.81 samples/sec Loss 13.8242 Epoch: 1 Global Step: 14465 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:46:47,617-Speed 128.72 samples/sec Loss 13.5914 Epoch: 1 Global Step: 14470 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:47:07,585-Speed 128.21 samples/sec Loss 13.7353 Epoch: 1 Global Step: 14475 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:47:27,523-Speed 128.41 samples/sec Loss 13.4679 Epoch: 1 Global Step: 14480 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:47:47,368-Speed 129.01 samples/sec Loss 13.4038 Epoch: 1 Global Step: 14485 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:48:07,483-Speed 127.27 samples/sec Loss 13.6751 Epoch: 1 Global Step: 14490 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:48:28,205-Speed 123.55 samples/sec Loss 13.3951 Epoch: 1 Global Step: 14495 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:48:48,027-Speed 129.15 samples/sec Loss 13.6588 Epoch: 1 Global Step: 14500 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:49:07,917-Speed 128.71 samples/sec Loss 13.5822 Epoch: 1 Global Step: 14505 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:49:27,959-Speed 127.74 samples/sec Loss 13.2198 Epoch: 1 Global Step: 14510 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:49:47,887-Speed 128.47 samples/sec Loss 13.3250 Epoch: 1 Global Step: 14515 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:50:07,937-Speed 127.69 samples/sec Loss 13.2592 Epoch: 1 Global Step: 14520 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:50:27,809-Speed 128.83 samples/sec Loss 13.3683 Epoch: 1 Global Step: 14525 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:50:47,970-Speed 126.99 samples/sec Loss 13.6383 Epoch: 1 Global Step: 14530 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:51:07,720-Speed 129.63 samples/sec Loss 13.3975 Epoch: 1 Global Step: 14535 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:51:28,285-Speed 124.49 samples/sec Loss 13.5165 Epoch: 1 Global Step: 14540 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:51:48,415-Speed 127.18 samples/sec Loss 13.4192 Epoch: 1 Global Step: 14545 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:52:08,502-Speed 127.45 samples/sec Loss 13.3139 Epoch: 1 Global Step: 14550 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:52:28,384-Speed 128.77 samples/sec Loss 13.3005 Epoch: 1 Global Step: 14555 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:52:48,409-Speed 127.85 samples/sec Loss 13.4907 Epoch: 1 Global Step: 14560 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:53:08,483-Speed 127.53 samples/sec Loss 13.4865 Epoch: 1 Global Step: 14565 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:53:28,392-Speed 128.60 samples/sec Loss 13.7263 Epoch: 1 Global Step: 14570 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:53:48,282-Speed 128.72 samples/sec Loss 13.2529 Epoch: 1 Global Step: 14575 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:54:08,305-Speed 127.86 samples/sec Loss 13.2445 Epoch: 1 Global Step: 14580 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:54:29,015-Speed 123.62 samples/sec Loss 13.6428 Epoch: 1 Global Step: 14585 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:54:48,903-Speed 128.73 samples/sec Loss 13.2697 Epoch: 1 Global Step: 14590 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:55:08,807-Speed 128.62 samples/sec Loss 13.2109 Epoch: 1 Global Step: 14595 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:55:28,796-Speed 128.08 samples/sec Loss 13.2959 Epoch: 1 Global Step: 14600 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:55:48,936-Speed 127.12 samples/sec Loss 13.8128 Epoch: 1 Global Step: 14605 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:56:08,605-Speed 130.16 samples/sec Loss 13.2978 Epoch: 1 Global Step: 14610 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:56:29,023-Speed 125.39 samples/sec Loss 13.7175 Epoch: 1 Global Step: 14615 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:56:48,977-Speed 128.31 samples/sec Loss 13.2296 Epoch: 1 Global Step: 14620 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:57:09,388-Speed 125.42 samples/sec Loss 13.7451 Epoch: 1 Global Step: 14625 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:57:29,802-Speed 125.41 samples/sec Loss 13.6518 Epoch: 1 Global Step: 14630 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:57:49,645-Speed 129.02 samples/sec Loss 13.5124 Epoch: 1 Global Step: 14635 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:58:09,473-Speed 129.11 samples/sec Loss 13.3856 Epoch: 1 Global Step: 14640 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:58:29,130-Speed 130.24 samples/sec Loss 13.6897 Epoch: 1 Global Step: 14645 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:58:48,961-Speed 129.10 samples/sec Loss 13.5215 Epoch: 1 Global Step: 14650 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:59:08,720-Speed 129.57 samples/sec Loss 13.6755 Epoch: 1 Global Step: 14655 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 11:59:28,716-Speed 128.04 samples/sec Loss 13.5230 Epoch: 1 Global Step: 14660 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 11:59:48,672-Speed 128.29 samples/sec Loss 13.5951 Epoch: 1 Global Step: 14665 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:00:08,498-Speed 129.13 samples/sec Loss 13.3685 Epoch: 1 Global Step: 14670 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:00:29,154-Speed 123.94 samples/sec Loss 13.5603 Epoch: 1 Global Step: 14675 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:00:48,828-Speed 130.12 samples/sec Loss 13.5932 Epoch: 1 Global Step: 14680 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:01:08,648-Speed 129.17 samples/sec Loss 13.4596 Epoch: 1 Global Step: 14685 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:01:28,502-Speed 128.95 samples/sec Loss 13.4716 Epoch: 1 Global Step: 14690 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:01:48,893-Speed 125.56 samples/sec Loss 13.5828 Epoch: 1 Global Step: 14695 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:02:08,936-Speed 127.73 samples/sec Loss 13.5012 Epoch: 1 Global Step: 14700 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:02:29,005-Speed 127.57 samples/sec Loss 13.4670 Epoch: 1 Global Step: 14705 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:02:49,089-Speed 127.48 samples/sec Loss 13.5548 Epoch: 1 Global Step: 14710 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:03:09,258-Speed 126.93 samples/sec Loss 13.6383 Epoch: 1 Global Step: 14715 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:03:29,974-Speed 123.58 samples/sec Loss 13.2554 Epoch: 1 Global Step: 14720 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:03:50,142-Speed 126.94 samples/sec Loss 13.7684 Epoch: 1 Global Step: 14725 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:04:10,837-Speed 123.71 samples/sec Loss 13.4449 Epoch: 1 Global Step: 14730 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:04:30,747-Speed 128.59 samples/sec Loss 13.4414 Epoch: 1 Global Step: 14735 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:04:50,564-Speed 129.19 samples/sec Loss 13.0457 Epoch: 1 Global Step: 14740 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:05:10,424-Speed 128.91 samples/sec Loss 13.5101 Epoch: 1 Global Step: 14745 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:05:30,113-Speed 130.03 samples/sec Loss 13.4045 Epoch: 1 Global Step: 14750 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:05:50,209-Speed 127.39 samples/sec Loss 13.2334 Epoch: 1 Global Step: 14755 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:06:09,964-Speed 129.60 samples/sec Loss 13.4025 Epoch: 1 Global Step: 14760 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:06:31,096-Speed 121.14 samples/sec Loss 13.3881 Epoch: 1 Global Step: 14765 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:06:50,746-Speed 130.29 samples/sec Loss 13.4068 Epoch: 1 Global Step: 14770 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:07:10,798-Speed 127.68 samples/sec Loss 13.4821 Epoch: 1 Global Step: 14775 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:07:30,660-Speed 128.89 samples/sec Loss 13.6861 Epoch: 1 Global Step: 14780 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:07:50,526-Speed 128.87 samples/sec Loss 13.5292 Epoch: 1 Global Step: 14785 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:08:10,462-Speed 128.42 samples/sec Loss 13.4038 Epoch: 1 Global Step: 14790 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:08:30,288-Speed 129.14 samples/sec Loss 13.4217 Epoch: 1 Global Step: 14795 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:08:50,439-Speed 127.05 samples/sec Loss 13.4717 Epoch: 1 Global Step: 14800 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:09:10,298-Speed 128.92 samples/sec Loss 13.5281 Epoch: 1 Global Step: 14805 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:09:30,970-Speed 123.84 samples/sec Loss 13.3130 Epoch: 1 Global Step: 14810 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:09:50,963-Speed 128.05 samples/sec Loss 13.3699 Epoch: 1 Global Step: 14815 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:10:10,775-Speed 129.22 samples/sec Loss 13.3963 Epoch: 1 Global Step: 14820 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:10:30,676-Speed 128.65 samples/sec Loss 13.2805 Epoch: 1 Global Step: 14825 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:10:50,534-Speed 128.92 samples/sec Loss 13.2785 Epoch: 1 Global Step: 14830 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:11:10,457-Speed 128.50 samples/sec Loss 13.2704 Epoch: 1 Global Step: 14835 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:11:30,619-Speed 126.98 samples/sec Loss 13.1954 Epoch: 1 Global Step: 14840 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:11:50,581-Speed 128.25 samples/sec Loss 13.3621 Epoch: 1 Global Step: 14845 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:12:10,381-Speed 129.30 samples/sec Loss 13.1934 Epoch: 1 Global Step: 14850 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:12:30,964-Speed 124.38 samples/sec Loss 13.3855 Epoch: 1 Global Step: 14855 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:12:50,835-Speed 128.84 samples/sec Loss 13.3351 Epoch: 1 Global Step: 14860 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:13:10,775-Speed 128.39 samples/sec Loss 13.4601 Epoch: 1 Global Step: 14865 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:13:30,604-Speed 129.11 samples/sec Loss 13.5909 Epoch: 1 Global Step: 14870 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:13:50,736-Speed 127.17 samples/sec Loss 13.2325 Epoch: 1 Global Step: 14875 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:14:10,549-Speed 129.21 samples/sec Loss 13.2157 Epoch: 1 Global Step: 14880 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:14:30,502-Speed 128.32 samples/sec Loss 13.5134 Epoch: 1 Global Step: 14885 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:14:50,438-Speed 128.41 samples/sec Loss 13.6940 Epoch: 1 Global Step: 14890 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:15:10,296-Speed 128.92 samples/sec Loss 13.2890 Epoch: 1 Global Step: 14895 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:15:30,859-Speed 124.50 samples/sec Loss 13.3817 Epoch: 1 Global Step: 14900 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:15:51,295-Speed 125.27 samples/sec Loss 13.5021 Epoch: 1 Global Step: 14905 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:16:11,099-Speed 129.28 samples/sec Loss 13.1471 Epoch: 1 Global Step: 14910 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:16:30,941-Speed 129.03 samples/sec Loss 13.0606 Epoch: 1 Global Step: 14915 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:16:50,850-Speed 128.59 samples/sec Loss 13.3203 Epoch: 1 Global Step: 14920 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:17:10,682-Speed 129.09 samples/sec Loss 13.4959 Epoch: 1 Global Step: 14925 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:17:30,518-Speed 129.07 samples/sec Loss 13.1455 Epoch: 1 Global Step: 14930 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:17:50,551-Speed 127.80 samples/sec Loss 13.5484 Epoch: 1 Global Step: 14935 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:18:10,311-Speed 129.56 samples/sec Loss 13.1228 Epoch: 1 Global Step: 14940 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:18:31,196-Speed 122.58 samples/sec Loss 13.6181 Epoch: 1 Global Step: 14945 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:18:51,367-Speed 126.92 samples/sec Loss 13.4395 Epoch: 1 Global Step: 14950 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:19:11,413-Speed 127.71 samples/sec Loss 13.2856 Epoch: 1 Global Step: 14955 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:19:31,582-Speed 126.93 samples/sec Loss 13.4066 Epoch: 1 Global Step: 14960 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:19:51,410-Speed 129.11 samples/sec Loss 13.5299 Epoch: 1 Global Step: 14965 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:20:11,490-Speed 127.50 samples/sec Loss 13.4149 Epoch: 1 Global Step: 14970 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:20:31,540-Speed 127.69 samples/sec Loss 13.0204 Epoch: 1 Global Step: 14975 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:20:51,715-Speed 126.90 samples/sec Loss 13.8333 Epoch: 1 Global Step: 14980 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:21:11,579-Speed 128.88 samples/sec Loss 13.2568 Epoch: 1 Global Step: 14985 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:21:32,252-Speed 123.84 samples/sec Loss 13.4878 Epoch: 1 Global Step: 14990 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:21:52,180-Speed 128.47 samples/sec Loss 13.4811 Epoch: 1 Global Step: 14995 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:22:11,951-Speed 129.49 samples/sec Loss 13.1145 Epoch: 1 Global Step: 15000 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:22:31,960-Speed 127.95 samples/sec Loss 13.6966 Epoch: 1 Global Step: 15005 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:22:51,681-Speed 129.82 samples/sec Loss 13.8228 Epoch: 1 Global Step: 15010 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:23:11,471-Speed 129.37 samples/sec Loss 13.4270 Epoch: 1 Global Step: 15015 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:23:31,076-Speed 130.59 samples/sec Loss 13.3694 Epoch: 1 Global Step: 15020 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:23:50,890-Speed 129.20 samples/sec Loss 13.2761 Epoch: 1 Global Step: 15025 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:24:10,675-Speed 129.40 samples/sec Loss 13.3494 Epoch: 1 Global Step: 15030 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:24:31,137-Speed 125.12 samples/sec Loss 13.4429 Epoch: 1 Global Step: 15035 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:24:51,348-Speed 126.67 samples/sec Loss 13.3004 Epoch: 1 Global Step: 15040 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:25:11,741-Speed 125.54 samples/sec Loss 13.3456 Epoch: 1 Global Step: 15045 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:25:31,725-Speed 128.11 samples/sec Loss 13.5199 Epoch: 1 Global Step: 15050 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:25:52,051-Speed 125.96 samples/sec Loss 13.6117 Epoch: 1 Global Step: 15055 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:26:12,212-Speed 126.99 samples/sec Loss 13.3881 Epoch: 1 Global Step: 15060 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:26:32,290-Speed 127.51 samples/sec Loss 13.5837 Epoch: 1 Global Step: 15065 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:26:52,442-Speed 127.04 samples/sec Loss 13.4213 Epoch: 1 Global Step: 15070 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:27:12,341-Speed 128.65 samples/sec Loss 13.5361 Epoch: 1 Global Step: 15075 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:27:32,932-Speed 124.33 samples/sec Loss 13.4061 Epoch: 1 Global Step: 15080 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:27:52,902-Speed 128.20 samples/sec Loss 13.4099 Epoch: 1 Global Step: 15085 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:28:12,772-Speed 128.85 samples/sec Loss 13.5911 Epoch: 1 Global Step: 15090 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:28:33,428-Speed 123.94 samples/sec Loss 13.4427 Epoch: 1 Global Step: 15095 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:28:53,282-Speed 128.95 samples/sec Loss 13.4891 Epoch: 1 Global Step: 15100 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:29:13,144-Speed 128.90 samples/sec Loss 13.2232 Epoch: 1 Global Step: 15105 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:29:33,510-Speed 125.70 samples/sec Loss 12.9971 Epoch: 1 Global Step: 15110 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:29:53,478-Speed 128.21 samples/sec Loss 13.2251 Epoch: 1 Global Step: 15115 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:30:13,564-Speed 127.46 samples/sec Loss 13.6451 Epoch: 1 Global Step: 15120 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:30:34,099-Speed 124.67 samples/sec Loss 12.9345 Epoch: 1 Global Step: 15125 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:30:53,926-Speed 129.12 samples/sec Loss 13.1618 Epoch: 1 Global Step: 15130 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:31:13,591-Speed 130.18 samples/sec Loss 13.0520 Epoch: 1 Global Step: 15135 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:31:33,233-Speed 130.34 samples/sec Loss 12.9981 Epoch: 1 Global Step: 15140 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:31:53,082-Speed 128.98 samples/sec Loss 13.2569 Epoch: 1 Global Step: 15145 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:32:12,931-Speed 128.98 samples/sec Loss 13.4804 Epoch: 1 Global Step: 15150 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:32:32,872-Speed 128.39 samples/sec Loss 13.1391 Epoch: 1 Global Step: 15155 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:32:52,825-Speed 128.31 samples/sec Loss 13.7073 Epoch: 1 Global Step: 15160 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:33:12,821-Speed 128.03 samples/sec Loss 13.2707 Epoch: 1 Global Step: 15165 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:33:33,488-Speed 123.88 samples/sec Loss 13.3461 Epoch: 1 Global Step: 15170 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:33:53,431-Speed 128.37 samples/sec Loss 13.4412 Epoch: 1 Global Step: 15175 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:34:13,531-Speed 127.37 samples/sec Loss 13.2873 Epoch: 1 Global Step: 15180 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:34:33,517-Speed 128.09 samples/sec Loss 13.0669 Epoch: 1 Global Step: 15185 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:34:53,210-Speed 130.00 samples/sec Loss 13.4817 Epoch: 1 Global Step: 15190 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:35:13,036-Speed 129.13 samples/sec Loss 13.2538 Epoch: 1 Global Step: 15195 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:35:33,000-Speed 128.24 samples/sec Loss 13.5086 Epoch: 1 Global Step: 15200 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:35:54,106-Speed 121.30 samples/sec Loss 13.4162 Epoch: 1 Global Step: 15205 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:36:13,938-Speed 129.09 samples/sec Loss 13.2319 Epoch: 1 Global Step: 15210 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:36:34,949-Speed 121.84 samples/sec Loss 13.6935 Epoch: 1 Global Step: 15215 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:36:55,137-Speed 126.82 samples/sec Loss 13.0923 Epoch: 1 Global Step: 15220 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:37:15,116-Speed 128.14 samples/sec Loss 13.0221 Epoch: 1 Global Step: 15225 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 12:37:35,122-Speed 127.97 samples/sec Loss 13.3569 Epoch: 1 Global Step: 15230 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:37:55,126-Speed 127.98 samples/sec Loss 13.2173 Epoch: 1 Global Step: 15235 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:38:14,964-Speed 129.05 samples/sec Loss 12.8208 Epoch: 1 Global Step: 15240 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:38:35,131-Speed 126.95 samples/sec Loss 13.2620 Epoch: 1 Global Step: 15245 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:38:54,866-Speed 129.73 samples/sec Loss 13.4686 Epoch: 1 Global Step: 15250 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:39:14,887-Speed 127.87 samples/sec Loss 13.0836 Epoch: 1 Global Step: 15255 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:39:35,434-Speed 124.60 samples/sec Loss 13.4565 Epoch: 1 Global Step: 15260 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:39:55,305-Speed 128.84 samples/sec Loss 13.0811 Epoch: 1 Global Step: 15265 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:40:15,142-Speed 129.05 samples/sec Loss 13.2185 Epoch: 1 Global Step: 15270 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:40:34,985-Speed 129.03 samples/sec Loss 13.2166 Epoch: 1 Global Step: 15275 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:40:55,067-Speed 127.48 samples/sec Loss 13.5577 Epoch: 1 Global Step: 15280 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:41:14,927-Speed 128.91 samples/sec Loss 13.4003 Epoch: 1 Global Step: 15285 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:41:34,872-Speed 128.36 samples/sec Loss 13.0031 Epoch: 1 Global Step: 15290 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:41:54,740-Speed 128.86 samples/sec Loss 13.1305 Epoch: 1 Global Step: 15295 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:42:14,920-Speed 126.87 samples/sec Loss 13.2649 Epoch: 1 Global Step: 15300 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:42:35,515-Speed 124.31 samples/sec Loss 13.2518 Epoch: 1 Global Step: 15305 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:42:55,334-Speed 129.18 samples/sec Loss 13.1628 Epoch: 1 Global Step: 15310 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:43:15,498-Speed 126.96 samples/sec Loss 13.1487 Epoch: 1 Global Step: 15315 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:43:36,229-Speed 123.49 samples/sec Loss 13.3041 Epoch: 1 Global Step: 15320 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:43:56,251-Speed 127.87 samples/sec Loss 13.2947 Epoch: 1 Global Step: 15325 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:44:16,091-Speed 129.04 samples/sec Loss 13.2513 Epoch: 1 Global Step: 15330 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:44:36,267-Speed 126.89 samples/sec Loss 13.2661 Epoch: 1 Global Step: 15335 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:44:56,299-Speed 127.81 samples/sec Loss 13.6454 Epoch: 1 Global Step: 15340 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:45:16,379-Speed 127.50 samples/sec Loss 13.6217 Epoch: 1 Global Step: 15345 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:45:37,766-Speed 119.70 samples/sec Loss 13.3644 Epoch: 1 Global Step: 15350 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:45:57,836-Speed 127.56 samples/sec Loss 13.4273 Epoch: 1 Global Step: 15355 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:46:17,906-Speed 127.56 samples/sec Loss 13.2850 Epoch: 1 Global Step: 15360 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:46:38,004-Speed 127.38 samples/sec Loss 13.4610 Epoch: 1 Global Step: 15365 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:46:57,808-Speed 129.28 samples/sec Loss 13.3535 Epoch: 1 Global Step: 15370 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:47:17,796-Speed 128.09 samples/sec Loss 13.3754 Epoch: 1 Global Step: 15375 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:47:37,726-Speed 128.46 samples/sec Loss 13.0983 Epoch: 1 Global Step: 15380 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:47:57,921-Speed 126.77 samples/sec Loss 13.3134 Epoch: 1 Global Step: 15385 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:48:17,867-Speed 128.35 samples/sec Loss 13.2501 Epoch: 1 Global Step: 15390 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:48:38,215-Speed 125.82 samples/sec Loss 13.4382 Epoch: 1 Global Step: 15395 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:48:58,384-Speed 126.93 samples/sec Loss 13.4743 Epoch: 1 Global Step: 15400 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:49:18,205-Speed 129.16 samples/sec Loss 13.5522 Epoch: 1 Global Step: 15405 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:49:38,299-Speed 127.41 samples/sec Loss 13.0104 Epoch: 1 Global Step: 15410 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:49:58,368-Speed 127.57 samples/sec Loss 13.3233 Epoch: 1 Global Step: 15415 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:50:18,314-Speed 128.35 samples/sec Loss 13.3825 Epoch: 1 Global Step: 15420 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:50:38,142-Speed 129.12 samples/sec Loss 13.6046 Epoch: 1 Global Step: 15425 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:50:58,115-Speed 128.18 samples/sec Loss 13.4594 Epoch: 1 Global Step: 15430 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:51:17,955-Speed 129.04 samples/sec Loss 13.0562 Epoch: 1 Global Step: 15435 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:51:38,920-Speed 122.11 samples/sec Loss 13.3731 Epoch: 1 Global Step: 15440 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:51:58,856-Speed 128.42 samples/sec Loss 13.3393 Epoch: 1 Global Step: 15445 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:52:19,162-Speed 126.08 samples/sec Loss 13.2270 Epoch: 1 Global Step: 15450 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:52:38,917-Speed 129.59 samples/sec Loss 13.2353 Epoch: 1 Global Step: 15455 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:52:58,756-Speed 129.05 samples/sec Loss 13.3158 Epoch: 1 Global Step: 15460 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:53:18,735-Speed 128.14 samples/sec Loss 13.2780 Epoch: 1 Global Step: 15465 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:53:38,833-Speed 127.38 samples/sec Loss 12.9888 Epoch: 1 Global Step: 15470 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:53:59,019-Speed 126.83 samples/sec Loss 13.3321 Epoch: 1 Global Step: 15475 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:54:18,977-Speed 128.28 samples/sec Loss 13.2215 Epoch: 1 Global Step: 15480 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:54:39,634-Speed 123.93 samples/sec Loss 13.3755 Epoch: 1 Global Step: 15485 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:54:59,678-Speed 127.73 samples/sec Loss 13.1859 Epoch: 1 Global Step: 15490 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:55:19,523-Speed 129.00 samples/sec Loss 13.1664 Epoch: 1 Global Step: 15495 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:55:39,497-Speed 128.18 samples/sec Loss 13.0500 Epoch: 1 Global Step: 15500 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:55:59,343-Speed 129.00 samples/sec Loss 13.4182 Epoch: 1 Global Step: 15505 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:56:19,747-Speed 125.47 samples/sec Loss 13.4412 Epoch: 1 Global Step: 15510 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:56:39,766-Speed 127.88 samples/sec Loss 13.1809 Epoch: 1 Global Step: 15515 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:56:59,704-Speed 128.41 samples/sec Loss 13.4178 Epoch: 1 Global Step: 15520 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:57:19,690-Speed 128.10 samples/sec Loss 13.4201 Epoch: 1 Global Step: 15525 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:57:40,343-Speed 123.96 samples/sec Loss 13.1386 Epoch: 1 Global Step: 15530 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:58:00,299-Speed 128.28 samples/sec Loss 13.0612 Epoch: 1 Global Step: 15535 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:58:20,423-Speed 127.22 samples/sec Loss 13.2300 Epoch: 1 Global Step: 15540 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:58:40,302-Speed 128.79 samples/sec Loss 13.0142 Epoch: 1 Global Step: 15545 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:59:00,208-Speed 128.61 samples/sec Loss 13.3104 Epoch: 1 Global Step: 15550 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:59:20,130-Speed 128.51 samples/sec Loss 13.4794 Epoch: 1 Global Step: 15555 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 12:59:40,065-Speed 128.43 samples/sec Loss 13.3242 Epoch: 1 Global Step: 15560 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:00:00,164-Speed 127.38 samples/sec Loss 13.0912 Epoch: 1 Global Step: 15565 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:00:19,856-Speed 130.01 samples/sec Loss 13.2142 Epoch: 1 Global Step: 15570 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:00:40,649-Speed 123.13 samples/sec Loss 13.0531 Epoch: 1 Global Step: 15575 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:01:00,481-Speed 129.09 samples/sec Loss 13.3068 Epoch: 1 Global Step: 15580 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:01:20,322-Speed 129.03 samples/sec Loss 13.2539 Epoch: 1 Global Step: 15585 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:01:40,261-Speed 128.40 samples/sec Loss 13.4831 Epoch: 1 Global Step: 15590 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:02:00,117-Speed 128.93 samples/sec Loss 13.1219 Epoch: 1 Global Step: 15595 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:02:20,235-Speed 127.26 samples/sec Loss 13.6041 Epoch: 1 Global Step: 15600 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:02:40,443-Speed 126.69 samples/sec Loss 13.2305 Epoch: 1 Global Step: 15605 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:03:00,429-Speed 128.10 samples/sec Loss 13.4214 Epoch: 1 Global Step: 15610 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:03:20,310-Speed 128.77 samples/sec Loss 13.6440 Epoch: 1 Global Step: 15615 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:03:41,252-Speed 122.25 samples/sec Loss 13.4122 Epoch: 1 Global Step: 15620 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:04:01,338-Speed 127.45 samples/sec Loss 13.3898 Epoch: 1 Global Step: 15625 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:04:21,319-Speed 128.13 samples/sec Loss 13.2719 Epoch: 1 Global Step: 15630 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:04:41,308-Speed 128.08 samples/sec Loss 13.2487 Epoch: 1 Global Step: 15635 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:05:01,303-Speed 128.04 samples/sec Loss 13.1354 Epoch: 1 Global Step: 15640 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:05:21,127-Speed 129.14 samples/sec Loss 13.4467 Epoch: 1 Global Step: 15645 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:05:41,292-Speed 126.96 samples/sec Loss 13.1228 Epoch: 1 Global Step: 15650 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:06:01,358-Speed 127.58 samples/sec Loss 13.2492 Epoch: 1 Global Step: 15655 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:06:21,854-Speed 124.91 samples/sec Loss 13.4450 Epoch: 1 Global Step: 15660 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:06:42,385-Speed 124.70 samples/sec Loss 13.3962 Epoch: 1 Global Step: 15665 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:07:02,121-Speed 129.71 samples/sec Loss 13.3315 Epoch: 1 Global Step: 15670 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:07:22,289-Speed 126.94 samples/sec Loss 13.1920 Epoch: 1 Global Step: 15675 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:07:42,212-Speed 128.50 samples/sec Loss 13.1009 Epoch: 1 Global Step: 15680 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:08:01,971-Speed 129.57 samples/sec Loss 13.2335 Epoch: 1 Global Step: 15685 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:08:21,848-Speed 128.80 samples/sec Loss 13.3703 Epoch: 1 Global Step: 15690 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:08:41,683-Speed 129.07 samples/sec Loss 12.9482 Epoch: 1 Global Step: 15695 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:09:01,520-Speed 129.06 samples/sec Loss 13.0920 Epoch: 1 Global Step: 15700 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:09:22,170-Speed 123.98 samples/sec Loss 13.4218 Epoch: 1 Global Step: 15705 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:09:42,818-Speed 123.99 samples/sec Loss 13.0134 Epoch: 1 Global Step: 15710 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:10:02,783-Speed 128.23 samples/sec Loss 13.1670 Epoch: 1 Global Step: 15715 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:10:22,701-Speed 128.53 samples/sec Loss 13.4454 Epoch: 1 Global Step: 15720 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:10:42,674-Speed 128.18 samples/sec Loss 13.4450 Epoch: 1 Global Step: 15725 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:11:02,775-Speed 127.36 samples/sec Loss 13.6375 Epoch: 1 Global Step: 15730 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:11:22,642-Speed 128.87 samples/sec Loss 13.3953 Epoch: 1 Global Step: 15735 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:11:42,819-Speed 126.89 samples/sec Loss 13.5959 Epoch: 1 Global Step: 15740 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:12:03,005-Speed 126.82 samples/sec Loss 13.2201 Epoch: 1 Global Step: 15745 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:12:23,360-Speed 125.77 samples/sec Loss 13.4542 Epoch: 1 Global Step: 15750 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:12:44,095-Speed 123.47 samples/sec Loss 13.2621 Epoch: 1 Global Step: 15755 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:13:04,208-Speed 127.29 samples/sec Loss 13.3757 Epoch: 1 Global Step: 15760 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:13:25,051-Speed 122.82 samples/sec Loss 13.1798 Epoch: 1 Global Step: 15765 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:13:45,311-Speed 126.37 samples/sec Loss 13.3956 Epoch: 1 Global Step: 15770 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:14:05,199-Speed 128.73 samples/sec Loss 13.2779 Epoch: 1 Global Step: 15775 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:14:25,430-Speed 126.55 samples/sec Loss 13.4281 Epoch: 1 Global Step: 15780 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:14:45,726-Speed 126.14 samples/sec Loss 14.0212 Epoch: 1 Global Step: 15785 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:15:05,932-Speed 126.70 samples/sec Loss 13.6900 Epoch: 1 Global Step: 15790 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:15:25,737-Speed 129.27 samples/sec Loss 13.0095 Epoch: 1 Global Step: 15795 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:15:46,254-Speed 124.78 samples/sec Loss 13.1693 Epoch: 1 Global Step: 15800 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:16:06,225-Speed 128.19 samples/sec Loss 13.0240 Epoch: 1 Global Step: 15805 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:16:26,059-Speed 129.07 samples/sec Loss 13.4734 Epoch: 1 Global Step: 15810 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:16:45,888-Speed 129.11 samples/sec Loss 12.8902 Epoch: 1 Global Step: 15815 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 13:17:05,971-Speed 127.47 samples/sec Loss 13.2630 Epoch: 1 Global Step: 15820 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:17:26,178-Speed 126.70 samples/sec Loss 13.5786 Epoch: 1 Global Step: 15825 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:17:46,508-Speed 125.93 samples/sec Loss 13.5737 Epoch: 1 Global Step: 15830 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:18:06,766-Speed 126.38 samples/sec Loss 13.4197 Epoch: 1 Global Step: 15835 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:18:26,782-Speed 127.90 samples/sec Loss 13.2867 Epoch: 1 Global Step: 15840 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:18:47,342-Speed 124.52 samples/sec Loss 13.3715 Epoch: 1 Global Step: 15845 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:19:07,364-Speed 127.86 samples/sec Loss 13.7641 Epoch: 1 Global Step: 15850 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:19:27,523-Speed 127.00 samples/sec Loss 13.8263 Epoch: 1 Global Step: 15855 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:19:47,398-Speed 128.81 samples/sec Loss 13.2453 Epoch: 1 Global Step: 15860 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:20:07,630-Speed 126.54 samples/sec Loss 13.2313 Epoch: 1 Global Step: 15865 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:20:27,484-Speed 128.95 samples/sec Loss 13.0791 Epoch: 1 Global Step: 15870 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:20:47,288-Speed 129.28 samples/sec Loss 13.4428 Epoch: 1 Global Step: 15875 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:21:07,349-Speed 127.62 samples/sec Loss 13.2215 Epoch: 1 Global Step: 15880 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:21:27,214-Speed 128.87 samples/sec Loss 13.2749 Epoch: 1 Global Step: 15885 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:21:47,838-Speed 124.13 samples/sec Loss 13.1941 Epoch: 1 Global Step: 15890 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:22:08,069-Speed 126.55 samples/sec Loss 13.4771 Epoch: 1 Global Step: 15895 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:22:28,028-Speed 128.27 samples/sec Loss 13.1310 Epoch: 1 Global Step: 15900 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:22:48,281-Speed 126.41 samples/sec Loss 13.6509 Epoch: 1 Global Step: 15905 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:23:09,008-Speed 123.52 samples/sec Loss 13.2562 Epoch: 1 Global Step: 15910 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:23:28,931-Speed 128.51 samples/sec Loss 13.3725 Epoch: 1 Global Step: 15915 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:23:48,857-Speed 128.48 samples/sec Loss 13.0777 Epoch: 1 Global Step: 15920 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:24:08,842-Speed 128.10 samples/sec Loss 13.1354 Epoch: 1 Global Step: 15925 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:24:28,770-Speed 128.47 samples/sec Loss 13.2714 Epoch: 1 Global Step: 15930 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:24:49,355-Speed 124.37 samples/sec Loss 13.7177 Epoch: 1 Global Step: 15935 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:25:09,132-Speed 129.45 samples/sec Loss 13.4725 Epoch: 1 Global Step: 15940 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:25:28,759-Speed 130.44 samples/sec Loss 13.2608 Epoch: 1 Global Step: 15945 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:25:49,065-Speed 126.08 samples/sec Loss 13.3382 Epoch: 1 Global Step: 15950 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:26:09,758-Speed 123.72 samples/sec Loss 13.0008 Epoch: 1 Global Step: 15955 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:26:29,637-Speed 128.78 samples/sec Loss 13.4089 Epoch: 1 Global Step: 15960 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:26:49,567-Speed 128.46 samples/sec Loss 13.2110 Epoch: 1 Global Step: 15965 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:27:09,723-Speed 127.02 samples/sec Loss 13.4041 Epoch: 1 Global Step: 15970 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:27:29,776-Speed 127.67 samples/sec Loss 13.2868 Epoch: 1 Global Step: 15975 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:27:50,457-Speed 123.79 samples/sec Loss 13.3959 Epoch: 1 Global Step: 15980 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:28:10,433-Speed 128.16 samples/sec Loss 13.2426 Epoch: 1 Global Step: 15985 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:28:30,315-Speed 128.77 samples/sec Loss 13.4637 Epoch: 1 Global Step: 15990 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:28:50,449-Speed 127.15 samples/sec Loss 13.2924 Epoch: 1 Global Step: 15995 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:29:10,739-Speed 126.18 samples/sec Loss 13.4957 Epoch: 1 Global Step: 16000 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:29:31,769-Speed 121.74 samples/sec Loss 13.4114 Epoch: 1 Global Step: 16005 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:29:52,038-Speed 126.31 samples/sec Loss 13.1318 Epoch: 1 Global Step: 16010 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:30:12,220-Speed 126.86 samples/sec Loss 13.3199 Epoch: 1 Global Step: 16015 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:30:32,032-Speed 129.22 samples/sec Loss 13.2979 Epoch: 1 Global Step: 16020 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:30:52,463-Speed 125.31 samples/sec Loss 12.9339 Epoch: 1 Global Step: 16025 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:31:12,463-Speed 128.00 samples/sec Loss 13.2140 Epoch: 1 Global Step: 16030 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:31:32,473-Speed 127.94 samples/sec Loss 13.1888 Epoch: 1 Global Step: 16035 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:31:52,549-Speed 127.52 samples/sec Loss 12.9411 Epoch: 1 Global Step: 16040 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:32:12,337-Speed 129.38 samples/sec Loss 13.1808 Epoch: 1 Global Step: 16045 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:32:32,717-Speed 125.62 samples/sec Loss 13.2298 Epoch: 1 Global Step: 16050 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:32:52,576-Speed 128.92 samples/sec Loss 13.0941 Epoch: 1 Global Step: 16055 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:33:12,827-Speed 126.42 samples/sec Loss 13.2077 Epoch: 1 Global Step: 16060 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:33:32,850-Speed 127.86 samples/sec Loss 13.2432 Epoch: 1 Global Step: 16065 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:33:53,534-Speed 123.78 samples/sec Loss 13.2193 Epoch: 1 Global Step: 16070 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:34:13,351-Speed 129.19 samples/sec Loss 13.1798 Epoch: 1 Global Step: 16075 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:34:33,476-Speed 127.21 samples/sec Loss 12.9050 Epoch: 1 Global Step: 16080 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:34:53,122-Speed 130.31 samples/sec Loss 13.3146 Epoch: 1 Global Step: 16085 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:35:13,015-Speed 128.70 samples/sec Loss 13.4018 Epoch: 1 Global Step: 16090 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:35:32,949-Speed 128.43 samples/sec Loss 13.2970 Epoch: 1 Global Step: 16095 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:35:52,912-Speed 128.25 samples/sec Loss 13.1174 Epoch: 1 Global Step: 16100 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:36:12,814-Speed 128.64 samples/sec Loss 13.0689 Epoch: 1 Global Step: 16105 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:36:32,891-Speed 127.52 samples/sec Loss 13.4031 Epoch: 1 Global Step: 16110 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:36:53,581-Speed 123.73 samples/sec Loss 13.4761 Epoch: 1 Global Step: 16115 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:37:13,394-Speed 129.22 samples/sec Loss 13.3776 Epoch: 1 Global Step: 16120 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:37:33,389-Speed 128.04 samples/sec Loss 13.2494 Epoch: 1 Global Step: 16125 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:37:53,344-Speed 128.29 samples/sec Loss 13.1504 Epoch: 1 Global Step: 16130 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:38:13,257-Speed 128.56 samples/sec Loss 13.5459 Epoch: 1 Global Step: 16135 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:38:33,267-Speed 127.95 samples/sec Loss 13.4015 Epoch: 1 Global Step: 16140 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:38:53,219-Speed 128.31 samples/sec Loss 13.4031 Epoch: 1 Global Step: 16145 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:39:12,923-Speed 129.93 samples/sec Loss 13.1230 Epoch: 1 Global Step: 16150 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:39:32,853-Speed 128.46 samples/sec Loss 13.2420 Epoch: 1 Global Step: 16155 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:39:53,536-Speed 123.78 samples/sec Loss 13.6024 Epoch: 1 Global Step: 16160 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:40:13,512-Speed 128.16 samples/sec Loss 13.0087 Epoch: 1 Global Step: 16165 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:40:33,532-Speed 127.88 samples/sec Loss 13.5888 Epoch: 1 Global Step: 16170 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:40:53,186-Speed 130.26 samples/sec Loss 13.0442 Epoch: 1 Global Step: 16175 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:41:13,118-Speed 128.45 samples/sec Loss 13.3920 Epoch: 1 Global Step: 16180 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:41:32,845-Speed 129.78 samples/sec Loss 13.1644 Epoch: 1 Global Step: 16185 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:41:52,959-Speed 127.29 samples/sec Loss 13.2293 Epoch: 1 Global Step: 16190 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:42:13,060-Speed 127.36 samples/sec Loss 13.5009 Epoch: 1 Global Step: 16195 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:42:32,841-Speed 129.42 samples/sec Loss 13.1255 Epoch: 1 Global Step: 16200 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:42:53,660-Speed 122.98 samples/sec Loss 13.1317 Epoch: 1 Global Step: 16205 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:43:13,780-Speed 127.24 samples/sec Loss 13.4699 Epoch: 1 Global Step: 16210 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:43:33,692-Speed 128.57 samples/sec Loss 13.1587 Epoch: 1 Global Step: 16215 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:43:53,821-Speed 127.19 samples/sec Loss 13.2662 Epoch: 1 Global Step: 16220 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:44:13,757-Speed 128.42 samples/sec Loss 13.2735 Epoch: 1 Global Step: 16225 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:44:33,674-Speed 128.54 samples/sec Loss 13.0964 Epoch: 1 Global Step: 16230 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:44:53,589-Speed 128.55 samples/sec Loss 13.1056 Epoch: 1 Global Step: 16235 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:45:13,372-Speed 129.41 samples/sec Loss 12.9710 Epoch: 1 Global Step: 16240 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:45:33,075-Speed 129.94 samples/sec Loss 13.1260 Epoch: 1 Global Step: 16245 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:45:54,017-Speed 122.25 samples/sec Loss 13.1816 Epoch: 1 Global Step: 16250 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:46:13,993-Speed 128.16 samples/sec Loss 13.1534 Epoch: 1 Global Step: 16255 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:46:33,940-Speed 128.34 samples/sec Loss 13.2852 Epoch: 1 Global Step: 16260 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:46:54,110-Speed 126.93 samples/sec Loss 13.2277 Epoch: 1 Global Step: 16265 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:47:14,228-Speed 127.25 samples/sec Loss 13.2724 Epoch: 1 Global Step: 16270 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:47:34,342-Speed 127.28 samples/sec Loss 13.3176 Epoch: 1 Global Step: 16275 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:47:54,443-Speed 127.37 samples/sec Loss 13.0349 Epoch: 1 Global Step: 16280 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:48:14,457-Speed 127.92 samples/sec Loss 12.9077 Epoch: 1 Global Step: 16285 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:48:34,418-Speed 128.25 samples/sec Loss 13.3492 Epoch: 1 Global Step: 16290 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:48:55,005-Speed 124.36 samples/sec Loss 13.2153 Epoch: 1 Global Step: 16295 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:49:15,352-Speed 125.82 samples/sec Loss 13.1499 Epoch: 1 Global Step: 16300 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:49:35,601-Speed 126.44 samples/sec Loss 13.2657 Epoch: 1 Global Step: 16305 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:49:55,670-Speed 127.56 samples/sec Loss 13.6246 Epoch: 1 Global Step: 16310 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:50:15,446-Speed 129.46 samples/sec Loss 13.6855 Epoch: 1 Global Step: 16315 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:50:35,359-Speed 128.57 samples/sec Loss 13.1714 Epoch: 1 Global Step: 16320 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:50:55,477-Speed 127.26 samples/sec Loss 13.4405 Epoch: 1 Global Step: 16325 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:51:15,852-Speed 125.65 samples/sec Loss 13.3408 Epoch: 1 Global Step: 16330 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:51:35,814-Speed 128.25 samples/sec Loss 13.4479 Epoch: 1 Global Step: 16335 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:51:56,322-Speed 124.85 samples/sec Loss 13.0140 Epoch: 1 Global Step: 16340 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:52:16,088-Speed 129.52 samples/sec Loss 12.7094 Epoch: 1 Global Step: 16345 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:52:35,849-Speed 129.55 samples/sec Loss 13.2072 Epoch: 1 Global Step: 16350 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:52:55,697-Speed 128.99 samples/sec Loss 12.9633 Epoch: 1 Global Step: 16355 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:53:15,408-Speed 129.88 samples/sec Loss 13.4112 Epoch: 1 Global Step: 16360 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:53:35,358-Speed 128.33 samples/sec Loss 13.0969 Epoch: 1 Global Step: 16365 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:53:55,245-Speed 128.73 samples/sec Loss 13.1128 Epoch: 1 Global Step: 16370 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:54:15,331-Speed 127.46 samples/sec Loss 13.1003 Epoch: 1 Global Step: 16375 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:54:35,295-Speed 128.24 samples/sec Loss 13.0095 Epoch: 1 Global Step: 16380 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:54:56,090-Speed 123.11 samples/sec Loss 13.1303 Epoch: 1 Global Step: 16385 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:55:16,383-Speed 126.16 samples/sec Loss 13.6373 Epoch: 1 Global Step: 16390 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:55:36,371-Speed 128.08 samples/sec Loss 13.0470 Epoch: 1 Global Step: 16395 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:55:56,499-Speed 127.20 samples/sec Loss 13.2681 Epoch: 1 Global Step: 16400 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:56:16,310-Speed 129.23 samples/sec Loss 13.2570 Epoch: 1 Global Step: 16405 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:56:35,936-Speed 130.45 samples/sec Loss 13.0423 Epoch: 1 Global Step: 16410 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:56:55,847-Speed 128.58 samples/sec Loss 13.4218 Epoch: 1 Global Step: 16415 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:57:15,677-Speed 129.11 samples/sec Loss 12.8450 Epoch: 1 Global Step: 16420 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 13:57:35,482-Speed 129.27 samples/sec Loss 13.0792 Epoch: 1 Global Step: 16425 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 13:57:56,246-Speed 123.30 samples/sec Loss 13.2691 Epoch: 1 Global Step: 16430 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 13:58:16,239-Speed 128.05 samples/sec Loss 12.8666 Epoch: 1 Global Step: 16435 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 13:58:36,251-Speed 127.93 samples/sec Loss 12.9998 Epoch: 1 Global Step: 16440 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 13:58:55,904-Speed 130.26 samples/sec Loss 13.4139 Epoch: 1 Global Step: 16445 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 13:59:16,000-Speed 127.40 samples/sec Loss 13.3196 Epoch: 1 Global Step: 16450 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 13:59:36,107-Speed 127.32 samples/sec Loss 13.0869 Epoch: 1 Global Step: 16455 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 13:59:56,024-Speed 128.54 samples/sec Loss 13.5220 Epoch: 1 Global Step: 16460 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:00:16,059-Speed 127.79 samples/sec Loss 13.3560 Epoch: 1 Global Step: 16465 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:00:35,972-Speed 128.57 samples/sec Loss 12.9992 Epoch: 1 Global Step: 16470 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:00:56,621-Speed 123.98 samples/sec Loss 13.0175 Epoch: 1 Global Step: 16475 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:01:16,520-Speed 128.65 samples/sec Loss 13.3198 Epoch: 1 Global Step: 16480 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:01:37,136-Speed 124.18 samples/sec Loss 13.0791 Epoch: 1 Global Step: 16485 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:01:57,263-Speed 127.20 samples/sec Loss 13.0796 Epoch: 1 Global Step: 16490 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:02:17,093-Speed 129.11 samples/sec Loss 13.2027 Epoch: 1 Global Step: 16495 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:02:37,049-Speed 128.29 samples/sec Loss 13.5067 Epoch: 1 Global Step: 16500 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:02:57,076-Speed 127.83 samples/sec Loss 12.9999 Epoch: 1 Global Step: 16505 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:03:16,874-Speed 129.31 samples/sec Loss 13.1633 Epoch: 1 Global Step: 16510 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:03:36,873-Speed 128.01 samples/sec Loss 13.3464 Epoch: 1 Global Step: 16515 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:03:57,696-Speed 122.95 samples/sec Loss 13.1398 Epoch: 1 Global Step: 16520 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:04:17,730-Speed 127.79 samples/sec Loss 13.6170 Epoch: 1 Global Step: 16525 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:04:37,851-Speed 127.23 samples/sec Loss 13.3536 Epoch: 1 Global Step: 16530 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:04:57,759-Speed 128.60 samples/sec Loss 13.3200 Epoch: 1 Global Step: 16535 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:05:17,841-Speed 127.48 samples/sec Loss 13.3731 Epoch: 1 Global Step: 16540 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:05:37,638-Speed 129.32 samples/sec Loss 13.0519 Epoch: 1 Global Step: 16545 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:05:57,541-Speed 128.63 samples/sec Loss 12.8800 Epoch: 1 Global Step: 16550 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:06:17,353-Speed 129.22 samples/sec Loss 13.2420 Epoch: 1 Global Step: 16555 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:06:37,406-Speed 127.67 samples/sec Loss 13.3676 Epoch: 1 Global Step: 16560 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:06:57,768-Speed 125.73 samples/sec Loss 12.5556 Epoch: 1 Global Step: 16565 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:07:17,692-Speed 128.49 samples/sec Loss 13.3063 Epoch: 1 Global Step: 16570 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:07:37,742-Speed 127.69 samples/sec Loss 13.1851 Epoch: 1 Global Step: 16575 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:07:57,649-Speed 128.60 samples/sec Loss 13.4565 Epoch: 1 Global Step: 16580 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:08:17,490-Speed 129.03 samples/sec Loss 13.0099 Epoch: 1 Global Step: 16585 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:08:37,295-Speed 129.27 samples/sec Loss 12.9530 Epoch: 1 Global Step: 16590 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:08:57,201-Speed 128.61 samples/sec Loss 13.2525 Epoch: 1 Global Step: 16595 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:09:17,021-Speed 129.17 samples/sec Loss 12.8225 Epoch: 1 Global Step: 16600 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:09:37,038-Speed 127.90 samples/sec Loss 13.0664 Epoch: 1 Global Step: 16605 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:09:57,674-Speed 124.06 samples/sec Loss 13.0805 Epoch: 1 Global Step: 16610 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:10:17,596-Speed 128.51 samples/sec Loss 13.2306 Epoch: 1 Global Step: 16615 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:10:37,324-Speed 129.77 samples/sec Loss 12.9549 Epoch: 1 Global Step: 16620 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:10:57,161-Speed 129.06 samples/sec Loss 12.8512 Epoch: 1 Global Step: 16625 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:11:17,054-Speed 128.70 samples/sec Loss 12.9897 Epoch: 1 Global Step: 16630 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:11:36,975-Speed 128.51 samples/sec Loss 13.2976 Epoch: 1 Global Step: 16635 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:11:57,111-Speed 127.15 samples/sec Loss 13.1707 Epoch: 1 Global Step: 16640 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:12:17,204-Speed 127.42 samples/sec Loss 13.3814 Epoch: 1 Global Step: 16645 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:12:37,202-Speed 128.02 samples/sec Loss 13.0022 Epoch: 1 Global Step: 16650 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:12:57,816-Speed 124.20 samples/sec Loss 13.0368 Epoch: 1 Global Step: 16655 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:13:17,577-Speed 129.55 samples/sec Loss 12.7638 Epoch: 1 Global Step: 16660 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:13:37,637-Speed 127.62 samples/sec Loss 13.3842 Epoch: 1 Global Step: 16665 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:13:57,508-Speed 128.84 samples/sec Loss 13.1091 Epoch: 1 Global Step: 16670 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:14:17,410-Speed 128.64 samples/sec Loss 13.1883 Epoch: 1 Global Step: 16675 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:14:37,610-Speed 126.74 samples/sec Loss 13.3827 Epoch: 1 Global Step: 16680 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:14:57,515-Speed 128.62 samples/sec Loss 13.0641 Epoch: 1 Global Step: 16685 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:15:17,561-Speed 127.71 samples/sec Loss 13.2311 Epoch: 1 Global Step: 16690 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:15:37,601-Speed 127.75 samples/sec Loss 13.4727 Epoch: 1 Global Step: 16695 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:15:58,168-Speed 124.48 samples/sec Loss 12.9109 Epoch: 1 Global Step: 16700 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:16:18,291-Speed 127.23 samples/sec Loss 13.1375 Epoch: 1 Global Step: 16705 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:16:38,384-Speed 127.41 samples/sec Loss 12.8686 Epoch: 1 Global Step: 16710 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:16:58,190-Speed 129.26 samples/sec Loss 13.1580 Epoch: 1 Global Step: 16715 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:17:18,207-Speed 127.90 samples/sec Loss 12.9244 Epoch: 1 Global Step: 16720 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:17:38,335-Speed 127.19 samples/sec Loss 13.0684 Epoch: 1 Global Step: 16725 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:17:58,282-Speed 128.35 samples/sec Loss 13.0952 Epoch: 1 Global Step: 16730 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:18:18,086-Speed 129.27 samples/sec Loss 12.8084 Epoch: 1 Global Step: 16735 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:18:37,912-Speed 129.13 samples/sec Loss 12.8591 Epoch: 1 Global Step: 16740 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:18:58,613-Speed 123.68 samples/sec Loss 12.7833 Epoch: 1 Global Step: 16745 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:19:18,838-Speed 126.58 samples/sec Loss 13.3089 Epoch: 1 Global Step: 16750 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:19:38,715-Speed 128.79 samples/sec Loss 13.0036 Epoch: 1 Global Step: 16755 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:19:58,623-Speed 128.60 samples/sec Loss 13.1984 Epoch: 1 Global Step: 16760 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:20:18,559-Speed 128.42 samples/sec Loss 12.8804 Epoch: 1 Global Step: 16765 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:20:38,328-Speed 129.50 samples/sec Loss 13.1111 Epoch: 1 Global Step: 16770 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:20:58,098-Speed 129.49 samples/sec Loss 13.0755 Epoch: 1 Global Step: 16775 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:21:18,219-Speed 127.24 samples/sec Loss 13.2667 Epoch: 1 Global Step: 16780 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:21:38,088-Speed 128.85 samples/sec Loss 12.6913 Epoch: 1 Global Step: 16785 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:21:58,924-Speed 122.87 samples/sec Loss 12.9909 Epoch: 1 Global Step: 16790 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:22:18,999-Speed 127.52 samples/sec Loss 12.8125 Epoch: 1 Global Step: 16795 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:22:39,033-Speed 127.79 samples/sec Loss 13.2280 Epoch: 1 Global Step: 16800 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:22:59,117-Speed 127.47 samples/sec Loss 12.9577 Epoch: 1 Global Step: 16805 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:23:19,320-Speed 126.72 samples/sec Loss 13.4072 Epoch: 1 Global Step: 16810 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:23:39,500-Speed 126.87 samples/sec Loss 13.1055 Epoch: 1 Global Step: 16815 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:23:59,714-Speed 126.65 samples/sec Loss 13.1354 Epoch: 1 Global Step: 16820 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:24:19,817-Speed 127.35 samples/sec Loss 13.1119 Epoch: 1 Global Step: 16825 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:24:39,826-Speed 127.95 samples/sec Loss 13.4647 Epoch: 1 Global Step: 16830 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:25:00,408-Speed 124.39 samples/sec Loss 13.5218 Epoch: 1 Global Step: 16835 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:25:20,288-Speed 128.78 samples/sec Loss 12.7763 Epoch: 1 Global Step: 16840 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:25:40,201-Speed 128.56 samples/sec Loss 13.3037 Epoch: 1 Global Step: 16845 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:26:00,151-Speed 128.33 samples/sec Loss 13.1585 Epoch: 1 Global Step: 16850 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:26:20,050-Speed 128.66 samples/sec Loss 13.0371 Epoch: 1 Global Step: 16855 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:26:40,059-Speed 127.95 samples/sec Loss 12.9694 Epoch: 1 Global Step: 16860 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:27:00,208-Speed 127.06 samples/sec Loss 12.9973 Epoch: 1 Global Step: 16865 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:27:20,058-Speed 128.98 samples/sec Loss 13.1499 Epoch: 1 Global Step: 16870 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:27:40,535-Speed 125.02 samples/sec Loss 13.1475 Epoch: 1 Global Step: 16875 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:28:01,517-Speed 122.02 samples/sec Loss 13.2737 Epoch: 1 Global Step: 16880 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:28:21,578-Speed 127.61 samples/sec Loss 13.2165 Epoch: 1 Global Step: 16885 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:28:41,712-Speed 127.15 samples/sec Loss 13.1222 Epoch: 1 Global Step: 16890 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:29:02,006-Speed 126.15 samples/sec Loss 13.6206 Epoch: 1 Global Step: 16895 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:29:22,067-Speed 127.62 samples/sec Loss 13.0412 Epoch: 1 Global Step: 16900 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:29:42,166-Speed 127.37 samples/sec Loss 13.2759 Epoch: 1 Global Step: 16905 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:30:02,085-Speed 128.53 samples/sec Loss 12.9865 Epoch: 1 Global Step: 16910 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:30:22,367-Speed 126.23 samples/sec Loss 13.4246 Epoch: 1 Global Step: 16915 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:30:42,381-Speed 127.93 samples/sec Loss 13.1112 Epoch: 1 Global Step: 16920 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:31:03,304-Speed 122.37 samples/sec Loss 13.0329 Epoch: 1 Global Step: 16925 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:31:23,102-Speed 129.31 samples/sec Loss 12.7111 Epoch: 1 Global Step: 16930 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:31:43,178-Speed 127.52 samples/sec Loss 12.8210 Epoch: 1 Global Step: 16935 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:32:03,265-Speed 127.45 samples/sec Loss 13.2298 Epoch: 1 Global Step: 16940 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:32:23,223-Speed 128.28 samples/sec Loss 13.4037 Epoch: 1 Global Step: 16945 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:32:43,290-Speed 127.58 samples/sec Loss 13.0632 Epoch: 1 Global Step: 16950 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:33:03,487-Speed 126.76 samples/sec Loss 13.2206 Epoch: 1 Global Step: 16955 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:33:23,590-Speed 127.35 samples/sec Loss 13.0220 Epoch: 1 Global Step: 16960 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:33:44,776-Speed 120.84 samples/sec Loss 12.6927 Epoch: 1 Global Step: 16965 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:34:05,560-Speed 123.18 samples/sec Loss 13.3673 Epoch: 1 Global Step: 16970 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:34:25,897-Speed 125.89 samples/sec Loss 13.1969 Epoch: 1 Global Step: 16975 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:34:45,937-Speed 127.75 samples/sec Loss 13.2066 Epoch: 1 Global Step: 16980 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:35:06,181-Speed 126.46 samples/sec Loss 13.2396 Epoch: 1 Global Step: 16985 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:35:26,357-Speed 126.89 samples/sec Loss 13.2834 Epoch: 1 Global Step: 16990 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:35:46,293-Speed 128.42 samples/sec Loss 13.1146 Epoch: 1 Global Step: 16995 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:36:06,351-Speed 127.64 samples/sec Loss 13.3263 Epoch: 1 Global Step: 17000 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:36:26,508-Speed 127.01 samples/sec Loss 13.0191 Epoch: 1 Global Step: 17005 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:36:46,525-Speed 127.90 samples/sec Loss 13.2408 Epoch: 1 Global Step: 17010 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:37:07,514-Speed 121.97 samples/sec Loss 13.3025 Epoch: 1 Global Step: 17015 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:37:27,447-Speed 128.44 samples/sec Loss 13.1377 Epoch: 1 Global Step: 17020 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:37:47,461-Speed 127.91 samples/sec Loss 13.1174 Epoch: 1 Global Step: 17025 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:38:07,336-Speed 128.81 samples/sec Loss 12.8772 Epoch: 1 Global Step: 17030 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:38:27,221-Speed 128.75 samples/sec Loss 13.2986 Epoch: 1 Global Step: 17035 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:38:47,080-Speed 128.92 samples/sec Loss 13.1364 Epoch: 1 Global Step: 17040 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:39:07,121-Speed 127.74 samples/sec Loss 13.2378 Epoch: 1 Global Step: 17045 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 14:39:27,205-Speed 127.47 samples/sec Loss 13.0931 Epoch: 1 Global Step: 17050 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:39:47,352-Speed 127.07 samples/sec Loss 12.8233 Epoch: 1 Global Step: 17055 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:40:08,062-Speed 123.62 samples/sec Loss 13.0172 Epoch: 1 Global Step: 17060 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:40:27,970-Speed 128.60 samples/sec Loss 13.1532 Epoch: 1 Global Step: 17065 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:40:48,201-Speed 126.54 samples/sec Loss 13.0060 Epoch: 1 Global Step: 17070 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:41:08,190-Speed 128.07 samples/sec Loss 13.1947 Epoch: 1 Global Step: 17075 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:41:28,025-Speed 129.08 samples/sec Loss 13.5461 Epoch: 1 Global Step: 17080 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:41:47,723-Speed 129.97 samples/sec Loss 12.7856 Epoch: 1 Global Step: 17085 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:42:07,477-Speed 129.60 samples/sec Loss 12.8134 Epoch: 1 Global Step: 17090 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:42:27,420-Speed 128.37 samples/sec Loss 13.0502 Epoch: 1 Global Step: 17095 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:42:47,460-Speed 127.75 samples/sec Loss 13.2613 Epoch: 1 Global Step: 17100 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:43:08,365-Speed 122.47 samples/sec Loss 13.2057 Epoch: 1 Global Step: 17105 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:43:28,367-Speed 127.99 samples/sec Loss 13.1251 Epoch: 1 Global Step: 17110 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:43:48,404-Speed 127.77 samples/sec Loss 12.7831 Epoch: 1 Global Step: 17115 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:44:08,621-Speed 126.64 samples/sec Loss 12.9311 Epoch: 1 Global Step: 17120 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:44:28,695-Speed 127.53 samples/sec Loss 12.7582 Epoch: 1 Global Step: 17125 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:44:49,048-Speed 125.79 samples/sec Loss 13.2324 Epoch: 1 Global Step: 17130 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:45:09,126-Speed 127.52 samples/sec Loss 12.7980 Epoch: 1 Global Step: 17135 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:45:29,248-Speed 127.23 samples/sec Loss 13.3979 Epoch: 1 Global Step: 17140 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:45:49,421-Speed 126.91 samples/sec Loss 13.1506 Epoch: 1 Global Step: 17145 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:46:10,417-Speed 121.94 samples/sec Loss 13.0414 Epoch: 1 Global Step: 17150 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:46:30,538-Speed 127.23 samples/sec Loss 13.2555 Epoch: 1 Global Step: 17155 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:46:50,682-Speed 127.09 samples/sec Loss 13.3243 Epoch: 1 Global Step: 17160 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:47:10,602-Speed 128.52 samples/sec Loss 12.9642 Epoch: 1 Global Step: 17165 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:47:30,703-Speed 127.37 samples/sec Loss 13.2934 Epoch: 1 Global Step: 17170 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:47:50,996-Speed 126.16 samples/sec Loss 13.2529 Epoch: 1 Global Step: 17175 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:48:10,951-Speed 128.30 samples/sec Loss 12.8989 Epoch: 1 Global Step: 17180 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:48:31,073-Speed 127.23 samples/sec Loss 12.8400 Epoch: 1 Global Step: 17185 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:48:50,953-Speed 128.78 samples/sec Loss 13.2124 Epoch: 1 Global Step: 17190 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:49:11,720-Speed 123.28 samples/sec Loss 13.1399 Epoch: 1 Global Step: 17195 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:49:31,770-Speed 127.68 samples/sec Loss 12.9801 Epoch: 1 Global Step: 17200 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:49:51,718-Speed 128.34 samples/sec Loss 13.1095 Epoch: 1 Global Step: 17205 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:50:11,634-Speed 128.55 samples/sec Loss 13.1977 Epoch: 1 Global Step: 17210 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:50:31,420-Speed 129.39 samples/sec Loss 13.1299 Epoch: 1 Global Step: 17215 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:50:51,667-Speed 126.45 samples/sec Loss 13.2368 Epoch: 1 Global Step: 17220 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:51:11,991-Speed 125.96 samples/sec Loss 13.2463 Epoch: 1 Global Step: 17225 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:51:32,011-Speed 127.88 samples/sec Loss 12.9400 Epoch: 1 Global Step: 17230 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:51:52,269-Speed 126.38 samples/sec Loss 13.0547 Epoch: 1 Global Step: 17235 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:52:12,977-Speed 123.63 samples/sec Loss 13.3436 Epoch: 1 Global Step: 17240 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:52:33,161-Speed 126.84 samples/sec Loss 13.4854 Epoch: 1 Global Step: 17245 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:52:53,077-Speed 128.55 samples/sec Loss 13.3038 Epoch: 1 Global Step: 17250 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:53:12,993-Speed 128.55 samples/sec Loss 13.0463 Epoch: 1 Global Step: 17255 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:53:32,973-Speed 128.13 samples/sec Loss 12.9591 Epoch: 1 Global Step: 17260 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:53:52,902-Speed 128.47 samples/sec Loss 12.8609 Epoch: 1 Global Step: 17265 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:54:13,194-Speed 126.17 samples/sec Loss 13.0025 Epoch: 1 Global Step: 17270 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:54:33,387-Speed 126.78 samples/sec Loss 13.1403 Epoch: 1 Global Step: 17275 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:54:53,277-Speed 128.72 samples/sec Loss 12.6019 Epoch: 1 Global Step: 17280 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:55:13,671-Speed 125.54 samples/sec Loss 13.2951 Epoch: 1 Global Step: 17285 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:55:33,692-Speed 127.87 samples/sec Loss 13.3472 Epoch: 1 Global Step: 17290 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:55:54,193-Speed 124.88 samples/sec Loss 13.3155 Epoch: 1 Global Step: 17295 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:56:14,027-Speed 129.08 samples/sec Loss 13.0534 Epoch: 1 Global Step: 17300 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:56:34,064-Speed 127.77 samples/sec Loss 13.1691 Epoch: 1 Global Step: 17305 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:56:54,055-Speed 128.07 samples/sec Loss 12.9278 Epoch: 1 Global Step: 17310 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:57:14,155-Speed 127.37 samples/sec Loss 13.2710 Epoch: 1 Global Step: 17315 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:57:34,181-Speed 127.84 samples/sec Loss 13.4536 Epoch: 1 Global Step: 17320 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:57:54,145-Speed 128.24 samples/sec Loss 13.2930 Epoch: 1 Global Step: 17325 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:58:14,769-Speed 124.14 samples/sec Loss 12.9887 Epoch: 1 Global Step: 17330 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:58:34,892-Speed 127.22 samples/sec Loss 13.0616 Epoch: 1 Global Step: 17335 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:58:55,007-Speed 127.27 samples/sec Loss 13.0732 Epoch: 1 Global Step: 17340 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:59:15,353-Speed 125.83 samples/sec Loss 13.1386 Epoch: 1 Global Step: 17345 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:59:35,409-Speed 127.65 samples/sec Loss 13.1142 Epoch: 1 Global Step: 17350 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 14:59:55,758-Speed 125.81 samples/sec Loss 13.0673 Epoch: 1 Global Step: 17355 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:00:15,834-Speed 127.52 samples/sec Loss 12.8418 Epoch: 1 Global Step: 17360 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:00:35,992-Speed 127.01 samples/sec Loss 13.0011 Epoch: 1 Global Step: 17365 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:00:55,888-Speed 128.68 samples/sec Loss 12.9455 Epoch: 1 Global Step: 17370 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:01:16,571-Speed 123.78 samples/sec Loss 13.1116 Epoch: 1 Global Step: 17375 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:01:36,488-Speed 128.54 samples/sec Loss 12.8959 Epoch: 1 Global Step: 17380 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:01:56,751-Speed 126.35 samples/sec Loss 13.0192 Epoch: 1 Global Step: 17385 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:02:16,547-Speed 129.33 samples/sec Loss 13.0502 Epoch: 1 Global Step: 17390 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:02:36,627-Speed 127.50 samples/sec Loss 12.7825 Epoch: 1 Global Step: 17395 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:02:56,775-Speed 127.06 samples/sec Loss 12.9835 Epoch: 1 Global Step: 17400 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:03:16,747-Speed 128.19 samples/sec Loss 13.3185 Epoch: 1 Global Step: 17405 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:03:37,017-Speed 126.30 samples/sec Loss 13.0206 Epoch: 1 Global Step: 17410 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:03:57,345-Speed 125.94 samples/sec Loss 12.9958 Epoch: 1 Global Step: 17415 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:04:18,127-Speed 123.19 samples/sec Loss 12.6298 Epoch: 1 Global Step: 17420 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:04:38,133-Speed 127.98 samples/sec Loss 12.9582 Epoch: 1 Global Step: 17425 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:04:58,026-Speed 128.70 samples/sec Loss 13.0689 Epoch: 1 Global Step: 17430 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:05:17,975-Speed 128.33 samples/sec Loss 13.1219 Epoch: 1 Global Step: 17435 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:05:37,974-Speed 128.02 samples/sec Loss 13.2953 Epoch: 1 Global Step: 17440 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:05:57,770-Speed 129.32 samples/sec Loss 12.8375 Epoch: 1 Global Step: 17445 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:06:17,762-Speed 128.06 samples/sec Loss 12.8557 Epoch: 1 Global Step: 17450 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:06:37,694-Speed 128.44 samples/sec Loss 12.9264 Epoch: 1 Global Step: 17455 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:06:57,645-Speed 128.32 samples/sec Loss 13.0866 Epoch: 1 Global Step: 17460 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:07:18,341-Speed 123.70 samples/sec Loss 12.6697 Epoch: 1 Global Step: 17465 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:07:38,556-Speed 126.65 samples/sec Loss 13.5312 Epoch: 1 Global Step: 17470 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:07:58,403-Speed 128.99 samples/sec Loss 12.9877 Epoch: 1 Global Step: 17475 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:08:18,384-Speed 128.13 samples/sec Loss 13.1129 Epoch: 1 Global Step: 17480 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:08:38,550-Speed 126.95 samples/sec Loss 12.8948 Epoch: 1 Global Step: 17485 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:08:58,680-Speed 127.18 samples/sec Loss 13.2016 Epoch: 1 Global Step: 17490 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:09:18,505-Speed 129.14 samples/sec Loss 12.9674 Epoch: 1 Global Step: 17495 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:09:38,701-Speed 126.77 samples/sec Loss 13.1810 Epoch: 1 Global Step: 17500 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:09:58,848-Speed 127.07 samples/sec Loss 13.2004 Epoch: 1 Global Step: 17505 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:10:19,550-Speed 123.66 samples/sec Loss 12.8723 Epoch: 1 Global Step: 17510 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:10:39,227-Speed 130.10 samples/sec Loss 12.7328 Epoch: 1 Global Step: 17515 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:10:59,328-Speed 127.36 samples/sec Loss 12.7289 Epoch: 1 Global Step: 17520 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:11:19,395-Speed 127.58 samples/sec Loss 13.1271 Epoch: 1 Global Step: 17525 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:11:39,478-Speed 127.48 samples/sec Loss 12.9553 Epoch: 1 Global Step: 17530 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:11:59,513-Speed 127.78 samples/sec Loss 13.0489 Epoch: 1 Global Step: 17535 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:12:19,357-Speed 129.02 samples/sec Loss 12.8117 Epoch: 1 Global Step: 17540 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:12:39,271-Speed 128.56 samples/sec Loss 12.7807 Epoch: 1 Global Step: 17545 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:12:59,239-Speed 128.21 samples/sec Loss 12.7221 Epoch: 1 Global Step: 17550 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:13:19,689-Speed 125.19 samples/sec Loss 13.1005 Epoch: 1 Global Step: 17555 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:13:39,640-Speed 128.32 samples/sec Loss 12.8440 Epoch: 1 Global Step: 17560 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:13:59,882-Speed 126.48 samples/sec Loss 12.9896 Epoch: 1 Global Step: 17565 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:14:20,048-Speed 126.95 samples/sec Loss 13.3161 Epoch: 1 Global Step: 17570 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:14:39,945-Speed 128.67 samples/sec Loss 13.1210 Epoch: 1 Global Step: 17575 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:14:59,914-Speed 128.21 samples/sec Loss 13.0276 Epoch: 1 Global Step: 17580 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:15:19,968-Speed 127.66 samples/sec Loss 12.5961 Epoch: 1 Global Step: 17585 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:15:40,139-Speed 126.93 samples/sec Loss 12.8685 Epoch: 1 Global Step: 17590 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:16:00,228-Speed 127.44 samples/sec Loss 12.9293 Epoch: 1 Global Step: 17595 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:16:20,836-Speed 124.23 samples/sec Loss 13.4113 Epoch: 1 Global Step: 17600 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:16:40,626-Speed 129.36 samples/sec Loss 13.1519 Epoch: 1 Global Step: 17605 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:17:00,507-Speed 128.77 samples/sec Loss 13.0015 Epoch: 1 Global Step: 17610 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:17:20,630-Speed 127.23 samples/sec Loss 12.9083 Epoch: 1 Global Step: 17615 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:17:41,028-Speed 125.51 samples/sec Loss 13.1821 Epoch: 1 Global Step: 17620 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:18:01,097-Speed 127.57 samples/sec Loss 12.6649 Epoch: 1 Global Step: 17625 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:18:21,085-Speed 128.08 samples/sec Loss 13.0329 Epoch: 1 Global Step: 17630 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:18:41,187-Speed 127.36 samples/sec Loss 12.8171 Epoch: 1 Global Step: 17635 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:19:01,488-Speed 126.11 samples/sec Loss 13.0107 Epoch: 1 Global Step: 17640 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:19:22,176-Speed 123.75 samples/sec Loss 13.0770 Epoch: 1 Global Step: 17645 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:19:42,347-Speed 126.92 samples/sec Loss 13.0531 Epoch: 1 Global Step: 17650 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:20:02,540-Speed 126.78 samples/sec Loss 12.8223 Epoch: 1 Global Step: 17655 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:20:22,929-Speed 125.57 samples/sec Loss 13.1640 Epoch: 1 Global Step: 17660 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:20:42,953-Speed 127.85 samples/sec Loss 13.0926 Epoch: 1 Global Step: 17665 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:21:03,051-Speed 127.39 samples/sec Loss 12.9696 Epoch: 1 Global Step: 17670 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:21:23,090-Speed 127.76 samples/sec Loss 13.2013 Epoch: 1 Global Step: 17675 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:21:43,239-Speed 127.06 samples/sec Loss 13.0414 Epoch: 1 Global Step: 17680 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:22:03,126-Speed 128.73 samples/sec Loss 12.8175 Epoch: 1 Global Step: 17685 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:22:23,676-Speed 124.59 samples/sec Loss 12.8591 Epoch: 1 Global Step: 17690 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:22:43,685-Speed 127.95 samples/sec Loss 12.9973 Epoch: 1 Global Step: 17695 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 15:23:03,447-Speed 129.55 samples/sec Loss 12.9144 Epoch: 1 Global Step: 17700 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:23:23,484-Speed 127.77 samples/sec Loss 12.8916 Epoch: 1 Global Step: 17705 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:23:43,122-Speed 130.37 samples/sec Loss 13.1735 Epoch: 1 Global Step: 17710 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:24:03,577-Speed 125.16 samples/sec Loss 13.2699 Epoch: 1 Global Step: 17715 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:24:23,878-Speed 126.11 samples/sec Loss 13.1062 Epoch: 1 Global Step: 17720 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:24:43,879-Speed 128.00 samples/sec Loss 12.9962 Epoch: 1 Global Step: 17725 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:25:03,898-Speed 127.89 samples/sec Loss 13.1734 Epoch: 1 Global Step: 17730 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:25:25,052-Speed 121.02 samples/sec Loss 13.2963 Epoch: 1 Global Step: 17735 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:25:45,189-Speed 127.13 samples/sec Loss 13.1086 Epoch: 1 Global Step: 17740 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:26:05,449-Speed 126.37 samples/sec Loss 13.1579 Epoch: 1 Global Step: 17745 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:26:25,559-Speed 127.30 samples/sec Loss 13.0709 Epoch: 1 Global Step: 17750 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:26:45,851-Speed 126.17 samples/sec Loss 13.3126 Epoch: 1 Global Step: 17755 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:27:05,781-Speed 128.46 samples/sec Loss 12.9726 Epoch: 1 Global Step: 17760 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:27:25,792-Speed 127.94 samples/sec Loss 12.7643 Epoch: 1 Global Step: 17765 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:27:45,857-Speed 127.59 samples/sec Loss 13.0379 Epoch: 1 Global Step: 17770 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:28:05,678-Speed 129.17 samples/sec Loss 12.8475 Epoch: 1 Global Step: 17775 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:28:26,292-Speed 124.19 samples/sec Loss 12.9962 Epoch: 1 Global Step: 17780 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:28:46,217-Speed 128.49 samples/sec Loss 13.0410 Epoch: 1 Global Step: 17785 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:29:06,498-Speed 126.24 samples/sec Loss 13.2046 Epoch: 1 Global Step: 17790 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:29:26,422-Speed 128.49 samples/sec Loss 13.0005 Epoch: 1 Global Step: 17795 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:29:46,578-Speed 127.02 samples/sec Loss 12.8522 Epoch: 1 Global Step: 17800 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:30:06,582-Speed 127.98 samples/sec Loss 12.9194 Epoch: 1 Global Step: 17805 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:30:26,580-Speed 128.02 samples/sec Loss 12.9553 Epoch: 1 Global Step: 17810 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:30:46,795-Speed 126.65 samples/sec Loss 13.1672 Epoch: 1 Global Step: 17815 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:31:06,923-Speed 127.19 samples/sec Loss 13.1834 Epoch: 1 Global Step: 17820 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:31:27,273-Speed 125.81 samples/sec Loss 13.2057 Epoch: 1 Global Step: 17825 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:31:47,296-Speed 127.85 samples/sec Loss 12.9903 Epoch: 1 Global Step: 17830 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:32:07,216-Speed 128.52 samples/sec Loss 12.8541 Epoch: 1 Global Step: 17835 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:32:27,393-Speed 126.89 samples/sec Loss 13.0234 Epoch: 1 Global Step: 17840 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:32:47,317-Speed 128.50 samples/sec Loss 13.0961 Epoch: 1 Global Step: 17845 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:33:07,424-Speed 127.32 samples/sec Loss 13.1493 Epoch: 1 Global Step: 17850 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:33:27,356-Speed 128.44 samples/sec Loss 13.0648 Epoch: 1 Global Step: 17855 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:33:47,548-Speed 126.79 samples/sec Loss 13.4828 Epoch: 1 Global Step: 17860 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:34:07,858-Speed 126.05 samples/sec Loss 12.9365 Epoch: 1 Global Step: 17865 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:34:28,793-Speed 122.29 samples/sec Loss 13.5425 Epoch: 1 Global Step: 17870 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:34:48,628-Speed 129.07 samples/sec Loss 12.7974 Epoch: 1 Global Step: 17875 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:35:08,352-Speed 129.80 samples/sec Loss 12.9600 Epoch: 1 Global Step: 17880 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:35:28,443-Speed 127.43 samples/sec Loss 12.6081 Epoch: 1 Global Step: 17885 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:35:48,420-Speed 128.16 samples/sec Loss 12.9521 Epoch: 1 Global Step: 17890 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:36:08,551-Speed 127.17 samples/sec Loss 12.9439 Epoch: 1 Global Step: 17895 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:36:28,896-Speed 125.84 samples/sec Loss 13.0792 Epoch: 1 Global Step: 17900 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:36:49,144-Speed 126.44 samples/sec Loss 12.7945 Epoch: 1 Global Step: 17905 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:37:09,254-Speed 127.31 samples/sec Loss 13.0092 Epoch: 1 Global Step: 17910 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:37:29,980-Speed 123.52 samples/sec Loss 12.7583 Epoch: 1 Global Step: 17915 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:37:49,852-Speed 128.83 samples/sec Loss 12.8970 Epoch: 1 Global Step: 17920 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:38:10,039-Speed 126.82 samples/sec Loss 12.8968 Epoch: 1 Global Step: 17925 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:38:30,321-Speed 126.23 samples/sec Loss 13.0238 Epoch: 1 Global Step: 17930 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:38:50,422-Speed 127.37 samples/sec Loss 12.9838 Epoch: 1 Global Step: 17935 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:39:10,872-Speed 125.19 samples/sec Loss 12.7894 Epoch: 1 Global Step: 17940 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:39:30,835-Speed 128.25 samples/sec Loss 12.8687 Epoch: 1 Global Step: 17945 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:39:51,068-Speed 126.53 samples/sec Loss 13.1323 Epoch: 1 Global Step: 17950 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:40:10,971-Speed 128.64 samples/sec Loss 12.8617 Epoch: 1 Global Step: 17955 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:40:31,734-Speed 123.31 samples/sec Loss 12.8477 Epoch: 1 Global Step: 17960 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:40:51,789-Speed 127.65 samples/sec Loss 12.9074 Epoch: 1 Global Step: 17965 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:41:11,873-Speed 127.47 samples/sec Loss 13.0632 Epoch: 1 Global Step: 17970 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:41:31,650-Speed 129.45 samples/sec Loss 12.7134 Epoch: 1 Global Step: 17975 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:41:51,602-Speed 128.31 samples/sec Loss 12.9841 Epoch: 1 Global Step: 17980 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:42:11,731-Speed 127.19 samples/sec Loss 13.4248 Epoch: 1 Global Step: 17985 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:42:31,764-Speed 127.80 samples/sec Loss 12.8545 Epoch: 1 Global Step: 17990 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:42:51,485-Speed 129.82 samples/sec Loss 12.5105 Epoch: 1 Global Step: 17995 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:43:11,468-Speed 128.11 samples/sec Loss 12.9189 Epoch: 1 Global Step: 18000 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:43:31,967-Speed 124.89 samples/sec Loss 12.9367 Epoch: 1 Global Step: 18005 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:43:52,293-Speed 125.96 samples/sec Loss 13.0615 Epoch: 1 Global Step: 18010 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:44:12,107-Speed 129.20 samples/sec Loss 12.9690 Epoch: 1 Global Step: 18015 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:44:32,308-Speed 126.74 samples/sec Loss 12.9313 Epoch: 1 Global Step: 18020 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:44:52,611-Speed 126.09 samples/sec Loss 12.9961 Epoch: 1 Global Step: 18025 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:45:12,949-Speed 125.88 samples/sec Loss 13.1447 Epoch: 1 Global Step: 18030 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:45:32,956-Speed 127.97 samples/sec Loss 12.4418 Epoch: 1 Global Step: 18035 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:45:53,016-Speed 127.63 samples/sec Loss 12.8239 Epoch: 1 Global Step: 18040 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:46:13,306-Speed 126.18 samples/sec Loss 13.0061 Epoch: 1 Global Step: 18045 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:46:33,841-Speed 124.68 samples/sec Loss 12.6542 Epoch: 1 Global Step: 18050 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:46:53,914-Speed 127.53 samples/sec Loss 13.2857 Epoch: 1 Global Step: 18055 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:47:14,356-Speed 125.24 samples/sec Loss 13.0975 Epoch: 1 Global Step: 18060 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:47:34,241-Speed 128.75 samples/sec Loss 12.8143 Epoch: 1 Global Step: 18065 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:47:54,343-Speed 127.36 samples/sec Loss 12.8999 Epoch: 1 Global Step: 18070 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:48:14,603-Speed 126.36 samples/sec Loss 12.9635 Epoch: 1 Global Step: 18075 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:48:34,534-Speed 128.45 samples/sec Loss 13.1274 Epoch: 1 Global Step: 18080 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:48:54,634-Speed 127.37 samples/sec Loss 12.7716 Epoch: 1 Global Step: 18085 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:49:14,552-Speed 128.54 samples/sec Loss 12.9267 Epoch: 1 Global Step: 18090 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:49:35,361-Speed 123.03 samples/sec Loss 12.9340 Epoch: 1 Global Step: 18095 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:49:55,473-Speed 127.29 samples/sec Loss 12.7444 Epoch: 1 Global Step: 18100 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:50:15,646-Speed 126.91 samples/sec Loss 13.1835 Epoch: 1 Global Step: 18105 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:50:35,900-Speed 126.40 samples/sec Loss 12.8961 Epoch: 1 Global Step: 18110 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:50:55,776-Speed 128.81 samples/sec Loss 12.8230 Epoch: 1 Global Step: 18115 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:51:16,009-Speed 126.53 samples/sec Loss 12.9597 Epoch: 1 Global Step: 18120 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:51:36,226-Speed 126.63 samples/sec Loss 13.0978 Epoch: 1 Global Step: 18125 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:51:56,214-Speed 128.09 samples/sec Loss 13.1329 Epoch: 1 Global Step: 18130 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:52:16,373-Speed 127.00 samples/sec Loss 13.2341 Epoch: 1 Global Step: 18135 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:52:36,847-Speed 125.04 samples/sec Loss 12.8469 Epoch: 1 Global Step: 18140 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:52:57,121-Speed 126.28 samples/sec Loss 12.9946 Epoch: 1 Global Step: 18145 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:53:17,235-Speed 127.28 samples/sec Loss 12.8966 Epoch: 1 Global Step: 18150 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:53:37,146-Speed 128.58 samples/sec Loss 12.7489 Epoch: 1 Global Step: 18155 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:53:57,241-Speed 127.40 samples/sec Loss 13.0823 Epoch: 1 Global Step: 18160 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:54:17,331-Speed 127.44 samples/sec Loss 13.0699 Epoch: 1 Global Step: 18165 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:54:37,468-Speed 127.14 samples/sec Loss 12.7870 Epoch: 1 Global Step: 18170 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:54:57,781-Speed 126.04 samples/sec Loss 12.9965 Epoch: 1 Global Step: 18175 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:55:17,804-Speed 127.86 samples/sec Loss 12.8107 Epoch: 1 Global Step: 18180 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:55:38,374-Speed 124.46 samples/sec Loss 12.8625 Epoch: 1 Global Step: 18185 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:55:58,434-Speed 127.62 samples/sec Loss 12.7744 Epoch: 1 Global Step: 18190 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:56:18,335-Speed 128.64 samples/sec Loss 12.8711 Epoch: 1 Global Step: 18195 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:56:38,252-Speed 128.54 samples/sec Loss 12.9135 Epoch: 1 Global Step: 18200 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:56:58,359-Speed 127.32 samples/sec Loss 13.0561 Epoch: 1 Global Step: 18205 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:57:18,357-Speed 128.02 samples/sec Loss 13.0158 Epoch: 1 Global Step: 18210 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:57:38,309-Speed 128.32 samples/sec Loss 12.7472 Epoch: 1 Global Step: 18215 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:57:58,399-Speed 127.43 samples/sec Loss 13.1574 Epoch: 1 Global Step: 18220 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:58:18,333-Speed 128.43 samples/sec Loss 12.8068 Epoch: 1 Global Step: 18225 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:58:38,840-Speed 124.84 samples/sec Loss 12.9965 Epoch: 1 Global Step: 18230 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:58:58,910-Speed 127.56 samples/sec Loss 13.1570 Epoch: 1 Global Step: 18235 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:59:19,105-Speed 126.77 samples/sec Loss 12.8091 Epoch: 1 Global Step: 18240 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:59:39,123-Speed 127.89 samples/sec Loss 12.9633 Epoch: 1 Global Step: 18245 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 15:59:58,954-Speed 129.10 samples/sec Loss 13.0990 Epoch: 1 Global Step: 18250 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:00:18,937-Speed 128.12 samples/sec Loss 13.1936 Epoch: 1 Global Step: 18255 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:00:38,889-Speed 128.31 samples/sec Loss 12.9191 Epoch: 1 Global Step: 18260 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:00:58,922-Speed 127.80 samples/sec Loss 12.9218 Epoch: 1 Global Step: 18265 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:01:18,877-Speed 128.29 samples/sec Loss 12.8054 Epoch: 1 Global Step: 18270 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:01:39,425-Speed 124.59 samples/sec Loss 13.0521 Epoch: 1 Global Step: 18275 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:01:59,683-Speed 126.37 samples/sec Loss 13.1086 Epoch: 1 Global Step: 18280 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:02:19,829-Speed 127.08 samples/sec Loss 13.2668 Epoch: 1 Global Step: 18285 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:02:40,018-Speed 126.81 samples/sec Loss 12.9968 Epoch: 1 Global Step: 18290 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:03:00,174-Speed 127.01 samples/sec Loss 13.2625 Epoch: 1 Global Step: 18295 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:03:20,055-Speed 128.78 samples/sec Loss 12.8694 Epoch: 1 Global Step: 18300 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:03:39,873-Speed 129.18 samples/sec Loss 12.6340 Epoch: 1 Global Step: 18305 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:04:00,206-Speed 125.91 samples/sec Loss 12.9690 Epoch: 1 Global Step: 18310 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:04:20,227-Speed 127.87 samples/sec Loss 13.1162 Epoch: 1 Global Step: 18315 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:04:41,267-Speed 121.68 samples/sec Loss 13.2638 Epoch: 1 Global Step: 18320 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:05:01,166-Speed 128.66 samples/sec Loss 12.8223 Epoch: 1 Global Step: 18325 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:05:20,959-Speed 129.34 samples/sec Loss 12.9808 Epoch: 1 Global Step: 18330 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:05:41,086-Speed 127.20 samples/sec Loss 13.1231 Epoch: 1 Global Step: 18335 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:06:01,033-Speed 128.35 samples/sec Loss 12.8401 Epoch: 1 Global Step: 18340 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:06:21,118-Speed 127.47 samples/sec Loss 13.2604 Epoch: 1 Global Step: 18345 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:06:41,118-Speed 128.01 samples/sec Loss 12.8762 Epoch: 1 Global Step: 18350 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:07:01,189-Speed 127.55 samples/sec Loss 13.2499 Epoch: 1 Global Step: 18355 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 16:07:21,410-Speed 126.61 samples/sec Loss 13.0444 Epoch: 1 Global Step: 18360 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:07:42,142-Speed 123.49 samples/sec Loss 13.0518 Epoch: 1 Global Step: 18365 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:08:01,992-Speed 128.97 samples/sec Loss 12.7038 Epoch: 1 Global Step: 18370 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:08:21,807-Speed 129.20 samples/sec Loss 13.0101 Epoch: 1 Global Step: 18375 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:08:41,476-Speed 130.17 samples/sec Loss 12.7883 Epoch: 1 Global Step: 18380 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:09:01,943-Speed 125.08 samples/sec Loss 12.9619 Epoch: 1 Global Step: 18385 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:09:21,776-Speed 129.09 samples/sec Loss 12.8176 Epoch: 1 Global Step: 18390 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:09:41,710-Speed 128.43 samples/sec Loss 13.1230 Epoch: 1 Global Step: 18395 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:10:01,840-Speed 127.18 samples/sec Loss 12.8721 Epoch: 1 Global Step: 18400 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:10:21,760-Speed 128.52 samples/sec Loss 12.9592 Epoch: 1 Global Step: 18405 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:10:42,614-Speed 122.77 samples/sec Loss 13.0210 Epoch: 1 Global Step: 18410 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:11:02,805-Speed 126.80 samples/sec Loss 12.8248 Epoch: 1 Global Step: 18415 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:11:22,685-Speed 128.78 samples/sec Loss 12.8174 Epoch: 1 Global Step: 18420 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:11:42,879-Speed 126.78 samples/sec Loss 13.0187 Epoch: 1 Global Step: 18425 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:12:02,662-Speed 129.41 samples/sec Loss 12.7243 Epoch: 1 Global Step: 18430 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:12:22,641-Speed 128.14 samples/sec Loss 12.6318 Epoch: 1 Global Step: 18435 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:12:42,905-Speed 126.34 samples/sec Loss 12.9813 Epoch: 1 Global Step: 18440 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:13:02,809-Speed 128.62 samples/sec Loss 12.7874 Epoch: 1 Global Step: 18445 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:13:23,013-Speed 126.71 samples/sec Loss 12.8343 Epoch: 1 Global Step: 18450 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:13:43,584-Speed 124.46 samples/sec Loss 12.7284 Epoch: 1 Global Step: 18455 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:14:03,489-Speed 128.62 samples/sec Loss 12.5213 Epoch: 1 Global Step: 18460 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:14:23,386-Speed 128.66 samples/sec Loss 13.0235 Epoch: 1 Global Step: 18465 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:14:43,580-Speed 126.78 samples/sec Loss 13.1262 Epoch: 1 Global Step: 18470 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:15:03,926-Speed 125.83 samples/sec Loss 12.5292 Epoch: 1 Global Step: 18475 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:15:23,993-Speed 127.58 samples/sec Loss 12.9759 Epoch: 1 Global Step: 18480 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:15:44,103-Speed 127.31 samples/sec Loss 12.8797 Epoch: 1 Global Step: 18485 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:16:04,429-Speed 125.95 samples/sec Loss 13.0090 Epoch: 1 Global Step: 18490 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:16:24,662-Speed 126.53 samples/sec Loss 13.1672 Epoch: 1 Global Step: 18495 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:16:45,522-Speed 122.73 samples/sec Loss 12.7489 Epoch: 1 Global Step: 18500 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:17:05,851-Speed 125.94 samples/sec Loss 12.9673 Epoch: 1 Global Step: 18505 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:17:26,169-Speed 126.00 samples/sec Loss 13.1433 Epoch: 1 Global Step: 18510 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:17:46,338-Speed 126.94 samples/sec Loss 12.8004 Epoch: 1 Global Step: 18515 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:18:06,660-Speed 125.98 samples/sec Loss 12.8663 Epoch: 1 Global Step: 18520 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:18:26,753-Speed 127.42 samples/sec Loss 12.7141 Epoch: 1 Global Step: 18525 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:18:46,565-Speed 129.23 samples/sec Loss 12.7225 Epoch: 1 Global Step: 18530 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:19:06,673-Speed 127.32 samples/sec Loss 12.5522 Epoch: 1 Global Step: 18535 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:19:26,649-Speed 128.16 samples/sec Loss 12.6202 Epoch: 1 Global Step: 18540 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:19:47,434-Speed 123.17 samples/sec Loss 12.8761 Epoch: 1 Global Step: 18545 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:20:07,472-Speed 127.76 samples/sec Loss 12.6619 Epoch: 1 Global Step: 18550 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:20:27,393-Speed 128.51 samples/sec Loss 12.9004 Epoch: 1 Global Step: 18555 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:20:47,768-Speed 125.65 samples/sec Loss 12.7088 Epoch: 1 Global Step: 18560 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:21:07,956-Speed 126.82 samples/sec Loss 12.6417 Epoch: 1 Global Step: 18565 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:21:28,035-Speed 127.50 samples/sec Loss 12.6182 Epoch: 1 Global Step: 18570 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:21:47,885-Speed 128.98 samples/sec Loss 12.7839 Epoch: 1 Global Step: 18575 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:22:07,705-Speed 129.17 samples/sec Loss 12.7048 Epoch: 1 Global Step: 18580 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:22:27,698-Speed 128.05 samples/sec Loss 12.9975 Epoch: 1 Global Step: 18585 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:22:48,552-Speed 122.76 samples/sec Loss 12.9430 Epoch: 1 Global Step: 18590 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:23:08,548-Speed 128.03 samples/sec Loss 13.1833 Epoch: 1 Global Step: 18595 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:23:28,423-Speed 128.81 samples/sec Loss 12.8271 Epoch: 1 Global Step: 18600 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:23:48,582-Speed 127.00 samples/sec Loss 12.4931 Epoch: 1 Global Step: 18605 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:24:08,560-Speed 128.15 samples/sec Loss 12.8742 Epoch: 1 Global Step: 18610 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:24:28,514-Speed 128.30 samples/sec Loss 12.7508 Epoch: 1 Global Step: 18615 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:24:48,683-Speed 126.94 samples/sec Loss 12.7582 Epoch: 1 Global Step: 18620 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:25:08,436-Speed 129.61 samples/sec Loss 12.6256 Epoch: 1 Global Step: 18625 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:25:28,440-Speed 127.98 samples/sec Loss 12.7174 Epoch: 1 Global Step: 18630 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:25:49,233-Speed 123.13 samples/sec Loss 12.8112 Epoch: 1 Global Step: 18635 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:26:09,246-Speed 127.92 samples/sec Loss 12.7873 Epoch: 1 Global Step: 18640 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:26:29,100-Speed 128.95 samples/sec Loss 12.8426 Epoch: 1 Global Step: 18645 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:26:49,256-Speed 127.02 samples/sec Loss 13.1700 Epoch: 1 Global Step: 18650 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:27:09,176-Speed 128.52 samples/sec Loss 12.8111 Epoch: 1 Global Step: 18655 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:27:29,353-Speed 126.89 samples/sec Loss 13.1043 Epoch: 1 Global Step: 18660 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:27:49,266-Speed 128.57 samples/sec Loss 12.8508 Epoch: 1 Global Step: 18665 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:28:09,638-Speed 125.67 samples/sec Loss 13.0515 Epoch: 1 Global Step: 18670 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:28:29,766-Speed 127.19 samples/sec Loss 12.9629 Epoch: 1 Global Step: 18675 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:28:50,092-Speed 125.96 samples/sec Loss 12.8624 Epoch: 1 Global Step: 18680 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:29:10,137-Speed 127.72 samples/sec Loss 12.8223 Epoch: 1 Global Step: 18685 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:29:30,255-Speed 127.26 samples/sec Loss 12.8454 Epoch: 1 Global Step: 18690 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:29:50,043-Speed 129.38 samples/sec Loss 12.5677 Epoch: 1 Global Step: 18695 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:30:10,162-Speed 127.25 samples/sec Loss 12.8152 Epoch: 1 Global Step: 18700 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:30:30,363-Speed 126.73 samples/sec Loss 12.9179 Epoch: 1 Global Step: 18705 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:30:50,519-Speed 127.02 samples/sec Loss 13.1450 Epoch: 1 Global Step: 18710 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:31:10,801-Speed 126.22 samples/sec Loss 12.9106 Epoch: 1 Global Step: 18715 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:31:31,121-Speed 126.00 samples/sec Loss 13.0013 Epoch: 1 Global Step: 18720 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:31:52,036-Speed 122.41 samples/sec Loss 12.8317 Epoch: 1 Global Step: 18725 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:32:12,135-Speed 127.38 samples/sec Loss 12.8095 Epoch: 1 Global Step: 18730 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:32:32,199-Speed 127.59 samples/sec Loss 12.5471 Epoch: 1 Global Step: 18735 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:32:52,446-Speed 126.45 samples/sec Loss 13.0104 Epoch: 1 Global Step: 18740 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:33:12,450-Speed 127.98 samples/sec Loss 12.7158 Epoch: 1 Global Step: 18745 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:33:32,773-Speed 125.97 samples/sec Loss 13.1963 Epoch: 1 Global Step: 18750 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:33:52,980-Speed 126.70 samples/sec Loss 12.8627 Epoch: 1 Global Step: 18755 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:34:12,958-Speed 128.14 samples/sec Loss 12.9010 Epoch: 1 Global Step: 18760 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:34:33,067-Speed 127.32 samples/sec Loss 12.8391 Epoch: 1 Global Step: 18765 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:34:53,802-Speed 123.47 samples/sec Loss 12.5617 Epoch: 1 Global Step: 18770 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:35:13,791-Speed 128.08 samples/sec Loss 12.6768 Epoch: 1 Global Step: 18775 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:35:33,944-Speed 127.03 samples/sec Loss 12.8979 Epoch: 1 Global Step: 18780 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:35:54,343-Speed 125.51 samples/sec Loss 13.1922 Epoch: 1 Global Step: 18785 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:36:14,416-Speed 127.54 samples/sec Loss 12.9586 Epoch: 1 Global Step: 18790 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:36:34,444-Speed 127.83 samples/sec Loss 12.7004 Epoch: 1 Global Step: 18795 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:36:54,341-Speed 128.67 samples/sec Loss 12.7708 Epoch: 1 Global Step: 18800 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:37:14,488-Speed 127.07 samples/sec Loss 12.8106 Epoch: 1 Global Step: 18805 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:37:34,524-Speed 127.77 samples/sec Loss 12.6986 Epoch: 1 Global Step: 18810 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:37:55,429-Speed 122.47 samples/sec Loss 12.5685 Epoch: 1 Global Step: 18815 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:38:15,630-Speed 126.73 samples/sec Loss 13.0961 Epoch: 1 Global Step: 18820 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:38:35,872-Speed 126.47 samples/sec Loss 13.0868 Epoch: 1 Global Step: 18825 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:38:55,765-Speed 128.70 samples/sec Loss 12.8007 Epoch: 1 Global Step: 18830 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:39:15,654-Speed 128.72 samples/sec Loss 12.5974 Epoch: 1 Global Step: 18835 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:39:36,001-Speed 125.82 samples/sec Loss 12.7131 Epoch: 1 Global Step: 18840 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:39:56,168-Speed 126.95 samples/sec Loss 12.7632 Epoch: 1 Global Step: 18845 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:40:16,116-Speed 128.35 samples/sec Loss 12.5920 Epoch: 1 Global Step: 18850 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:40:36,377-Speed 126.36 samples/sec Loss 12.7164 Epoch: 1 Global Step: 18855 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:40:56,826-Speed 125.20 samples/sec Loss 12.5441 Epoch: 1 Global Step: 18860 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:41:17,258-Speed 125.30 samples/sec Loss 13.3197 Epoch: 1 Global Step: 18865 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:41:37,399-Speed 127.11 samples/sec Loss 12.6223 Epoch: 1 Global Step: 18870 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:41:57,682-Speed 126.22 samples/sec Loss 12.9799 Epoch: 1 Global Step: 18875 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:42:17,601-Speed 128.53 samples/sec Loss 12.8676 Epoch: 1 Global Step: 18880 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:42:37,732-Speed 127.17 samples/sec Loss 12.9909 Epoch: 1 Global Step: 18885 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:42:57,738-Speed 127.97 samples/sec Loss 12.5330 Epoch: 1 Global Step: 18890 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:43:17,826-Speed 127.45 samples/sec Loss 12.7187 Epoch: 1 Global Step: 18895 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:43:37,883-Speed 127.64 samples/sec Loss 12.9074 Epoch: 1 Global Step: 18900 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:43:58,532-Speed 123.98 samples/sec Loss 12.5904 Epoch: 1 Global Step: 18905 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:44:18,528-Speed 128.03 samples/sec Loss 12.6150 Epoch: 1 Global Step: 18910 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:44:38,220-Speed 130.00 samples/sec Loss 13.2999 Epoch: 1 Global Step: 18915 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:44:58,624-Speed 125.47 samples/sec Loss 13.0346 Epoch: 1 Global Step: 18920 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:45:18,863-Speed 126.50 samples/sec Loss 13.0024 Epoch: 1 Global Step: 18925 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:45:38,857-Speed 128.04 samples/sec Loss 12.8980 Epoch: 1 Global Step: 18930 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:45:59,049-Speed 126.79 samples/sec Loss 12.8984 Epoch: 1 Global Step: 18935 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:46:19,163-Speed 127.28 samples/sec Loss 12.8720 Epoch: 1 Global Step: 18940 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:46:38,850-Speed 130.05 samples/sec Loss 12.5563 Epoch: 1 Global Step: 18945 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:46:59,365-Speed 124.79 samples/sec Loss 12.5123 Epoch: 1 Global Step: 18950 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:47:19,454-Speed 127.44 samples/sec Loss 13.0817 Epoch: 1 Global Step: 18955 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:47:39,431-Speed 128.15 samples/sec Loss 12.6461 Epoch: 1 Global Step: 18960 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:47:59,787-Speed 125.77 samples/sec Loss 13.0750 Epoch: 1 Global Step: 18965 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:48:19,696-Speed 128.59 samples/sec Loss 12.9000 Epoch: 1 Global Step: 18970 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:48:39,811-Speed 127.27 samples/sec Loss 12.8706 Epoch: 1 Global Step: 18975 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:48:59,907-Speed 127.40 samples/sec Loss 12.7131 Epoch: 1 Global Step: 18980 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:49:19,844-Speed 128.41 samples/sec Loss 12.5870 Epoch: 1 Global Step: 18985 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:49:40,220-Speed 125.65 samples/sec Loss 12.9898 Epoch: 1 Global Step: 18990 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:50:01,027-Speed 123.04 samples/sec Loss 12.7927 Epoch: 1 Global Step: 18995 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:50:20,963-Speed 128.41 samples/sec Loss 12.4705 Epoch: 1 Global Step: 19000 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:50:41,057-Speed 127.41 samples/sec Loss 12.8082 Epoch: 1 Global Step: 19005 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:51:00,972-Speed 128.55 samples/sec Loss 12.6494 Epoch: 1 Global Step: 19010 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:51:21,401-Speed 125.32 samples/sec Loss 13.2164 Epoch: 1 Global Step: 19015 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:51:41,564-Speed 126.97 samples/sec Loss 13.2130 Epoch: 1 Global Step: 19020 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:52:01,459-Speed 128.68 samples/sec Loss 12.8444 Epoch: 1 Global Step: 19025 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:52:21,572-Speed 127.29 samples/sec Loss 12.7867 Epoch: 1 Global Step: 19030 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:52:41,725-Speed 127.04 samples/sec Loss 12.8789 Epoch: 1 Global Step: 19035 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 16:53:02,414-Speed 123.75 samples/sec Loss 13.0137 Epoch: 1 Global Step: 19040 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:53:22,736-Speed 125.97 samples/sec Loss 12.9958 Epoch: 1 Global Step: 19045 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:53:42,981-Speed 126.46 samples/sec Loss 13.1402 Epoch: 1 Global Step: 19050 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:54:03,013-Speed 127.80 samples/sec Loss 12.8338 Epoch: 1 Global Step: 19055 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:54:23,446-Speed 125.30 samples/sec Loss 12.7807 Epoch: 1 Global Step: 19060 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:54:43,830-Speed 125.60 samples/sec Loss 12.7216 Epoch: 1 Global Step: 19065 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:55:04,116-Speed 126.20 samples/sec Loss 12.6954 Epoch: 1 Global Step: 19070 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:55:24,126-Speed 127.94 samples/sec Loss 12.9490 Epoch: 1 Global Step: 19075 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:55:44,136-Speed 127.95 samples/sec Loss 13.1776 Epoch: 1 Global Step: 19080 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:56:04,878-Speed 123.43 samples/sec Loss 12.8342 Epoch: 1 Global Step: 19085 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:56:24,890-Speed 127.93 samples/sec Loss 12.6876 Epoch: 1 Global Step: 19090 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:56:45,024-Speed 127.15 samples/sec Loss 12.7716 Epoch: 1 Global Step: 19095 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:57:05,452-Speed 125.33 samples/sec Loss 12.9345 Epoch: 1 Global Step: 19100 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:57:25,249-Speed 129.32 samples/sec Loss 12.6699 Epoch: 1 Global Step: 19105 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:57:45,339-Speed 127.43 samples/sec Loss 12.6225 Epoch: 1 Global Step: 19110 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:58:05,466-Speed 127.20 samples/sec Loss 12.5801 Epoch: 1 Global Step: 19115 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:58:25,689-Speed 126.60 samples/sec Loss 12.9362 Epoch: 1 Global Step: 19120 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:58:45,708-Speed 127.88 samples/sec Loss 12.9600 Epoch: 1 Global Step: 19125 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:59:05,981-Speed 126.28 samples/sec Loss 12.4309 Epoch: 1 Global Step: 19130 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:59:26,200-Speed 126.62 samples/sec Loss 12.8822 Epoch: 1 Global Step: 19135 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 16:59:46,128-Speed 128.46 samples/sec Loss 12.8301 Epoch: 1 Global Step: 19140 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:00:05,999-Speed 128.84 samples/sec Loss 12.7048 Epoch: 1 Global Step: 19145 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:00:26,178-Speed 126.87 samples/sec Loss 12.9846 Epoch: 1 Global Step: 19150 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:00:46,285-Speed 127.32 samples/sec Loss 12.8061 Epoch: 1 Global Step: 19155 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:01:06,159-Speed 128.82 samples/sec Loss 12.8960 Epoch: 1 Global Step: 19160 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:01:26,472-Speed 126.03 samples/sec Loss 12.6885 Epoch: 1 Global Step: 19165 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:01:47,507-Speed 121.71 samples/sec Loss 12.7621 Epoch: 1 Global Step: 19170 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:02:08,285-Speed 123.22 samples/sec Loss 13.2405 Epoch: 1 Global Step: 19175 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:02:28,158-Speed 128.82 samples/sec Loss 12.6382 Epoch: 1 Global Step: 19180 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:02:48,048-Speed 128.71 samples/sec Loss 12.8568 Epoch: 1 Global Step: 19185 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:03:08,054-Speed 127.97 samples/sec Loss 12.6700 Epoch: 1 Global Step: 19190 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:03:28,100-Speed 127.71 samples/sec Loss 12.7413 Epoch: 1 Global Step: 19195 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:03:49,354-Speed 120.45 samples/sec Loss 12.9342 Epoch: 1 Global Step: 19200 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:04:09,769-Speed 125.41 samples/sec Loss 13.0325 Epoch: 1 Global Step: 19205 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:04:30,221-Speed 125.18 samples/sec Loss 12.7586 Epoch: 1 Global Step: 19210 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:04:50,372-Speed 127.05 samples/sec Loss 12.9401 Epoch: 1 Global Step: 19215 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:05:11,214-Speed 122.84 samples/sec Loss 12.9347 Epoch: 1 Global Step: 19220 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:05:31,581-Speed 125.70 samples/sec Loss 12.7071 Epoch: 1 Global Step: 19225 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:05:51,920-Speed 125.87 samples/sec Loss 12.8719 Epoch: 1 Global Step: 19230 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:06:12,010-Speed 127.43 samples/sec Loss 12.7136 Epoch: 1 Global Step: 19235 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:06:31,978-Speed 128.21 samples/sec Loss 12.9634 Epoch: 1 Global Step: 19240 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:06:52,097-Speed 127.25 samples/sec Loss 12.6300 Epoch: 1 Global Step: 19245 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:07:12,039-Speed 128.38 samples/sec Loss 12.5070 Epoch: 1 Global Step: 19250 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:07:32,139-Speed 127.38 samples/sec Loss 12.6654 Epoch: 1 Global Step: 19255 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:07:52,028-Speed 128.72 samples/sec Loss 12.9400 Epoch: 1 Global Step: 19260 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:08:12,299-Speed 126.30 samples/sec Loss 12.6060 Epoch: 1 Global Step: 19265 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:08:32,322-Speed 127.86 samples/sec Loss 12.5984 Epoch: 1 Global Step: 19270 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:08:52,056-Speed 129.73 samples/sec Loss 12.8026 Epoch: 1 Global Step: 19275 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:09:12,271-Speed 126.65 samples/sec Loss 12.8169 Epoch: 1 Global Step: 19280 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:09:32,180-Speed 128.59 samples/sec Loss 12.8243 Epoch: 1 Global Step: 19285 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:09:52,608-Speed 125.32 samples/sec Loss 12.8243 Epoch: 1 Global Step: 19290 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:10:12,784-Speed 126.89 samples/sec Loss 12.7059 Epoch: 1 Global Step: 19295 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:10:33,045-Speed 126.36 samples/sec Loss 12.7323 Epoch: 1 Global Step: 19300 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:10:53,033-Speed 128.09 samples/sec Loss 12.8926 Epoch: 1 Global Step: 19305 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:11:13,765-Speed 123.48 samples/sec Loss 12.7504 Epoch: 1 Global Step: 19310 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:11:33,786-Speed 127.87 samples/sec Loss 13.1635 Epoch: 1 Global Step: 19315 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:11:53,859-Speed 127.54 samples/sec Loss 13.0454 Epoch: 1 Global Step: 19320 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:12:13,830-Speed 128.19 samples/sec Loss 12.7708 Epoch: 1 Global Step: 19325 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:12:33,700-Speed 128.84 samples/sec Loss 12.8511 Epoch: 1 Global Step: 19330 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:12:53,817-Speed 127.26 samples/sec Loss 12.7725 Epoch: 1 Global Step: 19335 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:13:13,882-Speed 127.59 samples/sec Loss 12.7586 Epoch: 1 Global Step: 19340 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:13:34,020-Speed 127.13 samples/sec Loss 12.4055 Epoch: 1 Global Step: 19345 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:13:54,343-Speed 125.97 samples/sec Loss 12.9496 Epoch: 1 Global Step: 19350 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:14:15,016-Speed 123.84 samples/sec Loss 12.9887 Epoch: 1 Global Step: 19355 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:14:35,399-Speed 125.60 samples/sec Loss 12.5801 Epoch: 1 Global Step: 19360 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:14:55,791-Speed 125.55 samples/sec Loss 12.7586 Epoch: 1 Global Step: 19365 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:15:15,913-Speed 127.23 samples/sec Loss 12.7648 Epoch: 1 Global Step: 19370 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:15:36,015-Speed 127.36 samples/sec Loss 12.5598 Epoch: 1 Global Step: 19375 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:15:56,363-Speed 125.82 samples/sec Loss 13.1485 Epoch: 1 Global Step: 19380 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:16:16,472-Speed 127.31 samples/sec Loss 12.6608 Epoch: 1 Global Step: 19385 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:16:36,471-Speed 128.01 samples/sec Loss 12.6349 Epoch: 1 Global Step: 19390 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:16:56,714-Speed 126.47 samples/sec Loss 12.7235 Epoch: 1 Global Step: 19395 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:17:17,455-Speed 123.43 samples/sec Loss 12.8548 Epoch: 1 Global Step: 19400 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:17:37,692-Speed 126.51 samples/sec Loss 13.0035 Epoch: 1 Global Step: 19405 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:17:57,707-Speed 127.91 samples/sec Loss 12.3224 Epoch: 1 Global Step: 19410 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:18:17,962-Speed 126.45 samples/sec Loss 12.5483 Epoch: 1 Global Step: 19415 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:18:37,968-Speed 127.97 samples/sec Loss 12.6983 Epoch: 1 Global Step: 19420 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:18:58,229-Speed 126.36 samples/sec Loss 12.8359 Epoch: 1 Global Step: 19425 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:19:18,233-Speed 127.98 samples/sec Loss 12.6950 Epoch: 1 Global Step: 19430 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:19:38,387-Speed 127.03 samples/sec Loss 12.6326 Epoch: 1 Global Step: 19435 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:19:58,338-Speed 128.32 samples/sec Loss 12.3987 Epoch: 1 Global Step: 19440 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:20:18,941-Speed 124.26 samples/sec Loss 12.7909 Epoch: 1 Global Step: 19445 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:20:38,720-Speed 129.43 samples/sec Loss 12.6495 Epoch: 1 Global Step: 19450 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:20:58,922-Speed 126.72 samples/sec Loss 12.5431 Epoch: 1 Global Step: 19455 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:21:18,908-Speed 128.10 samples/sec Loss 12.7621 Epoch: 1 Global Step: 19460 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:21:39,012-Speed 127.35 samples/sec Loss 12.9539 Epoch: 1 Global Step: 19465 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:21:59,048-Speed 127.77 samples/sec Loss 12.6700 Epoch: 1 Global Step: 19470 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:22:19,067-Speed 127.89 samples/sec Loss 12.8627 Epoch: 1 Global Step: 19475 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:22:39,176-Speed 127.31 samples/sec Loss 12.9751 Epoch: 1 Global Step: 19480 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:22:59,085-Speed 128.59 samples/sec Loss 12.5502 Epoch: 1 Global Step: 19485 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:23:19,724-Speed 124.04 samples/sec Loss 12.4655 Epoch: 1 Global Step: 19490 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:23:39,869-Speed 127.08 samples/sec Loss 12.8653 Epoch: 1 Global Step: 19495 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:24:00,866-Speed 121.93 samples/sec Loss 12.8692 Epoch: 1 Global Step: 19500 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:24:21,060-Speed 126.77 samples/sec Loss 13.1035 Epoch: 1 Global Step: 19505 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:24:41,284-Speed 126.59 samples/sec Loss 13.0215 Epoch: 1 Global Step: 19510 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:25:01,593-Speed 126.06 samples/sec Loss 12.5502 Epoch: 1 Global Step: 19515 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:25:21,797-Speed 126.71 samples/sec Loss 13.3190 Epoch: 1 Global Step: 19520 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:25:42,136-Speed 125.87 samples/sec Loss 13.0454 Epoch: 1 Global Step: 19525 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:26:02,608-Speed 125.06 samples/sec Loss 12.8378 Epoch: 1 Global Step: 19530 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:26:23,531-Speed 122.36 samples/sec Loss 12.7757 Epoch: 1 Global Step: 19535 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:26:43,860-Speed 125.93 samples/sec Loss 12.5225 Epoch: 1 Global Step: 19540 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:27:03,988-Speed 127.19 samples/sec Loss 12.5386 Epoch: 1 Global Step: 19545 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:27:24,025-Speed 127.77 samples/sec Loss 12.6694 Epoch: 1 Global Step: 19550 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:27:44,100-Speed 127.54 samples/sec Loss 12.5764 Epoch: 1 Global Step: 19555 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:28:04,290-Speed 126.80 samples/sec Loss 12.6193 Epoch: 1 Global Step: 19560 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:28:24,320-Speed 127.81 samples/sec Loss 12.6729 Epoch: 1 Global Step: 19565 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:28:44,465-Speed 127.09 samples/sec Loss 12.6063 Epoch: 1 Global Step: 19570 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:29:04,263-Speed 129.31 samples/sec Loss 12.7467 Epoch: 1 Global Step: 19575 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:29:25,288-Speed 121.77 samples/sec Loss 12.3703 Epoch: 1 Global Step: 19580 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:29:45,489-Speed 126.73 samples/sec Loss 12.6307 Epoch: 1 Global Step: 19585 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:30:05,631-Speed 127.10 samples/sec Loss 12.8855 Epoch: 1 Global Step: 19590 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:30:25,882-Speed 126.43 samples/sec Loss 12.7537 Epoch: 1 Global Step: 19595 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:30:45,677-Speed 129.33 samples/sec Loss 12.7034 Epoch: 1 Global Step: 19600 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:31:05,566-Speed 128.72 samples/sec Loss 12.7347 Epoch: 1 Global Step: 19605 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:31:25,694-Speed 127.19 samples/sec Loss 12.7605 Epoch: 1 Global Step: 19610 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:31:45,832-Speed 127.13 samples/sec Loss 12.8539 Epoch: 1 Global Step: 19615 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:32:06,201-Speed 125.69 samples/sec Loss 12.9449 Epoch: 1 Global Step: 19620 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:32:27,145-Speed 122.23 samples/sec Loss 12.7169 Epoch: 1 Global Step: 19625 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:32:47,364-Speed 126.62 samples/sec Loss 12.5505 Epoch: 1 Global Step: 19630 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:33:07,348-Speed 128.11 samples/sec Loss 12.7837 Epoch: 1 Global Step: 19635 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:33:27,452-Speed 127.34 samples/sec Loss 12.6768 Epoch: 1 Global Step: 19640 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:33:47,685-Speed 126.53 samples/sec Loss 12.9494 Epoch: 1 Global Step: 19645 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:34:07,929-Speed 126.47 samples/sec Loss 12.6168 Epoch: 1 Global Step: 19650 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:34:28,074-Speed 127.08 samples/sec Loss 13.0341 Epoch: 1 Global Step: 19655 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:34:48,384-Speed 126.05 samples/sec Loss 12.7436 Epoch: 1 Global Step: 19660 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:35:08,501-Speed 127.26 samples/sec Loss 12.8155 Epoch: 1 Global Step: 19665 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:35:29,372-Speed 122.67 samples/sec Loss 13.1425 Epoch: 1 Global Step: 19670 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:35:49,267-Speed 128.68 samples/sec Loss 12.3691 Epoch: 1 Global Step: 19675 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:36:09,620-Speed 125.79 samples/sec Loss 13.0019 Epoch: 1 Global Step: 19680 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:36:29,328-Speed 129.90 samples/sec Loss 12.4358 Epoch: 1 Global Step: 19685 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:36:49,465-Speed 127.14 samples/sec Loss 12.8346 Epoch: 1 Global Step: 19690 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:37:09,556-Speed 127.43 samples/sec Loss 12.7882 Epoch: 1 Global Step: 19695 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:37:29,901-Speed 125.83 samples/sec Loss 12.6928 Epoch: 1 Global Step: 19700 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:37:49,942-Speed 127.75 samples/sec Loss 12.9360 Epoch: 1 Global Step: 19705 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:38:10,456-Speed 124.79 samples/sec Loss 12.5637 Epoch: 1 Global Step: 19710 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:38:31,546-Speed 121.40 samples/sec Loss 13.0964 Epoch: 1 Global Step: 19715 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:38:51,513-Speed 128.21 samples/sec Loss 12.8233 Epoch: 1 Global Step: 19720 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:39:11,594-Speed 127.50 samples/sec Loss 12.7537 Epoch: 1 Global Step: 19725 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:39:31,862-Speed 126.31 samples/sec Loss 12.6528 Epoch: 1 Global Step: 19730 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:39:51,841-Speed 128.14 samples/sec Loss 12.5614 Epoch: 1 Global Step: 19735 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:40:11,855-Speed 127.91 samples/sec Loss 12.8292 Epoch: 1 Global Step: 19740 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:40:31,784-Speed 128.47 samples/sec Loss 12.8038 Epoch: 1 Global Step: 19745 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-07 17:40:52,081-Speed 126.13 samples/sec Loss 12.7698 Epoch: 1 Global Step: 19750 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:41:12,501-Speed 125.38 samples/sec Loss 13.0588 Epoch: 1 Global Step: 19755 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:41:33,317-Speed 122.99 samples/sec Loss 13.1553 Epoch: 1 Global Step: 19760 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:41:53,729-Speed 125.42 samples/sec Loss 13.0109 Epoch: 1 Global Step: 19765 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:42:13,822-Speed 127.41 samples/sec Loss 12.6810 Epoch: 1 Global Step: 19770 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:42:33,524-Speed 129.95 samples/sec Loss 12.4654 Epoch: 1 Global Step: 19775 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:42:53,494-Speed 128.20 samples/sec Loss 12.7598 Epoch: 1 Global Step: 19780 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:43:13,801-Speed 126.07 samples/sec Loss 12.7698 Epoch: 1 Global Step: 19785 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:43:33,716-Speed 128.56 samples/sec Loss 12.2744 Epoch: 1 Global Step: 19790 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:43:53,788-Speed 127.54 samples/sec Loss 12.7505 Epoch: 1 Global Step: 19795 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:44:14,072-Speed 126.22 samples/sec Loss 12.5642 Epoch: 1 Global Step: 19800 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:44:34,376-Speed 126.09 samples/sec Loss 12.6760 Epoch: 1 Global Step: 19805 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:44:54,548-Speed 126.92 samples/sec Loss 12.8375 Epoch: 1 Global Step: 19810 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:45:14,844-Speed 126.14 samples/sec Loss 13.0613 Epoch: 1 Global Step: 19815 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:45:35,318-Speed 125.04 samples/sec Loss 13.0476 Epoch: 1 Global Step: 19820 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:45:55,478-Speed 126.99 samples/sec Loss 12.9762 Epoch: 1 Global Step: 19825 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:46:15,737-Speed 126.37 samples/sec Loss 13.1906 Epoch: 1 Global Step: 19830 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:46:35,753-Speed 127.91 samples/sec Loss 12.7741 Epoch: 1 Global Step: 19835 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:46:55,769-Speed 127.91 samples/sec Loss 12.8413 Epoch: 1 Global Step: 19840 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:47:15,914-Speed 127.09 samples/sec Loss 12.6982 Epoch: 1 Global Step: 19845 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:47:36,812-Speed 122.51 samples/sec Loss 12.3574 Epoch: 1 Global Step: 19850 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:47:56,803-Speed 128.06 samples/sec Loss 12.5219 Epoch: 1 Global Step: 19855 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:48:16,676-Speed 128.83 samples/sec Loss 12.5252 Epoch: 1 Global Step: 19860 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:48:36,664-Speed 128.08 samples/sec Loss 12.2991 Epoch: 1 Global Step: 19865 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:48:56,700-Speed 127.77 samples/sec Loss 12.8520 Epoch: 1 Global Step: 19870 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:49:16,912-Speed 126.67 samples/sec Loss 12.9988 Epoch: 1 Global Step: 19875 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:49:37,120-Speed 126.69 samples/sec Loss 12.8928 Epoch: 1 Global Step: 19880 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:49:57,261-Speed 127.12 samples/sec Loss 12.5779 Epoch: 1 Global Step: 19885 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:50:17,714-Speed 125.17 samples/sec Loss 12.7287 Epoch: 1 Global Step: 19890 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:50:38,258-Speed 124.62 samples/sec Loss 12.6797 Epoch: 1 Global Step: 19895 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:50:58,210-Speed 128.32 samples/sec Loss 12.7044 Epoch: 1 Global Step: 19900 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:51:18,387-Speed 126.88 samples/sec Loss 12.7799 Epoch: 1 Global Step: 19905 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:51:38,611-Speed 126.59 samples/sec Loss 12.5335 Epoch: 1 Global Step: 19910 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:51:58,625-Speed 127.91 samples/sec Loss 12.6524 Epoch: 1 Global Step: 19915 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:52:18,812-Speed 126.82 samples/sec Loss 12.6937 Epoch: 1 Global Step: 19920 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:52:38,927-Speed 127.28 samples/sec Loss 12.7974 Epoch: 1 Global Step: 19925 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:53:26,658-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-07 17:53:41,807-Speed 40.71 samples/sec Loss 12.3477 Epoch: 2 Global Step: 19930 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:54:01,938-Speed 127.17 samples/sec Loss 12.1796 Epoch: 2 Global Step: 19935 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:54:22,643-Speed 123.65 samples/sec Loss 12.0252 Epoch: 2 Global Step: 19940 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:54:42,906-Speed 126.34 samples/sec Loss 11.8901 Epoch: 2 Global Step: 19945 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:55:03,205-Speed 126.12 samples/sec Loss 11.6509 Epoch: 2 Global Step: 19950 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:55:23,239-Speed 127.79 samples/sec Loss 11.8861 Epoch: 2 Global Step: 19955 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:55:43,516-Speed 126.26 samples/sec Loss 12.3033 Epoch: 2 Global Step: 19960 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:56:03,924-Speed 125.45 samples/sec Loss 12.0482 Epoch: 2 Global Step: 19965 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:56:23,774-Speed 128.97 samples/sec Loss 12.1790 Epoch: 2 Global Step: 19970 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:56:43,966-Speed 126.79 samples/sec Loss 12.3454 Epoch: 2 Global Step: 19975 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:57:03,996-Speed 127.82 samples/sec Loss 12.1517 Epoch: 2 Global Step: 19980 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:57:24,483-Speed 124.97 samples/sec Loss 12.0604 Epoch: 2 Global Step: 19985 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:57:44,584-Speed 127.36 samples/sec Loss 12.0501 Epoch: 2 Global Step: 19990 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:58:05,135-Speed 124.58 samples/sec Loss 12.0954 Epoch: 2 Global Step: 19995 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 17:58:25,533-Speed 125.51 samples/sec Loss 11.9608 Epoch: 2 Global Step: 20000 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-07 18:13:31,007-[lfw][20000]XNorm: 22.756209 Training: 2021-06-07 18:13:31,008-[lfw][20000]Accuracy-Flip: 0.99517+-0.00431 Training: 2021-06-07 18:13:31,008-[lfw][20000]Accuracy-Highest: 0.99517 Training: 2021-06-07 18:31:09,643-[cfp_fp][20000]XNorm: 20.709398 Training: 2021-06-07 18:31:09,644-[cfp_fp][20000]Accuracy-Flip: 0.96886+-0.00795 Training: 2021-06-07 18:31:09,644-[cfp_fp][20000]Accuracy-Highest: 0.96886 Training: 2021-06-07 18:46:17,764-[agedb_30][20000]XNorm: 22.704042 Training: 2021-06-07 18:46:17,765-[agedb_30][20000]Accuracy-Flip: 0.96433+-0.00692 Training: 2021-06-07 18:46:17,765-[agedb_30][20000]Accuracy-Highest: 0.96433 Training: 2021-06-07 18:46:37,985-Speed 0.89 samples/sec Loss 12.0248 Epoch: 2 Global Step: 20005 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:46:58,069-Speed 127.48 samples/sec Loss 11.9007 Epoch: 2 Global Step: 20010 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:47:18,053-Speed 128.11 samples/sec Loss 11.8182 Epoch: 2 Global Step: 20015 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:47:38,038-Speed 128.11 samples/sec Loss 11.9860 Epoch: 2 Global Step: 20020 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:47:58,332-Speed 126.15 samples/sec Loss 12.2176 Epoch: 2 Global Step: 20025 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:48:18,833-Speed 124.88 samples/sec Loss 12.2327 Epoch: 2 Global Step: 20030 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:48:38,860-Speed 127.84 samples/sec Loss 12.1530 Epoch: 2 Global Step: 20035 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:48:59,172-Speed 126.04 samples/sec Loss 12.2642 Epoch: 2 Global Step: 20040 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:49:19,312-Speed 127.12 samples/sec Loss 12.0802 Epoch: 2 Global Step: 20045 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:49:39,219-Speed 128.61 samples/sec Loss 12.1978 Epoch: 2 Global Step: 20050 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:49:59,440-Speed 126.61 samples/sec Loss 12.0470 Epoch: 2 Global Step: 20055 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:50:19,934-Speed 124.92 samples/sec Loss 12.2581 Epoch: 2 Global Step: 20060 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:50:40,074-Speed 127.12 samples/sec Loss 12.4342 Epoch: 2 Global Step: 20065 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:51:00,237-Speed 126.97 samples/sec Loss 12.1774 Epoch: 2 Global Step: 20070 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:51:21,011-Speed 123.24 samples/sec Loss 12.0757 Epoch: 2 Global Step: 20075 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:51:41,076-Speed 127.59 samples/sec Loss 12.2418 Epoch: 2 Global Step: 20080 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:52:01,340-Speed 126.34 samples/sec Loss 11.9094 Epoch: 2 Global Step: 20085 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:52:21,478-Speed 127.13 samples/sec Loss 12.0852 Epoch: 2 Global Step: 20090 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:52:41,237-Speed 129.57 samples/sec Loss 12.0767 Epoch: 2 Global Step: 20095 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:53:01,291-Speed 127.66 samples/sec Loss 12.2383 Epoch: 2 Global Step: 20100 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:53:21,413-Speed 127.23 samples/sec Loss 12.5200 Epoch: 2 Global Step: 20105 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:53:42,332-Speed 122.38 samples/sec Loss 12.2611 Epoch: 2 Global Step: 20110 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:54:02,298-Speed 128.22 samples/sec Loss 12.1351 Epoch: 2 Global Step: 20115 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:54:22,977-Speed 123.81 samples/sec Loss 11.9368 Epoch: 2 Global Step: 20120 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:54:42,926-Speed 128.33 samples/sec Loss 12.2649 Epoch: 2 Global Step: 20125 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:55:03,074-Speed 127.07 samples/sec Loss 12.3070 Epoch: 2 Global Step: 20130 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:55:23,262-Speed 126.81 samples/sec Loss 12.3075 Epoch: 2 Global Step: 20135 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:55:43,491-Speed 126.56 samples/sec Loss 12.3085 Epoch: 2 Global Step: 20140 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:56:03,254-Speed 129.54 samples/sec Loss 11.7164 Epoch: 2 Global Step: 20145 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:56:23,194-Speed 128.40 samples/sec Loss 12.0223 Epoch: 2 Global Step: 20150 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:56:43,010-Speed 129.20 samples/sec Loss 12.0457 Epoch: 2 Global Step: 20155 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:57:03,324-Speed 126.03 samples/sec Loss 12.6258 Epoch: 2 Global Step: 20160 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:57:23,867-Speed 124.62 samples/sec Loss 12.2934 Epoch: 2 Global Step: 20165 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:57:44,081-Speed 126.65 samples/sec Loss 12.0500 Epoch: 2 Global Step: 20170 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:58:04,412-Speed 125.92 samples/sec Loss 12.6070 Epoch: 2 Global Step: 20175 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:58:24,541-Speed 127.19 samples/sec Loss 12.0908 Epoch: 2 Global Step: 20180 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:58:44,775-Speed 126.53 samples/sec Loss 12.2480 Epoch: 2 Global Step: 20185 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:59:04,748-Speed 128.18 samples/sec Loss 12.2303 Epoch: 2 Global Step: 20190 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:59:24,680-Speed 128.45 samples/sec Loss 12.1587 Epoch: 2 Global Step: 20195 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 18:59:44,833-Speed 127.04 samples/sec Loss 12.1298 Epoch: 2 Global Step: 20200 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:00:04,988-Speed 127.02 samples/sec Loss 12.5114 Epoch: 2 Global Step: 20205 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:00:25,624-Speed 124.06 samples/sec Loss 12.2062 Epoch: 2 Global Step: 20210 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:00:45,907-Speed 126.22 samples/sec Loss 12.4550 Epoch: 2 Global Step: 20215 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:01:05,801-Speed 128.69 samples/sec Loss 12.0185 Epoch: 2 Global Step: 20220 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:01:25,915-Speed 127.28 samples/sec Loss 12.5030 Epoch: 2 Global Step: 20225 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:01:45,805-Speed 128.71 samples/sec Loss 12.4584 Epoch: 2 Global Step: 20230 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:02:05,832-Speed 127.83 samples/sec Loss 12.1473 Epoch: 2 Global Step: 20235 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:02:25,963-Speed 127.18 samples/sec Loss 12.2683 Epoch: 2 Global Step: 20240 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:02:46,022-Speed 127.63 samples/sec Loss 12.3519 Epoch: 2 Global Step: 20245 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:03:06,221-Speed 126.75 samples/sec Loss 12.2082 Epoch: 2 Global Step: 20250 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:03:26,960-Speed 123.44 samples/sec Loss 12.4353 Epoch: 2 Global Step: 20255 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:03:47,127-Speed 126.95 samples/sec Loss 12.4756 Epoch: 2 Global Step: 20260 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:04:07,449-Speed 125.98 samples/sec Loss 12.4677 Epoch: 2 Global Step: 20265 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:04:27,611-Speed 126.98 samples/sec Loss 12.4097 Epoch: 2 Global Step: 20270 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:04:47,539-Speed 128.48 samples/sec Loss 12.1009 Epoch: 2 Global Step: 20275 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:05:07,425-Speed 128.74 samples/sec Loss 11.9619 Epoch: 2 Global Step: 20280 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:05:27,417-Speed 128.06 samples/sec Loss 12.3423 Epoch: 2 Global Step: 20285 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:05:47,549-Speed 127.16 samples/sec Loss 12.0394 Epoch: 2 Global Step: 20290 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:06:07,678-Speed 127.19 samples/sec Loss 12.1704 Epoch: 2 Global Step: 20295 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:06:28,388-Speed 123.62 samples/sec Loss 12.1537 Epoch: 2 Global Step: 20300 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:06:48,418-Speed 127.81 samples/sec Loss 12.1904 Epoch: 2 Global Step: 20305 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:07:08,513-Speed 127.40 samples/sec Loss 12.3309 Epoch: 2 Global Step: 20310 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:07:28,397-Speed 128.75 samples/sec Loss 12.0693 Epoch: 2 Global Step: 20315 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:07:48,488-Speed 127.43 samples/sec Loss 12.6990 Epoch: 2 Global Step: 20320 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:08:08,433-Speed 128.37 samples/sec Loss 12.6421 Epoch: 2 Global Step: 20325 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:08:28,261-Speed 129.11 samples/sec Loss 12.3888 Epoch: 2 Global Step: 20330 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:08:48,257-Speed 128.04 samples/sec Loss 12.2729 Epoch: 2 Global Step: 20335 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:09:07,995-Speed 129.70 samples/sec Loss 12.1839 Epoch: 2 Global Step: 20340 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:09:29,026-Speed 121.73 samples/sec Loss 12.3441 Epoch: 2 Global Step: 20345 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:09:48,748-Speed 129.81 samples/sec Loss 12.3216 Epoch: 2 Global Step: 20350 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:10:08,857-Speed 127.32 samples/sec Loss 12.4565 Epoch: 2 Global Step: 20355 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:10:29,331-Speed 125.04 samples/sec Loss 12.4084 Epoch: 2 Global Step: 20360 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:10:49,393-Speed 127.61 samples/sec Loss 11.8758 Epoch: 2 Global Step: 20365 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:11:09,629-Speed 126.51 samples/sec Loss 12.2751 Epoch: 2 Global Step: 20370 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:11:29,761-Speed 127.17 samples/sec Loss 12.2439 Epoch: 2 Global Step: 20375 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:11:50,077-Speed 126.02 samples/sec Loss 12.4581 Epoch: 2 Global Step: 20380 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:12:10,376-Speed 126.12 samples/sec Loss 12.4803 Epoch: 2 Global Step: 20385 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:12:31,166-Speed 123.14 samples/sec Loss 12.3535 Epoch: 2 Global Step: 20390 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:12:51,391-Speed 126.58 samples/sec Loss 12.3333 Epoch: 2 Global Step: 20395 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:13:11,605-Speed 126.65 samples/sec Loss 12.4297 Epoch: 2 Global Step: 20400 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:13:31,966-Speed 125.74 samples/sec Loss 12.1121 Epoch: 2 Global Step: 20405 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:13:52,135-Speed 126.94 samples/sec Loss 12.5506 Epoch: 2 Global Step: 20410 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:14:12,074-Speed 128.40 samples/sec Loss 12.3599 Epoch: 2 Global Step: 20415 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:14:32,047-Speed 128.18 samples/sec Loss 12.5777 Epoch: 2 Global Step: 20420 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:14:52,092-Speed 127.72 samples/sec Loss 12.6491 Epoch: 2 Global Step: 20425 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:15:12,251-Speed 127.00 samples/sec Loss 12.3900 Epoch: 2 Global Step: 20430 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:15:33,142-Speed 122.55 samples/sec Loss 12.4859 Epoch: 2 Global Step: 20435 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:15:53,382-Speed 126.49 samples/sec Loss 12.3584 Epoch: 2 Global Step: 20440 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:16:13,620-Speed 126.50 samples/sec Loss 12.4964 Epoch: 2 Global Step: 20445 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:16:33,870-Speed 126.43 samples/sec Loss 12.0958 Epoch: 2 Global Step: 20450 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:16:54,093-Speed 126.60 samples/sec Loss 12.2423 Epoch: 2 Global Step: 20455 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:17:14,162-Speed 127.56 samples/sec Loss 12.8628 Epoch: 2 Global Step: 20460 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:17:34,300-Speed 127.14 samples/sec Loss 12.4322 Epoch: 2 Global Step: 20465 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:17:54,558-Speed 126.37 samples/sec Loss 12.7898 Epoch: 2 Global Step: 20470 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:18:14,678-Speed 127.25 samples/sec Loss 12.2769 Epoch: 2 Global Step: 20475 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:18:35,328-Speed 123.97 samples/sec Loss 12.3625 Epoch: 2 Global Step: 20480 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:18:55,118-Speed 129.36 samples/sec Loss 12.8046 Epoch: 2 Global Step: 20485 Fp16 Grad Scale: 8192 Required: 271 hours Training: 2021-06-07 19:19:15,106-Speed 128.09 samples/sec Loss 12.2879 Epoch: 2 Global Step: 20490 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:19:35,479-Speed 125.66 samples/sec Loss 12.8424 Epoch: 2 Global Step: 20495 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:19:55,796-Speed 126.01 samples/sec Loss 12.6146 Epoch: 2 Global Step: 20500 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:20:16,211-Speed 125.40 samples/sec Loss 12.3791 Epoch: 2 Global Step: 20505 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:20:36,420-Speed 126.69 samples/sec Loss 12.3663 Epoch: 2 Global Step: 20510 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:20:56,489-Speed 127.57 samples/sec Loss 12.2446 Epoch: 2 Global Step: 20515 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:21:16,676-Speed 126.82 samples/sec Loss 12.5706 Epoch: 2 Global Step: 20520 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:21:37,591-Speed 122.41 samples/sec Loss 12.6935 Epoch: 2 Global Step: 20525 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:21:57,829-Speed 126.50 samples/sec Loss 12.5013 Epoch: 2 Global Step: 20530 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:22:17,919-Speed 127.43 samples/sec Loss 12.4172 Epoch: 2 Global Step: 20535 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:22:38,136-Speed 126.63 samples/sec Loss 12.3984 Epoch: 2 Global Step: 20540 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:22:58,294-Speed 127.01 samples/sec Loss 12.2327 Epoch: 2 Global Step: 20545 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:23:18,532-Speed 126.50 samples/sec Loss 12.6915 Epoch: 2 Global Step: 20550 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:23:39,026-Speed 124.93 samples/sec Loss 12.4348 Epoch: 2 Global Step: 20555 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:23:59,257-Speed 126.54 samples/sec Loss 12.1930 Epoch: 2 Global Step: 20560 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:24:19,054-Speed 129.32 samples/sec Loss 12.7728 Epoch: 2 Global Step: 20565 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:24:39,653-Speed 124.29 samples/sec Loss 12.1252 Epoch: 2 Global Step: 20570 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:24:59,780-Speed 127.20 samples/sec Loss 12.7795 Epoch: 2 Global Step: 20575 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:25:19,526-Speed 129.65 samples/sec Loss 12.5671 Epoch: 2 Global Step: 20580 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:25:39,703-Speed 126.88 samples/sec Loss 12.5656 Epoch: 2 Global Step: 20585 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:25:59,718-Speed 127.91 samples/sec Loss 12.6309 Epoch: 2 Global Step: 20590 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:26:19,969-Speed 126.42 samples/sec Loss 12.4322 Epoch: 2 Global Step: 20595 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:26:40,374-Speed 125.47 samples/sec Loss 12.5736 Epoch: 2 Global Step: 20600 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:27:00,346-Speed 128.19 samples/sec Loss 12.6244 Epoch: 2 Global Step: 20605 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:27:20,724-Speed 125.63 samples/sec Loss 12.5847 Epoch: 2 Global Step: 20610 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:27:41,650-Speed 122.34 samples/sec Loss 12.4970 Epoch: 2 Global Step: 20615 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:28:01,694-Speed 127.72 samples/sec Loss 12.5928 Epoch: 2 Global Step: 20620 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:28:21,891-Speed 126.76 samples/sec Loss 12.7608 Epoch: 2 Global Step: 20625 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:28:41,768-Speed 128.79 samples/sec Loss 12.3317 Epoch: 2 Global Step: 20630 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:29:01,898-Speed 127.18 samples/sec Loss 12.5152 Epoch: 2 Global Step: 20635 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:29:22,050-Speed 127.04 samples/sec Loss 12.5125 Epoch: 2 Global Step: 20640 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:29:42,727-Speed 123.82 samples/sec Loss 12.6973 Epoch: 2 Global Step: 20645 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:30:02,786-Speed 127.63 samples/sec Loss 12.5883 Epoch: 2 Global Step: 20650 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:30:23,038-Speed 126.41 samples/sec Loss 12.7067 Epoch: 2 Global Step: 20655 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:30:44,087-Speed 121.63 samples/sec Loss 12.4577 Epoch: 2 Global Step: 20660 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:31:04,286-Speed 126.74 samples/sec Loss 12.5984 Epoch: 2 Global Step: 20665 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:31:24,204-Speed 128.53 samples/sec Loss 12.6748 Epoch: 2 Global Step: 20670 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:31:44,339-Speed 127.15 samples/sec Loss 12.1707 Epoch: 2 Global Step: 20675 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:32:04,732-Speed 125.54 samples/sec Loss 12.7413 Epoch: 2 Global Step: 20680 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:32:25,042-Speed 126.05 samples/sec Loss 12.5581 Epoch: 2 Global Step: 20685 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:32:45,204-Speed 126.98 samples/sec Loss 12.5618 Epoch: 2 Global Step: 20690 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:33:05,534-Speed 125.93 samples/sec Loss 12.8650 Epoch: 2 Global Step: 20695 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:33:25,681-Speed 127.08 samples/sec Loss 12.2769 Epoch: 2 Global Step: 20700 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:33:46,362-Speed 123.79 samples/sec Loss 12.0778 Epoch: 2 Global Step: 20705 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:34:06,383-Speed 127.87 samples/sec Loss 12.4380 Epoch: 2 Global Step: 20710 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:34:26,583-Speed 126.74 samples/sec Loss 12.4389 Epoch: 2 Global Step: 20715 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:34:47,113-Speed 124.70 samples/sec Loss 12.5288 Epoch: 2 Global Step: 20720 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:35:06,981-Speed 128.86 samples/sec Loss 12.3755 Epoch: 2 Global Step: 20725 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:35:27,247-Speed 126.33 samples/sec Loss 12.6437 Epoch: 2 Global Step: 20730 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:35:47,528-Speed 126.23 samples/sec Loss 12.7591 Epoch: 2 Global Step: 20735 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:36:07,902-Speed 125.66 samples/sec Loss 12.6585 Epoch: 2 Global Step: 20740 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:36:28,221-Speed 126.00 samples/sec Loss 12.6644 Epoch: 2 Global Step: 20745 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:36:49,315-Speed 121.36 samples/sec Loss 12.5838 Epoch: 2 Global Step: 20750 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:37:09,292-Speed 128.16 samples/sec Loss 12.6817 Epoch: 2 Global Step: 20755 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:37:29,422-Speed 127.17 samples/sec Loss 12.6415 Epoch: 2 Global Step: 20760 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:37:49,580-Speed 127.00 samples/sec Loss 12.9020 Epoch: 2 Global Step: 20765 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:38:09,708-Speed 127.20 samples/sec Loss 12.3503 Epoch: 2 Global Step: 20770 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:38:29,975-Speed 126.32 samples/sec Loss 12.6915 Epoch: 2 Global Step: 20775 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:38:50,097-Speed 127.24 samples/sec Loss 12.9414 Epoch: 2 Global Step: 20780 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:39:10,401-Speed 126.09 samples/sec Loss 12.7165 Epoch: 2 Global Step: 20785 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:39:30,444-Speed 127.74 samples/sec Loss 12.5640 Epoch: 2 Global Step: 20790 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:39:51,048-Speed 124.25 samples/sec Loss 12.4408 Epoch: 2 Global Step: 20795 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:40:11,166-Speed 127.25 samples/sec Loss 12.4839 Epoch: 2 Global Step: 20800 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:40:31,427-Speed 126.36 samples/sec Loss 12.6143 Epoch: 2 Global Step: 20805 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:40:51,726-Speed 126.12 samples/sec Loss 12.7974 Epoch: 2 Global Step: 20810 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:41:11,940-Speed 126.66 samples/sec Loss 12.6389 Epoch: 2 Global Step: 20815 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:41:31,914-Speed 128.17 samples/sec Loss 12.5009 Epoch: 2 Global Step: 20820 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:41:52,221-Speed 126.07 samples/sec Loss 12.4556 Epoch: 2 Global Step: 20825 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:42:12,372-Speed 127.05 samples/sec Loss 12.8556 Epoch: 2 Global Step: 20830 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:42:32,702-Speed 125.93 samples/sec Loss 12.8981 Epoch: 2 Global Step: 20835 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:42:53,313-Speed 124.21 samples/sec Loss 12.3328 Epoch: 2 Global Step: 20840 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:43:13,518-Speed 126.70 samples/sec Loss 12.5415 Epoch: 2 Global Step: 20845 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:43:33,622-Speed 127.34 samples/sec Loss 12.5944 Epoch: 2 Global Step: 20850 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:43:53,755-Speed 127.16 samples/sec Loss 12.2497 Epoch: 2 Global Step: 20855 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:44:13,853-Speed 127.39 samples/sec Loss 12.3423 Epoch: 2 Global Step: 20860 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:44:34,187-Speed 125.91 samples/sec Loss 12.4940 Epoch: 2 Global Step: 20865 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:44:55,088-Speed 122.48 samples/sec Loss 12.7278 Epoch: 2 Global Step: 20870 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:45:15,040-Speed 128.32 samples/sec Loss 12.8093 Epoch: 2 Global Step: 20875 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:45:35,365-Speed 125.96 samples/sec Loss 12.5980 Epoch: 2 Global Step: 20880 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:45:56,262-Speed 122.51 samples/sec Loss 12.5017 Epoch: 2 Global Step: 20885 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:46:16,484-Speed 126.60 samples/sec Loss 12.7726 Epoch: 2 Global Step: 20890 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:46:36,602-Speed 127.26 samples/sec Loss 12.1300 Epoch: 2 Global Step: 20895 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:46:56,523-Speed 128.51 samples/sec Loss 12.9538 Epoch: 2 Global Step: 20900 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:47:16,596-Speed 127.55 samples/sec Loss 12.5559 Epoch: 2 Global Step: 20905 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:47:36,675-Speed 127.50 samples/sec Loss 12.3104 Epoch: 2 Global Step: 20910 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:47:56,562-Speed 128.74 samples/sec Loss 12.8399 Epoch: 2 Global Step: 20915 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:48:16,679-Speed 127.26 samples/sec Loss 12.3364 Epoch: 2 Global Step: 20920 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:48:36,747-Speed 127.57 samples/sec Loss 12.4374 Epoch: 2 Global Step: 20925 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:48:57,574-Speed 122.92 samples/sec Loss 12.2873 Epoch: 2 Global Step: 20930 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:49:18,250-Speed 123.82 samples/sec Loss 12.4154 Epoch: 2 Global Step: 20935 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:49:38,464-Speed 126.65 samples/sec Loss 12.5780 Epoch: 2 Global Step: 20940 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:49:58,774-Speed 126.05 samples/sec Loss 12.6775 Epoch: 2 Global Step: 20945 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:50:19,486-Speed 123.61 samples/sec Loss 12.4230 Epoch: 2 Global Step: 20950 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:50:39,727-Speed 126.49 samples/sec Loss 12.4333 Epoch: 2 Global Step: 20955 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:51:00,003-Speed 126.27 samples/sec Loss 12.6269 Epoch: 2 Global Step: 20960 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:51:20,378-Speed 125.65 samples/sec Loss 12.5507 Epoch: 2 Global Step: 20965 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:51:40,859-Speed 125.00 samples/sec Loss 12.1992 Epoch: 2 Global Step: 20970 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:52:01,626-Speed 123.28 samples/sec Loss 12.3914 Epoch: 2 Global Step: 20975 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:52:21,771-Speed 127.08 samples/sec Loss 12.2936 Epoch: 2 Global Step: 20980 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:52:41,778-Speed 127.96 samples/sec Loss 12.3386 Epoch: 2 Global Step: 20985 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:53:01,673-Speed 128.68 samples/sec Loss 12.5101 Epoch: 2 Global Step: 20990 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:53:21,854-Speed 126.86 samples/sec Loss 12.8779 Epoch: 2 Global Step: 20995 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:53:41,891-Speed 127.78 samples/sec Loss 12.2728 Epoch: 2 Global Step: 21000 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:54:02,135-Speed 126.47 samples/sec Loss 12.6281 Epoch: 2 Global Step: 21005 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:54:22,240-Speed 127.34 samples/sec Loss 12.4867 Epoch: 2 Global Step: 21010 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:54:42,539-Speed 126.12 samples/sec Loss 12.8171 Epoch: 2 Global Step: 21015 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:55:03,276-Speed 123.46 samples/sec Loss 12.5262 Epoch: 2 Global Step: 21020 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:55:23,656-Speed 125.62 samples/sec Loss 12.6324 Epoch: 2 Global Step: 21025 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:55:43,776-Speed 127.24 samples/sec Loss 12.4817 Epoch: 2 Global Step: 21030 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:56:04,143-Speed 125.70 samples/sec Loss 12.4737 Epoch: 2 Global Step: 21035 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:56:24,484-Speed 125.86 samples/sec Loss 12.7924 Epoch: 2 Global Step: 21040 Fp16 Grad Scale: 8192 Required: 270 hours Training: 2021-06-07 19:56:44,777-Speed 126.16 samples/sec Loss 12.2679 Epoch: 2 Global Step: 21045 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 19:57:05,036-Speed 126.37 samples/sec Loss 12.6760 Epoch: 2 Global Step: 21050 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 19:57:25,235-Speed 126.75 samples/sec Loss 12.6619 Epoch: 2 Global Step: 21055 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 19:57:45,191-Speed 128.29 samples/sec Loss 12.5692 Epoch: 2 Global Step: 21060 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 19:58:06,025-Speed 122.89 samples/sec Loss 12.1881 Epoch: 2 Global Step: 21065 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 19:58:26,135-Speed 127.30 samples/sec Loss 12.6681 Epoch: 2 Global Step: 21070 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 19:58:46,267-Speed 127.16 samples/sec Loss 12.5737 Epoch: 2 Global Step: 21075 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 19:59:06,574-Speed 126.07 samples/sec Loss 12.8929 Epoch: 2 Global Step: 21080 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 19:59:26,892-Speed 126.01 samples/sec Loss 12.6782 Epoch: 2 Global Step: 21085 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 19:59:46,868-Speed 128.16 samples/sec Loss 12.1776 Epoch: 2 Global Step: 21090 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:00:06,938-Speed 127.56 samples/sec Loss 12.9878 Epoch: 2 Global Step: 21095 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:00:27,246-Speed 126.07 samples/sec Loss 12.9960 Epoch: 2 Global Step: 21100 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:00:47,751-Speed 124.86 samples/sec Loss 12.7311 Epoch: 2 Global Step: 21105 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:01:08,877-Speed 121.18 samples/sec Loss 12.2734 Epoch: 2 Global Step: 21110 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:01:29,273-Speed 125.52 samples/sec Loss 12.8443 Epoch: 2 Global Step: 21115 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:01:49,536-Speed 126.34 samples/sec Loss 12.5189 Epoch: 2 Global Step: 21120 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:02:10,369-Speed 122.89 samples/sec Loss 12.5441 Epoch: 2 Global Step: 21125 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:02:30,293-Speed 128.50 samples/sec Loss 12.4649 Epoch: 2 Global Step: 21130 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:02:50,486-Speed 126.78 samples/sec Loss 12.7939 Epoch: 2 Global Step: 21135 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:03:10,514-Speed 127.83 samples/sec Loss 12.6076 Epoch: 2 Global Step: 21140 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:03:30,763-Speed 126.43 samples/sec Loss 12.5184 Epoch: 2 Global Step: 21145 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:03:51,064-Speed 126.11 samples/sec Loss 12.2477 Epoch: 2 Global Step: 21150 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:04:12,348-Speed 120.28 samples/sec Loss 12.8889 Epoch: 2 Global Step: 21155 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:04:32,567-Speed 126.62 samples/sec Loss 12.5561 Epoch: 2 Global Step: 21160 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:04:52,888-Speed 125.98 samples/sec Loss 12.3245 Epoch: 2 Global Step: 21165 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:05:13,264-Speed 125.65 samples/sec Loss 12.5607 Epoch: 2 Global Step: 21170 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:05:33,591-Speed 125.95 samples/sec Loss 12.8474 Epoch: 2 Global Step: 21175 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:05:53,661-Speed 127.56 samples/sec Loss 12.5580 Epoch: 2 Global Step: 21180 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:06:14,045-Speed 125.60 samples/sec Loss 12.9435 Epoch: 2 Global Step: 21185 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:06:34,044-Speed 128.01 samples/sec Loss 12.3520 Epoch: 2 Global Step: 21190 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:06:54,153-Speed 127.32 samples/sec Loss 12.4790 Epoch: 2 Global Step: 21195 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:07:15,257-Speed 121.31 samples/sec Loss 12.5426 Epoch: 2 Global Step: 21200 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:07:36,160-Speed 122.48 samples/sec Loss 12.8861 Epoch: 2 Global Step: 21205 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:07:56,374-Speed 126.65 samples/sec Loss 12.7290 Epoch: 2 Global Step: 21210 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:08:16,581-Speed 126.70 samples/sec Loss 12.6446 Epoch: 2 Global Step: 21215 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:08:36,920-Speed 125.87 samples/sec Loss 12.1649 Epoch: 2 Global Step: 21220 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:08:57,223-Speed 126.10 samples/sec Loss 12.3121 Epoch: 2 Global Step: 21225 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:09:17,466-Speed 126.47 samples/sec Loss 12.5051 Epoch: 2 Global Step: 21230 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:09:37,879-Speed 125.41 samples/sec Loss 12.6712 Epoch: 2 Global Step: 21235 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:09:58,012-Speed 127.17 samples/sec Loss 12.5582 Epoch: 2 Global Step: 21240 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:10:18,791-Speed 123.21 samples/sec Loss 12.4687 Epoch: 2 Global Step: 21245 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:10:39,130-Speed 125.87 samples/sec Loss 12.7558 Epoch: 2 Global Step: 21250 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:10:59,333-Speed 126.72 samples/sec Loss 12.2995 Epoch: 2 Global Step: 21255 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:11:19,546-Speed 126.66 samples/sec Loss 12.3297 Epoch: 2 Global Step: 21260 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:11:39,724-Speed 126.88 samples/sec Loss 12.3416 Epoch: 2 Global Step: 21265 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:11:59,987-Speed 126.35 samples/sec Loss 12.7538 Epoch: 2 Global Step: 21270 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:12:20,125-Speed 127.13 samples/sec Loss 12.4554 Epoch: 2 Global Step: 21275 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:12:40,349-Speed 126.59 samples/sec Loss 12.4730 Epoch: 2 Global Step: 21280 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:13:00,732-Speed 125.60 samples/sec Loss 12.7835 Epoch: 2 Global Step: 21285 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:13:21,540-Speed 123.04 samples/sec Loss 12.5982 Epoch: 2 Global Step: 21290 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:13:41,757-Speed 126.63 samples/sec Loss 12.6589 Epoch: 2 Global Step: 21295 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:14:02,059-Speed 126.10 samples/sec Loss 12.3986 Epoch: 2 Global Step: 21300 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:14:21,955-Speed 128.68 samples/sec Loss 12.3840 Epoch: 2 Global Step: 21305 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:14:42,126-Speed 126.92 samples/sec Loss 12.7431 Epoch: 2 Global Step: 21310 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:15:01,977-Speed 128.97 samples/sec Loss 12.1939 Epoch: 2 Global Step: 21315 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:15:22,185-Speed 126.69 samples/sec Loss 12.5475 Epoch: 2 Global Step: 21320 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:15:42,340-Speed 127.02 samples/sec Loss 12.8016 Epoch: 2 Global Step: 21325 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:16:02,547-Speed 126.70 samples/sec Loss 12.5203 Epoch: 2 Global Step: 21330 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:16:23,247-Speed 123.68 samples/sec Loss 12.4790 Epoch: 2 Global Step: 21335 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:16:43,503-Speed 126.39 samples/sec Loss 12.5066 Epoch: 2 Global Step: 21340 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:17:03,701-Speed 126.75 samples/sec Loss 12.6065 Epoch: 2 Global Step: 21345 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:17:23,950-Speed 126.43 samples/sec Loss 12.4568 Epoch: 2 Global Step: 21350 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:17:44,172-Speed 126.60 samples/sec Loss 12.3884 Epoch: 2 Global Step: 21355 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:18:04,496-Speed 125.97 samples/sec Loss 12.5530 Epoch: 2 Global Step: 21360 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:18:24,903-Speed 125.46 samples/sec Loss 12.6304 Epoch: 2 Global Step: 21365 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:18:45,429-Speed 124.73 samples/sec Loss 12.4457 Epoch: 2 Global Step: 21370 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:19:05,651-Speed 126.61 samples/sec Loss 12.5107 Epoch: 2 Global Step: 21375 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:19:26,658-Speed 121.87 samples/sec Loss 12.4638 Epoch: 2 Global Step: 21380 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:19:46,920-Speed 126.35 samples/sec Loss 12.9616 Epoch: 2 Global Step: 21385 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:20:07,105-Speed 126.83 samples/sec Loss 12.6861 Epoch: 2 Global Step: 21390 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:20:27,261-Speed 127.02 samples/sec Loss 12.4735 Epoch: 2 Global Step: 21395 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:20:47,338-Speed 127.52 samples/sec Loss 12.3828 Epoch: 2 Global Step: 21400 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:21:07,851-Speed 124.81 samples/sec Loss 12.7163 Epoch: 2 Global Step: 21405 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:21:27,926-Speed 127.54 samples/sec Loss 12.8430 Epoch: 2 Global Step: 21410 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:21:47,960-Speed 127.79 samples/sec Loss 12.2950 Epoch: 2 Global Step: 21415 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:22:08,240-Speed 126.24 samples/sec Loss 12.6076 Epoch: 2 Global Step: 21420 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:22:29,270-Speed 121.74 samples/sec Loss 12.5683 Epoch: 2 Global Step: 21425 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:22:49,601-Speed 125.92 samples/sec Loss 13.0131 Epoch: 2 Global Step: 21430 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:23:09,691-Speed 127.43 samples/sec Loss 12.5740 Epoch: 2 Global Step: 21435 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:23:29,892-Speed 126.73 samples/sec Loss 12.6110 Epoch: 2 Global Step: 21440 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:23:50,035-Speed 127.10 samples/sec Loss 12.8429 Epoch: 2 Global Step: 21445 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:24:10,322-Speed 126.20 samples/sec Loss 12.5730 Epoch: 2 Global Step: 21450 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:24:30,567-Speed 126.46 samples/sec Loss 12.3715 Epoch: 2 Global Step: 21455 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:24:50,557-Speed 128.07 samples/sec Loss 12.2676 Epoch: 2 Global Step: 21460 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:25:10,653-Speed 127.39 samples/sec Loss 12.2156 Epoch: 2 Global Step: 21465 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:25:31,501-Speed 122.80 samples/sec Loss 12.7075 Epoch: 2 Global Step: 21470 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:25:51,775-Speed 126.28 samples/sec Loss 12.2187 Epoch: 2 Global Step: 21475 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:26:12,129-Speed 125.78 samples/sec Loss 12.2850 Epoch: 2 Global Step: 21480 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:26:32,356-Speed 126.57 samples/sec Loss 12.4244 Epoch: 2 Global Step: 21485 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:26:52,496-Speed 127.13 samples/sec Loss 12.3086 Epoch: 2 Global Step: 21490 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:27:12,897-Speed 125.49 samples/sec Loss 12.5247 Epoch: 2 Global Step: 21495 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:27:33,293-Speed 125.52 samples/sec Loss 12.6022 Epoch: 2 Global Step: 21500 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:27:53,667-Speed 125.66 samples/sec Loss 12.3843 Epoch: 2 Global Step: 21505 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:28:13,792-Speed 127.21 samples/sec Loss 12.5977 Epoch: 2 Global Step: 21510 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:28:34,930-Speed 121.12 samples/sec Loss 12.9442 Epoch: 2 Global Step: 21515 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:28:55,281-Speed 125.79 samples/sec Loss 12.7402 Epoch: 2 Global Step: 21520 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:29:15,499-Speed 126.63 samples/sec Loss 12.3573 Epoch: 2 Global Step: 21525 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:29:36,348-Speed 122.80 samples/sec Loss 12.7302 Epoch: 2 Global Step: 21530 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:29:56,459-Speed 127.30 samples/sec Loss 12.3387 Epoch: 2 Global Step: 21535 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:30:16,758-Speed 126.12 samples/sec Loss 12.6666 Epoch: 2 Global Step: 21540 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:30:36,995-Speed 126.51 samples/sec Loss 12.7556 Epoch: 2 Global Step: 21545 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:30:57,326-Speed 125.93 samples/sec Loss 12.2853 Epoch: 2 Global Step: 21550 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:31:17,750-Speed 125.34 samples/sec Loss 12.6123 Epoch: 2 Global Step: 21555 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:31:38,613-Speed 122.72 samples/sec Loss 12.4885 Epoch: 2 Global Step: 21560 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:31:58,911-Speed 126.12 samples/sec Loss 12.4312 Epoch: 2 Global Step: 21565 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:32:19,143-Speed 126.54 samples/sec Loss 12.3899 Epoch: 2 Global Step: 21570 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:32:39,380-Speed 126.51 samples/sec Loss 12.2950 Epoch: 2 Global Step: 21575 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:32:59,574-Speed 126.77 samples/sec Loss 12.5209 Epoch: 2 Global Step: 21580 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:33:19,707-Speed 127.17 samples/sec Loss 12.2775 Epoch: 2 Global Step: 21585 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:33:40,064-Speed 125.76 samples/sec Loss 12.6522 Epoch: 2 Global Step: 21590 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:34:00,623-Speed 124.53 samples/sec Loss 12.4116 Epoch: 2 Global Step: 21595 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:34:20,880-Speed 126.38 samples/sec Loss 12.6032 Epoch: 2 Global Step: 21600 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:34:41,599-Speed 123.56 samples/sec Loss 12.4725 Epoch: 2 Global Step: 21605 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:35:01,774-Speed 126.90 samples/sec Loss 12.3301 Epoch: 2 Global Step: 21610 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:35:21,673-Speed 128.65 samples/sec Loss 12.4023 Epoch: 2 Global Step: 21615 Fp16 Grad Scale: 8192 Required: 269 hours Training: 2021-06-07 20:35:42,054-Speed 125.62 samples/sec Loss 12.6256 Epoch: 2 Global Step: 21620 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:36:01,933-Speed 128.79 samples/sec Loss 12.4665 Epoch: 2 Global Step: 21625 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:36:22,073-Speed 127.12 samples/sec Loss 12.7609 Epoch: 2 Global Step: 21630 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:36:42,031-Speed 128.27 samples/sec Loss 12.5179 Epoch: 2 Global Step: 21635 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:37:02,196-Speed 126.97 samples/sec Loss 12.8388 Epoch: 2 Global Step: 21640 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:37:22,733-Speed 124.66 samples/sec Loss 12.5312 Epoch: 2 Global Step: 21645 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:37:43,446-Speed 123.60 samples/sec Loss 12.4000 Epoch: 2 Global Step: 21650 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:38:03,855-Speed 125.44 samples/sec Loss 12.8019 Epoch: 2 Global Step: 21655 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:38:24,199-Speed 125.84 samples/sec Loss 12.5238 Epoch: 2 Global Step: 21660 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:38:43,959-Speed 129.57 samples/sec Loss 12.3313 Epoch: 2 Global Step: 21665 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:39:04,140-Speed 126.86 samples/sec Loss 12.6037 Epoch: 2 Global Step: 21670 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:39:24,207-Speed 127.58 samples/sec Loss 12.7804 Epoch: 2 Global Step: 21675 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:39:44,539-Speed 125.91 samples/sec Loss 12.6527 Epoch: 2 Global Step: 21680 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:40:04,683-Speed 127.09 samples/sec Loss 12.5831 Epoch: 2 Global Step: 21685 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:40:24,928-Speed 126.46 samples/sec Loss 12.2524 Epoch: 2 Global Step: 21690 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:40:45,914-Speed 121.99 samples/sec Loss 12.6365 Epoch: 2 Global Step: 21695 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:41:05,995-Speed 127.49 samples/sec Loss 12.5470 Epoch: 2 Global Step: 21700 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:41:26,112-Speed 127.26 samples/sec Loss 12.1909 Epoch: 2 Global Step: 21705 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:41:46,958-Speed 122.82 samples/sec Loss 12.4180 Epoch: 2 Global Step: 21710 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:42:07,434-Speed 125.03 samples/sec Loss 12.6148 Epoch: 2 Global Step: 21715 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:42:27,728-Speed 126.15 samples/sec Loss 12.4665 Epoch: 2 Global Step: 21720 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:42:48,103-Speed 125.65 samples/sec Loss 12.7600 Epoch: 2 Global Step: 21725 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:43:08,247-Speed 127.09 samples/sec Loss 12.8219 Epoch: 2 Global Step: 21730 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:43:28,590-Speed 125.85 samples/sec Loss 12.5178 Epoch: 2 Global Step: 21735 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:43:49,482-Speed 122.54 samples/sec Loss 12.4701 Epoch: 2 Global Step: 21740 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:44:09,595-Speed 127.28 samples/sec Loss 12.4044 Epoch: 2 Global Step: 21745 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:44:29,813-Speed 126.62 samples/sec Loss 12.6237 Epoch: 2 Global Step: 21750 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:44:50,137-Speed 125.97 samples/sec Loss 12.5119 Epoch: 2 Global Step: 21755 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:45:10,225-Speed 127.44 samples/sec Loss 12.5082 Epoch: 2 Global Step: 21760 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:45:30,533-Speed 126.07 samples/sec Loss 12.4708 Epoch: 2 Global Step: 21765 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:45:50,767-Speed 126.53 samples/sec Loss 12.8748 Epoch: 2 Global Step: 21770 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:46:11,000-Speed 126.53 samples/sec Loss 12.4527 Epoch: 2 Global Step: 21775 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:46:30,966-Speed 128.23 samples/sec Loss 12.2249 Epoch: 2 Global Step: 21780 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:46:51,453-Speed 124.96 samples/sec Loss 12.5901 Epoch: 2 Global Step: 21785 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:47:11,577-Speed 127.21 samples/sec Loss 12.5663 Epoch: 2 Global Step: 21790 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:47:31,639-Speed 127.61 samples/sec Loss 12.6548 Epoch: 2 Global Step: 21795 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:47:51,912-Speed 126.28 samples/sec Loss 12.7691 Epoch: 2 Global Step: 21800 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:48:12,266-Speed 125.78 samples/sec Loss 12.7721 Epoch: 2 Global Step: 21805 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:48:32,285-Speed 127.88 samples/sec Loss 12.2996 Epoch: 2 Global Step: 21810 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:48:52,729-Speed 125.23 samples/sec Loss 12.2494 Epoch: 2 Global Step: 21815 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:49:12,906-Speed 126.88 samples/sec Loss 12.4286 Epoch: 2 Global Step: 21820 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:49:33,436-Speed 124.70 samples/sec Loss 12.8331 Epoch: 2 Global Step: 21825 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:49:54,098-Speed 123.91 samples/sec Loss 12.6457 Epoch: 2 Global Step: 21830 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:50:14,439-Speed 125.86 samples/sec Loss 12.6617 Epoch: 2 Global Step: 21835 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:50:34,557-Speed 127.25 samples/sec Loss 12.6091 Epoch: 2 Global Step: 21840 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:50:54,773-Speed 126.64 samples/sec Loss 12.4909 Epoch: 2 Global Step: 21845 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:51:14,973-Speed 126.74 samples/sec Loss 12.3349 Epoch: 2 Global Step: 21850 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:51:35,545-Speed 124.44 samples/sec Loss 12.6418 Epoch: 2 Global Step: 21855 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:51:55,789-Speed 126.47 samples/sec Loss 12.3551 Epoch: 2 Global Step: 21860 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:52:16,010-Speed 126.61 samples/sec Loss 12.6248 Epoch: 2 Global Step: 21865 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:52:36,249-Speed 126.50 samples/sec Loss 12.3971 Epoch: 2 Global Step: 21870 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:52:57,365-Speed 121.24 samples/sec Loss 12.4127 Epoch: 2 Global Step: 21875 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:53:17,597-Speed 126.54 samples/sec Loss 12.8284 Epoch: 2 Global Step: 21880 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:53:37,883-Speed 126.20 samples/sec Loss 12.5343 Epoch: 2 Global Step: 21885 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:53:58,015-Speed 127.17 samples/sec Loss 12.4743 Epoch: 2 Global Step: 21890 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:54:17,965-Speed 128.32 samples/sec Loss 12.0341 Epoch: 2 Global Step: 21895 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:54:37,905-Speed 128.39 samples/sec Loss 12.3330 Epoch: 2 Global Step: 21900 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:54:58,156-Speed 126.42 samples/sec Loss 12.7960 Epoch: 2 Global Step: 21905 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:55:18,500-Speed 125.85 samples/sec Loss 12.3796 Epoch: 2 Global Step: 21910 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:55:38,791-Speed 126.17 samples/sec Loss 12.5448 Epoch: 2 Global Step: 21915 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:56:00,072-Speed 120.30 samples/sec Loss 12.6924 Epoch: 2 Global Step: 21920 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:56:20,313-Speed 126.48 samples/sec Loss 12.0830 Epoch: 2 Global Step: 21925 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:56:40,469-Speed 127.02 samples/sec Loss 12.3000 Epoch: 2 Global Step: 21930 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:57:00,824-Speed 125.77 samples/sec Loss 12.2582 Epoch: 2 Global Step: 21935 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:57:21,162-Speed 125.88 samples/sec Loss 12.2686 Epoch: 2 Global Step: 21940 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:57:41,355-Speed 126.78 samples/sec Loss 12.4188 Epoch: 2 Global Step: 21945 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:58:01,540-Speed 126.83 samples/sec Loss 12.6439 Epoch: 2 Global Step: 21950 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:58:21,575-Speed 127.79 samples/sec Loss 12.0563 Epoch: 2 Global Step: 21955 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:58:41,748-Speed 126.91 samples/sec Loss 12.4825 Epoch: 2 Global Step: 21960 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:59:02,459-Speed 123.61 samples/sec Loss 12.2399 Epoch: 2 Global Step: 21965 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:59:22,768-Speed 126.05 samples/sec Loss 12.5073 Epoch: 2 Global Step: 21970 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 20:59:43,010-Speed 126.48 samples/sec Loss 12.3132 Epoch: 2 Global Step: 21975 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:00:03,455-Speed 125.22 samples/sec Loss 12.5668 Epoch: 2 Global Step: 21980 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:00:23,372-Speed 128.54 samples/sec Loss 12.3383 Epoch: 2 Global Step: 21985 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:00:43,273-Speed 128.65 samples/sec Loss 12.4563 Epoch: 2 Global Step: 21990 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:01:03,420-Speed 127.07 samples/sec Loss 12.6623 Epoch: 2 Global Step: 21995 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:01:23,698-Speed 126.25 samples/sec Loss 12.5475 Epoch: 2 Global Step: 22000 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:01:44,262-Speed 124.50 samples/sec Loss 12.4560 Epoch: 2 Global Step: 22005 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:02:05,510-Speed 120.49 samples/sec Loss 12.4781 Epoch: 2 Global Step: 22010 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:02:25,832-Speed 125.97 samples/sec Loss 12.2483 Epoch: 2 Global Step: 22015 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:02:46,190-Speed 125.76 samples/sec Loss 12.5450 Epoch: 2 Global Step: 22020 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:03:06,386-Speed 126.76 samples/sec Loss 12.7161 Epoch: 2 Global Step: 22025 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:03:26,720-Speed 125.91 samples/sec Loss 12.4043 Epoch: 2 Global Step: 22030 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:03:46,898-Speed 126.87 samples/sec Loss 12.7275 Epoch: 2 Global Step: 22035 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:04:07,055-Speed 127.01 samples/sec Loss 12.5992 Epoch: 2 Global Step: 22040 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:04:26,837-Speed 129.42 samples/sec Loss 12.1651 Epoch: 2 Global Step: 22045 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:04:47,070-Speed 126.53 samples/sec Loss 12.3116 Epoch: 2 Global Step: 22050 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:05:08,223-Speed 121.03 samples/sec Loss 12.2895 Epoch: 2 Global Step: 22055 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:05:28,704-Speed 125.00 samples/sec Loss 12.5226 Epoch: 2 Global Step: 22060 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:05:48,910-Speed 126.70 samples/sec Loss 12.7732 Epoch: 2 Global Step: 22065 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:06:09,309-Speed 125.51 samples/sec Loss 12.6290 Epoch: 2 Global Step: 22070 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:06:29,603-Speed 126.15 samples/sec Loss 12.4292 Epoch: 2 Global Step: 22075 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:06:49,844-Speed 126.49 samples/sec Loss 12.4581 Epoch: 2 Global Step: 22080 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:07:10,485-Speed 124.03 samples/sec Loss 12.5260 Epoch: 2 Global Step: 22085 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:07:30,677-Speed 126.79 samples/sec Loss 12.1112 Epoch: 2 Global Step: 22090 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:07:50,747-Speed 127.56 samples/sec Loss 12.4737 Epoch: 2 Global Step: 22095 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:08:11,370-Speed 124.14 samples/sec Loss 12.3917 Epoch: 2 Global Step: 22100 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:08:31,637-Speed 126.32 samples/sec Loss 12.5535 Epoch: 2 Global Step: 22105 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:08:51,987-Speed 125.80 samples/sec Loss 12.3243 Epoch: 2 Global Step: 22110 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:09:12,313-Speed 125.95 samples/sec Loss 12.2315 Epoch: 2 Global Step: 22115 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:09:32,536-Speed 126.60 samples/sec Loss 12.7215 Epoch: 2 Global Step: 22120 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:09:52,828-Speed 126.16 samples/sec Loss 12.6222 Epoch: 2 Global Step: 22125 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:10:13,252-Speed 125.35 samples/sec Loss 12.4686 Epoch: 2 Global Step: 22130 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:10:33,336-Speed 127.47 samples/sec Loss 12.3173 Epoch: 2 Global Step: 22135 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:10:53,483-Speed 127.07 samples/sec Loss 12.4987 Epoch: 2 Global Step: 22140 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:11:14,484-Speed 121.90 samples/sec Loss 12.3369 Epoch: 2 Global Step: 22145 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:11:34,848-Speed 125.72 samples/sec Loss 12.8500 Epoch: 2 Global Step: 22150 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:11:55,241-Speed 125.54 samples/sec Loss 12.5690 Epoch: 2 Global Step: 22155 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:12:15,608-Speed 125.70 samples/sec Loss 12.8327 Epoch: 2 Global Step: 22160 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:12:35,993-Speed 125.58 samples/sec Loss 12.4652 Epoch: 2 Global Step: 22165 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:12:56,342-Speed 125.81 samples/sec Loss 12.4024 Epoch: 2 Global Step: 22170 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:13:16,743-Speed 125.50 samples/sec Loss 12.6377 Epoch: 2 Global Step: 22175 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:13:37,414-Speed 123.85 samples/sec Loss 12.4634 Epoch: 2 Global Step: 22180 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:13:58,291-Speed 122.63 samples/sec Loss 12.5214 Epoch: 2 Global Step: 22185 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:14:19,281-Speed 121.96 samples/sec Loss 12.4881 Epoch: 2 Global Step: 22190 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:14:39,752-Speed 125.06 samples/sec Loss 12.4010 Epoch: 2 Global Step: 22195 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:14:59,876-Speed 127.22 samples/sec Loss 12.5039 Epoch: 2 Global Step: 22200 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:15:20,159-Speed 126.23 samples/sec Loss 12.5558 Epoch: 2 Global Step: 22205 Fp16 Grad Scale: 8192 Required: 268 hours Training: 2021-06-07 21:15:40,297-Speed 127.13 samples/sec Loss 12.2957 Epoch: 2 Global Step: 22210 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:16:00,631-Speed 125.91 samples/sec Loss 12.4399 Epoch: 2 Global Step: 22215 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:16:21,019-Speed 125.57 samples/sec Loss 12.6207 Epoch: 2 Global Step: 22220 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:16:41,159-Speed 127.12 samples/sec Loss 12.3272 Epoch: 2 Global Step: 22225 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:17:01,256-Speed 127.39 samples/sec Loss 12.3620 Epoch: 2 Global Step: 22230 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:17:22,243-Speed 121.99 samples/sec Loss 12.6427 Epoch: 2 Global Step: 22235 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:17:42,539-Speed 126.14 samples/sec Loss 12.4511 Epoch: 2 Global Step: 22240 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:18:02,888-Speed 125.81 samples/sec Loss 12.5962 Epoch: 2 Global Step: 22245 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:18:23,082-Speed 126.78 samples/sec Loss 12.6550 Epoch: 2 Global Step: 22250 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:18:43,109-Speed 127.83 samples/sec Loss 12.2333 Epoch: 2 Global Step: 22255 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:19:03,331-Speed 126.60 samples/sec Loss 12.4558 Epoch: 2 Global Step: 22260 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:19:23,655-Speed 125.97 samples/sec Loss 12.9587 Epoch: 2 Global Step: 22265 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:19:44,095-Speed 125.25 samples/sec Loss 12.5602 Epoch: 2 Global Step: 22270 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:20:04,402-Speed 126.07 samples/sec Loss 12.2495 Epoch: 2 Global Step: 22275 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:20:25,393-Speed 121.96 samples/sec Loss 12.5590 Epoch: 2 Global Step: 22280 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:20:45,901-Speed 124.83 samples/sec Loss 12.2338 Epoch: 2 Global Step: 22285 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:21:06,155-Speed 126.40 samples/sec Loss 12.6796 Epoch: 2 Global Step: 22290 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:21:26,468-Speed 126.04 samples/sec Loss 12.8186 Epoch: 2 Global Step: 22295 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:21:46,809-Speed 125.86 samples/sec Loss 12.5862 Epoch: 2 Global Step: 22300 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:22:07,127-Speed 126.00 samples/sec Loss 12.3356 Epoch: 2 Global Step: 22305 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:22:27,413-Speed 126.21 samples/sec Loss 12.3528 Epoch: 2 Global Step: 22310 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:22:47,739-Speed 125.95 samples/sec Loss 12.3271 Epoch: 2 Global Step: 22315 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:23:08,018-Speed 126.25 samples/sec Loss 12.6447 Epoch: 2 Global Step: 22320 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:23:28,500-Speed 124.99 samples/sec Loss 12.7614 Epoch: 2 Global Step: 22325 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:23:48,890-Speed 125.56 samples/sec Loss 12.9016 Epoch: 2 Global Step: 22330 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:24:09,589-Speed 123.68 samples/sec Loss 12.1956 Epoch: 2 Global Step: 22335 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:24:30,085-Speed 124.91 samples/sec Loss 12.3540 Epoch: 2 Global Step: 22340 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:24:50,329-Speed 126.46 samples/sec Loss 12.4829 Epoch: 2 Global Step: 22345 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:25:10,655-Speed 125.96 samples/sec Loss 12.6139 Epoch: 2 Global Step: 22350 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:25:31,046-Speed 125.55 samples/sec Loss 12.4745 Epoch: 2 Global Step: 22355 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:25:51,368-Speed 125.98 samples/sec Loss 12.6234 Epoch: 2 Global Step: 22360 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:26:11,710-Speed 125.86 samples/sec Loss 12.7744 Epoch: 2 Global Step: 22365 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:26:32,319-Speed 124.23 samples/sec Loss 12.7612 Epoch: 2 Global Step: 22370 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:26:52,476-Speed 127.01 samples/sec Loss 12.3614 Epoch: 2 Global Step: 22375 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:27:12,374-Speed 128.66 samples/sec Loss 12.2966 Epoch: 2 Global Step: 22380 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:27:32,837-Speed 125.11 samples/sec Loss 12.3705 Epoch: 2 Global Step: 22385 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:27:52,877-Speed 127.76 samples/sec Loss 12.6238 Epoch: 2 Global Step: 22390 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:28:12,848-Speed 128.19 samples/sec Loss 12.4596 Epoch: 2 Global Step: 22395 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:28:32,836-Speed 128.09 samples/sec Loss 12.1111 Epoch: 2 Global Step: 22400 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:28:53,100-Speed 126.34 samples/sec Loss 12.3375 Epoch: 2 Global Step: 22405 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:29:13,420-Speed 126.00 samples/sec Loss 12.8087 Epoch: 2 Global Step: 22410 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:29:34,280-Speed 122.73 samples/sec Loss 12.5296 Epoch: 2 Global Step: 22415 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:29:54,455-Speed 126.89 samples/sec Loss 12.1569 Epoch: 2 Global Step: 22420 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:30:14,744-Speed 126.18 samples/sec Loss 12.1570 Epoch: 2 Global Step: 22425 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:30:34,965-Speed 126.61 samples/sec Loss 12.4051 Epoch: 2 Global Step: 22430 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:30:55,598-Speed 124.08 samples/sec Loss 12.5014 Epoch: 2 Global Step: 22435 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:31:15,950-Speed 125.80 samples/sec Loss 12.5202 Epoch: 2 Global Step: 22440 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:31:36,877-Speed 122.34 samples/sec Loss 12.4625 Epoch: 2 Global Step: 22445 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:31:57,601-Speed 123.53 samples/sec Loss 12.3257 Epoch: 2 Global Step: 22450 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:32:17,819-Speed 126.63 samples/sec Loss 12.5028 Epoch: 2 Global Step: 22455 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:32:38,942-Speed 121.20 samples/sec Loss 12.3052 Epoch: 2 Global Step: 22460 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:32:59,083-Speed 127.11 samples/sec Loss 12.5415 Epoch: 2 Global Step: 22465 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:33:19,463-Speed 125.62 samples/sec Loss 12.4801 Epoch: 2 Global Step: 22470 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:33:39,398-Speed 128.42 samples/sec Loss 12.0101 Epoch: 2 Global Step: 22475 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:33:59,742-Speed 125.84 samples/sec Loss 12.4626 Epoch: 2 Global Step: 22480 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:34:19,840-Speed 127.38 samples/sec Loss 12.3195 Epoch: 2 Global Step: 22485 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:34:40,064-Speed 126.59 samples/sec Loss 12.2951 Epoch: 2 Global Step: 22490 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:35:00,172-Speed 127.32 samples/sec Loss 12.4664 Epoch: 2 Global Step: 22495 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:35:20,389-Speed 126.63 samples/sec Loss 12.4872 Epoch: 2 Global Step: 22500 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:35:41,319-Speed 122.32 samples/sec Loss 12.2944 Epoch: 2 Global Step: 22505 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:36:01,672-Speed 125.78 samples/sec Loss 12.0636 Epoch: 2 Global Step: 22510 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:36:22,015-Speed 125.84 samples/sec Loss 12.3796 Epoch: 2 Global Step: 22515 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:36:42,104-Speed 127.44 samples/sec Loss 12.3457 Epoch: 2 Global Step: 22520 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:37:02,494-Speed 125.56 samples/sec Loss 12.6253 Epoch: 2 Global Step: 22525 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:37:22,676-Speed 126.85 samples/sec Loss 12.2734 Epoch: 2 Global Step: 22530 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:37:42,966-Speed 126.18 samples/sec Loss 12.4972 Epoch: 2 Global Step: 22535 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:38:03,200-Speed 126.53 samples/sec Loss 12.2263 Epoch: 2 Global Step: 22540 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:38:23,453-Speed 126.41 samples/sec Loss 12.4213 Epoch: 2 Global Step: 22545 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:38:44,375-Speed 122.37 samples/sec Loss 12.4199 Epoch: 2 Global Step: 22550 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:39:04,664-Speed 126.18 samples/sec Loss 12.2606 Epoch: 2 Global Step: 22555 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:39:25,113-Speed 125.20 samples/sec Loss 12.5526 Epoch: 2 Global Step: 22560 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:39:45,418-Speed 126.09 samples/sec Loss 12.6331 Epoch: 2 Global Step: 22565 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:40:05,741-Speed 125.97 samples/sec Loss 12.7918 Epoch: 2 Global Step: 22570 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:40:26,178-Speed 125.27 samples/sec Loss 12.3266 Epoch: 2 Global Step: 22575 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:40:46,217-Speed 127.76 samples/sec Loss 12.2717 Epoch: 2 Global Step: 22580 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:41:06,510-Speed 126.16 samples/sec Loss 12.5756 Epoch: 2 Global Step: 22585 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:41:27,528-Speed 121.81 samples/sec Loss 12.5657 Epoch: 2 Global Step: 22590 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:41:48,222-Speed 123.71 samples/sec Loss 12.5959 Epoch: 2 Global Step: 22595 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:42:08,350-Speed 127.19 samples/sec Loss 12.2312 Epoch: 2 Global Step: 22600 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:42:28,657-Speed 126.07 samples/sec Loss 12.3173 Epoch: 2 Global Step: 22605 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:42:49,003-Speed 125.83 samples/sec Loss 12.9312 Epoch: 2 Global Step: 22610 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:43:08,887-Speed 128.75 samples/sec Loss 12.4825 Epoch: 2 Global Step: 22615 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:43:29,204-Speed 126.01 samples/sec Loss 12.5213 Epoch: 2 Global Step: 22620 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:43:49,625-Speed 125.37 samples/sec Loss 12.5996 Epoch: 2 Global Step: 22625 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:44:09,873-Speed 126.44 samples/sec Loss 12.2047 Epoch: 2 Global Step: 22630 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:44:30,036-Speed 126.98 samples/sec Loss 12.2977 Epoch: 2 Global Step: 22635 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:44:51,187-Speed 121.04 samples/sec Loss 12.4722 Epoch: 2 Global Step: 22640 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:45:11,427-Speed 126.48 samples/sec Loss 12.2538 Epoch: 2 Global Step: 22645 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:45:31,902-Speed 125.04 samples/sec Loss 12.5673 Epoch: 2 Global Step: 22650 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:45:51,987-Speed 127.47 samples/sec Loss 12.5218 Epoch: 2 Global Step: 22655 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:46:12,081-Speed 127.41 samples/sec Loss 12.5505 Epoch: 2 Global Step: 22660 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:46:32,233-Speed 127.04 samples/sec Loss 12.4538 Epoch: 2 Global Step: 22665 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:46:52,654-Speed 125.37 samples/sec Loss 12.2734 Epoch: 2 Global Step: 22670 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:47:13,067-Speed 125.42 samples/sec Loss 12.3152 Epoch: 2 Global Step: 22675 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:47:33,368-Speed 126.12 samples/sec Loss 12.4324 Epoch: 2 Global Step: 22680 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:47:54,649-Speed 120.30 samples/sec Loss 12.7759 Epoch: 2 Global Step: 22685 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:48:14,576-Speed 128.47 samples/sec Loss 12.1471 Epoch: 2 Global Step: 22690 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:48:34,891-Speed 126.02 samples/sec Loss 12.5737 Epoch: 2 Global Step: 22695 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:48:55,002-Speed 127.30 samples/sec Loss 12.7172 Epoch: 2 Global Step: 22700 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:49:15,199-Speed 126.76 samples/sec Loss 12.4821 Epoch: 2 Global Step: 22705 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:49:35,358-Speed 127.00 samples/sec Loss 12.4110 Epoch: 2 Global Step: 22710 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:49:55,679-Speed 125.99 samples/sec Loss 12.2165 Epoch: 2 Global Step: 22715 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:50:16,100-Speed 125.37 samples/sec Loss 12.7073 Epoch: 2 Global Step: 22720 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:50:36,435-Speed 125.90 samples/sec Loss 12.6296 Epoch: 2 Global Step: 22725 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:50:57,467-Speed 121.72 samples/sec Loss 12.4069 Epoch: 2 Global Step: 22730 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:51:18,122-Speed 123.95 samples/sec Loss 12.5483 Epoch: 2 Global Step: 22735 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:51:38,260-Speed 127.13 samples/sec Loss 12.6260 Epoch: 2 Global Step: 22740 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:51:58,475-Speed 126.65 samples/sec Loss 12.4486 Epoch: 2 Global Step: 22745 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:52:18,799-Speed 125.97 samples/sec Loss 12.5098 Epoch: 2 Global Step: 22750 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:52:39,303-Speed 124.86 samples/sec Loss 12.7689 Epoch: 2 Global Step: 22755 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:52:59,510-Speed 126.69 samples/sec Loss 12.6448 Epoch: 2 Global Step: 22760 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:53:19,751-Speed 126.49 samples/sec Loss 12.2348 Epoch: 2 Global Step: 22765 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:53:39,854-Speed 127.35 samples/sec Loss 11.9382 Epoch: 2 Global Step: 22770 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:54:01,055-Speed 120.75 samples/sec Loss 12.5164 Epoch: 2 Global Step: 22775 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:54:21,495-Speed 125.25 samples/sec Loss 12.6855 Epoch: 2 Global Step: 22780 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:54:42,541-Speed 121.65 samples/sec Loss 12.2756 Epoch: 2 Global Step: 22785 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:55:02,933-Speed 125.54 samples/sec Loss 12.3189 Epoch: 2 Global Step: 22790 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:55:23,125-Speed 126.79 samples/sec Loss 12.4025 Epoch: 2 Global Step: 22795 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:55:43,132-Speed 127.96 samples/sec Loss 12.4952 Epoch: 2 Global Step: 22800 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:56:03,617-Speed 124.97 samples/sec Loss 12.5536 Epoch: 2 Global Step: 22805 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:56:23,824-Speed 126.70 samples/sec Loss 12.5295 Epoch: 2 Global Step: 22810 Fp16 Grad Scale: 8192 Required: 267 hours Training: 2021-06-07 21:56:44,049-Speed 126.58 samples/sec Loss 12.4706 Epoch: 2 Global Step: 22815 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 21:57:05,057-Speed 121.87 samples/sec Loss 12.5483 Epoch: 2 Global Step: 22820 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 21:57:25,431-Speed 125.65 samples/sec Loss 12.7865 Epoch: 2 Global Step: 22825 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 21:57:45,629-Speed 126.75 samples/sec Loss 12.6400 Epoch: 2 Global Step: 22830 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 21:58:05,985-Speed 125.77 samples/sec Loss 12.4436 Epoch: 2 Global Step: 22835 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 21:58:26,338-Speed 125.78 samples/sec Loss 12.6576 Epoch: 2 Global Step: 22840 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 21:58:46,618-Speed 126.24 samples/sec Loss 12.4536 Epoch: 2 Global Step: 22845 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 21:59:06,972-Speed 125.78 samples/sec Loss 12.5048 Epoch: 2 Global Step: 22850 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 21:59:27,369-Speed 125.51 samples/sec Loss 12.2808 Epoch: 2 Global Step: 22855 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 21:59:47,581-Speed 126.67 samples/sec Loss 12.5465 Epoch: 2 Global Step: 22860 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:00:08,508-Speed 122.34 samples/sec Loss 12.3521 Epoch: 2 Global Step: 22865 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:00:28,615-Speed 127.32 samples/sec Loss 12.3636 Epoch: 2 Global Step: 22870 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:00:48,922-Speed 126.07 samples/sec Loss 12.3523 Epoch: 2 Global Step: 22875 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:01:09,290-Speed 125.70 samples/sec Loss 12.4610 Epoch: 2 Global Step: 22880 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:01:29,677-Speed 125.58 samples/sec Loss 12.7868 Epoch: 2 Global Step: 22885 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:01:49,795-Speed 127.26 samples/sec Loss 12.2902 Epoch: 2 Global Step: 22890 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:02:10,020-Speed 126.58 samples/sec Loss 12.4691 Epoch: 2 Global Step: 22895 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:02:30,240-Speed 126.61 samples/sec Loss 12.3624 Epoch: 2 Global Step: 22900 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:02:50,568-Speed 125.94 samples/sec Loss 12.3039 Epoch: 2 Global Step: 22905 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:03:11,363-Speed 123.11 samples/sec Loss 12.5731 Epoch: 2 Global Step: 22910 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:03:31,586-Speed 126.60 samples/sec Loss 12.4392 Epoch: 2 Global Step: 22915 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:03:52,120-Speed 124.68 samples/sec Loss 12.3839 Epoch: 2 Global Step: 22920 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:04:12,456-Speed 125.89 samples/sec Loss 12.6199 Epoch: 2 Global Step: 22925 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:04:32,840-Speed 125.59 samples/sec Loss 12.4021 Epoch: 2 Global Step: 22930 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:04:53,259-Speed 125.38 samples/sec Loss 12.5934 Epoch: 2 Global Step: 22935 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:05:13,576-Speed 126.01 samples/sec Loss 12.2062 Epoch: 2 Global Step: 22940 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:05:33,884-Speed 126.07 samples/sec Loss 12.3852 Epoch: 2 Global Step: 22945 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:05:54,104-Speed 126.61 samples/sec Loss 12.5030 Epoch: 2 Global Step: 22950 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:06:15,129-Speed 121.77 samples/sec Loss 12.8530 Epoch: 2 Global Step: 22955 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:06:35,425-Speed 126.14 samples/sec Loss 12.6666 Epoch: 2 Global Step: 22960 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:06:55,899-Speed 125.04 samples/sec Loss 12.5814 Epoch: 2 Global Step: 22965 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:07:16,350-Speed 125.19 samples/sec Loss 12.5443 Epoch: 2 Global Step: 22970 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:07:36,605-Speed 126.39 samples/sec Loss 12.5619 Epoch: 2 Global Step: 22975 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:07:56,951-Speed 125.83 samples/sec Loss 12.8090 Epoch: 2 Global Step: 22980 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:08:17,439-Speed 124.96 samples/sec Loss 12.7080 Epoch: 2 Global Step: 22985 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:08:37,831-Speed 125.55 samples/sec Loss 12.3842 Epoch: 2 Global Step: 22990 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:08:58,110-Speed 126.25 samples/sec Loss 12.4298 Epoch: 2 Global Step: 22995 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:09:19,022-Speed 122.42 samples/sec Loss 12.4207 Epoch: 2 Global Step: 23000 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:09:39,203-Speed 126.86 samples/sec Loss 11.9864 Epoch: 2 Global Step: 23005 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:09:59,514-Speed 126.04 samples/sec Loss 12.4098 Epoch: 2 Global Step: 23010 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:10:19,844-Speed 125.93 samples/sec Loss 11.9762 Epoch: 2 Global Step: 23015 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:10:40,167-Speed 125.97 samples/sec Loss 12.4912 Epoch: 2 Global Step: 23020 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:11:00,510-Speed 125.85 samples/sec Loss 12.4391 Epoch: 2 Global Step: 23025 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:11:20,850-Speed 125.87 samples/sec Loss 12.5277 Epoch: 2 Global Step: 23030 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:11:41,598-Speed 123.39 samples/sec Loss 12.3566 Epoch: 2 Global Step: 23035 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:12:01,743-Speed 127.09 samples/sec Loss 12.6491 Epoch: 2 Global Step: 23040 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:12:22,552-Speed 123.03 samples/sec Loss 12.3744 Epoch: 2 Global Step: 23045 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:12:42,755-Speed 126.72 samples/sec Loss 12.5760 Epoch: 2 Global Step: 23050 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:13:03,162-Speed 125.46 samples/sec Loss 12.6979 Epoch: 2 Global Step: 23055 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:13:23,600-Speed 125.26 samples/sec Loss 12.4259 Epoch: 2 Global Step: 23060 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:13:44,111-Speed 124.82 samples/sec Loss 12.7102 Epoch: 2 Global Step: 23065 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:14:04,435-Speed 125.97 samples/sec Loss 12.4494 Epoch: 2 Global Step: 23070 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:14:24,778-Speed 125.85 samples/sec Loss 12.8252 Epoch: 2 Global Step: 23075 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:14:45,771-Speed 121.95 samples/sec Loss 12.4897 Epoch: 2 Global Step: 23080 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:15:06,056-Speed 126.21 samples/sec Loss 12.4161 Epoch: 2 Global Step: 23085 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:15:27,116-Speed 121.56 samples/sec Loss 12.3282 Epoch: 2 Global Step: 23090 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:15:47,525-Speed 125.44 samples/sec Loss 12.4625 Epoch: 2 Global Step: 23095 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:16:07,810-Speed 126.21 samples/sec Loss 12.4844 Epoch: 2 Global Step: 23100 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:16:28,129-Speed 125.99 samples/sec Loss 12.3989 Epoch: 2 Global Step: 23105 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:16:48,762-Speed 124.08 samples/sec Loss 12.6818 Epoch: 2 Global Step: 23110 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:17:09,278-Speed 124.79 samples/sec Loss 12.5720 Epoch: 2 Global Step: 23115 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:17:29,922-Speed 124.01 samples/sec Loss 12.6396 Epoch: 2 Global Step: 23120 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:17:50,108-Speed 126.83 samples/sec Loss 12.4759 Epoch: 2 Global Step: 23125 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:18:10,226-Speed 127.26 samples/sec Loss 12.3255 Epoch: 2 Global Step: 23130 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:18:31,203-Speed 122.04 samples/sec Loss 12.4942 Epoch: 2 Global Step: 23135 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:18:51,617-Speed 125.41 samples/sec Loss 12.4749 Epoch: 2 Global Step: 23140 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:19:12,235-Speed 124.17 samples/sec Loss 13.0078 Epoch: 2 Global Step: 23145 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:19:32,620-Speed 125.59 samples/sec Loss 12.5956 Epoch: 2 Global Step: 23150 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:19:52,975-Speed 125.77 samples/sec Loss 12.7833 Epoch: 2 Global Step: 23155 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:20:13,298-Speed 125.98 samples/sec Loss 12.8375 Epoch: 2 Global Step: 23160 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:20:33,715-Speed 125.39 samples/sec Loss 12.7142 Epoch: 2 Global Step: 23165 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:20:54,112-Speed 125.52 samples/sec Loss 12.4734 Epoch: 2 Global Step: 23170 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:21:15,003-Speed 122.55 samples/sec Loss 11.9761 Epoch: 2 Global Step: 23175 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:21:35,878-Speed 122.65 samples/sec Loss 12.6529 Epoch: 2 Global Step: 23180 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:21:56,061-Speed 126.84 samples/sec Loss 12.4754 Epoch: 2 Global Step: 23185 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:22:16,301-Speed 126.49 samples/sec Loss 12.2472 Epoch: 2 Global Step: 23190 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:22:36,624-Speed 125.97 samples/sec Loss 12.6677 Epoch: 2 Global Step: 23195 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:22:57,042-Speed 125.39 samples/sec Loss 12.3583 Epoch: 2 Global Step: 23200 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:23:17,372-Speed 125.93 samples/sec Loss 12.4309 Epoch: 2 Global Step: 23205 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:23:37,739-Speed 125.71 samples/sec Loss 12.8079 Epoch: 2 Global Step: 23210 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:23:57,825-Speed 127.46 samples/sec Loss 12.4780 Epoch: 2 Global Step: 23215 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:24:18,086-Speed 126.36 samples/sec Loss 12.3094 Epoch: 2 Global Step: 23220 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:24:38,934-Speed 122.80 samples/sec Loss 12.7007 Epoch: 2 Global Step: 23225 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:24:59,080-Speed 127.08 samples/sec Loss 12.4541 Epoch: 2 Global Step: 23230 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:25:19,183-Speed 127.35 samples/sec Loss 12.2399 Epoch: 2 Global Step: 23235 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:25:39,222-Speed 127.76 samples/sec Loss 12.4394 Epoch: 2 Global Step: 23240 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:25:59,316-Speed 127.41 samples/sec Loss 12.4023 Epoch: 2 Global Step: 23245 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:26:19,559-Speed 126.47 samples/sec Loss 12.4784 Epoch: 2 Global Step: 23250 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:26:39,882-Speed 125.97 samples/sec Loss 12.1593 Epoch: 2 Global Step: 23255 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:27:00,103-Speed 126.61 samples/sec Loss 12.4576 Epoch: 2 Global Step: 23260 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:27:20,305-Speed 126.73 samples/sec Loss 12.5095 Epoch: 2 Global Step: 23265 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:27:41,400-Speed 121.36 samples/sec Loss 12.6472 Epoch: 2 Global Step: 23270 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:28:01,729-Speed 125.93 samples/sec Loss 12.8167 Epoch: 2 Global Step: 23275 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:28:22,079-Speed 125.80 samples/sec Loss 12.6381 Epoch: 2 Global Step: 23280 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:28:42,382-Speed 126.10 samples/sec Loss 12.2430 Epoch: 2 Global Step: 23285 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:29:02,605-Speed 126.60 samples/sec Loss 12.4809 Epoch: 2 Global Step: 23290 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:29:22,823-Speed 126.63 samples/sec Loss 12.1688 Epoch: 2 Global Step: 23295 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:29:43,209-Speed 125.59 samples/sec Loss 12.4307 Epoch: 2 Global Step: 23300 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:30:03,343-Speed 127.16 samples/sec Loss 12.3142 Epoch: 2 Global Step: 23305 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:30:23,641-Speed 126.13 samples/sec Loss 12.7709 Epoch: 2 Global Step: 23310 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:30:44,803-Speed 120.98 samples/sec Loss 12.7430 Epoch: 2 Global Step: 23315 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:31:05,777-Speed 122.06 samples/sec Loss 12.5688 Epoch: 2 Global Step: 23320 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:31:26,112-Speed 125.90 samples/sec Loss 12.2248 Epoch: 2 Global Step: 23325 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:31:46,444-Speed 125.92 samples/sec Loss 12.3349 Epoch: 2 Global Step: 23330 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:32:06,731-Speed 126.20 samples/sec Loss 12.3115 Epoch: 2 Global Step: 23335 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:32:27,061-Speed 125.93 samples/sec Loss 12.1353 Epoch: 2 Global Step: 23340 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:32:47,034-Speed 128.18 samples/sec Loss 12.3629 Epoch: 2 Global Step: 23345 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:33:07,385-Speed 125.80 samples/sec Loss 12.3810 Epoch: 2 Global Step: 23350 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:33:27,953-Speed 124.48 samples/sec Loss 12.4093 Epoch: 2 Global Step: 23355 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:33:48,963-Speed 121.85 samples/sec Loss 12.3108 Epoch: 2 Global Step: 23360 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:34:09,274-Speed 126.05 samples/sec Loss 12.1950 Epoch: 2 Global Step: 23365 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:34:29,595-Speed 125.98 samples/sec Loss 12.4839 Epoch: 2 Global Step: 23370 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:34:49,963-Speed 125.69 samples/sec Loss 12.0580 Epoch: 2 Global Step: 23375 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:35:10,225-Speed 126.35 samples/sec Loss 12.6635 Epoch: 2 Global Step: 23380 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:35:30,569-Speed 125.85 samples/sec Loss 12.5258 Epoch: 2 Global Step: 23385 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:35:50,999-Speed 125.31 samples/sec Loss 12.3279 Epoch: 2 Global Step: 23390 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:36:11,449-Speed 125.19 samples/sec Loss 12.3128 Epoch: 2 Global Step: 23395 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:36:31,734-Speed 126.21 samples/sec Loss 12.5239 Epoch: 2 Global Step: 23400 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:36:52,672-Speed 122.27 samples/sec Loss 12.5700 Epoch: 2 Global Step: 23405 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:37:12,808-Speed 127.14 samples/sec Loss 12.5400 Epoch: 2 Global Step: 23410 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:37:33,228-Speed 125.37 samples/sec Loss 12.6926 Epoch: 2 Global Step: 23415 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:37:53,763-Speed 124.67 samples/sec Loss 12.4309 Epoch: 2 Global Step: 23420 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:38:13,954-Speed 126.80 samples/sec Loss 12.5298 Epoch: 2 Global Step: 23425 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:38:34,306-Speed 125.79 samples/sec Loss 12.3844 Epoch: 2 Global Step: 23430 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:38:54,654-Speed 125.82 samples/sec Loss 12.4921 Epoch: 2 Global Step: 23435 Fp16 Grad Scale: 8192 Required: 266 hours Training: 2021-06-07 22:39:15,032-Speed 125.63 samples/sec Loss 12.4733 Epoch: 2 Global Step: 23440 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:39:35,439-Speed 125.46 samples/sec Loss 12.7889 Epoch: 2 Global Step: 23445 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:39:56,185-Speed 123.40 samples/sec Loss 12.2428 Epoch: 2 Global Step: 23450 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:40:16,488-Speed 126.10 samples/sec Loss 12.4042 Epoch: 2 Global Step: 23455 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:40:36,563-Speed 127.52 samples/sec Loss 11.8511 Epoch: 2 Global Step: 23460 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:40:56,757-Speed 126.78 samples/sec Loss 12.5357 Epoch: 2 Global Step: 23465 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:41:16,996-Speed 126.50 samples/sec Loss 12.5572 Epoch: 2 Global Step: 23470 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:41:37,417-Speed 125.37 samples/sec Loss 12.6349 Epoch: 2 Global Step: 23475 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:41:57,806-Speed 125.56 samples/sec Loss 12.5965 Epoch: 2 Global Step: 23480 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:42:18,100-Speed 126.15 samples/sec Loss 12.5102 Epoch: 2 Global Step: 23485 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:42:38,453-Speed 125.79 samples/sec Loss 12.1306 Epoch: 2 Global Step: 23490 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:42:59,544-Speed 121.38 samples/sec Loss 12.4297 Epoch: 2 Global Step: 23495 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:43:20,223-Speed 123.80 samples/sec Loss 12.6381 Epoch: 2 Global Step: 23500 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:43:41,001-Speed 123.21 samples/sec Loss 12.3574 Epoch: 2 Global Step: 23505 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:44:01,632-Speed 124.09 samples/sec Loss 12.4552 Epoch: 2 Global Step: 23510 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:44:21,867-Speed 126.52 samples/sec Loss 12.4295 Epoch: 2 Global Step: 23515 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:44:42,179-Speed 126.04 samples/sec Loss 11.8817 Epoch: 2 Global Step: 23520 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:45:02,525-Speed 125.83 samples/sec Loss 12.3683 Epoch: 2 Global Step: 23525 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:45:22,880-Speed 125.77 samples/sec Loss 12.3515 Epoch: 2 Global Step: 23530 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:45:43,386-Speed 124.85 samples/sec Loss 12.6038 Epoch: 2 Global Step: 23535 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:46:04,180-Speed 123.11 samples/sec Loss 12.2035 Epoch: 2 Global Step: 23540 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:46:24,925-Speed 123.41 samples/sec Loss 12.4904 Epoch: 2 Global Step: 23545 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:46:45,326-Speed 125.49 samples/sec Loss 12.6651 Epoch: 2 Global Step: 23550 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:47:05,428-Speed 127.36 samples/sec Loss 12.3987 Epoch: 2 Global Step: 23555 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:47:25,839-Speed 125.43 samples/sec Loss 12.4248 Epoch: 2 Global Step: 23560 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:47:46,270-Speed 125.31 samples/sec Loss 12.2992 Epoch: 2 Global Step: 23565 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:48:06,497-Speed 126.58 samples/sec Loss 12.1796 Epoch: 2 Global Step: 23570 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:48:26,765-Speed 126.31 samples/sec Loss 12.2085 Epoch: 2 Global Step: 23575 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:48:47,140-Speed 125.65 samples/sec Loss 12.6280 Epoch: 2 Global Step: 23580 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:49:07,908-Speed 123.27 samples/sec Loss 12.1553 Epoch: 2 Global Step: 23585 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:49:29,207-Speed 120.20 samples/sec Loss 12.4190 Epoch: 2 Global Step: 23590 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:49:49,419-Speed 126.67 samples/sec Loss 12.6152 Epoch: 2 Global Step: 23595 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:50:09,751-Speed 125.92 samples/sec Loss 12.4077 Epoch: 2 Global Step: 23600 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:50:30,170-Speed 125.38 samples/sec Loss 12.2858 Epoch: 2 Global Step: 23605 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:50:50,484-Speed 126.03 samples/sec Loss 12.3970 Epoch: 2 Global Step: 23610 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:51:10,823-Speed 125.87 samples/sec Loss 12.7784 Epoch: 2 Global Step: 23615 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:51:31,054-Speed 126.55 samples/sec Loss 12.1169 Epoch: 2 Global Step: 23620 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:51:51,175-Speed 127.24 samples/sec Loss 12.2149 Epoch: 2 Global Step: 23625 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:52:12,139-Speed 122.13 samples/sec Loss 12.5636 Epoch: 2 Global Step: 23630 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:52:32,939-Speed 123.08 samples/sec Loss 13.0247 Epoch: 2 Global Step: 23635 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:52:53,344-Speed 125.47 samples/sec Loss 12.5302 Epoch: 2 Global Step: 23640 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:53:13,471-Speed 127.20 samples/sec Loss 12.2594 Epoch: 2 Global Step: 23645 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:53:33,690-Speed 126.63 samples/sec Loss 12.3256 Epoch: 2 Global Step: 23650 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:53:54,465-Speed 123.23 samples/sec Loss 12.2467 Epoch: 2 Global Step: 23655 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:54:14,643-Speed 126.87 samples/sec Loss 12.3611 Epoch: 2 Global Step: 23660 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:54:35,004-Speed 125.74 samples/sec Loss 12.3949 Epoch: 2 Global Step: 23665 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:54:55,368-Speed 125.72 samples/sec Loss 12.3493 Epoch: 2 Global Step: 23670 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:55:16,490-Speed 121.20 samples/sec Loss 12.7465 Epoch: 2 Global Step: 23675 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:55:36,692-Speed 126.73 samples/sec Loss 12.0884 Epoch: 2 Global Step: 23680 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:55:56,625-Speed 128.43 samples/sec Loss 12.1269 Epoch: 2 Global Step: 23685 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:56:17,025-Speed 125.49 samples/sec Loss 11.9907 Epoch: 2 Global Step: 23690 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:56:37,322-Speed 126.14 samples/sec Loss 12.2622 Epoch: 2 Global Step: 23695 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:56:57,536-Speed 126.65 samples/sec Loss 11.8803 Epoch: 2 Global Step: 23700 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:57:17,712-Speed 126.89 samples/sec Loss 11.9914 Epoch: 2 Global Step: 23705 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:57:37,814-Speed 127.36 samples/sec Loss 12.3865 Epoch: 2 Global Step: 23710 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:57:58,307-Speed 124.93 samples/sec Loss 12.3565 Epoch: 2 Global Step: 23715 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:58:19,389-Speed 121.44 samples/sec Loss 12.0223 Epoch: 2 Global Step: 23720 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:58:39,739-Speed 125.80 samples/sec Loss 12.2947 Epoch: 2 Global Step: 23725 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:59:00,122-Speed 125.60 samples/sec Loss 12.3408 Epoch: 2 Global Step: 23730 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:59:20,478-Speed 125.77 samples/sec Loss 12.2975 Epoch: 2 Global Step: 23735 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 22:59:40,914-Speed 125.28 samples/sec Loss 12.3043 Epoch: 2 Global Step: 23740 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:00:01,475-Speed 124.51 samples/sec Loss 12.5354 Epoch: 2 Global Step: 23745 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:00:21,860-Speed 125.61 samples/sec Loss 12.3175 Epoch: 2 Global Step: 23750 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:00:41,945-Speed 127.46 samples/sec Loss 12.1468 Epoch: 2 Global Step: 23755 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:01:02,263-Speed 126.01 samples/sec Loss 12.6257 Epoch: 2 Global Step: 23760 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:01:23,406-Speed 121.09 samples/sec Loss 12.2780 Epoch: 2 Global Step: 23765 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:01:44,079-Speed 123.84 samples/sec Loss 12.4309 Epoch: 2 Global Step: 23770 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:02:04,036-Speed 128.28 samples/sec Loss 12.3644 Epoch: 2 Global Step: 23775 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:02:24,352-Speed 126.02 samples/sec Loss 12.6144 Epoch: 2 Global Step: 23780 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:02:44,553-Speed 126.73 samples/sec Loss 12.4911 Epoch: 2 Global Step: 23785 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:03:04,690-Speed 127.13 samples/sec Loss 12.4174 Epoch: 2 Global Step: 23790 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:03:24,960-Speed 126.31 samples/sec Loss 11.9739 Epoch: 2 Global Step: 23795 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:03:45,173-Speed 126.66 samples/sec Loss 12.2705 Epoch: 2 Global Step: 23800 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:04:05,288-Speed 127.28 samples/sec Loss 12.3936 Epoch: 2 Global Step: 23805 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:04:26,013-Speed 123.53 samples/sec Loss 12.5449 Epoch: 2 Global Step: 23810 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:04:46,362-Speed 125.81 samples/sec Loss 12.3678 Epoch: 2 Global Step: 23815 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:05:06,356-Speed 128.04 samples/sec Loss 12.4962 Epoch: 2 Global Step: 23820 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:05:26,577-Speed 126.60 samples/sec Loss 12.3692 Epoch: 2 Global Step: 23825 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:05:47,017-Speed 125.25 samples/sec Loss 12.4679 Epoch: 2 Global Step: 23830 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:06:07,072-Speed 127.66 samples/sec Loss 12.2280 Epoch: 2 Global Step: 23835 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:06:27,343-Speed 126.29 samples/sec Loss 12.4498 Epoch: 2 Global Step: 23840 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:06:47,792-Speed 125.20 samples/sec Loss 12.4492 Epoch: 2 Global Step: 23845 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:07:08,160-Speed 125.70 samples/sec Loss 12.1136 Epoch: 2 Global Step: 23850 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:07:29,311-Speed 121.04 samples/sec Loss 12.2062 Epoch: 2 Global Step: 23855 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:07:49,825-Speed 124.79 samples/sec Loss 12.5664 Epoch: 2 Global Step: 23860 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:08:10,067-Speed 126.48 samples/sec Loss 12.5004 Epoch: 2 Global Step: 23865 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:08:30,295-Speed 126.57 samples/sec Loss 12.4614 Epoch: 2 Global Step: 23870 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:08:50,643-Speed 125.81 samples/sec Loss 12.3485 Epoch: 2 Global Step: 23875 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:09:10,966-Speed 125.98 samples/sec Loss 12.4097 Epoch: 2 Global Step: 23880 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:09:31,386-Speed 125.38 samples/sec Loss 12.1947 Epoch: 2 Global Step: 23885 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:09:51,645-Speed 126.37 samples/sec Loss 12.4791 Epoch: 2 Global Step: 23890 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:10:12,197-Speed 124.57 samples/sec Loss 12.2618 Epoch: 2 Global Step: 23895 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:10:33,165-Speed 122.09 samples/sec Loss 12.4325 Epoch: 2 Global Step: 23900 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:10:53,415-Speed 126.43 samples/sec Loss 12.3870 Epoch: 2 Global Step: 23905 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:11:13,621-Speed 126.70 samples/sec Loss 12.2169 Epoch: 2 Global Step: 23910 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:11:33,990-Speed 125.69 samples/sec Loss 12.2612 Epoch: 2 Global Step: 23915 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:11:54,330-Speed 125.87 samples/sec Loss 12.4928 Epoch: 2 Global Step: 23920 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:12:14,458-Speed 127.20 samples/sec Loss 12.3375 Epoch: 2 Global Step: 23925 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:12:34,701-Speed 126.47 samples/sec Loss 12.4877 Epoch: 2 Global Step: 23930 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:12:55,094-Speed 125.54 samples/sec Loss 11.8474 Epoch: 2 Global Step: 23935 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:13:15,277-Speed 126.85 samples/sec Loss 12.5315 Epoch: 2 Global Step: 23940 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:13:36,466-Speed 120.82 samples/sec Loss 12.3320 Epoch: 2 Global Step: 23945 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:13:56,470-Speed 127.98 samples/sec Loss 12.1923 Epoch: 2 Global Step: 23950 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:14:16,864-Speed 125.53 samples/sec Loss 12.4257 Epoch: 2 Global Step: 23955 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:14:37,118-Speed 126.41 samples/sec Loss 12.2624 Epoch: 2 Global Step: 23960 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:14:57,330-Speed 126.66 samples/sec Loss 12.3886 Epoch: 2 Global Step: 23965 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:15:17,640-Speed 126.06 samples/sec Loss 12.3043 Epoch: 2 Global Step: 23970 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:15:38,116-Speed 125.03 samples/sec Loss 12.6657 Epoch: 2 Global Step: 23975 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:15:58,512-Speed 125.52 samples/sec Loss 12.4515 Epoch: 2 Global Step: 23980 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:16:18,780-Speed 126.31 samples/sec Loss 12.5037 Epoch: 2 Global Step: 23985 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:16:39,938-Speed 121.00 samples/sec Loss 12.1974 Epoch: 2 Global Step: 23990 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:17:00,098-Speed 126.99 samples/sec Loss 12.8716 Epoch: 2 Global Step: 23995 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:17:20,157-Speed 127.63 samples/sec Loss 12.1521 Epoch: 2 Global Step: 24000 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:17:40,335-Speed 126.88 samples/sec Loss 12.1465 Epoch: 2 Global Step: 24005 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:18:00,629-Speed 126.15 samples/sec Loss 12.3699 Epoch: 2 Global Step: 24010 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:18:20,900-Speed 126.29 samples/sec Loss 12.3311 Epoch: 2 Global Step: 24015 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:18:41,249-Speed 125.82 samples/sec Loss 12.2103 Epoch: 2 Global Step: 24020 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:19:01,386-Speed 127.13 samples/sec Loss 12.7442 Epoch: 2 Global Step: 24025 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:19:21,762-Speed 125.65 samples/sec Loss 12.5258 Epoch: 2 Global Step: 24030 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:19:42,615-Speed 122.77 samples/sec Loss 12.5480 Epoch: 2 Global Step: 24035 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:20:02,725-Speed 127.30 samples/sec Loss 12.1846 Epoch: 2 Global Step: 24040 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:20:23,065-Speed 125.87 samples/sec Loss 12.4790 Epoch: 2 Global Step: 24045 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:20:43,281-Speed 126.64 samples/sec Loss 12.2140 Epoch: 2 Global Step: 24050 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:21:03,877-Speed 124.30 samples/sec Loss 12.3878 Epoch: 2 Global Step: 24055 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:21:24,155-Speed 126.25 samples/sec Loss 12.2772 Epoch: 2 Global Step: 24060 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:21:44,594-Speed 125.26 samples/sec Loss 12.2479 Epoch: 2 Global Step: 24065 Fp16 Grad Scale: 8192 Required: 265 hours Training: 2021-06-07 23:22:04,911-Speed 126.01 samples/sec Loss 12.6870 Epoch: 2 Global Step: 24070 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:22:25,553-Speed 124.02 samples/sec Loss 12.4828 Epoch: 2 Global Step: 24075 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:22:46,573-Speed 121.80 samples/sec Loss 12.6864 Epoch: 2 Global Step: 24080 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:23:06,724-Speed 127.04 samples/sec Loss 12.3733 Epoch: 2 Global Step: 24085 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:23:26,989-Speed 126.33 samples/sec Loss 12.6755 Epoch: 2 Global Step: 24090 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:23:47,204-Speed 126.65 samples/sec Loss 12.4324 Epoch: 2 Global Step: 24095 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:24:07,488-Speed 126.21 samples/sec Loss 12.3068 Epoch: 2 Global Step: 24100 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:24:27,833-Speed 125.84 samples/sec Loss 12.6554 Epoch: 2 Global Step: 24105 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:24:48,270-Speed 125.27 samples/sec Loss 12.4052 Epoch: 2 Global Step: 24110 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:25:08,935-Speed 123.89 samples/sec Loss 12.3411 Epoch: 2 Global Step: 24115 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:25:29,282-Speed 125.82 samples/sec Loss 11.7833 Epoch: 2 Global Step: 24120 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:25:50,183-Speed 122.49 samples/sec Loss 12.5717 Epoch: 2 Global Step: 24125 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:26:10,314-Speed 127.17 samples/sec Loss 12.5687 Epoch: 2 Global Step: 24130 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:26:30,751-Speed 125.27 samples/sec Loss 12.7724 Epoch: 2 Global Step: 24135 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:26:50,915-Speed 126.97 samples/sec Loss 12.4819 Epoch: 2 Global Step: 24140 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:27:11,208-Speed 126.16 samples/sec Loss 12.7376 Epoch: 2 Global Step: 24145 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:27:31,734-Speed 124.73 samples/sec Loss 12.3041 Epoch: 2 Global Step: 24150 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:27:52,144-Speed 125.43 samples/sec Loss 12.7500 Epoch: 2 Global Step: 24155 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:28:12,428-Speed 126.22 samples/sec Loss 12.1656 Epoch: 2 Global Step: 24160 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:28:32,797-Speed 125.68 samples/sec Loss 12.3875 Epoch: 2 Global Step: 24165 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:28:53,906-Speed 121.28 samples/sec Loss 12.3942 Epoch: 2 Global Step: 24170 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:29:14,282-Speed 125.65 samples/sec Loss 12.2130 Epoch: 2 Global Step: 24175 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:29:34,482-Speed 126.73 samples/sec Loss 12.1974 Epoch: 2 Global Step: 24180 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:29:54,760-Speed 126.25 samples/sec Loss 12.4762 Epoch: 2 Global Step: 24185 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:30:14,886-Speed 127.21 samples/sec Loss 12.4979 Epoch: 2 Global Step: 24190 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:30:35,205-Speed 126.00 samples/sec Loss 12.2584 Epoch: 2 Global Step: 24195 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:30:55,601-Speed 125.52 samples/sec Loss 12.4042 Epoch: 2 Global Step: 24200 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:31:16,123-Speed 124.75 samples/sec Loss 12.1528 Epoch: 2 Global Step: 24205 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:31:36,649-Speed 124.73 samples/sec Loss 12.2485 Epoch: 2 Global Step: 24210 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:31:57,592-Speed 122.24 samples/sec Loss 12.6841 Epoch: 2 Global Step: 24215 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:32:17,749-Speed 127.00 samples/sec Loss 11.9818 Epoch: 2 Global Step: 24220 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:32:38,233-Speed 124.98 samples/sec Loss 12.6702 Epoch: 2 Global Step: 24225 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:32:58,439-Speed 126.70 samples/sec Loss 11.9643 Epoch: 2 Global Step: 24230 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:33:19,086-Speed 124.00 samples/sec Loss 12.2087 Epoch: 2 Global Step: 24235 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:33:39,474-Speed 125.57 samples/sec Loss 12.2251 Epoch: 2 Global Step: 24240 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:33:59,901-Speed 125.33 samples/sec Loss 12.0484 Epoch: 2 Global Step: 24245 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:34:20,245-Speed 125.85 samples/sec Loss 12.1162 Epoch: 2 Global Step: 24250 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:34:40,573-Speed 125.94 samples/sec Loss 12.4821 Epoch: 2 Global Step: 24255 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:35:01,011-Speed 125.27 samples/sec Loss 12.0165 Epoch: 2 Global Step: 24260 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:35:20,965-Speed 128.30 samples/sec Loss 12.2233 Epoch: 2 Global Step: 24265 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:35:41,218-Speed 126.40 samples/sec Loss 12.2093 Epoch: 2 Global Step: 24270 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:36:01,378-Speed 126.99 samples/sec Loss 12.5109 Epoch: 2 Global Step: 24275 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:36:21,700-Speed 125.98 samples/sec Loss 12.4305 Epoch: 2 Global Step: 24280 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:36:41,837-Speed 127.14 samples/sec Loss 12.3169 Epoch: 2 Global Step: 24285 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:37:02,165-Speed 125.95 samples/sec Loss 12.2376 Epoch: 2 Global Step: 24290 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:37:22,952-Speed 123.17 samples/sec Loss 12.3323 Epoch: 2 Global Step: 24295 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:37:43,263-Speed 126.05 samples/sec Loss 12.1699 Epoch: 2 Global Step: 24300 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:38:04,322-Speed 121.57 samples/sec Loss 12.3153 Epoch: 2 Global Step: 24305 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:38:24,801-Speed 125.01 samples/sec Loss 12.0968 Epoch: 2 Global Step: 24310 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:38:45,109-Speed 126.06 samples/sec Loss 12.3385 Epoch: 2 Global Step: 24315 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:39:05,640-Speed 124.69 samples/sec Loss 11.9750 Epoch: 2 Global Step: 24320 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:39:26,087-Speed 125.21 samples/sec Loss 12.3757 Epoch: 2 Global Step: 24325 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:39:46,401-Speed 126.03 samples/sec Loss 12.4118 Epoch: 2 Global Step: 24330 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:40:06,610-Speed 126.68 samples/sec Loss 12.1861 Epoch: 2 Global Step: 24335 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:40:26,924-Speed 126.03 samples/sec Loss 12.0769 Epoch: 2 Global Step: 24340 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:40:47,206-Speed 126.23 samples/sec Loss 12.3138 Epoch: 2 Global Step: 24345 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:41:08,300-Speed 121.37 samples/sec Loss 11.9662 Epoch: 2 Global Step: 24350 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:41:28,720-Speed 125.37 samples/sec Loss 12.3948 Epoch: 2 Global Step: 24355 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:41:49,127-Speed 125.45 samples/sec Loss 12.2588 Epoch: 2 Global Step: 24360 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:42:09,519-Speed 125.55 samples/sec Loss 12.1411 Epoch: 2 Global Step: 24365 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:42:29,711-Speed 126.79 samples/sec Loss 12.4624 Epoch: 2 Global Step: 24370 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:42:50,295-Speed 124.38 samples/sec Loss 12.4107 Epoch: 2 Global Step: 24375 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:43:10,518-Speed 126.60 samples/sec Loss 12.3454 Epoch: 2 Global Step: 24380 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:43:30,928-Speed 125.45 samples/sec Loss 12.3016 Epoch: 2 Global Step: 24385 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:43:51,312-Speed 125.60 samples/sec Loss 12.1826 Epoch: 2 Global Step: 24390 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:44:12,420-Speed 121.29 samples/sec Loss 12.2891 Epoch: 2 Global Step: 24395 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:44:32,872-Speed 125.17 samples/sec Loss 12.1093 Epoch: 2 Global Step: 24400 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:44:53,070-Speed 126.75 samples/sec Loss 12.3251 Epoch: 2 Global Step: 24405 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:45:13,704-Speed 124.08 samples/sec Loss 12.2195 Epoch: 2 Global Step: 24410 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:45:34,067-Speed 125.73 samples/sec Loss 12.5597 Epoch: 2 Global Step: 24415 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:45:54,403-Speed 125.89 samples/sec Loss 12.3195 Epoch: 2 Global Step: 24420 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:46:14,476-Speed 127.55 samples/sec Loss 12.1641 Epoch: 2 Global Step: 24425 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:46:34,776-Speed 126.12 samples/sec Loss 12.3913 Epoch: 2 Global Step: 24430 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:46:55,472-Speed 123.70 samples/sec Loss 12.4316 Epoch: 2 Global Step: 24435 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:47:16,536-Speed 121.54 samples/sec Loss 12.3838 Epoch: 2 Global Step: 24440 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:47:36,835-Speed 126.12 samples/sec Loss 12.4540 Epoch: 2 Global Step: 24445 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:47:57,203-Speed 125.70 samples/sec Loss 12.1304 Epoch: 2 Global Step: 24450 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:48:17,608-Speed 125.47 samples/sec Loss 12.4933 Epoch: 2 Global Step: 24455 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:48:37,934-Speed 125.95 samples/sec Loss 12.1020 Epoch: 2 Global Step: 24460 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:48:58,116-Speed 126.85 samples/sec Loss 12.3064 Epoch: 2 Global Step: 24465 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:49:18,550-Speed 125.29 samples/sec Loss 12.1986 Epoch: 2 Global Step: 24470 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:49:38,934-Speed 125.60 samples/sec Loss 12.2138 Epoch: 2 Global Step: 24475 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:49:59,134-Speed 126.74 samples/sec Loss 11.9475 Epoch: 2 Global Step: 24480 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:50:20,433-Speed 120.20 samples/sec Loss 12.4094 Epoch: 2 Global Step: 24485 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:50:40,856-Speed 125.35 samples/sec Loss 12.2523 Epoch: 2 Global Step: 24490 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:51:01,155-Speed 126.11 samples/sec Loss 12.2795 Epoch: 2 Global Step: 24495 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:51:21,498-Speed 125.85 samples/sec Loss 12.4591 Epoch: 2 Global Step: 24500 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:51:42,016-Speed 124.78 samples/sec Loss 12.3649 Epoch: 2 Global Step: 24505 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:52:02,353-Speed 125.88 samples/sec Loss 12.5055 Epoch: 2 Global Step: 24510 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:52:22,831-Speed 125.02 samples/sec Loss 12.6437 Epoch: 2 Global Step: 24515 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:52:43,379-Speed 124.59 samples/sec Loss 12.1333 Epoch: 2 Global Step: 24520 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:53:03,766-Speed 125.58 samples/sec Loss 12.0724 Epoch: 2 Global Step: 24525 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:53:24,803-Speed 121.70 samples/sec Loss 12.1370 Epoch: 2 Global Step: 24530 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:53:45,195-Speed 125.54 samples/sec Loss 12.5900 Epoch: 2 Global Step: 24535 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:54:05,404-Speed 126.69 samples/sec Loss 12.1800 Epoch: 2 Global Step: 24540 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:54:25,759-Speed 125.77 samples/sec Loss 11.9663 Epoch: 2 Global Step: 24545 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:54:45,987-Speed 126.57 samples/sec Loss 12.3318 Epoch: 2 Global Step: 24550 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:55:06,257-Speed 126.30 samples/sec Loss 12.1102 Epoch: 2 Global Step: 24555 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:55:26,716-Speed 125.14 samples/sec Loss 12.2915 Epoch: 2 Global Step: 24560 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:55:47,127-Speed 125.43 samples/sec Loss 12.1901 Epoch: 2 Global Step: 24565 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:56:07,282-Speed 127.02 samples/sec Loss 12.3424 Epoch: 2 Global Step: 24570 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:56:28,303-Speed 121.79 samples/sec Loss 12.2888 Epoch: 2 Global Step: 24575 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:56:48,457-Speed 127.03 samples/sec Loss 12.0537 Epoch: 2 Global Step: 24580 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:57:08,717-Speed 126.36 samples/sec Loss 12.2118 Epoch: 2 Global Step: 24585 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:57:29,222-Speed 124.85 samples/sec Loss 12.2267 Epoch: 2 Global Step: 24590 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:57:49,558-Speed 125.90 samples/sec Loss 12.5435 Epoch: 2 Global Step: 24595 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:58:09,952-Speed 125.54 samples/sec Loss 12.4764 Epoch: 2 Global Step: 24600 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:58:30,217-Speed 126.33 samples/sec Loss 12.2742 Epoch: 2 Global Step: 24605 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:58:50,311-Speed 127.41 samples/sec Loss 12.0589 Epoch: 2 Global Step: 24610 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:59:10,650-Speed 125.87 samples/sec Loss 12.2084 Epoch: 2 Global Step: 24615 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:59:31,688-Speed 121.69 samples/sec Loss 12.4877 Epoch: 2 Global Step: 24620 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-07 23:59:52,324-Speed 124.06 samples/sec Loss 12.4807 Epoch: 2 Global Step: 24625 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:00:12,533-Speed 126.68 samples/sec Loss 12.1583 Epoch: 2 Global Step: 24630 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:00:32,797-Speed 126.34 samples/sec Loss 12.0385 Epoch: 2 Global Step: 24635 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:00:52,928-Speed 127.18 samples/sec Loss 12.0769 Epoch: 2 Global Step: 24640 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:01:13,152-Speed 126.59 samples/sec Loss 12.2169 Epoch: 2 Global Step: 24645 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:01:33,553-Speed 125.50 samples/sec Loss 12.4678 Epoch: 2 Global Step: 24650 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:01:53,857-Speed 126.09 samples/sec Loss 12.2973 Epoch: 2 Global Step: 24655 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:02:14,174-Speed 126.01 samples/sec Loss 12.1643 Epoch: 2 Global Step: 24660 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:02:34,931-Speed 123.34 samples/sec Loss 12.2696 Epoch: 2 Global Step: 24665 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:02:55,295-Speed 125.72 samples/sec Loss 12.1735 Epoch: 2 Global Step: 24670 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:03:15,904-Speed 124.22 samples/sec Loss 12.3910 Epoch: 2 Global Step: 24675 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:03:36,279-Speed 125.65 samples/sec Loss 12.3193 Epoch: 2 Global Step: 24680 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:03:56,451-Speed 126.91 samples/sec Loss 12.5679 Epoch: 2 Global Step: 24685 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:04:16,563-Speed 127.30 samples/sec Loss 12.3705 Epoch: 2 Global Step: 24690 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:04:36,984-Speed 125.37 samples/sec Loss 11.9378 Epoch: 2 Global Step: 24695 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:04:57,427-Speed 125.24 samples/sec Loss 12.2368 Epoch: 2 Global Step: 24700 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:05:17,800-Speed 125.66 samples/sec Loss 12.0201 Epoch: 2 Global Step: 24705 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:05:38,808-Speed 121.87 samples/sec Loss 12.4498 Epoch: 2 Global Step: 24710 Fp16 Grad Scale: 8192 Required: 264 hours Training: 2021-06-08 00:05:59,187-Speed 125.62 samples/sec Loss 12.2093 Epoch: 2 Global Step: 24715 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:06:19,764-Speed 124.41 samples/sec Loss 12.6411 Epoch: 2 Global Step: 24720 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:06:40,079-Speed 126.02 samples/sec Loss 12.1485 Epoch: 2 Global Step: 24725 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:07:00,241-Speed 126.98 samples/sec Loss 12.1168 Epoch: 2 Global Step: 24730 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:07:20,511-Speed 126.30 samples/sec Loss 12.2567 Epoch: 2 Global Step: 24735 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:07:40,889-Speed 125.64 samples/sec Loss 12.0171 Epoch: 2 Global Step: 24740 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:08:01,218-Speed 125.93 samples/sec Loss 12.4259 Epoch: 2 Global Step: 24745 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:08:21,448-Speed 126.56 samples/sec Loss 12.1314 Epoch: 2 Global Step: 24750 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:08:42,688-Speed 120.53 samples/sec Loss 12.2081 Epoch: 2 Global Step: 24755 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:09:03,003-Speed 126.02 samples/sec Loss 12.2400 Epoch: 2 Global Step: 24760 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:09:23,321-Speed 126.00 samples/sec Loss 12.3500 Epoch: 2 Global Step: 24765 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:09:44,214-Speed 122.54 samples/sec Loss 12.4351 Epoch: 2 Global Step: 24770 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:10:04,653-Speed 125.25 samples/sec Loss 12.1109 Epoch: 2 Global Step: 24775 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:10:25,096-Speed 125.24 samples/sec Loss 12.3901 Epoch: 2 Global Step: 24780 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:10:45,413-Speed 126.01 samples/sec Loss 11.9656 Epoch: 2 Global Step: 24785 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:11:05,809-Speed 125.52 samples/sec Loss 11.6831 Epoch: 2 Global Step: 24790 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:11:26,126-Speed 126.01 samples/sec Loss 12.1293 Epoch: 2 Global Step: 24795 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:11:47,019-Speed 122.53 samples/sec Loss 12.3566 Epoch: 2 Global Step: 24800 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:12:07,193-Speed 126.90 samples/sec Loss 12.5177 Epoch: 2 Global Step: 24805 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:12:27,523-Speed 125.93 samples/sec Loss 12.4730 Epoch: 2 Global Step: 24810 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:12:47,802-Speed 126.25 samples/sec Loss 12.3952 Epoch: 2 Global Step: 24815 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:13:08,120-Speed 126.01 samples/sec Loss 12.1066 Epoch: 2 Global Step: 24820 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:13:28,447-Speed 125.94 samples/sec Loss 12.3888 Epoch: 2 Global Step: 24825 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:13:48,718-Speed 126.30 samples/sec Loss 12.7559 Epoch: 2 Global Step: 24830 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:14:09,052-Speed 125.90 samples/sec Loss 12.1453 Epoch: 2 Global Step: 24835 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:14:29,461-Speed 125.44 samples/sec Loss 12.1671 Epoch: 2 Global Step: 24840 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:14:50,502-Speed 121.67 samples/sec Loss 12.2331 Epoch: 2 Global Step: 24845 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:15:10,852-Speed 125.80 samples/sec Loss 12.3341 Epoch: 2 Global Step: 24850 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:15:31,352-Speed 124.89 samples/sec Loss 12.3439 Epoch: 2 Global Step: 24855 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:15:51,732-Speed 125.62 samples/sec Loss 12.0814 Epoch: 2 Global Step: 24860 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:16:11,747-Speed 127.91 samples/sec Loss 11.7790 Epoch: 2 Global Step: 24865 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:16:31,965-Speed 126.63 samples/sec Loss 12.2537 Epoch: 2 Global Step: 24870 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:16:52,319-Speed 125.78 samples/sec Loss 12.2803 Epoch: 2 Global Step: 24875 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:17:12,751-Speed 125.30 samples/sec Loss 12.4898 Epoch: 2 Global Step: 24880 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:17:33,105-Speed 125.79 samples/sec Loss 12.2484 Epoch: 2 Global Step: 24885 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:17:54,190-Speed 121.42 samples/sec Loss 12.5239 Epoch: 2 Global Step: 24890 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:18:14,547-Speed 125.76 samples/sec Loss 12.1314 Epoch: 2 Global Step: 24895 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:18:34,760-Speed 126.66 samples/sec Loss 12.3290 Epoch: 2 Global Step: 24900 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:18:55,057-Speed 126.14 samples/sec Loss 12.4765 Epoch: 2 Global Step: 24905 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:19:15,214-Speed 127.01 samples/sec Loss 12.3578 Epoch: 2 Global Step: 24910 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:19:35,560-Speed 125.83 samples/sec Loss 12.6207 Epoch: 2 Global Step: 24915 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:19:55,590-Speed 127.81 samples/sec Loss 12.0411 Epoch: 2 Global Step: 24920 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:20:16,017-Speed 125.33 samples/sec Loss 12.3801 Epoch: 2 Global Step: 24925 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:20:36,295-Speed 126.25 samples/sec Loss 12.5464 Epoch: 2 Global Step: 24930 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:20:57,410-Speed 121.25 samples/sec Loss 12.0232 Epoch: 2 Global Step: 24935 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:21:17,727-Speed 126.01 samples/sec Loss 12.2849 Epoch: 2 Global Step: 24940 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:21:38,317-Speed 124.33 samples/sec Loss 12.3586 Epoch: 2 Global Step: 24945 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:21:59,388-Speed 121.50 samples/sec Loss 12.2284 Epoch: 2 Global Step: 24950 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:22:19,539-Speed 127.05 samples/sec Loss 12.1251 Epoch: 2 Global Step: 24955 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:22:39,763-Speed 126.59 samples/sec Loss 12.3822 Epoch: 2 Global Step: 24960 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:23:00,287-Speed 124.74 samples/sec Loss 12.3575 Epoch: 2 Global Step: 24965 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:23:20,569-Speed 126.23 samples/sec Loss 12.1855 Epoch: 2 Global Step: 24970 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:23:41,050-Speed 125.00 samples/sec Loss 12.6064 Epoch: 2 Global Step: 24975 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:24:02,085-Speed 121.71 samples/sec Loss 12.1558 Epoch: 2 Global Step: 24980 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:24:22,396-Speed 126.04 samples/sec Loss 12.2331 Epoch: 2 Global Step: 24985 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:24:42,516-Speed 127.24 samples/sec Loss 12.5684 Epoch: 2 Global Step: 24990 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:25:02,737-Speed 126.61 samples/sec Loss 12.5078 Epoch: 2 Global Step: 24995 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:25:23,033-Speed 126.14 samples/sec Loss 12.2565 Epoch: 2 Global Step: 25000 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:25:43,270-Speed 126.51 samples/sec Loss 12.3355 Epoch: 2 Global Step: 25005 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:26:03,412-Speed 127.11 samples/sec Loss 12.3855 Epoch: 2 Global Step: 25010 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:26:23,719-Speed 126.07 samples/sec Loss 12.2220 Epoch: 2 Global Step: 25015 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:26:44,047-Speed 125.95 samples/sec Loss 12.4107 Epoch: 2 Global Step: 25020 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:27:05,142-Speed 121.36 samples/sec Loss 12.4359 Epoch: 2 Global Step: 25025 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:27:25,866-Speed 123.53 samples/sec Loss 12.1765 Epoch: 2 Global Step: 25030 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:27:46,241-Speed 125.65 samples/sec Loss 12.5724 Epoch: 2 Global Step: 25035 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:28:06,632-Speed 125.55 samples/sec Loss 12.2195 Epoch: 2 Global Step: 25040 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:28:26,841-Speed 126.68 samples/sec Loss 12.0001 Epoch: 2 Global Step: 25045 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:28:47,158-Speed 126.01 samples/sec Loss 11.8720 Epoch: 2 Global Step: 25050 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:29:07,465-Speed 126.08 samples/sec Loss 12.3089 Epoch: 2 Global Step: 25055 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:29:27,705-Speed 126.49 samples/sec Loss 12.5508 Epoch: 2 Global Step: 25060 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:29:47,879-Speed 126.90 samples/sec Loss 12.1268 Epoch: 2 Global Step: 25065 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:30:08,682-Speed 123.06 samples/sec Loss 12.2545 Epoch: 2 Global Step: 25070 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:30:28,621-Speed 128.40 samples/sec Loss 12.0892 Epoch: 2 Global Step: 25075 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:30:48,926-Speed 126.08 samples/sec Loss 12.5760 Epoch: 2 Global Step: 25080 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:31:09,161-Speed 126.52 samples/sec Loss 12.3178 Epoch: 2 Global Step: 25085 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:31:29,535-Speed 125.66 samples/sec Loss 11.7333 Epoch: 2 Global Step: 25090 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:31:49,692-Speed 127.01 samples/sec Loss 12.2850 Epoch: 2 Global Step: 25095 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:32:09,872-Speed 126.86 samples/sec Loss 12.1424 Epoch: 2 Global Step: 25100 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:32:30,073-Speed 126.73 samples/sec Loss 12.3772 Epoch: 2 Global Step: 25105 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:32:50,182-Speed 127.31 samples/sec Loss 12.1842 Epoch: 2 Global Step: 25110 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:33:11,178-Speed 121.93 samples/sec Loss 12.3941 Epoch: 2 Global Step: 25115 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:33:31,241-Speed 127.60 samples/sec Loss 12.0618 Epoch: 2 Global Step: 25120 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:33:51,513-Speed 126.29 samples/sec Loss 12.0058 Epoch: 2 Global Step: 25125 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:34:11,882-Speed 125.69 samples/sec Loss 12.2746 Epoch: 2 Global Step: 25130 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:34:32,478-Speed 124.30 samples/sec Loss 12.2590 Epoch: 2 Global Step: 25135 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:34:53,189-Speed 123.61 samples/sec Loss 12.4401 Epoch: 2 Global Step: 25140 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:35:13,747-Speed 124.53 samples/sec Loss 12.7058 Epoch: 2 Global Step: 25145 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:35:34,137-Speed 125.57 samples/sec Loss 12.2207 Epoch: 2 Global Step: 25150 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:35:54,460-Speed 125.97 samples/sec Loss 12.2370 Epoch: 2 Global Step: 25155 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:36:15,608-Speed 121.06 samples/sec Loss 12.3777 Epoch: 2 Global Step: 25160 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:36:35,856-Speed 126.44 samples/sec Loss 12.2211 Epoch: 2 Global Step: 25165 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:36:56,240-Speed 125.59 samples/sec Loss 12.1949 Epoch: 2 Global Step: 25170 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:37:16,618-Speed 125.64 samples/sec Loss 11.9392 Epoch: 2 Global Step: 25175 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:37:37,008-Speed 125.56 samples/sec Loss 12.3019 Epoch: 2 Global Step: 25180 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:37:57,360-Speed 125.79 samples/sec Loss 12.0207 Epoch: 2 Global Step: 25185 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:38:17,920-Speed 124.52 samples/sec Loss 11.9835 Epoch: 2 Global Step: 25190 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:38:38,146-Speed 126.57 samples/sec Loss 11.8312 Epoch: 2 Global Step: 25195 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:38:58,621-Speed 125.04 samples/sec Loss 12.2759 Epoch: 2 Global Step: 25200 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:39:19,649-Speed 121.74 samples/sec Loss 12.4876 Epoch: 2 Global Step: 25205 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:39:40,076-Speed 125.34 samples/sec Loss 12.3450 Epoch: 2 Global Step: 25210 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:40:00,442-Speed 125.70 samples/sec Loss 12.2465 Epoch: 2 Global Step: 25215 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:40:20,825-Speed 125.60 samples/sec Loss 12.1488 Epoch: 2 Global Step: 25220 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:40:40,851-Speed 127.84 samples/sec Loss 12.2510 Epoch: 2 Global Step: 25225 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:41:01,215-Speed 125.72 samples/sec Loss 12.0571 Epoch: 2 Global Step: 25230 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:41:21,377-Speed 126.98 samples/sec Loss 12.3041 Epoch: 2 Global Step: 25235 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:41:41,698-Speed 125.98 samples/sec Loss 12.1875 Epoch: 2 Global Step: 25240 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:42:01,679-Speed 128.13 samples/sec Loss 11.9538 Epoch: 2 Global Step: 25245 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:42:22,656-Speed 122.05 samples/sec Loss 12.3811 Epoch: 2 Global Step: 25250 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:42:43,084-Speed 125.32 samples/sec Loss 12.5862 Epoch: 2 Global Step: 25255 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:43:03,606-Speed 124.75 samples/sec Loss 12.3753 Epoch: 2 Global Step: 25260 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:43:23,890-Speed 126.21 samples/sec Loss 12.0720 Epoch: 2 Global Step: 25265 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:43:44,019-Speed 127.19 samples/sec Loss 12.1828 Epoch: 2 Global Step: 25270 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:44:04,421-Speed 125.48 samples/sec Loss 12.2405 Epoch: 2 Global Step: 25275 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:44:24,996-Speed 124.43 samples/sec Loss 12.6644 Epoch: 2 Global Step: 25280 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:44:45,238-Speed 126.47 samples/sec Loss 12.6255 Epoch: 2 Global Step: 25285 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:45:05,530-Speed 126.17 samples/sec Loss 12.1846 Epoch: 2 Global Step: 25290 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:45:26,403-Speed 122.65 samples/sec Loss 12.5403 Epoch: 2 Global Step: 25295 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:45:46,901-Speed 124.89 samples/sec Loss 12.3311 Epoch: 2 Global Step: 25300 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:46:07,303-Speed 125.49 samples/sec Loss 12.2596 Epoch: 2 Global Step: 25305 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:46:27,406-Speed 127.35 samples/sec Loss 11.9904 Epoch: 2 Global Step: 25310 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:46:47,921-Speed 124.79 samples/sec Loss 12.5703 Epoch: 2 Global Step: 25315 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:47:08,277-Speed 125.77 samples/sec Loss 12.4636 Epoch: 2 Global Step: 25320 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:47:28,468-Speed 126.80 samples/sec Loss 12.1582 Epoch: 2 Global Step: 25325 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:47:48,573-Speed 127.34 samples/sec Loss 12.0548 Epoch: 2 Global Step: 25330 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:48:09,107-Speed 124.68 samples/sec Loss 12.2399 Epoch: 2 Global Step: 25335 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:48:30,078-Speed 122.08 samples/sec Loss 12.2772 Epoch: 2 Global Step: 25340 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:48:50,246-Speed 126.94 samples/sec Loss 12.3524 Epoch: 2 Global Step: 25345 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:49:10,520-Speed 126.28 samples/sec Loss 12.1422 Epoch: 2 Global Step: 25350 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:49:31,043-Speed 124.74 samples/sec Loss 12.2329 Epoch: 2 Global Step: 25355 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:49:51,312-Speed 126.31 samples/sec Loss 11.9061 Epoch: 2 Global Step: 25360 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:50:11,691-Speed 125.63 samples/sec Loss 12.1378 Epoch: 2 Global Step: 25365 Fp16 Grad Scale: 8192 Required: 263 hours Training: 2021-06-08 00:50:31,932-Speed 126.49 samples/sec Loss 12.4280 Epoch: 2 Global Step: 25370 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:50:52,219-Speed 126.19 samples/sec Loss 12.2559 Epoch: 2 Global Step: 25375 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:51:12,338-Speed 127.26 samples/sec Loss 12.2952 Epoch: 2 Global Step: 25380 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:51:33,387-Speed 121.62 samples/sec Loss 12.1143 Epoch: 2 Global Step: 25385 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:51:53,718-Speed 125.92 samples/sec Loss 12.4165 Epoch: 2 Global Step: 25390 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:52:13,973-Speed 126.39 samples/sec Loss 12.1199 Epoch: 2 Global Step: 25395 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:52:34,342-Speed 125.68 samples/sec Loss 11.9656 Epoch: 2 Global Step: 25400 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:52:54,648-Speed 126.08 samples/sec Loss 12.2482 Epoch: 2 Global Step: 25405 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:53:14,817-Speed 126.94 samples/sec Loss 12.1436 Epoch: 2 Global Step: 25410 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:53:35,262-Speed 125.22 samples/sec Loss 12.0930 Epoch: 2 Global Step: 25415 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:53:55,450-Speed 126.82 samples/sec Loss 12.3215 Epoch: 2 Global Step: 25420 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:54:15,744-Speed 126.15 samples/sec Loss 12.2955 Epoch: 2 Global Step: 25425 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:54:36,794-Speed 121.62 samples/sec Loss 12.5707 Epoch: 2 Global Step: 25430 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:54:57,272-Speed 125.02 samples/sec Loss 12.3333 Epoch: 2 Global Step: 25435 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:55:17,695-Speed 125.35 samples/sec Loss 12.1479 Epoch: 2 Global Step: 25440 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:55:38,123-Speed 125.33 samples/sec Loss 12.2536 Epoch: 2 Global Step: 25445 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:55:58,336-Speed 126.66 samples/sec Loss 12.2226 Epoch: 2 Global Step: 25450 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:56:18,410-Speed 127.55 samples/sec Loss 12.1986 Epoch: 2 Global Step: 25455 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:56:38,747-Speed 125.88 samples/sec Loss 12.3901 Epoch: 2 Global Step: 25460 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:56:58,967-Speed 126.61 samples/sec Loss 12.3079 Epoch: 2 Global Step: 25465 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:57:19,331-Speed 125.72 samples/sec Loss 12.5874 Epoch: 2 Global Step: 25470 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:57:40,473-Speed 121.09 samples/sec Loss 12.6074 Epoch: 2 Global Step: 25475 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:58:01,066-Speed 124.32 samples/sec Loss 12.6010 Epoch: 2 Global Step: 25480 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:58:21,215-Speed 127.06 samples/sec Loss 12.4678 Epoch: 2 Global Step: 25485 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:58:41,494-Speed 126.25 samples/sec Loss 12.4613 Epoch: 2 Global Step: 25490 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:59:01,708-Speed 126.65 samples/sec Loss 12.1471 Epoch: 2 Global Step: 25495 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:59:21,722-Speed 127.92 samples/sec Loss 12.3537 Epoch: 2 Global Step: 25500 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 00:59:41,728-Speed 127.97 samples/sec Loss 12.1810 Epoch: 2 Global Step: 25505 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:00:02,150-Speed 125.36 samples/sec Loss 12.3639 Epoch: 2 Global Step: 25510 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:00:22,148-Speed 128.02 samples/sec Loss 12.2746 Epoch: 2 Global Step: 25515 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:00:43,224-Speed 121.47 samples/sec Loss 12.4322 Epoch: 2 Global Step: 25520 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:01:03,433-Speed 126.68 samples/sec Loss 12.1871 Epoch: 2 Global Step: 25525 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:01:24,121-Speed 123.75 samples/sec Loss 12.1813 Epoch: 2 Global Step: 25530 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:01:44,488-Speed 125.70 samples/sec Loss 12.2144 Epoch: 2 Global Step: 25535 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:02:04,848-Speed 125.75 samples/sec Loss 11.8106 Epoch: 2 Global Step: 25540 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:02:25,264-Speed 125.40 samples/sec Loss 12.1985 Epoch: 2 Global Step: 25545 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:02:45,595-Speed 125.92 samples/sec Loss 12.3090 Epoch: 2 Global Step: 25550 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:03:05,935-Speed 125.87 samples/sec Loss 12.4846 Epoch: 2 Global Step: 25555 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:03:26,220-Speed 126.21 samples/sec Loss 11.9239 Epoch: 2 Global Step: 25560 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:03:47,219-Speed 121.92 samples/sec Loss 12.0678 Epoch: 2 Global Step: 25565 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:04:07,346-Speed 127.20 samples/sec Loss 12.1782 Epoch: 2 Global Step: 25570 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:04:27,611-Speed 126.33 samples/sec Loss 12.0574 Epoch: 2 Global Step: 25575 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:04:47,957-Speed 125.83 samples/sec Loss 12.3481 Epoch: 2 Global Step: 25580 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:05:08,230-Speed 126.29 samples/sec Loss 12.2359 Epoch: 2 Global Step: 25585 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:05:28,906-Speed 123.82 samples/sec Loss 12.4981 Epoch: 2 Global Step: 25590 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:05:49,145-Speed 126.49 samples/sec Loss 11.9756 Epoch: 2 Global Step: 25595 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:06:09,472-Speed 125.95 samples/sec Loss 12.0657 Epoch: 2 Global Step: 25600 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:06:29,891-Speed 125.39 samples/sec Loss 12.0053 Epoch: 2 Global Step: 25605 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:06:50,684-Speed 123.12 samples/sec Loss 12.4945 Epoch: 2 Global Step: 25610 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:07:11,057-Speed 125.67 samples/sec Loss 12.2484 Epoch: 2 Global Step: 25615 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:07:31,412-Speed 125.77 samples/sec Loss 12.2427 Epoch: 2 Global Step: 25620 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:07:51,604-Speed 126.79 samples/sec Loss 12.1884 Epoch: 2 Global Step: 25625 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:08:12,108-Speed 124.87 samples/sec Loss 12.3107 Epoch: 2 Global Step: 25630 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:08:32,418-Speed 126.05 samples/sec Loss 12.2032 Epoch: 2 Global Step: 25635 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:08:52,682-Speed 126.34 samples/sec Loss 12.0522 Epoch: 2 Global Step: 25640 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:09:12,744-Speed 127.61 samples/sec Loss 11.9521 Epoch: 2 Global Step: 25645 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:09:33,076-Speed 125.92 samples/sec Loss 12.3970 Epoch: 2 Global Step: 25650 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:09:54,166-Speed 121.39 samples/sec Loss 11.8759 Epoch: 2 Global Step: 25655 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:10:14,428-Speed 126.38 samples/sec Loss 12.2077 Epoch: 2 Global Step: 25660 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:10:34,673-Speed 126.45 samples/sec Loss 12.2208 Epoch: 2 Global Step: 25665 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:10:54,889-Speed 126.64 samples/sec Loss 11.9456 Epoch: 2 Global Step: 25670 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:11:15,123-Speed 126.53 samples/sec Loss 11.9770 Epoch: 2 Global Step: 25675 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:11:35,489-Speed 125.71 samples/sec Loss 12.2283 Epoch: 2 Global Step: 25680 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:11:55,912-Speed 125.36 samples/sec Loss 12.2752 Epoch: 2 Global Step: 25685 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:12:16,826-Speed 122.41 samples/sec Loss 12.3857 Epoch: 2 Global Step: 25690 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:12:37,082-Speed 126.39 samples/sec Loss 12.3108 Epoch: 2 Global Step: 25695 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:12:58,079-Speed 121.93 samples/sec Loss 12.2111 Epoch: 2 Global Step: 25700 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:13:18,300-Speed 126.60 samples/sec Loss 11.9754 Epoch: 2 Global Step: 25705 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:13:38,607-Speed 126.07 samples/sec Loss 12.5439 Epoch: 2 Global Step: 25710 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:13:59,007-Speed 125.50 samples/sec Loss 12.2338 Epoch: 2 Global Step: 25715 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:14:19,341-Speed 125.90 samples/sec Loss 12.2741 Epoch: 2 Global Step: 25720 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:14:39,762-Speed 125.37 samples/sec Loss 12.1281 Epoch: 2 Global Step: 25725 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:15:00,018-Speed 126.39 samples/sec Loss 11.9997 Epoch: 2 Global Step: 25730 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:15:20,345-Speed 125.95 samples/sec Loss 12.2038 Epoch: 2 Global Step: 25735 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:15:40,538-Speed 126.79 samples/sec Loss 11.9732 Epoch: 2 Global Step: 25740 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:16:01,338-Speed 123.08 samples/sec Loss 12.4490 Epoch: 2 Global Step: 25745 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:16:21,603-Speed 126.33 samples/sec Loss 12.2673 Epoch: 2 Global Step: 25750 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:16:41,801-Speed 126.75 samples/sec Loss 12.1984 Epoch: 2 Global Step: 25755 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:17:01,891-Speed 127.43 samples/sec Loss 12.0441 Epoch: 2 Global Step: 25760 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:17:22,044-Speed 127.03 samples/sec Loss 12.4253 Epoch: 2 Global Step: 25765 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:17:42,481-Speed 125.27 samples/sec Loss 11.9824 Epoch: 2 Global Step: 25770 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:18:02,787-Speed 126.08 samples/sec Loss 12.4747 Epoch: 2 Global Step: 25775 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:18:23,132-Speed 125.84 samples/sec Loss 12.3940 Epoch: 2 Global Step: 25780 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:18:43,422-Speed 126.18 samples/sec Loss 12.2319 Epoch: 2 Global Step: 25785 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:19:04,423-Speed 121.90 samples/sec Loss 12.1223 Epoch: 2 Global Step: 25790 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:19:24,728-Speed 126.08 samples/sec Loss 11.9496 Epoch: 2 Global Step: 25795 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:19:45,620-Speed 122.54 samples/sec Loss 12.2610 Epoch: 2 Global Step: 25800 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:20:05,967-Speed 125.82 samples/sec Loss 12.1191 Epoch: 2 Global Step: 25805 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:20:26,257-Speed 126.18 samples/sec Loss 12.2375 Epoch: 2 Global Step: 25810 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:20:46,274-Speed 127.90 samples/sec Loss 12.2271 Epoch: 2 Global Step: 25815 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:21:06,456-Speed 126.85 samples/sec Loss 12.1956 Epoch: 2 Global Step: 25820 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:21:26,745-Speed 126.19 samples/sec Loss 11.8568 Epoch: 2 Global Step: 25825 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:21:46,898-Speed 127.04 samples/sec Loss 11.9267 Epoch: 2 Global Step: 25830 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:22:07,718-Speed 122.96 samples/sec Loss 12.2399 Epoch: 2 Global Step: 25835 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:22:27,936-Speed 126.63 samples/sec Loss 12.2621 Epoch: 2 Global Step: 25840 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:22:48,178-Speed 126.48 samples/sec Loss 12.2469 Epoch: 2 Global Step: 25845 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:23:08,574-Speed 125.53 samples/sec Loss 12.1210 Epoch: 2 Global Step: 25850 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:23:28,866-Speed 126.16 samples/sec Loss 12.2874 Epoch: 2 Global Step: 25855 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:23:48,933-Speed 127.58 samples/sec Loss 12.2132 Epoch: 2 Global Step: 25860 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:24:09,325-Speed 125.55 samples/sec Loss 12.2259 Epoch: 2 Global Step: 25865 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:24:29,700-Speed 125.65 samples/sec Loss 12.0855 Epoch: 2 Global Step: 25870 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:24:50,302-Speed 124.27 samples/sec Loss 12.5988 Epoch: 2 Global Step: 25875 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:25:11,327-Speed 121.76 samples/sec Loss 12.3234 Epoch: 2 Global Step: 25880 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:25:31,758-Speed 125.31 samples/sec Loss 12.5440 Epoch: 2 Global Step: 25885 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:25:52,086-Speed 125.94 samples/sec Loss 12.3589 Epoch: 2 Global Step: 25890 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:26:12,447-Speed 125.74 samples/sec Loss 12.4993 Epoch: 2 Global Step: 25895 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:26:32,597-Speed 127.05 samples/sec Loss 12.2445 Epoch: 2 Global Step: 25900 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:26:53,205-Speed 124.23 samples/sec Loss 12.4302 Epoch: 2 Global Step: 25905 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:27:13,413-Speed 126.69 samples/sec Loss 12.2476 Epoch: 2 Global Step: 25910 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:27:33,578-Speed 126.96 samples/sec Loss 12.1344 Epoch: 2 Global Step: 25915 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:27:53,854-Speed 126.26 samples/sec Loss 12.2493 Epoch: 2 Global Step: 25920 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:28:14,699-Speed 122.82 samples/sec Loss 12.4616 Epoch: 2 Global Step: 25925 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:28:35,092-Speed 125.54 samples/sec Loss 12.4173 Epoch: 2 Global Step: 25930 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:28:55,305-Speed 126.66 samples/sec Loss 11.9171 Epoch: 2 Global Step: 25935 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:29:15,682-Speed 125.64 samples/sec Loss 12.0004 Epoch: 2 Global Step: 25940 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:29:35,945-Speed 126.34 samples/sec Loss 12.0559 Epoch: 2 Global Step: 25945 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:29:56,242-Speed 126.13 samples/sec Loss 12.4350 Epoch: 2 Global Step: 25950 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:30:16,626-Speed 125.59 samples/sec Loss 12.1107 Epoch: 2 Global Step: 25955 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:30:36,966-Speed 125.87 samples/sec Loss 12.3997 Epoch: 2 Global Step: 25960 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:30:57,278-Speed 126.05 samples/sec Loss 12.3215 Epoch: 2 Global Step: 25965 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:31:18,175-Speed 122.51 samples/sec Loss 11.9676 Epoch: 2 Global Step: 25970 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:31:38,453-Speed 126.25 samples/sec Loss 12.0037 Epoch: 2 Global Step: 25975 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:31:58,769-Speed 126.01 samples/sec Loss 12.1390 Epoch: 2 Global Step: 25980 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:32:18,999-Speed 126.55 samples/sec Loss 12.4614 Epoch: 2 Global Step: 25985 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:32:39,191-Speed 126.79 samples/sec Loss 12.1491 Epoch: 2 Global Step: 25990 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:32:59,324-Speed 127.16 samples/sec Loss 12.0133 Epoch: 2 Global Step: 25995 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:33:19,974-Speed 123.98 samples/sec Loss 11.6923 Epoch: 2 Global Step: 26000 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:33:40,262-Speed 126.19 samples/sec Loss 12.2508 Epoch: 2 Global Step: 26005 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:34:00,724-Speed 125.12 samples/sec Loss 12.2017 Epoch: 2 Global Step: 26010 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:34:21,555-Speed 122.90 samples/sec Loss 12.3056 Epoch: 2 Global Step: 26015 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:34:41,892-Speed 125.88 samples/sec Loss 12.0028 Epoch: 2 Global Step: 26020 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 01:35:02,233-Speed 125.86 samples/sec Loss 12.3953 Epoch: 2 Global Step: 26025 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:35:22,328-Speed 127.40 samples/sec Loss 12.1150 Epoch: 2 Global Step: 26030 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:35:42,657-Speed 125.93 samples/sec Loss 12.3806 Epoch: 2 Global Step: 26035 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:36:02,988-Speed 125.93 samples/sec Loss 12.1562 Epoch: 2 Global Step: 26040 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:36:23,338-Speed 125.80 samples/sec Loss 12.2646 Epoch: 2 Global Step: 26045 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:36:43,782-Speed 125.23 samples/sec Loss 12.2821 Epoch: 2 Global Step: 26050 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:37:04,268-Speed 124.97 samples/sec Loss 12.1377 Epoch: 2 Global Step: 26055 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:37:25,391-Speed 121.20 samples/sec Loss 11.8779 Epoch: 2 Global Step: 26060 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:37:45,830-Speed 125.25 samples/sec Loss 12.3430 Epoch: 2 Global Step: 26065 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:38:06,454-Speed 124.13 samples/sec Loss 11.9474 Epoch: 2 Global Step: 26070 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:38:26,812-Speed 125.76 samples/sec Loss 12.1626 Epoch: 2 Global Step: 26075 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:38:47,101-Speed 126.18 samples/sec Loss 12.0014 Epoch: 2 Global Step: 26080 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:39:07,403-Speed 126.11 samples/sec Loss 12.0799 Epoch: 2 Global Step: 26085 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:39:27,653-Speed 126.43 samples/sec Loss 12.1800 Epoch: 2 Global Step: 26090 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:39:48,109-Speed 125.15 samples/sec Loss 12.1190 Epoch: 2 Global Step: 26095 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:40:08,640-Speed 124.70 samples/sec Loss 12.1947 Epoch: 2 Global Step: 26100 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:40:30,410-Speed 117.60 samples/sec Loss 12.2822 Epoch: 2 Global Step: 26105 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:40:50,630-Speed 126.61 samples/sec Loss 12.0224 Epoch: 2 Global Step: 26110 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:41:11,144-Speed 124.80 samples/sec Loss 12.2064 Epoch: 2 Global Step: 26115 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:41:31,361-Speed 126.63 samples/sec Loss 11.8517 Epoch: 2 Global Step: 26120 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:41:51,501-Speed 127.12 samples/sec Loss 12.1187 Epoch: 2 Global Step: 26125 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:42:11,682-Speed 126.86 samples/sec Loss 12.1317 Epoch: 2 Global Step: 26130 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:42:31,849-Speed 126.95 samples/sec Loss 12.1487 Epoch: 2 Global Step: 26135 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:42:52,225-Speed 125.65 samples/sec Loss 12.1882 Epoch: 2 Global Step: 26140 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:43:13,011-Speed 123.17 samples/sec Loss 12.3067 Epoch: 2 Global Step: 26145 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:43:34,254-Speed 120.51 samples/sec Loss 12.4409 Epoch: 2 Global Step: 26150 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:43:54,691-Speed 125.27 samples/sec Loss 12.2736 Epoch: 2 Global Step: 26155 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:44:15,154-Speed 125.10 samples/sec Loss 12.3741 Epoch: 2 Global Step: 26160 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:44:35,471-Speed 126.01 samples/sec Loss 11.9490 Epoch: 2 Global Step: 26165 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:44:55,816-Speed 125.84 samples/sec Loss 12.2740 Epoch: 2 Global Step: 26170 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:45:16,135-Speed 126.00 samples/sec Loss 12.0172 Epoch: 2 Global Step: 26175 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:45:36,590-Speed 125.16 samples/sec Loss 12.1389 Epoch: 2 Global Step: 26180 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:45:56,876-Speed 126.20 samples/sec Loss 12.5451 Epoch: 2 Global Step: 26185 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:46:17,451-Speed 124.44 samples/sec Loss 12.3345 Epoch: 2 Global Step: 26190 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:46:38,483-Speed 121.72 samples/sec Loss 12.5337 Epoch: 2 Global Step: 26195 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:46:58,714-Speed 126.54 samples/sec Loss 12.1830 Epoch: 2 Global Step: 26200 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:47:18,840-Speed 127.20 samples/sec Loss 12.0998 Epoch: 2 Global Step: 26205 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:47:39,146-Speed 126.08 samples/sec Loss 12.2205 Epoch: 2 Global Step: 26210 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:47:59,402-Speed 126.39 samples/sec Loss 12.2927 Epoch: 2 Global Step: 26215 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:48:19,691-Speed 126.19 samples/sec Loss 12.1823 Epoch: 2 Global Step: 26220 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:48:39,997-Speed 126.08 samples/sec Loss 12.3737 Epoch: 2 Global Step: 26225 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:49:00,774-Speed 123.22 samples/sec Loss 11.9774 Epoch: 2 Global Step: 26230 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:49:21,074-Speed 126.12 samples/sec Loss 12.1038 Epoch: 2 Global Step: 26235 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:49:42,458-Speed 119.72 samples/sec Loss 12.1984 Epoch: 2 Global Step: 26240 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:50:02,729-Speed 126.29 samples/sec Loss 12.3900 Epoch: 2 Global Step: 26245 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:50:23,127-Speed 125.51 samples/sec Loss 12.0963 Epoch: 2 Global Step: 26250 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:50:43,518-Speed 125.55 samples/sec Loss 12.4675 Epoch: 2 Global Step: 26255 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:51:03,867-Speed 125.81 samples/sec Loss 12.2331 Epoch: 2 Global Step: 26260 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:51:24,202-Speed 125.90 samples/sec Loss 12.0148 Epoch: 2 Global Step: 26265 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:51:44,547-Speed 125.84 samples/sec Loss 12.0330 Epoch: 2 Global Step: 26270 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:52:05,061-Speed 124.80 samples/sec Loss 12.2943 Epoch: 2 Global Step: 26275 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:52:25,507-Speed 125.22 samples/sec Loss 11.8194 Epoch: 2 Global Step: 26280 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:52:46,712-Speed 120.73 samples/sec Loss 12.2633 Epoch: 2 Global Step: 26285 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:53:07,223-Speed 124.81 samples/sec Loss 12.2457 Epoch: 2 Global Step: 26290 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:53:27,595-Speed 125.67 samples/sec Loss 12.1094 Epoch: 2 Global Step: 26295 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:53:47,872-Speed 126.26 samples/sec Loss 12.1496 Epoch: 2 Global Step: 26300 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:54:08,093-Speed 126.61 samples/sec Loss 12.2141 Epoch: 2 Global Step: 26305 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:54:28,402-Speed 126.06 samples/sec Loss 12.1598 Epoch: 2 Global Step: 26310 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:54:48,437-Speed 127.78 samples/sec Loss 12.1816 Epoch: 2 Global Step: 26315 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:55:08,794-Speed 125.77 samples/sec Loss 12.3003 Epoch: 2 Global Step: 26320 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:55:29,676-Speed 122.60 samples/sec Loss 11.9024 Epoch: 2 Global Step: 26325 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:55:50,556-Speed 122.61 samples/sec Loss 12.2859 Epoch: 2 Global Step: 26330 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:56:10,868-Speed 126.04 samples/sec Loss 12.2878 Epoch: 2 Global Step: 26335 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:56:31,248-Speed 125.62 samples/sec Loss 12.0710 Epoch: 2 Global Step: 26340 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:56:51,501-Speed 126.41 samples/sec Loss 12.1885 Epoch: 2 Global Step: 26345 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:57:11,656-Speed 127.02 samples/sec Loss 12.6505 Epoch: 2 Global Step: 26350 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:57:31,775-Speed 127.25 samples/sec Loss 12.2933 Epoch: 2 Global Step: 26355 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:57:52,200-Speed 125.34 samples/sec Loss 11.9513 Epoch: 2 Global Step: 26360 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:58:12,571-Speed 125.68 samples/sec Loss 11.9493 Epoch: 2 Global Step: 26365 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:58:33,119-Speed 124.59 samples/sec Loss 12.1506 Epoch: 2 Global Step: 26370 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:58:54,516-Speed 119.65 samples/sec Loss 12.1444 Epoch: 2 Global Step: 26375 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:59:14,800-Speed 126.21 samples/sec Loss 12.3021 Epoch: 2 Global Step: 26380 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:59:34,922-Speed 127.23 samples/sec Loss 11.9393 Epoch: 2 Global Step: 26385 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 01:59:55,320-Speed 125.51 samples/sec Loss 12.3519 Epoch: 2 Global Step: 26390 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:00:15,625-Speed 126.09 samples/sec Loss 12.0276 Epoch: 2 Global Step: 26395 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:00:36,009-Speed 125.60 samples/sec Loss 12.1077 Epoch: 2 Global Step: 26400 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:00:56,382-Speed 125.67 samples/sec Loss 11.8882 Epoch: 2 Global Step: 26405 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:01:16,802-Speed 125.38 samples/sec Loss 12.3616 Epoch: 2 Global Step: 26410 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:01:37,047-Speed 126.46 samples/sec Loss 12.0176 Epoch: 2 Global Step: 26415 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:01:57,939-Speed 122.55 samples/sec Loss 11.9923 Epoch: 2 Global Step: 26420 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:02:18,125-Speed 126.83 samples/sec Loss 12.2565 Epoch: 2 Global Step: 26425 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:02:38,401-Speed 126.26 samples/sec Loss 12.3571 Epoch: 2 Global Step: 26430 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:02:58,521-Speed 127.25 samples/sec Loss 12.2238 Epoch: 2 Global Step: 26435 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:03:18,832-Speed 126.04 samples/sec Loss 11.8234 Epoch: 2 Global Step: 26440 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:03:38,992-Speed 126.99 samples/sec Loss 12.2609 Epoch: 2 Global Step: 26445 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:03:59,257-Speed 126.33 samples/sec Loss 12.1820 Epoch: 2 Global Step: 26450 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:04:19,336-Speed 127.50 samples/sec Loss 12.1324 Epoch: 2 Global Step: 26455 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:04:39,523-Speed 126.82 samples/sec Loss 12.1489 Epoch: 2 Global Step: 26460 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:05:00,533-Speed 121.85 samples/sec Loss 11.7725 Epoch: 2 Global Step: 26465 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:05:21,148-Speed 124.19 samples/sec Loss 12.0306 Epoch: 2 Global Step: 26470 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:05:41,434-Speed 126.20 samples/sec Loss 12.1345 Epoch: 2 Global Step: 26475 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:06:01,819-Speed 125.59 samples/sec Loss 12.3880 Epoch: 2 Global Step: 26480 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:06:22,212-Speed 125.54 samples/sec Loss 12.0918 Epoch: 2 Global Step: 26485 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:06:42,418-Speed 126.70 samples/sec Loss 12.2378 Epoch: 2 Global Step: 26490 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:07:02,586-Speed 126.94 samples/sec Loss 12.3214 Epoch: 2 Global Step: 26495 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:07:22,887-Speed 126.11 samples/sec Loss 12.2634 Epoch: 2 Global Step: 26500 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:07:43,217-Speed 125.93 samples/sec Loss 12.4029 Epoch: 2 Global Step: 26505 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:08:04,377-Speed 120.99 samples/sec Loss 11.9986 Epoch: 2 Global Step: 26510 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:08:25,176-Speed 123.09 samples/sec Loss 12.5629 Epoch: 2 Global Step: 26515 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:08:45,302-Speed 127.20 samples/sec Loss 12.0518 Epoch: 2 Global Step: 26520 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:09:05,463-Speed 126.98 samples/sec Loss 11.8660 Epoch: 2 Global Step: 26525 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:09:25,597-Speed 127.16 samples/sec Loss 12.1852 Epoch: 2 Global Step: 26530 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:09:45,893-Speed 126.14 samples/sec Loss 12.0561 Epoch: 2 Global Step: 26535 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:10:06,074-Speed 126.86 samples/sec Loss 11.9318 Epoch: 2 Global Step: 26540 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:10:26,369-Speed 126.15 samples/sec Loss 12.2004 Epoch: 2 Global Step: 26545 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:10:46,242-Speed 128.83 samples/sec Loss 11.6682 Epoch: 2 Global Step: 26550 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:11:07,439-Speed 120.78 samples/sec Loss 12.2664 Epoch: 2 Global Step: 26555 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:11:27,727-Speed 126.19 samples/sec Loss 11.9567 Epoch: 2 Global Step: 26560 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:11:48,052-Speed 125.96 samples/sec Loss 12.5550 Epoch: 2 Global Step: 26565 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:12:08,637-Speed 124.37 samples/sec Loss 12.3157 Epoch: 2 Global Step: 26570 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:12:28,807-Speed 126.92 samples/sec Loss 11.8741 Epoch: 2 Global Step: 26575 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:12:49,132-Speed 125.99 samples/sec Loss 12.2740 Epoch: 2 Global Step: 26580 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:13:09,623-Speed 124.94 samples/sec Loss 11.9816 Epoch: 2 Global Step: 26585 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:13:30,004-Speed 125.61 samples/sec Loss 12.2363 Epoch: 2 Global Step: 26590 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:13:50,396-Speed 125.55 samples/sec Loss 12.0596 Epoch: 2 Global Step: 26595 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:14:11,336-Speed 122.26 samples/sec Loss 12.2138 Epoch: 2 Global Step: 26600 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:14:31,643-Speed 126.07 samples/sec Loss 11.9427 Epoch: 2 Global Step: 26605 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:14:52,079-Speed 125.27 samples/sec Loss 11.6202 Epoch: 2 Global Step: 26610 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:15:12,413-Speed 125.91 samples/sec Loss 12.1904 Epoch: 2 Global Step: 26615 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:15:32,753-Speed 125.87 samples/sec Loss 12.1644 Epoch: 2 Global Step: 26620 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:15:53,088-Speed 125.90 samples/sec Loss 12.2254 Epoch: 2 Global Step: 26625 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:16:13,246-Speed 127.00 samples/sec Loss 11.9540 Epoch: 2 Global Step: 26630 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:16:33,450-Speed 126.72 samples/sec Loss 11.7505 Epoch: 2 Global Step: 26635 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:16:53,793-Speed 125.85 samples/sec Loss 11.9272 Epoch: 2 Global Step: 26640 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:17:15,006-Speed 120.69 samples/sec Loss 12.1226 Epoch: 2 Global Step: 26645 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:17:35,461-Speed 125.16 samples/sec Loss 12.4059 Epoch: 2 Global Step: 26650 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:17:56,016-Speed 124.55 samples/sec Loss 12.3962 Epoch: 2 Global Step: 26655 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:18:16,443-Speed 125.33 samples/sec Loss 12.1387 Epoch: 2 Global Step: 26660 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:18:36,776-Speed 125.91 samples/sec Loss 12.3185 Epoch: 2 Global Step: 26665 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:18:57,151-Speed 125.65 samples/sec Loss 12.3580 Epoch: 2 Global Step: 26670 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:19:17,351-Speed 126.75 samples/sec Loss 12.1380 Epoch: 2 Global Step: 26675 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:19:37,699-Speed 125.81 samples/sec Loss 12.1664 Epoch: 2 Global Step: 26680 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:19:58,278-Speed 124.41 samples/sec Loss 12.2323 Epoch: 2 Global Step: 26685 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:20:19,260-Speed 122.02 samples/sec Loss 11.9504 Epoch: 2 Global Step: 26690 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:20:39,538-Speed 126.25 samples/sec Loss 12.1866 Epoch: 2 Global Step: 26695 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 02:20:59,808-Speed 126.30 samples/sec Loss 12.3988 Epoch: 2 Global Step: 26700 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:21:20,119-Speed 126.04 samples/sec Loss 12.0679 Epoch: 2 Global Step: 26705 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:21:40,258-Speed 127.12 samples/sec Loss 12.1979 Epoch: 2 Global Step: 26710 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:22:00,376-Speed 127.26 samples/sec Loss 11.9692 Epoch: 2 Global Step: 26715 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:22:20,619-Speed 126.47 samples/sec Loss 12.5603 Epoch: 2 Global Step: 26720 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:22:40,742-Speed 127.23 samples/sec Loss 12.5583 Epoch: 2 Global Step: 26725 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:23:01,016-Speed 126.28 samples/sec Loss 12.2446 Epoch: 2 Global Step: 26730 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:23:21,752-Speed 123.46 samples/sec Loss 12.2719 Epoch: 2 Global Step: 26735 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:23:41,916-Speed 126.96 samples/sec Loss 12.3148 Epoch: 2 Global Step: 26740 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:24:02,100-Speed 126.84 samples/sec Loss 12.0595 Epoch: 2 Global Step: 26745 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:24:22,439-Speed 125.87 samples/sec Loss 12.1447 Epoch: 2 Global Step: 26750 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:24:42,696-Speed 126.39 samples/sec Loss 12.1614 Epoch: 2 Global Step: 26755 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:25:03,098-Speed 125.49 samples/sec Loss 12.5908 Epoch: 2 Global Step: 26760 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:25:23,509-Speed 125.43 samples/sec Loss 12.0445 Epoch: 2 Global Step: 26765 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:25:43,911-Speed 125.48 samples/sec Loss 12.0240 Epoch: 2 Global Step: 26770 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:26:04,491-Speed 124.40 samples/sec Loss 12.1530 Epoch: 2 Global Step: 26775 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:26:25,252-Speed 123.32 samples/sec Loss 11.8781 Epoch: 2 Global Step: 26780 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:26:45,540-Speed 126.19 samples/sec Loss 12.3794 Epoch: 2 Global Step: 26785 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:27:05,881-Speed 125.86 samples/sec Loss 12.0961 Epoch: 2 Global Step: 26790 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:27:26,374-Speed 124.93 samples/sec Loss 12.2573 Epoch: 2 Global Step: 26795 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:27:46,699-Speed 125.96 samples/sec Loss 12.2097 Epoch: 2 Global Step: 26800 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:28:07,018-Speed 126.00 samples/sec Loss 11.9866 Epoch: 2 Global Step: 26805 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:28:27,491-Speed 125.05 samples/sec Loss 12.4129 Epoch: 2 Global Step: 26810 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:28:47,976-Speed 124.98 samples/sec Loss 12.2554 Epoch: 2 Global Step: 26815 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:29:07,996-Speed 127.87 samples/sec Loss 11.6392 Epoch: 2 Global Step: 26820 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:29:28,916-Speed 122.38 samples/sec Loss 12.4620 Epoch: 2 Global Step: 26825 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:29:49,205-Speed 126.18 samples/sec Loss 12.1108 Epoch: 2 Global Step: 26830 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:30:09,307-Speed 127.36 samples/sec Loss 12.0516 Epoch: 2 Global Step: 26835 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:30:29,812-Speed 124.86 samples/sec Loss 12.1616 Epoch: 2 Global Step: 26840 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:30:49,972-Speed 126.99 samples/sec Loss 12.3151 Epoch: 2 Global Step: 26845 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:31:10,297-Speed 125.96 samples/sec Loss 12.1459 Epoch: 2 Global Step: 26850 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:31:30,468-Speed 126.92 samples/sec Loss 12.5135 Epoch: 2 Global Step: 26855 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:31:50,794-Speed 125.96 samples/sec Loss 12.1355 Epoch: 2 Global Step: 26860 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:32:11,172-Speed 125.63 samples/sec Loss 12.1089 Epoch: 2 Global Step: 26865 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:32:32,568-Speed 119.66 samples/sec Loss 12.1539 Epoch: 2 Global Step: 26870 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:32:52,884-Speed 126.02 samples/sec Loss 12.1315 Epoch: 2 Global Step: 26875 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:33:13,148-Speed 126.34 samples/sec Loss 12.2019 Epoch: 2 Global Step: 26880 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:33:33,502-Speed 125.78 samples/sec Loss 12.5670 Epoch: 2 Global Step: 26885 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:33:53,675-Speed 126.91 samples/sec Loss 12.1625 Epoch: 2 Global Step: 26890 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:34:13,957-Speed 126.22 samples/sec Loss 12.0799 Epoch: 2 Global Step: 26895 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:34:34,272-Speed 126.02 samples/sec Loss 12.4084 Epoch: 2 Global Step: 26900 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:34:54,584-Speed 126.04 samples/sec Loss 12.0260 Epoch: 2 Global Step: 26905 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:35:14,867-Speed 126.23 samples/sec Loss 12.3366 Epoch: 2 Global Step: 26910 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:35:35,716-Speed 122.79 samples/sec Loss 12.3073 Epoch: 2 Global Step: 26915 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:35:56,244-Speed 124.71 samples/sec Loss 12.1807 Epoch: 2 Global Step: 26920 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:36:16,440-Speed 126.76 samples/sec Loss 12.2765 Epoch: 2 Global Step: 26925 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:36:37,237-Speed 123.10 samples/sec Loss 12.1594 Epoch: 2 Global Step: 26930 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:36:57,562-Speed 125.96 samples/sec Loss 12.1916 Epoch: 2 Global Step: 26935 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:37:17,759-Speed 126.76 samples/sec Loss 12.3219 Epoch: 2 Global Step: 26940 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:37:38,046-Speed 126.20 samples/sec Loss 12.1515 Epoch: 2 Global Step: 26945 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:37:58,228-Speed 126.85 samples/sec Loss 12.0619 Epoch: 2 Global Step: 26950 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:38:18,519-Speed 126.17 samples/sec Loss 12.3409 Epoch: 2 Global Step: 26955 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:38:39,500-Speed 122.02 samples/sec Loss 12.0162 Epoch: 2 Global Step: 26960 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:38:59,793-Speed 126.16 samples/sec Loss 11.9365 Epoch: 2 Global Step: 26965 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:39:19,984-Speed 126.79 samples/sec Loss 12.2170 Epoch: 2 Global Step: 26970 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:39:40,273-Speed 126.19 samples/sec Loss 12.1482 Epoch: 2 Global Step: 26975 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:40:00,933-Speed 123.92 samples/sec Loss 11.5370 Epoch: 2 Global Step: 26980 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:40:21,220-Speed 126.20 samples/sec Loss 11.9503 Epoch: 2 Global Step: 26985 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:40:41,535-Speed 126.02 samples/sec Loss 12.3665 Epoch: 2 Global Step: 26990 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:41:01,853-Speed 126.00 samples/sec Loss 11.9879 Epoch: 2 Global Step: 26995 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:41:22,149-Speed 126.14 samples/sec Loss 12.3550 Epoch: 2 Global Step: 27000 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:41:43,208-Speed 121.57 samples/sec Loss 12.2242 Epoch: 2 Global Step: 27005 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:42:03,519-Speed 126.05 samples/sec Loss 12.0717 Epoch: 2 Global Step: 27010 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:42:24,210-Speed 123.73 samples/sec Loss 11.9797 Epoch: 2 Global Step: 27015 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:42:44,468-Speed 126.38 samples/sec Loss 12.2324 Epoch: 2 Global Step: 27020 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:43:04,544-Speed 127.52 samples/sec Loss 12.2580 Epoch: 2 Global Step: 27025 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:43:25,103-Speed 124.53 samples/sec Loss 12.1725 Epoch: 2 Global Step: 27030 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:43:45,289-Speed 126.83 samples/sec Loss 11.9661 Epoch: 2 Global Step: 27035 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:44:05,784-Speed 124.91 samples/sec Loss 11.9246 Epoch: 2 Global Step: 27040 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:44:26,019-Speed 126.52 samples/sec Loss 12.0963 Epoch: 2 Global Step: 27045 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:44:47,074-Speed 121.59 samples/sec Loss 12.0387 Epoch: 2 Global Step: 27050 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:45:07,758-Speed 123.77 samples/sec Loss 12.2153 Epoch: 2 Global Step: 27055 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:45:28,432-Speed 123.83 samples/sec Loss 11.9623 Epoch: 2 Global Step: 27060 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:45:48,747-Speed 126.02 samples/sec Loss 11.9745 Epoch: 2 Global Step: 27065 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:46:09,009-Speed 126.35 samples/sec Loss 11.9342 Epoch: 2 Global Step: 27070 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:46:29,332-Speed 125.98 samples/sec Loss 12.5145 Epoch: 2 Global Step: 27075 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:46:49,639-Speed 126.08 samples/sec Loss 12.2925 Epoch: 2 Global Step: 27080 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:47:09,970-Speed 125.93 samples/sec Loss 12.1793 Epoch: 2 Global Step: 27085 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:47:30,272-Speed 126.10 samples/sec Loss 12.0558 Epoch: 2 Global Step: 27090 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:47:51,313-Speed 121.67 samples/sec Loss 11.7715 Epoch: 2 Global Step: 27095 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:48:11,612-Speed 126.12 samples/sec Loss 11.7149 Epoch: 2 Global Step: 27100 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:48:31,926-Speed 126.03 samples/sec Loss 12.0586 Epoch: 2 Global Step: 27105 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:48:52,188-Speed 126.35 samples/sec Loss 12.4800 Epoch: 2 Global Step: 27110 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:49:12,496-Speed 126.07 samples/sec Loss 12.5202 Epoch: 2 Global Step: 27115 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:49:32,770-Speed 126.28 samples/sec Loss 12.2150 Epoch: 2 Global Step: 27120 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:49:53,092-Speed 125.98 samples/sec Loss 11.9198 Epoch: 2 Global Step: 27125 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:50:13,145-Speed 127.67 samples/sec Loss 11.6799 Epoch: 2 Global Step: 27130 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:50:33,493-Speed 125.82 samples/sec Loss 12.2707 Epoch: 2 Global Step: 27135 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:50:54,619-Speed 121.18 samples/sec Loss 12.1774 Epoch: 2 Global Step: 27140 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:51:14,954-Speed 125.90 samples/sec Loss 11.8912 Epoch: 2 Global Step: 27145 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:51:35,177-Speed 126.59 samples/sec Loss 12.2282 Epoch: 2 Global Step: 27150 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:51:55,276-Speed 127.38 samples/sec Loss 12.2091 Epoch: 2 Global Step: 27155 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:52:15,635-Speed 125.75 samples/sec Loss 12.3598 Epoch: 2 Global Step: 27160 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:52:35,971-Speed 125.90 samples/sec Loss 12.1048 Epoch: 2 Global Step: 27165 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:52:56,114-Speed 127.10 samples/sec Loss 12.0555 Epoch: 2 Global Step: 27170 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:53:16,463-Speed 125.81 samples/sec Loss 12.3355 Epoch: 2 Global Step: 27175 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:53:36,866-Speed 125.49 samples/sec Loss 12.0564 Epoch: 2 Global Step: 27180 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:53:57,909-Speed 121.66 samples/sec Loss 12.0068 Epoch: 2 Global Step: 27185 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:54:18,310-Speed 125.49 samples/sec Loss 12.4472 Epoch: 2 Global Step: 27190 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:54:38,566-Speed 126.39 samples/sec Loss 12.1796 Epoch: 2 Global Step: 27195 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:54:59,269-Speed 123.66 samples/sec Loss 11.9699 Epoch: 2 Global Step: 27200 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:55:20,234-Speed 122.11 samples/sec Loss 11.9795 Epoch: 2 Global Step: 27205 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:55:40,497-Speed 126.34 samples/sec Loss 12.3024 Epoch: 2 Global Step: 27210 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:56:00,949-Speed 125.18 samples/sec Loss 12.1118 Epoch: 2 Global Step: 27215 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:56:21,637-Speed 123.75 samples/sec Loss 12.0567 Epoch: 2 Global Step: 27220 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:56:42,159-Speed 124.75 samples/sec Loss 12.1098 Epoch: 2 Global Step: 27225 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:57:03,224-Speed 121.54 samples/sec Loss 12.2386 Epoch: 2 Global Step: 27230 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:57:23,602-Speed 125.63 samples/sec Loss 12.3822 Epoch: 2 Global Step: 27235 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:57:43,871-Speed 126.31 samples/sec Loss 12.1156 Epoch: 2 Global Step: 27240 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:58:04,239-Speed 125.70 samples/sec Loss 12.3216 Epoch: 2 Global Step: 27245 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:58:24,524-Speed 126.21 samples/sec Loss 12.0807 Epoch: 2 Global Step: 27250 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:58:44,820-Speed 126.14 samples/sec Loss 12.0285 Epoch: 2 Global Step: 27255 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:59:05,089-Speed 126.30 samples/sec Loss 12.0589 Epoch: 2 Global Step: 27260 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:59:25,414-Speed 125.96 samples/sec Loss 11.9529 Epoch: 2 Global Step: 27265 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 02:59:45,677-Speed 126.34 samples/sec Loss 11.9370 Epoch: 2 Global Step: 27270 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:00:06,483-Speed 123.05 samples/sec Loss 12.0665 Epoch: 2 Global Step: 27275 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:00:26,762-Speed 126.24 samples/sec Loss 12.1471 Epoch: 2 Global Step: 27280 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:00:47,093-Speed 125.92 samples/sec Loss 12.3628 Epoch: 2 Global Step: 27285 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:01:07,538-Speed 125.22 samples/sec Loss 12.0129 Epoch: 2 Global Step: 27290 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:01:27,999-Speed 125.12 samples/sec Loss 12.0694 Epoch: 2 Global Step: 27295 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:01:48,383-Speed 125.59 samples/sec Loss 12.5642 Epoch: 2 Global Step: 27300 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:02:08,635-Speed 126.42 samples/sec Loss 12.5869 Epoch: 2 Global Step: 27305 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:02:29,080-Speed 125.22 samples/sec Loss 12.0622 Epoch: 2 Global Step: 27310 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:02:49,278-Speed 126.75 samples/sec Loss 11.8666 Epoch: 2 Global Step: 27315 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:03:10,168-Speed 122.55 samples/sec Loss 12.0590 Epoch: 2 Global Step: 27320 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:03:30,250-Speed 127.48 samples/sec Loss 12.1295 Epoch: 2 Global Step: 27325 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:03:50,876-Speed 124.12 samples/sec Loss 12.2173 Epoch: 2 Global Step: 27330 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:04:11,204-Speed 125.94 samples/sec Loss 12.1565 Epoch: 2 Global Step: 27335 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:04:31,543-Speed 125.87 samples/sec Loss 12.3725 Epoch: 2 Global Step: 27340 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:04:51,903-Speed 125.75 samples/sec Loss 12.3441 Epoch: 2 Global Step: 27345 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:05:12,162-Speed 126.37 samples/sec Loss 11.9348 Epoch: 2 Global Step: 27350 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:05:32,826-Speed 123.89 samples/sec Loss 12.1981 Epoch: 2 Global Step: 27355 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:05:53,021-Speed 126.77 samples/sec Loss 12.0508 Epoch: 2 Global Step: 27360 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:06:14,049-Speed 121.75 samples/sec Loss 12.1371 Epoch: 2 Global Step: 27365 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:06:34,464-Speed 125.40 samples/sec Loss 12.0893 Epoch: 2 Global Step: 27370 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 03:06:54,876-Speed 125.43 samples/sec Loss 12.2632 Epoch: 2 Global Step: 27375 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:07:15,238-Speed 125.73 samples/sec Loss 12.0043 Epoch: 2 Global Step: 27380 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:07:35,630-Speed 125.55 samples/sec Loss 12.3050 Epoch: 2 Global Step: 27385 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:07:56,284-Speed 123.96 samples/sec Loss 11.8542 Epoch: 2 Global Step: 27390 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:08:16,731-Speed 125.21 samples/sec Loss 12.2542 Epoch: 2 Global Step: 27395 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:08:37,086-Speed 125.77 samples/sec Loss 11.9750 Epoch: 2 Global Step: 27400 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:08:57,431-Speed 125.84 samples/sec Loss 12.0505 Epoch: 2 Global Step: 27405 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:09:18,409-Speed 122.04 samples/sec Loss 12.0757 Epoch: 2 Global Step: 27410 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:09:38,752-Speed 125.85 samples/sec Loss 12.1105 Epoch: 2 Global Step: 27415 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:09:58,765-Speed 127.92 samples/sec Loss 12.1284 Epoch: 2 Global Step: 27420 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:10:19,247-Speed 124.99 samples/sec Loss 12.2903 Epoch: 2 Global Step: 27425 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:10:39,480-Speed 126.54 samples/sec Loss 12.1369 Epoch: 2 Global Step: 27430 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:11:00,109-Speed 124.11 samples/sec Loss 12.3846 Epoch: 2 Global Step: 27435 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:11:20,450-Speed 125.86 samples/sec Loss 12.1437 Epoch: 2 Global Step: 27440 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:11:40,724-Speed 126.28 samples/sec Loss 12.3370 Epoch: 2 Global Step: 27445 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:12:01,101-Speed 125.64 samples/sec Loss 12.2188 Epoch: 2 Global Step: 27450 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:12:22,486-Speed 119.72 samples/sec Loss 12.1206 Epoch: 2 Global Step: 27455 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:12:42,784-Speed 126.12 samples/sec Loss 11.9273 Epoch: 2 Global Step: 27460 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:13:03,148-Speed 125.72 samples/sec Loss 12.1865 Epoch: 2 Global Step: 27465 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:13:23,355-Speed 126.69 samples/sec Loss 11.9418 Epoch: 2 Global Step: 27470 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:13:43,990-Speed 124.07 samples/sec Loss 12.4602 Epoch: 2 Global Step: 27475 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:14:04,309-Speed 126.00 samples/sec Loss 12.2607 Epoch: 2 Global Step: 27480 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:14:25,088-Speed 123.21 samples/sec Loss 12.0054 Epoch: 2 Global Step: 27485 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:14:45,399-Speed 126.05 samples/sec Loss 11.8764 Epoch: 2 Global Step: 27490 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:15:05,708-Speed 126.06 samples/sec Loss 12.1965 Epoch: 2 Global Step: 27495 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:15:26,801-Speed 121.38 samples/sec Loss 12.3671 Epoch: 2 Global Step: 27500 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:15:47,107-Speed 126.07 samples/sec Loss 12.1314 Epoch: 2 Global Step: 27505 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:16:07,919-Speed 123.01 samples/sec Loss 12.1467 Epoch: 2 Global Step: 27510 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:16:28,387-Speed 125.09 samples/sec Loss 11.9486 Epoch: 2 Global Step: 27515 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:16:48,694-Speed 126.07 samples/sec Loss 11.8185 Epoch: 2 Global Step: 27520 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:17:09,066-Speed 125.67 samples/sec Loss 11.9703 Epoch: 2 Global Step: 27525 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:17:29,473-Speed 125.46 samples/sec Loss 12.1351 Epoch: 2 Global Step: 27530 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:17:49,821-Speed 125.82 samples/sec Loss 12.1525 Epoch: 2 Global Step: 27535 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:18:10,109-Speed 126.19 samples/sec Loss 12.2096 Epoch: 2 Global Step: 27540 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:18:31,192-Speed 121.43 samples/sec Loss 12.1080 Epoch: 2 Global Step: 27545 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:18:51,507-Speed 126.02 samples/sec Loss 11.8644 Epoch: 2 Global Step: 27550 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:19:11,769-Speed 126.35 samples/sec Loss 11.8201 Epoch: 2 Global Step: 27555 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:19:32,115-Speed 125.83 samples/sec Loss 12.2251 Epoch: 2 Global Step: 27560 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:19:52,340-Speed 126.58 samples/sec Loss 12.1463 Epoch: 2 Global Step: 27565 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:20:12,639-Speed 126.13 samples/sec Loss 12.1987 Epoch: 2 Global Step: 27570 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:20:32,701-Speed 127.61 samples/sec Loss 11.6595 Epoch: 2 Global Step: 27575 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:20:52,994-Speed 126.16 samples/sec Loss 11.7401 Epoch: 2 Global Step: 27580 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:21:13,202-Speed 126.69 samples/sec Loss 12.2032 Epoch: 2 Global Step: 27585 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:21:34,073-Speed 122.67 samples/sec Loss 11.9495 Epoch: 2 Global Step: 27590 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:21:54,283-Speed 126.68 samples/sec Loss 12.1905 Epoch: 2 Global Step: 27595 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:22:14,575-Speed 126.16 samples/sec Loss 11.9893 Epoch: 2 Global Step: 27600 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:22:34,691-Speed 127.27 samples/sec Loss 12.1761 Epoch: 2 Global Step: 27605 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:22:55,094-Speed 125.48 samples/sec Loss 11.9096 Epoch: 2 Global Step: 27610 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:23:16,129-Speed 121.71 samples/sec Loss 12.0109 Epoch: 2 Global Step: 27615 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:23:36,434-Speed 126.09 samples/sec Loss 11.9078 Epoch: 2 Global Step: 27620 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:23:56,626-Speed 126.79 samples/sec Loss 11.9993 Epoch: 2 Global Step: 27625 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:24:17,194-Speed 124.48 samples/sec Loss 12.3986 Epoch: 2 Global Step: 27630 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:24:38,266-Speed 121.49 samples/sec Loss 12.3086 Epoch: 2 Global Step: 27635 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:24:58,666-Speed 125.49 samples/sec Loss 12.4817 Epoch: 2 Global Step: 27640 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:25:18,788-Speed 127.23 samples/sec Loss 11.8457 Epoch: 2 Global Step: 27645 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:25:39,120-Speed 125.91 samples/sec Loss 12.1053 Epoch: 2 Global Step: 27650 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:25:59,517-Speed 125.52 samples/sec Loss 12.0145 Epoch: 2 Global Step: 27655 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:26:19,839-Speed 125.99 samples/sec Loss 12.0771 Epoch: 2 Global Step: 27660 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:26:40,023-Speed 126.84 samples/sec Loss 12.1159 Epoch: 2 Global Step: 27665 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:27:00,510-Speed 124.97 samples/sec Loss 12.1214 Epoch: 2 Global Step: 27670 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:27:20,961-Speed 125.19 samples/sec Loss 12.3733 Epoch: 2 Global Step: 27675 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:27:41,968-Speed 121.87 samples/sec Loss 12.0701 Epoch: 2 Global Step: 27680 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:28:02,072-Speed 127.34 samples/sec Loss 12.5248 Epoch: 2 Global Step: 27685 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:28:22,364-Speed 126.16 samples/sec Loss 12.0477 Epoch: 2 Global Step: 27690 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:28:42,656-Speed 126.17 samples/sec Loss 11.9392 Epoch: 2 Global Step: 27695 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:29:02,989-Speed 125.91 samples/sec Loss 11.9559 Epoch: 2 Global Step: 27700 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:29:23,331-Speed 125.86 samples/sec Loss 11.8616 Epoch: 2 Global Step: 27705 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:29:43,822-Speed 124.94 samples/sec Loss 12.3227 Epoch: 2 Global Step: 27710 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:30:04,150-Speed 125.94 samples/sec Loss 11.8288 Epoch: 2 Global Step: 27715 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:30:24,433-Speed 126.22 samples/sec Loss 11.8951 Epoch: 2 Global Step: 27720 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:30:45,286-Speed 122.77 samples/sec Loss 12.1741 Epoch: 2 Global Step: 27725 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:31:05,421-Speed 127.15 samples/sec Loss 12.2867 Epoch: 2 Global Step: 27730 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:31:25,786-Speed 125.71 samples/sec Loss 12.2427 Epoch: 2 Global Step: 27735 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:31:46,323-Speed 124.66 samples/sec Loss 11.9985 Epoch: 2 Global Step: 27740 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:32:06,889-Speed 124.48 samples/sec Loss 12.2270 Epoch: 2 Global Step: 27745 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:32:27,185-Speed 126.14 samples/sec Loss 12.2441 Epoch: 2 Global Step: 27750 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:32:47,352-Speed 126.95 samples/sec Loss 12.0311 Epoch: 2 Global Step: 27755 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:33:07,650-Speed 126.13 samples/sec Loss 12.1689 Epoch: 2 Global Step: 27760 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:33:28,006-Speed 125.77 samples/sec Loss 11.8741 Epoch: 2 Global Step: 27765 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:33:49,694-Speed 118.04 samples/sec Loss 11.9421 Epoch: 2 Global Step: 27770 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:34:10,035-Speed 125.86 samples/sec Loss 12.4273 Epoch: 2 Global Step: 27775 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:34:30,418-Speed 125.60 samples/sec Loss 12.1015 Epoch: 2 Global Step: 27780 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:34:50,794-Speed 125.65 samples/sec Loss 11.9438 Epoch: 2 Global Step: 27785 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:35:11,145-Speed 125.80 samples/sec Loss 12.3281 Epoch: 2 Global Step: 27790 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:35:31,564-Speed 125.38 samples/sec Loss 11.9791 Epoch: 2 Global Step: 27795 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:35:51,858-Speed 126.15 samples/sec Loss 12.3713 Epoch: 2 Global Step: 27800 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:36:12,694-Speed 122.88 samples/sec Loss 11.8206 Epoch: 2 Global Step: 27805 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:36:32,942-Speed 126.44 samples/sec Loss 12.2289 Epoch: 2 Global Step: 27810 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:36:53,721-Speed 123.21 samples/sec Loss 12.0890 Epoch: 2 Global Step: 27815 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:37:14,011-Speed 126.18 samples/sec Loss 12.2909 Epoch: 2 Global Step: 27820 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:37:34,217-Speed 126.70 samples/sec Loss 12.3438 Epoch: 2 Global Step: 27825 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:37:54,547-Speed 125.93 samples/sec Loss 12.2834 Epoch: 2 Global Step: 27830 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:38:14,825-Speed 126.26 samples/sec Loss 12.4261 Epoch: 2 Global Step: 27835 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:38:34,976-Speed 127.05 samples/sec Loss 11.9915 Epoch: 2 Global Step: 27840 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:38:55,126-Speed 127.06 samples/sec Loss 12.2730 Epoch: 2 Global Step: 27845 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:39:15,660-Speed 124.69 samples/sec Loss 12.2632 Epoch: 2 Global Step: 27850 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:39:36,118-Speed 125.14 samples/sec Loss 12.1868 Epoch: 2 Global Step: 27855 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:39:56,907-Speed 123.15 samples/sec Loss 11.9851 Epoch: 2 Global Step: 27860 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:40:17,025-Speed 127.26 samples/sec Loss 12.1487 Epoch: 2 Global Step: 27865 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:40:37,314-Speed 126.18 samples/sec Loss 12.1865 Epoch: 2 Global Step: 27870 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:40:57,905-Speed 124.33 samples/sec Loss 12.3372 Epoch: 2 Global Step: 27875 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:41:18,192-Speed 126.21 samples/sec Loss 11.9564 Epoch: 2 Global Step: 27880 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:41:38,457-Speed 126.33 samples/sec Loss 12.1651 Epoch: 2 Global Step: 27885 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:41:58,494-Speed 127.78 samples/sec Loss 12.0681 Epoch: 2 Global Step: 27890 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:42:18,935-Speed 125.25 samples/sec Loss 12.1504 Epoch: 2 Global Step: 27895 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:42:39,285-Speed 125.80 samples/sec Loss 12.0765 Epoch: 2 Global Step: 27900 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:43:00,286-Speed 121.91 samples/sec Loss 11.9135 Epoch: 2 Global Step: 27905 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:43:21,240-Speed 122.18 samples/sec Loss 12.0900 Epoch: 2 Global Step: 27910 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:43:41,518-Speed 126.25 samples/sec Loss 11.8889 Epoch: 2 Global Step: 27915 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:44:01,996-Speed 125.02 samples/sec Loss 12.0014 Epoch: 2 Global Step: 27920 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:44:22,247-Speed 126.42 samples/sec Loss 12.0821 Epoch: 2 Global Step: 27925 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:44:42,622-Speed 125.66 samples/sec Loss 11.8512 Epoch: 2 Global Step: 27930 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:45:02,931-Speed 126.06 samples/sec Loss 11.8658 Epoch: 2 Global Step: 27935 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:45:23,212-Speed 126.23 samples/sec Loss 11.8008 Epoch: 2 Global Step: 27940 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:45:43,495-Speed 126.22 samples/sec Loss 11.8947 Epoch: 2 Global Step: 27945 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:46:04,625-Speed 121.16 samples/sec Loss 11.8337 Epoch: 2 Global Step: 27950 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:46:24,776-Speed 127.05 samples/sec Loss 11.6081 Epoch: 2 Global Step: 27955 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:46:45,230-Speed 125.16 samples/sec Loss 11.9127 Epoch: 2 Global Step: 27960 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:47:05,509-Speed 126.25 samples/sec Loss 12.4493 Epoch: 2 Global Step: 27965 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:47:25,795-Speed 126.20 samples/sec Loss 12.3001 Epoch: 2 Global Step: 27970 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:47:46,109-Speed 126.03 samples/sec Loss 12.1345 Epoch: 2 Global Step: 27975 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:48:06,425-Speed 126.02 samples/sec Loss 12.0514 Epoch: 2 Global Step: 27980 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:48:26,649-Speed 126.59 samples/sec Loss 11.9602 Epoch: 2 Global Step: 27985 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:48:46,959-Speed 126.06 samples/sec Loss 12.0858 Epoch: 2 Global Step: 27990 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:49:08,003-Speed 121.66 samples/sec Loss 12.1220 Epoch: 2 Global Step: 27995 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:49:28,358-Speed 125.77 samples/sec Loss 11.9315 Epoch: 2 Global Step: 28000 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:49:48,845-Speed 124.96 samples/sec Loss 12.3305 Epoch: 2 Global Step: 28005 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:50:09,221-Speed 125.65 samples/sec Loss 12.2116 Epoch: 2 Global Step: 28010 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:50:29,602-Speed 125.61 samples/sec Loss 12.1302 Epoch: 2 Global Step: 28015 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:50:49,909-Speed 126.07 samples/sec Loss 12.1357 Epoch: 2 Global Step: 28020 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:51:10,120-Speed 126.67 samples/sec Loss 12.1106 Epoch: 2 Global Step: 28025 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:51:30,223-Speed 127.35 samples/sec Loss 12.0032 Epoch: 2 Global Step: 28030 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:51:50,514-Speed 126.17 samples/sec Loss 12.0935 Epoch: 2 Global Step: 28035 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:52:11,635-Speed 121.21 samples/sec Loss 12.1384 Epoch: 2 Global Step: 28040 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:52:31,986-Speed 125.80 samples/sec Loss 12.2061 Epoch: 2 Global Step: 28045 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:52:52,749-Speed 123.30 samples/sec Loss 12.0405 Epoch: 2 Global Step: 28050 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:53:13,342-Speed 124.32 samples/sec Loss 12.0583 Epoch: 2 Global Step: 28055 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:53:33,713-Speed 125.67 samples/sec Loss 11.8866 Epoch: 2 Global Step: 28060 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 03:53:54,067-Speed 125.79 samples/sec Loss 12.4266 Epoch: 2 Global Step: 28065 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:54:14,351-Speed 126.22 samples/sec Loss 12.0230 Epoch: 2 Global Step: 28070 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:54:34,774-Speed 125.35 samples/sec Loss 11.9002 Epoch: 2 Global Step: 28075 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:54:55,133-Speed 125.75 samples/sec Loss 12.2058 Epoch: 2 Global Step: 28080 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:55:16,894-Speed 117.65 samples/sec Loss 11.7998 Epoch: 2 Global Step: 28085 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:55:37,210-Speed 126.01 samples/sec Loss 12.3431 Epoch: 2 Global Step: 28090 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:55:57,583-Speed 125.66 samples/sec Loss 12.0467 Epoch: 2 Global Step: 28095 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:56:18,093-Speed 124.83 samples/sec Loss 11.9367 Epoch: 2 Global Step: 28100 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:56:38,440-Speed 125.82 samples/sec Loss 12.2975 Epoch: 2 Global Step: 28105 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:56:58,723-Speed 126.22 samples/sec Loss 11.9628 Epoch: 2 Global Step: 28110 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:57:18,960-Speed 126.51 samples/sec Loss 11.8767 Epoch: 2 Global Step: 28115 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:57:39,303-Speed 125.85 samples/sec Loss 12.1894 Epoch: 2 Global Step: 28120 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:57:59,704-Speed 125.49 samples/sec Loss 12.3498 Epoch: 2 Global Step: 28125 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:58:20,689-Speed 122.00 samples/sec Loss 12.0245 Epoch: 2 Global Step: 28130 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:58:41,221-Speed 124.69 samples/sec Loss 12.2518 Epoch: 2 Global Step: 28135 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:59:01,652-Speed 125.31 samples/sec Loss 12.3051 Epoch: 2 Global Step: 28140 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:59:22,007-Speed 125.77 samples/sec Loss 12.0495 Epoch: 2 Global Step: 28145 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 03:59:42,488-Speed 125.00 samples/sec Loss 12.2455 Epoch: 2 Global Step: 28150 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:00:02,792-Speed 126.09 samples/sec Loss 12.1010 Epoch: 2 Global Step: 28155 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:00:23,141-Speed 125.81 samples/sec Loss 12.0267 Epoch: 2 Global Step: 28160 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:00:43,707-Speed 124.48 samples/sec Loss 12.0742 Epoch: 2 Global Step: 28165 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:01:04,054-Speed 125.83 samples/sec Loss 12.3279 Epoch: 2 Global Step: 28170 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:01:25,083-Speed 121.74 samples/sec Loss 12.1014 Epoch: 2 Global Step: 28175 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:01:45,338-Speed 126.39 samples/sec Loss 11.9594 Epoch: 2 Global Step: 28180 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:02:05,647-Speed 126.06 samples/sec Loss 12.4220 Epoch: 2 Global Step: 28185 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:02:25,903-Speed 126.39 samples/sec Loss 12.0391 Epoch: 2 Global Step: 28190 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:02:45,968-Speed 127.59 samples/sec Loss 12.1574 Epoch: 2 Global Step: 28195 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:03:06,273-Speed 126.09 samples/sec Loss 12.1208 Epoch: 2 Global Step: 28200 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:03:26,558-Speed 126.21 samples/sec Loss 12.0847 Epoch: 2 Global Step: 28205 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:03:46,934-Speed 125.64 samples/sec Loss 11.8358 Epoch: 2 Global Step: 28210 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:04:07,136-Speed 126.73 samples/sec Loss 12.1594 Epoch: 2 Global Step: 28215 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:04:28,131-Speed 121.94 samples/sec Loss 12.2000 Epoch: 2 Global Step: 28220 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:04:48,433-Speed 126.10 samples/sec Loss 11.8424 Epoch: 2 Global Step: 28225 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:05:08,586-Speed 127.03 samples/sec Loss 12.2900 Epoch: 2 Global Step: 28230 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:05:28,980-Speed 125.54 samples/sec Loss 11.8410 Epoch: 2 Global Step: 28235 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:05:49,387-Speed 125.45 samples/sec Loss 11.8548 Epoch: 2 Global Step: 28240 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:06:09,448-Speed 127.62 samples/sec Loss 11.8912 Epoch: 2 Global Step: 28245 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:06:29,603-Speed 127.02 samples/sec Loss 11.8833 Epoch: 2 Global Step: 28250 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:06:49,932-Speed 125.94 samples/sec Loss 11.9933 Epoch: 2 Global Step: 28255 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:07:10,236-Speed 126.09 samples/sec Loss 12.1752 Epoch: 2 Global Step: 28260 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:07:31,487-Speed 120.47 samples/sec Loss 12.0848 Epoch: 2 Global Step: 28265 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:07:51,798-Speed 126.04 samples/sec Loss 12.2693 Epoch: 2 Global Step: 28270 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:08:12,113-Speed 126.02 samples/sec Loss 12.1334 Epoch: 2 Global Step: 28275 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:08:32,531-Speed 125.39 samples/sec Loss 12.1768 Epoch: 2 Global Step: 28280 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:08:52,865-Speed 125.91 samples/sec Loss 12.0531 Epoch: 2 Global Step: 28285 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:09:13,016-Speed 127.05 samples/sec Loss 11.8636 Epoch: 2 Global Step: 28290 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:09:33,009-Speed 128.05 samples/sec Loss 11.8788 Epoch: 2 Global Step: 28295 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:09:53,339-Speed 125.93 samples/sec Loss 12.5764 Epoch: 2 Global Step: 28300 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:10:13,713-Speed 125.66 samples/sec Loss 11.9471 Epoch: 2 Global Step: 28305 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:10:34,832-Speed 121.22 samples/sec Loss 12.3190 Epoch: 2 Global Step: 28310 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:10:55,121-Speed 126.18 samples/sec Loss 12.1652 Epoch: 2 Global Step: 28315 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:11:15,492-Speed 125.67 samples/sec Loss 12.2744 Epoch: 2 Global Step: 28320 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:11:36,749-Speed 120.44 samples/sec Loss 12.3386 Epoch: 2 Global Step: 28325 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:11:57,294-Speed 124.62 samples/sec Loss 12.3888 Epoch: 2 Global Step: 28330 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:12:17,555-Speed 126.36 samples/sec Loss 11.9201 Epoch: 2 Global Step: 28335 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:12:37,936-Speed 125.61 samples/sec Loss 11.9134 Epoch: 2 Global Step: 28340 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:12:58,245-Speed 126.06 samples/sec Loss 12.2000 Epoch: 2 Global Step: 28345 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:13:18,501-Speed 126.40 samples/sec Loss 12.3555 Epoch: 2 Global Step: 28350 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:13:39,628-Speed 121.18 samples/sec Loss 12.2660 Epoch: 2 Global Step: 28355 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:13:59,789-Speed 126.98 samples/sec Loss 11.8702 Epoch: 2 Global Step: 28360 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:14:20,059-Speed 126.30 samples/sec Loss 12.9665 Epoch: 2 Global Step: 28365 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:14:40,866-Speed 123.05 samples/sec Loss 11.9653 Epoch: 2 Global Step: 28370 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:15:01,292-Speed 125.33 samples/sec Loss 12.3004 Epoch: 2 Global Step: 28375 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:15:21,596-Speed 126.10 samples/sec Loss 11.6958 Epoch: 2 Global Step: 28380 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:15:41,860-Speed 126.34 samples/sec Loss 12.0709 Epoch: 2 Global Step: 28385 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:16:02,237-Speed 125.64 samples/sec Loss 12.0798 Epoch: 2 Global Step: 28390 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:16:22,644-Speed 125.46 samples/sec Loss 12.3793 Epoch: 2 Global Step: 28395 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:16:43,546-Speed 122.48 samples/sec Loss 12.0726 Epoch: 2 Global Step: 28400 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:17:03,723-Speed 126.88 samples/sec Loss 11.9904 Epoch: 2 Global Step: 28405 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:17:23,992-Speed 126.31 samples/sec Loss 12.1006 Epoch: 2 Global Step: 28410 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:17:44,296-Speed 126.09 samples/sec Loss 11.7668 Epoch: 2 Global Step: 28415 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:18:04,439-Speed 127.10 samples/sec Loss 11.9719 Epoch: 2 Global Step: 28420 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:18:24,484-Speed 127.73 samples/sec Loss 12.0316 Epoch: 2 Global Step: 28425 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:18:44,739-Speed 126.40 samples/sec Loss 12.0286 Epoch: 2 Global Step: 28430 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:19:05,062-Speed 125.97 samples/sec Loss 12.3055 Epoch: 2 Global Step: 28435 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:19:25,280-Speed 126.63 samples/sec Loss 12.0917 Epoch: 2 Global Step: 28440 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:19:46,458-Speed 120.89 samples/sec Loss 11.9755 Epoch: 2 Global Step: 28445 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:20:06,781-Speed 125.97 samples/sec Loss 12.0767 Epoch: 2 Global Step: 28450 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:20:27,015-Speed 126.53 samples/sec Loss 12.0698 Epoch: 2 Global Step: 28455 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:20:47,298-Speed 126.22 samples/sec Loss 12.2866 Epoch: 2 Global Step: 28460 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:21:07,591-Speed 126.16 samples/sec Loss 11.9277 Epoch: 2 Global Step: 28465 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:21:27,768-Speed 126.89 samples/sec Loss 11.9984 Epoch: 2 Global Step: 28470 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:21:48,040-Speed 126.29 samples/sec Loss 11.9628 Epoch: 2 Global Step: 28475 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:22:08,267-Speed 126.57 samples/sec Loss 12.3068 Epoch: 2 Global Step: 28480 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:22:28,250-Speed 128.11 samples/sec Loss 11.9652 Epoch: 2 Global Step: 28485 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:22:48,901-Speed 123.97 samples/sec Loss 11.9817 Epoch: 2 Global Step: 28490 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:23:09,480-Speed 124.40 samples/sec Loss 12.1324 Epoch: 2 Global Step: 28495 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:23:29,761-Speed 126.23 samples/sec Loss 12.4469 Epoch: 2 Global Step: 28500 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:23:50,099-Speed 125.88 samples/sec Loss 12.0230 Epoch: 2 Global Step: 28505 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:24:10,384-Speed 126.20 samples/sec Loss 11.9225 Epoch: 2 Global Step: 28510 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:24:30,744-Speed 125.75 samples/sec Loss 11.9881 Epoch: 2 Global Step: 28515 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:24:50,832-Speed 127.44 samples/sec Loss 11.7370 Epoch: 2 Global Step: 28520 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:25:11,141-Speed 126.07 samples/sec Loss 12.3416 Epoch: 2 Global Step: 28525 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:25:31,452-Speed 126.04 samples/sec Loss 12.2019 Epoch: 2 Global Step: 28530 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:25:52,479-Speed 121.76 samples/sec Loss 11.8669 Epoch: 2 Global Step: 28535 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:26:13,398-Speed 122.38 samples/sec Loss 12.3306 Epoch: 2 Global Step: 28540 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:26:33,731-Speed 125.91 samples/sec Loss 12.2324 Epoch: 2 Global Step: 28545 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:26:54,047-Speed 126.01 samples/sec Loss 11.8701 Epoch: 2 Global Step: 28550 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:27:14,377-Speed 125.93 samples/sec Loss 11.9554 Epoch: 2 Global Step: 28555 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:27:34,562-Speed 126.83 samples/sec Loss 12.0869 Epoch: 2 Global Step: 28560 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:27:55,160-Speed 124.29 samples/sec Loss 12.1284 Epoch: 2 Global Step: 28565 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:28:15,236-Speed 127.52 samples/sec Loss 11.9274 Epoch: 2 Global Step: 28570 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:28:35,551-Speed 126.02 samples/sec Loss 12.1895 Epoch: 2 Global Step: 28575 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:28:56,252-Speed 123.67 samples/sec Loss 12.1807 Epoch: 2 Global Step: 28580 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:29:16,582-Speed 125.93 samples/sec Loss 12.1992 Epoch: 2 Global Step: 28585 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:29:37,069-Speed 124.96 samples/sec Loss 12.0718 Epoch: 2 Global Step: 28590 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:29:57,391-Speed 125.98 samples/sec Loss 11.7468 Epoch: 2 Global Step: 28595 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:30:17,434-Speed 127.73 samples/sec Loss 12.2896 Epoch: 2 Global Step: 28600 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:30:37,692-Speed 126.38 samples/sec Loss 12.0260 Epoch: 2 Global Step: 28605 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:30:57,841-Speed 127.06 samples/sec Loss 11.7229 Epoch: 2 Global Step: 28610 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:31:17,993-Speed 127.04 samples/sec Loss 12.0208 Epoch: 2 Global Step: 28615 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:31:38,314-Speed 125.98 samples/sec Loss 12.0394 Epoch: 2 Global Step: 28620 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:31:59,238-Speed 122.35 samples/sec Loss 12.0787 Epoch: 2 Global Step: 28625 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:32:19,546-Speed 126.07 samples/sec Loss 12.0409 Epoch: 2 Global Step: 28630 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:32:39,903-Speed 125.76 samples/sec Loss 11.9903 Epoch: 2 Global Step: 28635 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:33:00,259-Speed 125.76 samples/sec Loss 11.9735 Epoch: 2 Global Step: 28640 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:33:20,525-Speed 126.33 samples/sec Loss 12.0758 Epoch: 2 Global Step: 28645 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:33:40,869-Speed 125.84 samples/sec Loss 11.7935 Epoch: 2 Global Step: 28650 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:34:01,151-Speed 126.23 samples/sec Loss 12.0882 Epoch: 2 Global Step: 28655 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:34:21,529-Speed 125.63 samples/sec Loss 12.1283 Epoch: 2 Global Step: 28660 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:34:41,817-Speed 126.19 samples/sec Loss 12.3291 Epoch: 2 Global Step: 28665 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:35:02,721-Speed 122.47 samples/sec Loss 11.8682 Epoch: 2 Global Step: 28670 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:35:23,092-Speed 125.67 samples/sec Loss 12.1199 Epoch: 2 Global Step: 28675 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:35:43,205-Speed 127.29 samples/sec Loss 12.1488 Epoch: 2 Global Step: 28680 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:36:03,461-Speed 126.39 samples/sec Loss 12.2656 Epoch: 2 Global Step: 28685 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:36:23,812-Speed 125.80 samples/sec Loss 11.8112 Epoch: 2 Global Step: 28690 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:36:44,079-Speed 126.32 samples/sec Loss 11.6717 Epoch: 2 Global Step: 28695 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:37:04,464-Speed 125.59 samples/sec Loss 12.1108 Epoch: 2 Global Step: 28700 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:37:24,629-Speed 126.96 samples/sec Loss 11.7177 Epoch: 2 Global Step: 28705 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:37:45,054-Speed 125.34 samples/sec Loss 12.1423 Epoch: 2 Global Step: 28710 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:38:06,129-Speed 121.48 samples/sec Loss 12.0359 Epoch: 2 Global Step: 28715 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:38:26,554-Speed 125.34 samples/sec Loss 12.1553 Epoch: 2 Global Step: 28720 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:38:46,878-Speed 125.96 samples/sec Loss 12.1297 Epoch: 2 Global Step: 28725 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:39:07,225-Speed 125.82 samples/sec Loss 12.1135 Epoch: 2 Global Step: 28730 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:39:27,628-Speed 125.48 samples/sec Loss 12.2614 Epoch: 2 Global Step: 28735 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:39:48,147-Speed 124.77 samples/sec Loss 12.0977 Epoch: 2 Global Step: 28740 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:40:08,501-Speed 125.80 samples/sec Loss 11.9373 Epoch: 2 Global Step: 28745 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:40:28,827-Speed 125.95 samples/sec Loss 12.3869 Epoch: 2 Global Step: 28750 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:40:49,389-Speed 124.51 samples/sec Loss 11.7206 Epoch: 2 Global Step: 28755 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 04:41:10,332-Speed 122.25 samples/sec Loss 11.9964 Epoch: 2 Global Step: 28760 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:41:30,750-Speed 125.38 samples/sec Loss 11.7539 Epoch: 2 Global Step: 28765 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:41:51,188-Speed 125.26 samples/sec Loss 11.9382 Epoch: 2 Global Step: 28770 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:42:11,613-Speed 125.34 samples/sec Loss 12.2641 Epoch: 2 Global Step: 28775 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:42:32,878-Speed 120.39 samples/sec Loss 11.6231 Epoch: 2 Global Step: 28780 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:42:53,352-Speed 125.05 samples/sec Loss 12.0673 Epoch: 2 Global Step: 28785 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:43:13,561-Speed 126.69 samples/sec Loss 11.9681 Epoch: 2 Global Step: 28790 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:43:33,717-Speed 127.02 samples/sec Loss 11.8300 Epoch: 2 Global Step: 28795 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:43:54,021-Speed 126.09 samples/sec Loss 11.9033 Epoch: 2 Global Step: 28800 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:44:15,199-Speed 120.89 samples/sec Loss 11.7690 Epoch: 2 Global Step: 28805 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:44:36,147-Speed 122.21 samples/sec Loss 12.2284 Epoch: 2 Global Step: 28810 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:44:56,530-Speed 125.60 samples/sec Loss 12.2880 Epoch: 2 Global Step: 28815 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:45:17,313-Speed 123.18 samples/sec Loss 12.0218 Epoch: 2 Global Step: 28820 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:45:37,392-Speed 127.50 samples/sec Loss 12.0471 Epoch: 2 Global Step: 28825 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:45:57,746-Speed 125.78 samples/sec Loss 12.3996 Epoch: 2 Global Step: 28830 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:46:18,012-Speed 126.33 samples/sec Loss 12.1232 Epoch: 2 Global Step: 28835 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:46:38,753-Speed 123.44 samples/sec Loss 12.0519 Epoch: 2 Global Step: 28840 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:46:58,966-Speed 126.66 samples/sec Loss 11.8186 Epoch: 2 Global Step: 28845 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:47:19,881-Speed 122.41 samples/sec Loss 11.7963 Epoch: 2 Global Step: 28850 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:47:40,166-Speed 126.20 samples/sec Loss 11.9950 Epoch: 2 Global Step: 28855 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:48:00,407-Speed 126.48 samples/sec Loss 12.2890 Epoch: 2 Global Step: 28860 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:48:20,727-Speed 125.99 samples/sec Loss 11.9116 Epoch: 2 Global Step: 28865 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:48:40,899-Speed 126.91 samples/sec Loss 11.7308 Epoch: 2 Global Step: 28870 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:49:02,054-Speed 121.02 samples/sec Loss 11.8412 Epoch: 2 Global Step: 28875 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:49:22,493-Speed 125.26 samples/sec Loss 11.9102 Epoch: 2 Global Step: 28880 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:49:42,900-Speed 125.46 samples/sec Loss 11.6888 Epoch: 2 Global Step: 28885 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:50:03,253-Speed 125.79 samples/sec Loss 11.9104 Epoch: 2 Global Step: 28890 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:50:24,506-Speed 120.46 samples/sec Loss 11.7495 Epoch: 2 Global Step: 28895 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:50:44,911-Speed 125.46 samples/sec Loss 12.2793 Epoch: 2 Global Step: 28900 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:51:05,325-Speed 125.41 samples/sec Loss 12.1079 Epoch: 2 Global Step: 28905 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:51:25,650-Speed 125.96 samples/sec Loss 11.7747 Epoch: 2 Global Step: 28910 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:51:46,043-Speed 125.54 samples/sec Loss 11.9675 Epoch: 2 Global Step: 28915 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:52:06,314-Speed 126.29 samples/sec Loss 11.9393 Epoch: 2 Global Step: 28920 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:52:26,636-Speed 125.98 samples/sec Loss 11.8486 Epoch: 2 Global Step: 28925 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:52:47,216-Speed 124.40 samples/sec Loss 12.0681 Epoch: 2 Global Step: 28930 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:53:07,622-Speed 125.46 samples/sec Loss 11.5453 Epoch: 2 Global Step: 28935 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:53:28,670-Speed 121.63 samples/sec Loss 11.9119 Epoch: 2 Global Step: 28940 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:53:49,072-Speed 125.48 samples/sec Loss 12.0033 Epoch: 2 Global Step: 28945 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:54:09,387-Speed 126.02 samples/sec Loss 12.0455 Epoch: 2 Global Step: 28950 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:54:29,690-Speed 126.09 samples/sec Loss 12.0464 Epoch: 2 Global Step: 28955 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:54:50,222-Speed 124.69 samples/sec Loss 11.9938 Epoch: 2 Global Step: 28960 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:55:10,530-Speed 126.07 samples/sec Loss 12.0215 Epoch: 2 Global Step: 28965 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:55:30,581-Speed 127.69 samples/sec Loss 11.9807 Epoch: 2 Global Step: 28970 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:55:50,936-Speed 125.77 samples/sec Loss 11.8804 Epoch: 2 Global Step: 28975 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:56:11,286-Speed 125.81 samples/sec Loss 11.9310 Epoch: 2 Global Step: 28980 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:56:32,098-Speed 123.02 samples/sec Loss 11.8689 Epoch: 2 Global Step: 28985 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:56:52,672-Speed 124.43 samples/sec Loss 12.4663 Epoch: 2 Global Step: 28990 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:57:12,648-Speed 128.16 samples/sec Loss 12.1368 Epoch: 2 Global Step: 28995 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:57:32,863-Speed 126.65 samples/sec Loss 11.9896 Epoch: 2 Global Step: 29000 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:57:53,158-Speed 126.15 samples/sec Loss 12.1346 Epoch: 2 Global Step: 29005 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:58:13,411-Speed 126.41 samples/sec Loss 12.5919 Epoch: 2 Global Step: 29010 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:58:33,956-Speed 124.61 samples/sec Loss 12.2889 Epoch: 2 Global Step: 29015 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:58:54,229-Speed 126.28 samples/sec Loss 12.2123 Epoch: 2 Global Step: 29020 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:59:14,549-Speed 126.00 samples/sec Loss 12.1225 Epoch: 2 Global Step: 29025 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:59:35,550-Speed 121.91 samples/sec Loss 12.1027 Epoch: 2 Global Step: 29030 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 04:59:55,741-Speed 126.80 samples/sec Loss 11.9562 Epoch: 2 Global Step: 29035 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:00:16,085-Speed 125.84 samples/sec Loss 12.4310 Epoch: 2 Global Step: 29040 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:00:36,327-Speed 126.48 samples/sec Loss 12.0685 Epoch: 2 Global Step: 29045 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:00:56,524-Speed 126.76 samples/sec Loss 12.0863 Epoch: 2 Global Step: 29050 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:01:16,734-Speed 126.68 samples/sec Loss 11.7587 Epoch: 2 Global Step: 29055 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:01:36,891-Speed 127.01 samples/sec Loss 12.0992 Epoch: 2 Global Step: 29060 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:01:57,221-Speed 125.93 samples/sec Loss 12.1537 Epoch: 2 Global Step: 29065 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:02:17,671-Speed 125.19 samples/sec Loss 11.8653 Epoch: 2 Global Step: 29070 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:02:38,653-Speed 122.01 samples/sec Loss 12.1821 Epoch: 2 Global Step: 29075 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:02:59,018-Speed 125.72 samples/sec Loss 11.9591 Epoch: 2 Global Step: 29080 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:03:19,309-Speed 126.17 samples/sec Loss 12.3817 Epoch: 2 Global Step: 29085 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:03:39,645-Speed 125.90 samples/sec Loss 12.4235 Epoch: 2 Global Step: 29090 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:03:59,730-Speed 127.47 samples/sec Loss 12.4328 Epoch: 2 Global Step: 29095 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:04:19,986-Speed 126.39 samples/sec Loss 12.1439 Epoch: 2 Global Step: 29100 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:04:40,064-Speed 127.51 samples/sec Loss 11.9504 Epoch: 2 Global Step: 29105 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:05:00,367-Speed 126.09 samples/sec Loss 11.8593 Epoch: 2 Global Step: 29110 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:05:20,352-Speed 128.10 samples/sec Loss 12.3411 Epoch: 2 Global Step: 29115 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:05:41,185-Speed 122.89 samples/sec Loss 12.1236 Epoch: 2 Global Step: 29120 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:06:01,471-Speed 126.20 samples/sec Loss 12.1109 Epoch: 2 Global Step: 29125 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:06:21,638-Speed 126.94 samples/sec Loss 11.9030 Epoch: 2 Global Step: 29130 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:06:41,979-Speed 125.87 samples/sec Loss 12.0675 Epoch: 2 Global Step: 29135 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:07:02,200-Speed 126.61 samples/sec Loss 12.1593 Epoch: 2 Global Step: 29140 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:07:22,542-Speed 125.86 samples/sec Loss 12.2301 Epoch: 2 Global Step: 29145 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:07:43,003-Speed 125.13 samples/sec Loss 12.1611 Epoch: 2 Global Step: 29150 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:08:03,390-Speed 125.58 samples/sec Loss 11.6759 Epoch: 2 Global Step: 29155 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:08:23,694-Speed 126.09 samples/sec Loss 12.0326 Epoch: 2 Global Step: 29160 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:08:44,526-Speed 122.89 samples/sec Loss 11.8631 Epoch: 2 Global Step: 29165 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:09:04,872-Speed 125.83 samples/sec Loss 11.9096 Epoch: 2 Global Step: 29170 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:09:25,148-Speed 126.26 samples/sec Loss 12.0135 Epoch: 2 Global Step: 29175 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:09:45,447-Speed 126.12 samples/sec Loss 11.9484 Epoch: 2 Global Step: 29180 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:10:05,941-Speed 124.92 samples/sec Loss 12.0069 Epoch: 2 Global Step: 29185 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:10:25,888-Speed 128.35 samples/sec Loss 11.8078 Epoch: 2 Global Step: 29190 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:10:46,168-Speed 126.24 samples/sec Loss 12.2629 Epoch: 2 Global Step: 29195 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:11:06,304-Speed 127.15 samples/sec Loss 12.1420 Epoch: 2 Global Step: 29200 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:11:26,595-Speed 126.17 samples/sec Loss 12.4422 Epoch: 2 Global Step: 29205 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:11:47,623-Speed 121.75 samples/sec Loss 12.1320 Epoch: 2 Global Step: 29210 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:12:07,891-Speed 126.32 samples/sec Loss 12.0613 Epoch: 2 Global Step: 29215 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:12:28,272-Speed 125.61 samples/sec Loss 12.3523 Epoch: 2 Global Step: 29220 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:12:48,598-Speed 125.96 samples/sec Loss 12.2874 Epoch: 2 Global Step: 29225 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:13:09,034-Speed 125.27 samples/sec Loss 11.9323 Epoch: 2 Global Step: 29230 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:13:29,303-Speed 126.31 samples/sec Loss 11.9838 Epoch: 2 Global Step: 29235 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:13:49,637-Speed 125.90 samples/sec Loss 11.8135 Epoch: 2 Global Step: 29240 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:14:10,043-Speed 125.46 samples/sec Loss 11.9785 Epoch: 2 Global Step: 29245 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:14:30,296-Speed 126.41 samples/sec Loss 12.0752 Epoch: 2 Global Step: 29250 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:14:51,371-Speed 121.48 samples/sec Loss 11.7482 Epoch: 2 Global Step: 29255 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:15:11,739-Speed 125.69 samples/sec Loss 12.1383 Epoch: 2 Global Step: 29260 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:15:31,847-Speed 127.32 samples/sec Loss 12.3735 Epoch: 2 Global Step: 29265 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:15:52,086-Speed 126.50 samples/sec Loss 11.8556 Epoch: 2 Global Step: 29270 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:16:12,445-Speed 125.75 samples/sec Loss 12.4763 Epoch: 2 Global Step: 29275 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:16:32,732-Speed 126.20 samples/sec Loss 12.2517 Epoch: 2 Global Step: 29280 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:16:52,961-Speed 126.56 samples/sec Loss 11.7886 Epoch: 2 Global Step: 29285 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:17:13,254-Speed 126.16 samples/sec Loss 12.1221 Epoch: 2 Global Step: 29290 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:17:33,542-Speed 126.19 samples/sec Loss 12.1021 Epoch: 2 Global Step: 29295 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:17:54,530-Speed 121.98 samples/sec Loss 12.2865 Epoch: 2 Global Step: 29300 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:18:14,810-Speed 126.24 samples/sec Loss 12.0269 Epoch: 2 Global Step: 29305 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:18:35,142-Speed 125.92 samples/sec Loss 11.9382 Epoch: 2 Global Step: 29310 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:18:55,555-Speed 125.42 samples/sec Loss 12.3489 Epoch: 2 Global Step: 29315 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:19:15,811-Speed 126.39 samples/sec Loss 11.7980 Epoch: 2 Global Step: 29320 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:19:36,201-Speed 125.56 samples/sec Loss 11.9700 Epoch: 2 Global Step: 29325 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:19:56,221-Speed 127.88 samples/sec Loss 11.8752 Epoch: 2 Global Step: 29330 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:20:16,551-Speed 125.93 samples/sec Loss 11.7783 Epoch: 2 Global Step: 29335 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:20:36,834-Speed 126.22 samples/sec Loss 11.7674 Epoch: 2 Global Step: 29340 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:20:57,913-Speed 121.45 samples/sec Loss 12.1203 Epoch: 2 Global Step: 29345 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:21:18,041-Speed 127.19 samples/sec Loss 12.0846 Epoch: 2 Global Step: 29350 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:21:38,333-Speed 126.16 samples/sec Loss 12.0490 Epoch: 2 Global Step: 29355 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:21:58,458-Speed 127.21 samples/sec Loss 12.0892 Epoch: 2 Global Step: 29360 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:22:18,793-Speed 125.90 samples/sec Loss 11.9317 Epoch: 2 Global Step: 29365 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:22:39,038-Speed 126.46 samples/sec Loss 11.9154 Epoch: 2 Global Step: 29370 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:22:59,268-Speed 126.55 samples/sec Loss 12.0315 Epoch: 2 Global Step: 29375 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:23:19,256-Speed 128.08 samples/sec Loss 11.7700 Epoch: 2 Global Step: 29380 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:23:39,616-Speed 125.74 samples/sec Loss 12.1787 Epoch: 2 Global Step: 29385 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:24:00,556-Speed 122.26 samples/sec Loss 12.1297 Epoch: 2 Global Step: 29390 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:24:20,787-Speed 126.55 samples/sec Loss 11.5868 Epoch: 2 Global Step: 29395 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:24:41,086-Speed 126.12 samples/sec Loss 11.9756 Epoch: 2 Global Step: 29400 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:25:01,387-Speed 126.11 samples/sec Loss 12.1588 Epoch: 2 Global Step: 29405 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:25:21,688-Speed 126.11 samples/sec Loss 11.8501 Epoch: 2 Global Step: 29410 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:25:41,954-Speed 126.33 samples/sec Loss 12.0379 Epoch: 2 Global Step: 29415 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:26:02,265-Speed 126.05 samples/sec Loss 12.1338 Epoch: 2 Global Step: 29420 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:26:22,592-Speed 125.94 samples/sec Loss 12.0831 Epoch: 2 Global Step: 29425 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:26:43,021-Speed 125.32 samples/sec Loss 11.6639 Epoch: 2 Global Step: 29430 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:27:03,907-Speed 122.57 samples/sec Loss 11.8590 Epoch: 2 Global Step: 29435 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:27:24,050-Speed 127.11 samples/sec Loss 12.2538 Epoch: 2 Global Step: 29440 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:27:44,336-Speed 126.20 samples/sec Loss 12.1587 Epoch: 2 Global Step: 29445 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:28:04,721-Speed 125.59 samples/sec Loss 12.4165 Epoch: 2 Global Step: 29450 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 05:28:25,135-Speed 125.41 samples/sec Loss 11.9543 Epoch: 2 Global Step: 29455 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:28:45,403-Speed 126.32 samples/sec Loss 11.9007 Epoch: 2 Global Step: 29460 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:29:05,520-Speed 127.26 samples/sec Loss 12.0320 Epoch: 2 Global Step: 29465 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:29:25,649-Speed 127.19 samples/sec Loss 11.4755 Epoch: 2 Global Step: 29470 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:29:45,958-Speed 126.05 samples/sec Loss 11.8647 Epoch: 2 Global Step: 29475 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:30:06,944-Speed 121.99 samples/sec Loss 12.2583 Epoch: 2 Global Step: 29480 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:30:27,218-Speed 126.27 samples/sec Loss 11.9782 Epoch: 2 Global Step: 29485 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:30:47,251-Speed 127.80 samples/sec Loss 11.7539 Epoch: 2 Global Step: 29490 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:31:07,559-Speed 126.06 samples/sec Loss 11.7733 Epoch: 2 Global Step: 29495 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:31:27,819-Speed 126.37 samples/sec Loss 11.9671 Epoch: 2 Global Step: 29500 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:31:48,026-Speed 126.70 samples/sec Loss 11.7103 Epoch: 2 Global Step: 29505 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:32:08,243-Speed 126.64 samples/sec Loss 11.8134 Epoch: 2 Global Step: 29510 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:32:28,533-Speed 126.18 samples/sec Loss 12.1062 Epoch: 2 Global Step: 29515 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:32:49,019-Speed 124.97 samples/sec Loss 11.8308 Epoch: 2 Global Step: 29520 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:33:09,924-Speed 122.46 samples/sec Loss 12.0574 Epoch: 2 Global Step: 29525 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:33:30,262-Speed 125.87 samples/sec Loss 11.9967 Epoch: 2 Global Step: 29530 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:33:50,695-Speed 125.29 samples/sec Loss 11.8959 Epoch: 2 Global Step: 29535 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:34:10,873-Speed 126.88 samples/sec Loss 11.9631 Epoch: 2 Global Step: 29540 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:34:31,192-Speed 126.00 samples/sec Loss 12.0919 Epoch: 2 Global Step: 29545 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:34:51,451-Speed 126.37 samples/sec Loss 12.0197 Epoch: 2 Global Step: 29550 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:35:11,805-Speed 125.78 samples/sec Loss 12.2844 Epoch: 2 Global Step: 29555 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:35:31,990-Speed 126.83 samples/sec Loss 12.1252 Epoch: 2 Global Step: 29560 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:35:52,225-Speed 126.52 samples/sec Loss 12.2713 Epoch: 2 Global Step: 29565 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:36:12,968-Speed 123.42 samples/sec Loss 12.0299 Epoch: 2 Global Step: 29570 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:36:33,284-Speed 126.01 samples/sec Loss 12.0999 Epoch: 2 Global Step: 29575 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:36:53,431-Speed 127.07 samples/sec Loss 11.9205 Epoch: 2 Global Step: 29580 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:37:13,671-Speed 126.49 samples/sec Loss 11.7632 Epoch: 2 Global Step: 29585 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:37:33,789-Speed 127.26 samples/sec Loss 11.9838 Epoch: 2 Global Step: 29590 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:37:54,067-Speed 126.25 samples/sec Loss 11.9961 Epoch: 2 Global Step: 29595 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:38:14,312-Speed 126.46 samples/sec Loss 11.5249 Epoch: 2 Global Step: 29600 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:38:35,060-Speed 123.39 samples/sec Loss 12.0055 Epoch: 2 Global Step: 29605 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:38:55,440-Speed 125.62 samples/sec Loss 12.0614 Epoch: 2 Global Step: 29610 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:39:16,606-Speed 120.95 samples/sec Loss 12.0051 Epoch: 2 Global Step: 29615 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:39:36,886-Speed 126.24 samples/sec Loss 12.1570 Epoch: 2 Global Step: 29620 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:39:57,302-Speed 125.41 samples/sec Loss 11.9335 Epoch: 2 Global Step: 29625 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:40:17,606-Speed 126.09 samples/sec Loss 11.9148 Epoch: 2 Global Step: 29630 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:40:37,924-Speed 126.00 samples/sec Loss 11.5984 Epoch: 2 Global Step: 29635 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:40:58,273-Speed 125.81 samples/sec Loss 12.1592 Epoch: 2 Global Step: 29640 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:41:18,893-Speed 124.16 samples/sec Loss 12.0412 Epoch: 2 Global Step: 29645 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:41:39,259-Speed 125.71 samples/sec Loss 12.4393 Epoch: 2 Global Step: 29650 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:41:59,743-Speed 124.98 samples/sec Loss 12.1165 Epoch: 2 Global Step: 29655 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:42:20,977-Speed 120.57 samples/sec Loss 11.9076 Epoch: 2 Global Step: 29660 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:42:41,135-Speed 127.00 samples/sec Loss 11.7291 Epoch: 2 Global Step: 29665 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:43:01,933-Speed 123.10 samples/sec Loss 12.3822 Epoch: 2 Global Step: 29670 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:43:22,251-Speed 126.00 samples/sec Loss 11.8005 Epoch: 2 Global Step: 29675 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:43:43,086-Speed 122.88 samples/sec Loss 11.5638 Epoch: 2 Global Step: 29680 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:44:03,444-Speed 125.75 samples/sec Loss 11.7151 Epoch: 2 Global Step: 29685 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:44:23,746-Speed 126.10 samples/sec Loss 12.0225 Epoch: 2 Global Step: 29690 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:44:44,227-Speed 125.01 samples/sec Loss 11.7544 Epoch: 2 Global Step: 29695 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:45:04,593-Speed 125.70 samples/sec Loss 12.1758 Epoch: 2 Global Step: 29700 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:45:25,714-Speed 121.22 samples/sec Loss 12.1274 Epoch: 2 Global Step: 29705 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:45:46,022-Speed 126.06 samples/sec Loss 12.2197 Epoch: 2 Global Step: 29710 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:46:06,732-Speed 123.62 samples/sec Loss 11.8455 Epoch: 2 Global Step: 29715 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:46:27,324-Speed 124.32 samples/sec Loss 11.9856 Epoch: 2 Global Step: 29720 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:46:47,776-Speed 125.18 samples/sec Loss 12.0440 Epoch: 2 Global Step: 29725 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:47:08,650-Speed 122.65 samples/sec Loss 11.9935 Epoch: 2 Global Step: 29730 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:47:28,949-Speed 126.12 samples/sec Loss 11.8961 Epoch: 2 Global Step: 29735 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:47:49,280-Speed 125.92 samples/sec Loss 12.2942 Epoch: 2 Global Step: 29740 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:48:10,060-Speed 123.20 samples/sec Loss 11.7942 Epoch: 2 Global Step: 29745 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:48:31,174-Speed 121.25 samples/sec Loss 11.8690 Epoch: 2 Global Step: 29750 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:48:51,769-Speed 124.31 samples/sec Loss 12.1710 Epoch: 2 Global Step: 29755 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:49:12,002-Speed 126.53 samples/sec Loss 12.0962 Epoch: 2 Global Step: 29760 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:49:32,169-Speed 126.95 samples/sec Loss 12.0221 Epoch: 2 Global Step: 29765 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:49:52,572-Speed 125.48 samples/sec Loss 11.7919 Epoch: 2 Global Step: 29770 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:50:12,943-Speed 125.67 samples/sec Loss 11.8069 Epoch: 2 Global Step: 29775 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:50:33,435-Speed 124.93 samples/sec Loss 11.8206 Epoch: 2 Global Step: 29780 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:50:53,952-Speed 124.78 samples/sec Loss 12.1507 Epoch: 2 Global Step: 29785 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:51:14,297-Speed 125.84 samples/sec Loss 12.0990 Epoch: 2 Global Step: 29790 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:51:35,327-Speed 121.74 samples/sec Loss 11.9216 Epoch: 2 Global Step: 29795 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:51:56,202-Speed 122.65 samples/sec Loss 12.3208 Epoch: 2 Global Step: 29800 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:52:16,330-Speed 127.19 samples/sec Loss 11.9989 Epoch: 2 Global Step: 29805 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:52:36,557-Speed 126.57 samples/sec Loss 12.1232 Epoch: 2 Global Step: 29810 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:52:57,184-Speed 124.12 samples/sec Loss 11.6406 Epoch: 2 Global Step: 29815 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:53:18,008-Speed 122.95 samples/sec Loss 11.9185 Epoch: 2 Global Step: 29820 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:53:38,313-Speed 126.08 samples/sec Loss 11.8753 Epoch: 2 Global Step: 29825 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:53:58,618-Speed 126.09 samples/sec Loss 12.1877 Epoch: 2 Global Step: 29830 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:54:18,839-Speed 126.61 samples/sec Loss 11.8610 Epoch: 2 Global Step: 29835 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:54:39,863-Speed 121.78 samples/sec Loss 12.1523 Epoch: 2 Global Step: 29840 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:55:00,038-Speed 126.90 samples/sec Loss 12.2048 Epoch: 2 Global Step: 29845 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:55:20,277-Speed 126.49 samples/sec Loss 12.2544 Epoch: 2 Global Step: 29850 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:55:40,590-Speed 126.04 samples/sec Loss 12.1509 Epoch: 2 Global Step: 29855 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:56:01,070-Speed 125.01 samples/sec Loss 11.9907 Epoch: 2 Global Step: 29860 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:56:21,338-Speed 126.32 samples/sec Loss 12.0815 Epoch: 2 Global Step: 29865 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:56:41,591-Speed 126.41 samples/sec Loss 12.1968 Epoch: 2 Global Step: 29870 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:57:02,318-Speed 123.52 samples/sec Loss 11.9303 Epoch: 2 Global Step: 29875 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:57:22,456-Speed 127.13 samples/sec Loss 12.0184 Epoch: 2 Global Step: 29880 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:57:43,793-Speed 119.99 samples/sec Loss 11.9272 Epoch: 2 Global Step: 29885 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:58:03,948-Speed 127.03 samples/sec Loss 12.4461 Epoch: 2 Global Step: 29890 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:58:47,651-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-08 05:59:07,722-Speed 40.14 samples/sec Loss 11.5435 Epoch: 3 Global Step: 29895 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:59:28,007-Speed 126.21 samples/sec Loss 11.3041 Epoch: 3 Global Step: 29900 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 05:59:48,286-Speed 126.24 samples/sec Loss 11.4263 Epoch: 3 Global Step: 29905 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:00:08,578-Speed 126.16 samples/sec Loss 11.2958 Epoch: 3 Global Step: 29910 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:00:28,913-Speed 125.90 samples/sec Loss 11.2329 Epoch: 3 Global Step: 29915 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:00:49,250-Speed 125.89 samples/sec Loss 11.4689 Epoch: 3 Global Step: 29920 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:01:09,622-Speed 125.67 samples/sec Loss 11.1569 Epoch: 3 Global Step: 29925 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:01:30,186-Speed 124.50 samples/sec Loss 11.4735 Epoch: 3 Global Step: 29930 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:01:50,414-Speed 126.57 samples/sec Loss 11.4392 Epoch: 3 Global Step: 29935 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:02:11,089-Speed 123.83 samples/sec Loss 11.3512 Epoch: 3 Global Step: 29940 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:02:31,491-Speed 125.48 samples/sec Loss 11.2881 Epoch: 3 Global Step: 29945 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:02:51,688-Speed 126.76 samples/sec Loss 11.2613 Epoch: 3 Global Step: 29950 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:03:11,708-Speed 127.88 samples/sec Loss 11.3914 Epoch: 3 Global Step: 29955 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:03:31,967-Speed 126.37 samples/sec Loss 11.3267 Epoch: 3 Global Step: 29960 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:03:52,268-Speed 126.11 samples/sec Loss 11.1978 Epoch: 3 Global Step: 29965 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:04:12,458-Speed 126.80 samples/sec Loss 11.2084 Epoch: 3 Global Step: 29970 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:04:33,063-Speed 124.25 samples/sec Loss 11.2851 Epoch: 3 Global Step: 29975 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:04:53,388-Speed 125.96 samples/sec Loss 11.3923 Epoch: 3 Global Step: 29980 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:05:14,140-Speed 123.37 samples/sec Loss 11.7081 Epoch: 3 Global Step: 29985 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:05:34,683-Speed 124.62 samples/sec Loss 11.1104 Epoch: 3 Global Step: 29990 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:05:55,096-Speed 125.42 samples/sec Loss 11.4122 Epoch: 3 Global Step: 29995 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:06:15,438-Speed 125.85 samples/sec Loss 11.2886 Epoch: 3 Global Step: 30000 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 06:21:08,752-[lfw][30000]XNorm: 21.791840 Training: 2021-06-08 06:21:08,752-[lfw][30000]Accuracy-Flip: 0.99700+-0.00287 Training: 2021-06-08 06:21:08,752-[lfw][30000]Accuracy-Highest: 0.99700 Training: 2021-06-08 06:38:33,642-[cfp_fp][30000]XNorm: 19.667135 Training: 2021-06-08 06:38:33,643-[cfp_fp][30000]Accuracy-Flip: 0.96714+-0.00795 Training: 2021-06-08 06:38:33,643-[cfp_fp][30000]Accuracy-Highest: 0.96886 Training: 2021-06-08 06:53:27,908-[agedb_30][30000]XNorm: 21.442568 Training: 2021-06-08 06:53:27,908-[agedb_30][30000]Accuracy-Flip: 0.96750+-0.00716 Training: 2021-06-08 06:53:27,908-[agedb_30][30000]Accuracy-Highest: 0.96750 Training: 2021-06-08 06:53:48,351-Speed 0.90 samples/sec Loss 11.1788 Epoch: 3 Global Step: 30005 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 06:54:08,566-Speed 126.65 samples/sec Loss 11.4346 Epoch: 3 Global Step: 30010 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 06:54:28,972-Speed 125.46 samples/sec Loss 11.6207 Epoch: 3 Global Step: 30015 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 06:54:50,106-Speed 121.14 samples/sec Loss 11.3718 Epoch: 3 Global Step: 30020 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 06:55:10,275-Speed 126.93 samples/sec Loss 11.2339 Epoch: 3 Global Step: 30025 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 06:55:30,552-Speed 126.26 samples/sec Loss 11.3281 Epoch: 3 Global Step: 30030 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 06:55:50,802-Speed 126.43 samples/sec Loss 11.5127 Epoch: 3 Global Step: 30035 Fp16 Grad Scale: 8192 Required: 262 hours Training: 2021-06-08 06:56:11,187-Speed 125.59 samples/sec Loss 11.1785 Epoch: 3 Global Step: 30040 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:56:31,434-Speed 126.45 samples/sec Loss 11.5587 Epoch: 3 Global Step: 30045 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:56:51,833-Speed 125.51 samples/sec Loss 11.4489 Epoch: 3 Global Step: 30050 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:57:11,923-Speed 127.43 samples/sec Loss 10.8927 Epoch: 3 Global Step: 30055 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:57:32,653-Speed 123.50 samples/sec Loss 11.3585 Epoch: 3 Global Step: 30060 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:57:53,574-Speed 122.37 samples/sec Loss 11.2204 Epoch: 3 Global Step: 30065 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:58:13,867-Speed 126.15 samples/sec Loss 11.7399 Epoch: 3 Global Step: 30070 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:58:34,175-Speed 126.07 samples/sec Loss 11.4604 Epoch: 3 Global Step: 30075 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:58:54,535-Speed 125.75 samples/sec Loss 11.5902 Epoch: 3 Global Step: 30080 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:59:14,703-Speed 126.94 samples/sec Loss 11.3562 Epoch: 3 Global Step: 30085 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:59:35,096-Speed 125.54 samples/sec Loss 11.5258 Epoch: 3 Global Step: 30090 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 06:59:55,334-Speed 126.51 samples/sec Loss 11.4487 Epoch: 3 Global Step: 30095 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:00:15,617-Speed 126.23 samples/sec Loss 11.9471 Epoch: 3 Global Step: 30100 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:00:35,753-Speed 127.14 samples/sec Loss 11.1975 Epoch: 3 Global Step: 30105 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:00:56,553-Speed 123.09 samples/sec Loss 11.7102 Epoch: 3 Global Step: 30110 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:01:18,217-Speed 118.17 samples/sec Loss 11.8578 Epoch: 3 Global Step: 30115 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:01:38,252-Speed 127.78 samples/sec Loss 11.4875 Epoch: 3 Global Step: 30120 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:01:58,431-Speed 126.88 samples/sec Loss 11.3828 Epoch: 3 Global Step: 30125 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:02:18,699-Speed 126.31 samples/sec Loss 11.2893 Epoch: 3 Global Step: 30130 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:02:38,854-Speed 127.02 samples/sec Loss 11.4864 Epoch: 3 Global Step: 30135 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:02:59,452-Speed 124.29 samples/sec Loss 11.0953 Epoch: 3 Global Step: 30140 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:03:19,763-Speed 126.05 samples/sec Loss 11.7642 Epoch: 3 Global Step: 30145 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:03:40,219-Speed 125.15 samples/sec Loss 11.3759 Epoch: 3 Global Step: 30150 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:04:01,234-Speed 121.82 samples/sec Loss 11.5074 Epoch: 3 Global Step: 30155 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:04:21,610-Speed 125.65 samples/sec Loss 11.6615 Epoch: 3 Global Step: 30160 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:04:41,685-Speed 127.53 samples/sec Loss 11.5312 Epoch: 3 Global Step: 30165 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:05:01,924-Speed 126.50 samples/sec Loss 11.5091 Epoch: 3 Global Step: 30170 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:05:21,975-Speed 127.68 samples/sec Loss 11.4749 Epoch: 3 Global Step: 30175 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:05:42,201-Speed 126.58 samples/sec Loss 11.6478 Epoch: 3 Global Step: 30180 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:06:02,357-Speed 127.02 samples/sec Loss 11.7070 Epoch: 3 Global Step: 30185 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:06:22,585-Speed 126.56 samples/sec Loss 11.5192 Epoch: 3 Global Step: 30190 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:06:43,272-Speed 123.75 samples/sec Loss 11.3881 Epoch: 3 Global Step: 30195 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:07:04,075-Speed 123.07 samples/sec Loss 11.5035 Epoch: 3 Global Step: 30200 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:07:24,442-Speed 125.70 samples/sec Loss 11.8525 Epoch: 3 Global Step: 30205 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:07:44,389-Speed 128.35 samples/sec Loss 11.2969 Epoch: 3 Global Step: 30210 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:08:04,891-Speed 124.88 samples/sec Loss 11.6325 Epoch: 3 Global Step: 30215 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:08:25,003-Speed 127.29 samples/sec Loss 11.2740 Epoch: 3 Global Step: 30220 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:08:45,431-Speed 125.32 samples/sec Loss 11.6684 Epoch: 3 Global Step: 30225 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:09:05,693-Speed 126.35 samples/sec Loss 11.7259 Epoch: 3 Global Step: 30230 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:09:26,067-Speed 125.66 samples/sec Loss 11.5921 Epoch: 3 Global Step: 30235 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:09:46,365-Speed 126.13 samples/sec Loss 11.8767 Epoch: 3 Global Step: 30240 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:10:07,209-Speed 122.83 samples/sec Loss 11.3539 Epoch: 3 Global Step: 30245 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:10:27,564-Speed 125.77 samples/sec Loss 11.4641 Epoch: 3 Global Step: 30250 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:10:47,838-Speed 126.28 samples/sec Loss 11.4871 Epoch: 3 Global Step: 30255 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:11:08,006-Speed 126.94 samples/sec Loss 11.7999 Epoch: 3 Global Step: 30260 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:11:28,402-Speed 125.52 samples/sec Loss 11.3522 Epoch: 3 Global Step: 30265 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:11:48,779-Speed 125.63 samples/sec Loss 11.8011 Epoch: 3 Global Step: 30270 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:12:09,165-Speed 125.59 samples/sec Loss 11.5696 Epoch: 3 Global Step: 30275 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:12:29,483-Speed 126.01 samples/sec Loss 11.3165 Epoch: 3 Global Step: 30280 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:12:49,830-Speed 125.83 samples/sec Loss 11.8366 Epoch: 3 Global Step: 30285 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:13:10,754-Speed 122.35 samples/sec Loss 11.5592 Epoch: 3 Global Step: 30290 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:13:31,069-Speed 126.02 samples/sec Loss 11.6714 Epoch: 3 Global Step: 30295 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:13:51,531-Speed 125.12 samples/sec Loss 11.7281 Epoch: 3 Global Step: 30300 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:14:11,788-Speed 126.38 samples/sec Loss 11.5745 Epoch: 3 Global Step: 30305 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:14:32,143-Speed 125.77 samples/sec Loss 12.0138 Epoch: 3 Global Step: 30310 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:14:52,660-Speed 124.78 samples/sec Loss 11.7619 Epoch: 3 Global Step: 30315 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:15:12,851-Speed 126.80 samples/sec Loss 11.4719 Epoch: 3 Global Step: 30320 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:15:33,072-Speed 126.61 samples/sec Loss 11.5279 Epoch: 3 Global Step: 30325 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:15:53,274-Speed 126.73 samples/sec Loss 11.3730 Epoch: 3 Global Step: 30330 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:16:14,037-Speed 123.31 samples/sec Loss 11.5600 Epoch: 3 Global Step: 30335 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:16:34,134-Speed 127.39 samples/sec Loss 11.8303 Epoch: 3 Global Step: 30340 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:16:54,352-Speed 126.62 samples/sec Loss 11.7259 Epoch: 3 Global Step: 30345 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:17:14,724-Speed 125.67 samples/sec Loss 11.2083 Epoch: 3 Global Step: 30350 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:17:34,744-Speed 127.88 samples/sec Loss 11.6077 Epoch: 3 Global Step: 30355 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:17:56,101-Speed 119.87 samples/sec Loss 11.8969 Epoch: 3 Global Step: 30360 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:18:16,725-Speed 124.13 samples/sec Loss 11.6633 Epoch: 3 Global Step: 30365 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:18:37,072-Speed 125.83 samples/sec Loss 11.7342 Epoch: 3 Global Step: 30370 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:18:57,651-Speed 124.40 samples/sec Loss 11.2338 Epoch: 3 Global Step: 30375 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:19:18,631-Speed 122.03 samples/sec Loss 11.1740 Epoch: 3 Global Step: 30380 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:19:38,983-Speed 125.79 samples/sec Loss 11.9896 Epoch: 3 Global Step: 30385 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:19:59,341-Speed 125.76 samples/sec Loss 11.3838 Epoch: 3 Global Step: 30390 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:20:19,708-Speed 125.70 samples/sec Loss 11.9992 Epoch: 3 Global Step: 30395 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:20:39,984-Speed 126.27 samples/sec Loss 11.7370 Epoch: 3 Global Step: 30400 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:21:00,442-Speed 125.14 samples/sec Loss 11.5629 Epoch: 3 Global Step: 30405 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:21:20,697-Speed 126.40 samples/sec Loss 11.7432 Epoch: 3 Global Step: 30410 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:21:41,216-Speed 124.77 samples/sec Loss 11.4948 Epoch: 3 Global Step: 30415 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:22:01,289-Speed 127.55 samples/sec Loss 11.7506 Epoch: 3 Global Step: 30420 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:22:22,035-Speed 123.40 samples/sec Loss 11.7563 Epoch: 3 Global Step: 30425 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:22:42,310-Speed 126.27 samples/sec Loss 11.7012 Epoch: 3 Global Step: 30430 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:23:02,685-Speed 125.65 samples/sec Loss 11.5472 Epoch: 3 Global Step: 30435 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:23:22,972-Speed 126.20 samples/sec Loss 11.7194 Epoch: 3 Global Step: 30440 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:23:43,233-Speed 126.36 samples/sec Loss 11.5873 Epoch: 3 Global Step: 30445 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:24:03,529-Speed 126.14 samples/sec Loss 11.7270 Epoch: 3 Global Step: 30450 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:24:23,567-Speed 127.76 samples/sec Loss 11.8903 Epoch: 3 Global Step: 30455 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:24:43,918-Speed 125.80 samples/sec Loss 11.7680 Epoch: 3 Global Step: 30460 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:25:04,514-Speed 124.30 samples/sec Loss 11.6453 Epoch: 3 Global Step: 30465 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:25:25,547-Speed 121.72 samples/sec Loss 11.6888 Epoch: 3 Global Step: 30470 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:25:46,214-Speed 123.87 samples/sec Loss 11.6886 Epoch: 3 Global Step: 30475 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:26:06,327-Speed 127.28 samples/sec Loss 12.0590 Epoch: 3 Global Step: 30480 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:26:26,583-Speed 126.39 samples/sec Loss 11.7424 Epoch: 3 Global Step: 30485 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:26:46,616-Speed 127.80 samples/sec Loss 11.3828 Epoch: 3 Global Step: 30490 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:27:06,748-Speed 127.16 samples/sec Loss 11.6673 Epoch: 3 Global Step: 30495 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:27:26,975-Speed 126.58 samples/sec Loss 11.9710 Epoch: 3 Global Step: 30500 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:27:47,325-Speed 125.81 samples/sec Loss 11.6940 Epoch: 3 Global Step: 30505 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:28:07,549-Speed 126.59 samples/sec Loss 11.6422 Epoch: 3 Global Step: 30510 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:28:28,467-Speed 122.39 samples/sec Loss 11.6615 Epoch: 3 Global Step: 30515 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:28:49,231-Speed 123.30 samples/sec Loss 11.5811 Epoch: 3 Global Step: 30520 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:29:09,531-Speed 126.11 samples/sec Loss 11.7638 Epoch: 3 Global Step: 30525 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:29:30,052-Speed 124.76 samples/sec Loss 11.7655 Epoch: 3 Global Step: 30530 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:29:50,171-Speed 127.25 samples/sec Loss 11.5353 Epoch: 3 Global Step: 30535 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:30:10,440-Speed 126.31 samples/sec Loss 11.8694 Epoch: 3 Global Step: 30540 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:30:30,702-Speed 126.35 samples/sec Loss 11.9493 Epoch: 3 Global Step: 30545 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:30:51,070-Speed 125.69 samples/sec Loss 11.4449 Epoch: 3 Global Step: 30550 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:31:11,400-Speed 125.93 samples/sec Loss 11.2854 Epoch: 3 Global Step: 30555 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:31:32,516-Speed 121.24 samples/sec Loss 11.8119 Epoch: 3 Global Step: 30560 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:31:52,729-Speed 126.66 samples/sec Loss 11.3167 Epoch: 3 Global Step: 30565 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:32:13,200-Speed 125.06 samples/sec Loss 11.4634 Epoch: 3 Global Step: 30570 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:32:33,265-Speed 127.59 samples/sec Loss 11.4178 Epoch: 3 Global Step: 30575 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:32:53,509-Speed 126.47 samples/sec Loss 11.4192 Epoch: 3 Global Step: 30580 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:33:13,779-Speed 126.30 samples/sec Loss 11.9812 Epoch: 3 Global Step: 30585 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:33:34,118-Speed 125.88 samples/sec Loss 11.8579 Epoch: 3 Global Step: 30590 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:33:54,368-Speed 126.42 samples/sec Loss 11.9361 Epoch: 3 Global Step: 30595 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:34:14,606-Speed 126.50 samples/sec Loss 12.0744 Epoch: 3 Global Step: 30600 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:34:35,574-Speed 122.10 samples/sec Loss 11.9691 Epoch: 3 Global Step: 30605 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:34:55,803-Speed 126.56 samples/sec Loss 11.4768 Epoch: 3 Global Step: 30610 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:35:16,229-Speed 125.34 samples/sec Loss 12.0171 Epoch: 3 Global Step: 30615 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:35:36,481-Speed 126.41 samples/sec Loss 11.5044 Epoch: 3 Global Step: 30620 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:35:56,769-Speed 126.19 samples/sec Loss 11.9308 Epoch: 3 Global Step: 30625 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:36:17,069-Speed 126.12 samples/sec Loss 11.6585 Epoch: 3 Global Step: 30630 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:36:37,425-Speed 125.77 samples/sec Loss 11.7442 Epoch: 3 Global Step: 30635 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:36:57,786-Speed 125.74 samples/sec Loss 11.9074 Epoch: 3 Global Step: 30640 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:37:17,984-Speed 126.75 samples/sec Loss 11.4088 Epoch: 3 Global Step: 30645 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:37:38,926-Speed 122.25 samples/sec Loss 11.7297 Epoch: 3 Global Step: 30650 Fp16 Grad Scale: 8192 Required: 261 hours Training: 2021-06-08 07:37:59,147-Speed 126.61 samples/sec Loss 11.5758 Epoch: 3 Global Step: 30655 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:38:19,414-Speed 126.32 samples/sec Loss 11.5779 Epoch: 3 Global Step: 30660 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:38:39,526-Speed 127.30 samples/sec Loss 11.6162 Epoch: 3 Global Step: 30665 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:38:59,600-Speed 127.53 samples/sec Loss 11.7167 Epoch: 3 Global Step: 30670 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:39:19,883-Speed 126.22 samples/sec Loss 12.0507 Epoch: 3 Global Step: 30675 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:39:40,138-Speed 126.39 samples/sec Loss 11.6342 Epoch: 3 Global Step: 30680 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:40:00,655-Speed 124.78 samples/sec Loss 12.0191 Epoch: 3 Global Step: 30685 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:40:21,111-Speed 125.16 samples/sec Loss 12.0193 Epoch: 3 Global Step: 30690 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:40:42,009-Speed 122.51 samples/sec Loss 11.8662 Epoch: 3 Global Step: 30695 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:41:02,238-Speed 126.55 samples/sec Loss 11.8779 Epoch: 3 Global Step: 30700 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:41:22,444-Speed 126.70 samples/sec Loss 11.7506 Epoch: 3 Global Step: 30705 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:41:43,186-Speed 123.43 samples/sec Loss 11.8217 Epoch: 3 Global Step: 30710 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:42:03,449-Speed 126.34 samples/sec Loss 11.8669 Epoch: 3 Global Step: 30715 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:42:23,726-Speed 126.26 samples/sec Loss 11.9847 Epoch: 3 Global Step: 30720 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:42:43,969-Speed 126.47 samples/sec Loss 11.8813 Epoch: 3 Global Step: 30725 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:43:04,214-Speed 126.46 samples/sec Loss 11.7137 Epoch: 3 Global Step: 30730 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:43:24,563-Speed 125.81 samples/sec Loss 11.9203 Epoch: 3 Global Step: 30735 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:43:45,601-Speed 121.69 samples/sec Loss 11.5084 Epoch: 3 Global Step: 30740 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:44:05,938-Speed 125.88 samples/sec Loss 11.5534 Epoch: 3 Global Step: 30745 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:44:26,128-Speed 126.80 samples/sec Loss 11.6287 Epoch: 3 Global Step: 30750 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:44:46,376-Speed 126.44 samples/sec Loss 11.7747 Epoch: 3 Global Step: 30755 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:45:06,633-Speed 126.38 samples/sec Loss 11.9430 Epoch: 3 Global Step: 30760 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:45:27,045-Speed 125.43 samples/sec Loss 11.8808 Epoch: 3 Global Step: 30765 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:45:47,393-Speed 125.82 samples/sec Loss 11.6157 Epoch: 3 Global Step: 30770 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:46:07,669-Speed 126.27 samples/sec Loss 11.7090 Epoch: 3 Global Step: 30775 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:46:28,170-Speed 124.88 samples/sec Loss 11.7197 Epoch: 3 Global Step: 30780 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:46:49,113-Speed 122.24 samples/sec Loss 11.9326 Epoch: 3 Global Step: 30785 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:47:09,516-Speed 125.47 samples/sec Loss 11.7838 Epoch: 3 Global Step: 30790 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:47:29,940-Speed 125.35 samples/sec Loss 11.5641 Epoch: 3 Global Step: 30795 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:47:50,214-Speed 126.27 samples/sec Loss 11.7677 Epoch: 3 Global Step: 30800 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:48:10,305-Speed 127.43 samples/sec Loss 11.9569 Epoch: 3 Global Step: 30805 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:48:30,651-Speed 125.83 samples/sec Loss 11.5558 Epoch: 3 Global Step: 30810 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:48:50,926-Speed 126.27 samples/sec Loss 11.7590 Epoch: 3 Global Step: 30815 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:49:11,049-Speed 127.23 samples/sec Loss 11.4843 Epoch: 3 Global Step: 30820 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:49:31,328-Speed 126.25 samples/sec Loss 11.7358 Epoch: 3 Global Step: 30825 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:49:52,288-Speed 122.14 samples/sec Loss 11.9967 Epoch: 3 Global Step: 30830 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:50:12,610-Speed 125.98 samples/sec Loss 11.7998 Epoch: 3 Global Step: 30835 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:50:33,022-Speed 125.42 samples/sec Loss 11.6999 Epoch: 3 Global Step: 30840 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:50:53,285-Speed 126.34 samples/sec Loss 11.9351 Epoch: 3 Global Step: 30845 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:51:13,258-Speed 128.18 samples/sec Loss 11.8940 Epoch: 3 Global Step: 30850 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:51:33,781-Speed 124.75 samples/sec Loss 11.8171 Epoch: 3 Global Step: 30855 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:51:54,118-Speed 125.89 samples/sec Loss 11.9378 Epoch: 3 Global Step: 30860 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:52:14,383-Speed 126.33 samples/sec Loss 11.7639 Epoch: 3 Global Step: 30865 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:52:34,696-Speed 126.03 samples/sec Loss 11.6540 Epoch: 3 Global Step: 30870 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:52:55,624-Speed 122.33 samples/sec Loss 11.7106 Epoch: 3 Global Step: 30875 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:53:15,925-Speed 126.11 samples/sec Loss 11.7635 Epoch: 3 Global Step: 30880 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:53:36,367-Speed 125.24 samples/sec Loss 12.0821 Epoch: 3 Global Step: 30885 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:53:56,623-Speed 126.39 samples/sec Loss 12.0348 Epoch: 3 Global Step: 30890 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:54:17,235-Speed 124.21 samples/sec Loss 11.9007 Epoch: 3 Global Step: 30895 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:54:37,560-Speed 125.96 samples/sec Loss 12.1903 Epoch: 3 Global Step: 30900 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:54:57,950-Speed 125.56 samples/sec Loss 11.9639 Epoch: 3 Global Step: 30905 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:55:18,257-Speed 126.07 samples/sec Loss 11.5223 Epoch: 3 Global Step: 30910 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:55:38,516-Speed 126.37 samples/sec Loss 11.7008 Epoch: 3 Global Step: 30915 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:55:59,521-Speed 121.88 samples/sec Loss 11.8545 Epoch: 3 Global Step: 30920 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:56:19,945-Speed 125.35 samples/sec Loss 11.7850 Epoch: 3 Global Step: 30925 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:56:40,259-Speed 126.02 samples/sec Loss 11.9792 Epoch: 3 Global Step: 30930 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:57:00,599-Speed 125.87 samples/sec Loss 11.6199 Epoch: 3 Global Step: 30935 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:57:20,837-Speed 126.50 samples/sec Loss 12.0593 Epoch: 3 Global Step: 30940 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:57:41,334-Speed 124.90 samples/sec Loss 12.0203 Epoch: 3 Global Step: 30945 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:58:01,593-Speed 126.37 samples/sec Loss 11.9776 Epoch: 3 Global Step: 30950 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:58:21,970-Speed 125.64 samples/sec Loss 11.7160 Epoch: 3 Global Step: 30955 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:58:42,147-Speed 126.88 samples/sec Loss 11.6498 Epoch: 3 Global Step: 30960 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:59:03,207-Speed 121.57 samples/sec Loss 11.7726 Epoch: 3 Global Step: 30965 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:59:23,496-Speed 126.18 samples/sec Loss 11.8320 Epoch: 3 Global Step: 30970 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 07:59:43,806-Speed 126.05 samples/sec Loss 11.9903 Epoch: 3 Global Step: 30975 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:00:04,031-Speed 126.58 samples/sec Loss 11.7960 Epoch: 3 Global Step: 30980 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:00:24,336-Speed 126.09 samples/sec Loss 11.8881 Epoch: 3 Global Step: 30985 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:00:44,713-Speed 125.64 samples/sec Loss 12.0783 Epoch: 3 Global Step: 30990 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:01:04,958-Speed 126.46 samples/sec Loss 12.0845 Epoch: 3 Global Step: 30995 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:01:25,088-Speed 127.18 samples/sec Loss 11.7983 Epoch: 3 Global Step: 31000 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:01:45,174-Speed 127.46 samples/sec Loss 11.7968 Epoch: 3 Global Step: 31005 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:02:06,064-Speed 122.55 samples/sec Loss 11.7648 Epoch: 3 Global Step: 31010 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:02:26,302-Speed 126.50 samples/sec Loss 11.8848 Epoch: 3 Global Step: 31015 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:02:46,434-Speed 127.16 samples/sec Loss 12.0210 Epoch: 3 Global Step: 31020 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:03:06,687-Speed 126.41 samples/sec Loss 12.2398 Epoch: 3 Global Step: 31025 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:03:26,757-Speed 127.56 samples/sec Loss 12.1362 Epoch: 3 Global Step: 31030 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:03:46,935-Speed 126.88 samples/sec Loss 11.7675 Epoch: 3 Global Step: 31035 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:04:07,123-Speed 126.81 samples/sec Loss 11.8602 Epoch: 3 Global Step: 31040 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:04:27,205-Speed 127.49 samples/sec Loss 11.9057 Epoch: 3 Global Step: 31045 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:04:47,453-Speed 126.44 samples/sec Loss 11.9304 Epoch: 3 Global Step: 31050 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:05:08,471-Speed 121.81 samples/sec Loss 11.6555 Epoch: 3 Global Step: 31055 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:05:29,092-Speed 124.15 samples/sec Loss 11.6660 Epoch: 3 Global Step: 31060 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:05:49,166-Speed 127.53 samples/sec Loss 11.8004 Epoch: 3 Global Step: 31065 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:06:09,920-Speed 123.36 samples/sec Loss 11.9855 Epoch: 3 Global Step: 31070 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:06:30,139-Speed 126.62 samples/sec Loss 11.7421 Epoch: 3 Global Step: 31075 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:06:50,466-Speed 125.94 samples/sec Loss 11.8716 Epoch: 3 Global Step: 31080 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:07:10,645-Speed 126.88 samples/sec Loss 11.5832 Epoch: 3 Global Step: 31085 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:07:30,960-Speed 126.03 samples/sec Loss 11.9072 Epoch: 3 Global Step: 31090 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:07:51,335-Speed 125.65 samples/sec Loss 11.7523 Epoch: 3 Global Step: 31095 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:08:12,477-Speed 121.09 samples/sec Loss 12.0791 Epoch: 3 Global Step: 31100 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:08:32,839-Speed 125.73 samples/sec Loss 11.6234 Epoch: 3 Global Step: 31105 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:08:53,243-Speed 125.47 samples/sec Loss 11.9578 Epoch: 3 Global Step: 31110 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:09:13,734-Speed 124.94 samples/sec Loss 12.0953 Epoch: 3 Global Step: 31115 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:09:34,029-Speed 126.14 samples/sec Loss 11.5993 Epoch: 3 Global Step: 31120 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:09:54,304-Speed 126.27 samples/sec Loss 11.7295 Epoch: 3 Global Step: 31125 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:10:14,604-Speed 126.12 samples/sec Loss 11.7149 Epoch: 3 Global Step: 31130 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:10:34,926-Speed 125.98 samples/sec Loss 11.7402 Epoch: 3 Global Step: 31135 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:10:55,200-Speed 126.28 samples/sec Loss 12.0835 Epoch: 3 Global Step: 31140 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:11:16,242-Speed 121.67 samples/sec Loss 11.7593 Epoch: 3 Global Step: 31145 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:11:36,512-Speed 126.30 samples/sec Loss 11.9054 Epoch: 3 Global Step: 31150 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:11:56,794-Speed 126.23 samples/sec Loss 11.9656 Epoch: 3 Global Step: 31155 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:12:17,099-Speed 126.08 samples/sec Loss 11.8139 Epoch: 3 Global Step: 31160 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:12:37,452-Speed 125.79 samples/sec Loss 11.5759 Epoch: 3 Global Step: 31165 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:12:57,731-Speed 126.24 samples/sec Loss 11.8603 Epoch: 3 Global Step: 31170 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:13:18,170-Speed 125.26 samples/sec Loss 12.1311 Epoch: 3 Global Step: 31175 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:13:38,480-Speed 126.05 samples/sec Loss 11.9189 Epoch: 3 Global Step: 31180 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:13:58,740-Speed 126.36 samples/sec Loss 11.8839 Epoch: 3 Global Step: 31185 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:14:19,898-Speed 121.01 samples/sec Loss 12.1329 Epoch: 3 Global Step: 31190 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:14:40,059-Speed 126.98 samples/sec Loss 12.0297 Epoch: 3 Global Step: 31195 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:15:00,287-Speed 126.56 samples/sec Loss 11.9677 Epoch: 3 Global Step: 31200 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:15:20,569-Speed 126.23 samples/sec Loss 12.1030 Epoch: 3 Global Step: 31205 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:15:41,348-Speed 123.21 samples/sec Loss 11.8654 Epoch: 3 Global Step: 31210 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:16:01,748-Speed 125.50 samples/sec Loss 11.7551 Epoch: 3 Global Step: 31215 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:16:21,845-Speed 127.39 samples/sec Loss 11.6640 Epoch: 3 Global Step: 31220 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:16:42,055-Speed 126.68 samples/sec Loss 11.7312 Epoch: 3 Global Step: 31225 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:17:02,272-Speed 126.63 samples/sec Loss 11.6810 Epoch: 3 Global Step: 31230 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:17:23,282-Speed 121.85 samples/sec Loss 12.0487 Epoch: 3 Global Step: 31235 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:17:43,540-Speed 126.38 samples/sec Loss 11.9812 Epoch: 3 Global Step: 31240 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:18:03,972-Speed 125.30 samples/sec Loss 11.8480 Epoch: 3 Global Step: 31245 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:18:25,060-Speed 121.40 samples/sec Loss 12.2423 Epoch: 3 Global Step: 31250 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:18:45,330-Speed 126.30 samples/sec Loss 11.9440 Epoch: 3 Global Step: 31255 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:19:05,593-Speed 126.35 samples/sec Loss 11.9141 Epoch: 3 Global Step: 31260 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:19:25,851-Speed 126.37 samples/sec Loss 11.8785 Epoch: 3 Global Step: 31265 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:19:46,086-Speed 126.52 samples/sec Loss 11.5900 Epoch: 3 Global Step: 31270 Fp16 Grad Scale: 8192 Required: 260 hours Training: 2021-06-08 08:20:06,609-Speed 124.74 samples/sec Loss 11.7279 Epoch: 3 Global Step: 31275 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:20:27,319-Speed 123.62 samples/sec Loss 11.8738 Epoch: 3 Global Step: 31280 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:20:47,559-Speed 126.49 samples/sec Loss 11.8588 Epoch: 3 Global Step: 31285 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:21:07,836-Speed 126.26 samples/sec Loss 11.7332 Epoch: 3 Global Step: 31290 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:21:28,126-Speed 126.17 samples/sec Loss 11.6430 Epoch: 3 Global Step: 31295 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:21:48,296-Speed 126.93 samples/sec Loss 11.7549 Epoch: 3 Global Step: 31300 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:22:08,761-Speed 125.10 samples/sec Loss 12.0701 Epoch: 3 Global Step: 31305 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:22:29,575-Speed 123.00 samples/sec Loss 11.8515 Epoch: 3 Global Step: 31310 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:22:49,790-Speed 126.64 samples/sec Loss 11.7287 Epoch: 3 Global Step: 31315 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:23:10,022-Speed 126.54 samples/sec Loss 12.4005 Epoch: 3 Global Step: 31320 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:23:31,030-Speed 121.87 samples/sec Loss 11.9488 Epoch: 3 Global Step: 31325 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:23:51,341-Speed 126.04 samples/sec Loss 12.0967 Epoch: 3 Global Step: 31330 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:24:11,497-Speed 127.02 samples/sec Loss 12.3335 Epoch: 3 Global Step: 31335 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:24:31,800-Speed 126.10 samples/sec Loss 11.6311 Epoch: 3 Global Step: 31340 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:24:52,056-Speed 126.39 samples/sec Loss 11.8177 Epoch: 3 Global Step: 31345 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:25:12,305-Speed 126.43 samples/sec Loss 12.0844 Epoch: 3 Global Step: 31350 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:25:32,577-Speed 126.29 samples/sec Loss 11.4523 Epoch: 3 Global Step: 31355 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:25:52,933-Speed 125.77 samples/sec Loss 11.9824 Epoch: 3 Global Step: 31360 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:26:13,332-Speed 125.51 samples/sec Loss 11.5802 Epoch: 3 Global Step: 31365 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:26:34,315-Speed 122.01 samples/sec Loss 12.2485 Epoch: 3 Global Step: 31370 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:26:55,244-Speed 122.32 samples/sec Loss 11.7496 Epoch: 3 Global Step: 31375 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:27:15,497-Speed 126.40 samples/sec Loss 11.9448 Epoch: 3 Global Step: 31380 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:27:36,022-Speed 124.73 samples/sec Loss 11.9897 Epoch: 3 Global Step: 31385 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:27:56,294-Speed 126.29 samples/sec Loss 12.0698 Epoch: 3 Global Step: 31390 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:28:16,778-Speed 124.98 samples/sec Loss 11.6785 Epoch: 3 Global Step: 31395 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:28:37,331-Speed 124.56 samples/sec Loss 11.8283 Epoch: 3 Global Step: 31400 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:28:57,566-Speed 126.52 samples/sec Loss 11.9903 Epoch: 3 Global Step: 31405 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:29:17,679-Speed 127.29 samples/sec Loss 11.5143 Epoch: 3 Global Step: 31410 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:29:38,805-Speed 121.18 samples/sec Loss 12.3329 Epoch: 3 Global Step: 31415 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:29:59,328-Speed 124.74 samples/sec Loss 11.6110 Epoch: 3 Global Step: 31420 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:30:19,645-Speed 126.01 samples/sec Loss 11.8682 Epoch: 3 Global Step: 31425 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:30:40,123-Speed 125.02 samples/sec Loss 11.9193 Epoch: 3 Global Step: 31430 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:31:00,517-Speed 125.53 samples/sec Loss 11.8449 Epoch: 3 Global Step: 31435 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:31:21,026-Speed 124.83 samples/sec Loss 11.9260 Epoch: 3 Global Step: 31440 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:31:41,293-Speed 126.32 samples/sec Loss 11.8099 Epoch: 3 Global Step: 31445 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:32:01,591-Speed 126.12 samples/sec Loss 12.0122 Epoch: 3 Global Step: 31450 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:32:21,863-Speed 126.29 samples/sec Loss 11.9215 Epoch: 3 Global Step: 31455 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:32:42,546-Speed 123.78 samples/sec Loss 11.9794 Epoch: 3 Global Step: 31460 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:33:02,849-Speed 126.10 samples/sec Loss 11.6884 Epoch: 3 Global Step: 31465 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:33:23,102-Speed 126.40 samples/sec Loss 12.2774 Epoch: 3 Global Step: 31470 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:33:43,406-Speed 126.09 samples/sec Loss 11.6375 Epoch: 3 Global Step: 31475 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:34:03,697-Speed 126.17 samples/sec Loss 11.6315 Epoch: 3 Global Step: 31480 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:34:24,014-Speed 126.01 samples/sec Loss 11.7617 Epoch: 3 Global Step: 31485 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:34:44,272-Speed 126.38 samples/sec Loss 12.0320 Epoch: 3 Global Step: 31490 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:35:04,625-Speed 125.79 samples/sec Loss 11.9263 Epoch: 3 Global Step: 31495 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:35:24,859-Speed 126.53 samples/sec Loss 12.2688 Epoch: 3 Global Step: 31500 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:35:46,090-Speed 120.59 samples/sec Loss 11.7592 Epoch: 3 Global Step: 31505 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:36:06,352-Speed 126.35 samples/sec Loss 11.8125 Epoch: 3 Global Step: 31510 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:36:27,125-Speed 123.24 samples/sec Loss 11.7186 Epoch: 3 Global Step: 31515 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:36:47,622-Speed 124.90 samples/sec Loss 11.7395 Epoch: 3 Global Step: 31520 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:37:08,056-Speed 125.29 samples/sec Loss 11.4597 Epoch: 3 Global Step: 31525 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:37:28,314-Speed 126.38 samples/sec Loss 11.8811 Epoch: 3 Global Step: 31530 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:37:48,397-Speed 127.48 samples/sec Loss 12.1777 Epoch: 3 Global Step: 31535 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:38:08,671-Speed 126.28 samples/sec Loss 11.8206 Epoch: 3 Global Step: 31540 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:38:28,860-Speed 126.81 samples/sec Loss 12.0231 Epoch: 3 Global Step: 31545 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:38:49,789-Speed 122.32 samples/sec Loss 11.5618 Epoch: 3 Global Step: 31550 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:39:09,896-Speed 127.32 samples/sec Loss 11.7846 Epoch: 3 Global Step: 31555 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:39:30,542-Speed 124.00 samples/sec Loss 11.8123 Epoch: 3 Global Step: 31560 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:39:50,782-Speed 126.49 samples/sec Loss 11.8098 Epoch: 3 Global Step: 31565 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:40:11,033-Speed 126.42 samples/sec Loss 11.9990 Epoch: 3 Global Step: 31570 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:40:31,289-Speed 126.39 samples/sec Loss 12.0760 Epoch: 3 Global Step: 31575 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:40:51,606-Speed 126.01 samples/sec Loss 11.8557 Epoch: 3 Global Step: 31580 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:41:11,828-Speed 126.61 samples/sec Loss 11.5170 Epoch: 3 Global Step: 31585 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:41:32,292-Speed 125.10 samples/sec Loss 11.6335 Epoch: 3 Global Step: 31590 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:41:53,352-Speed 121.56 samples/sec Loss 11.9102 Epoch: 3 Global Step: 31595 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:42:13,657-Speed 126.08 samples/sec Loss 12.1542 Epoch: 3 Global Step: 31600 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:42:33,869-Speed 126.66 samples/sec Loss 12.1261 Epoch: 3 Global Step: 31605 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:42:54,091-Speed 126.60 samples/sec Loss 12.3452 Epoch: 3 Global Step: 31610 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:43:14,150-Speed 127.63 samples/sec Loss 12.0711 Epoch: 3 Global Step: 31615 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:43:34,436-Speed 126.20 samples/sec Loss 11.9714 Epoch: 3 Global Step: 31620 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:43:54,490-Speed 127.66 samples/sec Loss 11.8032 Epoch: 3 Global Step: 31625 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:44:14,641-Speed 127.05 samples/sec Loss 11.6173 Epoch: 3 Global Step: 31630 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:44:34,810-Speed 126.93 samples/sec Loss 11.8404 Epoch: 3 Global Step: 31635 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:44:55,564-Speed 123.36 samples/sec Loss 11.8692 Epoch: 3 Global Step: 31640 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:45:15,621-Speed 127.64 samples/sec Loss 11.9753 Epoch: 3 Global Step: 31645 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:45:35,912-Speed 126.17 samples/sec Loss 12.4761 Epoch: 3 Global Step: 31650 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:45:56,201-Speed 126.19 samples/sec Loss 11.7697 Epoch: 3 Global Step: 31655 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:46:16,366-Speed 126.96 samples/sec Loss 11.7670 Epoch: 3 Global Step: 31660 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:46:37,011-Speed 124.01 samples/sec Loss 11.8278 Epoch: 3 Global Step: 31665 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:46:57,276-Speed 126.34 samples/sec Loss 11.8427 Epoch: 3 Global Step: 31670 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:47:17,611-Speed 125.90 samples/sec Loss 11.8636 Epoch: 3 Global Step: 31675 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:47:37,834-Speed 126.60 samples/sec Loss 11.5001 Epoch: 3 Global Step: 31680 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:47:58,791-Speed 122.16 samples/sec Loss 11.4391 Epoch: 3 Global Step: 31685 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:48:19,247-Speed 125.15 samples/sec Loss 11.9328 Epoch: 3 Global Step: 31690 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:48:39,504-Speed 126.38 samples/sec Loss 11.7792 Epoch: 3 Global Step: 31695 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:48:59,749-Speed 126.46 samples/sec Loss 11.6224 Epoch: 3 Global Step: 31700 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:49:20,120-Speed 125.68 samples/sec Loss 11.9303 Epoch: 3 Global Step: 31705 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:49:40,388-Speed 126.32 samples/sec Loss 11.5748 Epoch: 3 Global Step: 31710 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:50:00,714-Speed 125.96 samples/sec Loss 12.1543 Epoch: 3 Global Step: 31715 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:50:20,929-Speed 126.64 samples/sec Loss 11.5434 Epoch: 3 Global Step: 31720 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:50:41,108-Speed 126.87 samples/sec Loss 11.6832 Epoch: 3 Global Step: 31725 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:51:02,027-Speed 122.39 samples/sec Loss 11.7791 Epoch: 3 Global Step: 31730 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:51:22,305-Speed 126.25 samples/sec Loss 11.8749 Epoch: 3 Global Step: 31735 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:51:42,530-Speed 126.58 samples/sec Loss 12.3272 Epoch: 3 Global Step: 31740 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:52:03,009-Speed 125.02 samples/sec Loss 11.9334 Epoch: 3 Global Step: 31745 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:52:23,177-Speed 126.94 samples/sec Loss 11.8463 Epoch: 3 Global Step: 31750 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:52:43,429-Speed 126.41 samples/sec Loss 11.5722 Epoch: 3 Global Step: 31755 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:53:03,640-Speed 126.67 samples/sec Loss 11.8095 Epoch: 3 Global Step: 31760 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:53:23,734-Speed 127.41 samples/sec Loss 11.8016 Epoch: 3 Global Step: 31765 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:53:43,864-Speed 127.18 samples/sec Loss 11.6486 Epoch: 3 Global Step: 31770 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:54:05,033-Speed 120.94 samples/sec Loss 11.6648 Epoch: 3 Global Step: 31775 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:54:25,329-Speed 126.13 samples/sec Loss 11.8515 Epoch: 3 Global Step: 31780 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:54:46,078-Speed 123.38 samples/sec Loss 12.0267 Epoch: 3 Global Step: 31785 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:55:06,454-Speed 125.65 samples/sec Loss 11.7847 Epoch: 3 Global Step: 31790 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:55:26,669-Speed 126.64 samples/sec Loss 11.6176 Epoch: 3 Global Step: 31795 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:55:46,867-Speed 126.75 samples/sec Loss 11.8512 Epoch: 3 Global Step: 31800 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:56:07,261-Speed 125.53 samples/sec Loss 11.9622 Epoch: 3 Global Step: 31805 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:56:27,350-Speed 127.44 samples/sec Loss 12.1589 Epoch: 3 Global Step: 31810 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:56:47,539-Speed 126.81 samples/sec Loss 11.6933 Epoch: 3 Global Step: 31815 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:57:08,373-Speed 122.88 samples/sec Loss 11.8763 Epoch: 3 Global Step: 31820 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:57:28,629-Speed 126.39 samples/sec Loss 11.7115 Epoch: 3 Global Step: 31825 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:57:48,735-Speed 127.33 samples/sec Loss 11.8187 Epoch: 3 Global Step: 31830 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:58:08,985-Speed 126.43 samples/sec Loss 11.5816 Epoch: 3 Global Step: 31835 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:58:29,091-Speed 127.33 samples/sec Loss 11.9197 Epoch: 3 Global Step: 31840 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:58:49,403-Speed 126.04 samples/sec Loss 11.9685 Epoch: 3 Global Step: 31845 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:59:09,743-Speed 125.87 samples/sec Loss 11.7885 Epoch: 3 Global Step: 31850 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:59:29,952-Speed 126.69 samples/sec Loss 11.6321 Epoch: 3 Global Step: 31855 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 08:59:50,050-Speed 127.38 samples/sec Loss 11.5998 Epoch: 3 Global Step: 31860 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 09:00:11,027-Speed 122.05 samples/sec Loss 11.8856 Epoch: 3 Global Step: 31865 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 09:00:31,273-Speed 126.45 samples/sec Loss 11.5602 Epoch: 3 Global Step: 31870 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 09:00:51,307-Speed 127.79 samples/sec Loss 11.7353 Epoch: 3 Global Step: 31875 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 09:01:11,707-Speed 125.49 samples/sec Loss 11.4299 Epoch: 3 Global Step: 31880 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 09:01:32,034-Speed 125.95 samples/sec Loss 11.6534 Epoch: 3 Global Step: 31885 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 09:01:52,372-Speed 125.88 samples/sec Loss 12.0416 Epoch: 3 Global Step: 31890 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 09:02:12,476-Speed 127.35 samples/sec Loss 11.7094 Epoch: 3 Global Step: 31895 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 09:02:32,877-Speed 125.49 samples/sec Loss 12.1861 Epoch: 3 Global Step: 31900 Fp16 Grad Scale: 8192 Required: 259 hours Training: 2021-06-08 09:02:53,280-Speed 125.48 samples/sec Loss 11.8773 Epoch: 3 Global Step: 31905 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:03:14,158-Speed 122.63 samples/sec Loss 11.6699 Epoch: 3 Global Step: 31910 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:03:34,654-Speed 124.90 samples/sec Loss 11.7363 Epoch: 3 Global Step: 31915 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:03:54,958-Speed 126.09 samples/sec Loss 12.0535 Epoch: 3 Global Step: 31920 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:04:15,203-Speed 126.46 samples/sec Loss 11.6858 Epoch: 3 Global Step: 31925 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:04:35,301-Speed 127.39 samples/sec Loss 11.9775 Epoch: 3 Global Step: 31930 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:04:55,818-Speed 124.78 samples/sec Loss 11.6568 Epoch: 3 Global Step: 31935 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:05:16,250-Speed 125.30 samples/sec Loss 11.9952 Epoch: 3 Global Step: 31940 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:05:36,672-Speed 125.36 samples/sec Loss 11.5471 Epoch: 3 Global Step: 31945 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:05:57,036-Speed 125.72 samples/sec Loss 11.9654 Epoch: 3 Global Step: 31950 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:06:18,004-Speed 122.10 samples/sec Loss 11.5706 Epoch: 3 Global Step: 31955 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:06:38,351-Speed 125.83 samples/sec Loss 11.8279 Epoch: 3 Global Step: 31960 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:06:58,666-Speed 126.02 samples/sec Loss 11.6167 Epoch: 3 Global Step: 31965 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:07:18,932-Speed 126.33 samples/sec Loss 11.7792 Epoch: 3 Global Step: 31970 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:07:39,281-Speed 125.81 samples/sec Loss 12.2445 Epoch: 3 Global Step: 31975 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:07:59,568-Speed 126.20 samples/sec Loss 11.8710 Epoch: 3 Global Step: 31980 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:08:19,879-Speed 126.05 samples/sec Loss 12.0112 Epoch: 3 Global Step: 31985 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:08:40,005-Speed 127.21 samples/sec Loss 11.5949 Epoch: 3 Global Step: 31990 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:09:00,692-Speed 123.76 samples/sec Loss 11.8939 Epoch: 3 Global Step: 31995 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:09:21,730-Speed 121.69 samples/sec Loss 11.8646 Epoch: 3 Global Step: 32000 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:09:42,041-Speed 126.05 samples/sec Loss 11.9971 Epoch: 3 Global Step: 32005 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:10:02,321-Speed 126.24 samples/sec Loss 12.0587 Epoch: 3 Global Step: 32010 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:10:22,590-Speed 126.31 samples/sec Loss 12.0510 Epoch: 3 Global Step: 32015 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:10:42,879-Speed 126.18 samples/sec Loss 11.7619 Epoch: 3 Global Step: 32020 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:11:03,180-Speed 126.12 samples/sec Loss 11.7811 Epoch: 3 Global Step: 32025 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:11:23,482-Speed 126.10 samples/sec Loss 11.8967 Epoch: 3 Global Step: 32030 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:11:43,789-Speed 126.08 samples/sec Loss 12.0435 Epoch: 3 Global Step: 32035 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:12:04,071-Speed 126.22 samples/sec Loss 11.8896 Epoch: 3 Global Step: 32040 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:12:25,050-Speed 122.03 samples/sec Loss 11.7457 Epoch: 3 Global Step: 32045 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:12:45,347-Speed 126.13 samples/sec Loss 11.8238 Epoch: 3 Global Step: 32050 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:13:05,718-Speed 125.67 samples/sec Loss 11.5688 Epoch: 3 Global Step: 32055 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:13:26,086-Speed 125.69 samples/sec Loss 12.1410 Epoch: 3 Global Step: 32060 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:13:46,440-Speed 125.79 samples/sec Loss 11.7937 Epoch: 3 Global Step: 32065 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:14:06,789-Speed 125.81 samples/sec Loss 11.9317 Epoch: 3 Global Step: 32070 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:14:27,115-Speed 125.96 samples/sec Loss 11.9338 Epoch: 3 Global Step: 32075 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:14:47,441-Speed 125.95 samples/sec Loss 11.8635 Epoch: 3 Global Step: 32080 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:15:07,795-Speed 125.79 samples/sec Loss 12.1068 Epoch: 3 Global Step: 32085 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:15:28,714-Speed 122.38 samples/sec Loss 11.5356 Epoch: 3 Global Step: 32090 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:15:49,003-Speed 126.18 samples/sec Loss 12.1855 Epoch: 3 Global Step: 32095 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:16:09,345-Speed 125.86 samples/sec Loss 12.0117 Epoch: 3 Global Step: 32100 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:16:29,764-Speed 125.38 samples/sec Loss 12.1362 Epoch: 3 Global Step: 32105 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:16:50,077-Speed 126.04 samples/sec Loss 11.9064 Epoch: 3 Global Step: 32110 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:17:10,573-Speed 124.91 samples/sec Loss 11.8251 Epoch: 3 Global Step: 32115 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:17:31,235-Speed 123.91 samples/sec Loss 11.7642 Epoch: 3 Global Step: 32120 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:17:51,683-Speed 125.21 samples/sec Loss 12.0606 Epoch: 3 Global Step: 32125 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:18:11,955-Speed 126.29 samples/sec Loss 11.9102 Epoch: 3 Global Step: 32130 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:18:32,903-Speed 122.21 samples/sec Loss 11.8543 Epoch: 3 Global Step: 32135 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:18:53,156-Speed 126.41 samples/sec Loss 11.6303 Epoch: 3 Global Step: 32140 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:19:13,290-Speed 127.15 samples/sec Loss 11.1205 Epoch: 3 Global Step: 32145 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:19:33,565-Speed 126.27 samples/sec Loss 11.7094 Epoch: 3 Global Step: 32150 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:19:53,947-Speed 125.61 samples/sec Loss 11.5438 Epoch: 3 Global Step: 32155 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:20:14,275-Speed 125.94 samples/sec Loss 11.7642 Epoch: 3 Global Step: 32160 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:20:34,644-Speed 125.69 samples/sec Loss 11.5821 Epoch: 3 Global Step: 32165 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:20:54,928-Speed 126.22 samples/sec Loss 11.6690 Epoch: 3 Global Step: 32170 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:21:15,206-Speed 126.26 samples/sec Loss 12.0694 Epoch: 3 Global Step: 32175 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:21:36,200-Speed 121.95 samples/sec Loss 11.7491 Epoch: 3 Global Step: 32180 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:21:56,517-Speed 126.00 samples/sec Loss 11.9806 Epoch: 3 Global Step: 32185 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:22:16,885-Speed 125.69 samples/sec Loss 11.9990 Epoch: 3 Global Step: 32190 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:22:37,195-Speed 126.05 samples/sec Loss 12.1748 Epoch: 3 Global Step: 32195 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:22:57,480-Speed 126.21 samples/sec Loss 12.2205 Epoch: 3 Global Step: 32200 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:23:17,771-Speed 126.17 samples/sec Loss 11.9631 Epoch: 3 Global Step: 32205 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:23:38,184-Speed 125.42 samples/sec Loss 11.9459 Epoch: 3 Global Step: 32210 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:23:58,460-Speed 126.27 samples/sec Loss 11.8793 Epoch: 3 Global Step: 32215 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:24:18,750-Speed 126.18 samples/sec Loss 11.7828 Epoch: 3 Global Step: 32220 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:24:39,720-Speed 122.09 samples/sec Loss 12.1714 Epoch: 3 Global Step: 32225 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:25:00,014-Speed 126.15 samples/sec Loss 11.4700 Epoch: 3 Global Step: 32230 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:25:20,507-Speed 124.92 samples/sec Loss 11.6739 Epoch: 3 Global Step: 32235 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:25:40,692-Speed 126.84 samples/sec Loss 11.3656 Epoch: 3 Global Step: 32240 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:26:01,106-Speed 125.41 samples/sec Loss 11.8487 Epoch: 3 Global Step: 32245 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:26:21,375-Speed 126.31 samples/sec Loss 12.0362 Epoch: 3 Global Step: 32250 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:26:41,660-Speed 126.21 samples/sec Loss 11.6826 Epoch: 3 Global Step: 32255 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:27:01,850-Speed 126.81 samples/sec Loss 11.9030 Epoch: 3 Global Step: 32260 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:27:22,140-Speed 126.18 samples/sec Loss 12.0170 Epoch: 3 Global Step: 32265 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:27:43,111-Speed 122.08 samples/sec Loss 11.7825 Epoch: 3 Global Step: 32270 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:28:03,377-Speed 126.32 samples/sec Loss 11.7938 Epoch: 3 Global Step: 32275 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:28:23,631-Speed 126.40 samples/sec Loss 11.9989 Epoch: 3 Global Step: 32280 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:28:43,905-Speed 126.28 samples/sec Loss 11.8244 Epoch: 3 Global Step: 32285 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:29:04,371-Speed 125.09 samples/sec Loss 11.8896 Epoch: 3 Global Step: 32290 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:29:24,616-Speed 126.46 samples/sec Loss 11.7822 Epoch: 3 Global Step: 32295 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:29:44,884-Speed 126.32 samples/sec Loss 11.7885 Epoch: 3 Global Step: 32300 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:30:05,152-Speed 126.32 samples/sec Loss 11.7642 Epoch: 3 Global Step: 32305 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:30:25,464-Speed 126.05 samples/sec Loss 11.9037 Epoch: 3 Global Step: 32310 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:30:46,461-Speed 121.93 samples/sec Loss 11.9280 Epoch: 3 Global Step: 32315 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:31:06,800-Speed 125.87 samples/sec Loss 11.8575 Epoch: 3 Global Step: 32320 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:31:27,116-Speed 126.01 samples/sec Loss 11.7572 Epoch: 3 Global Step: 32325 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:31:47,981-Speed 122.70 samples/sec Loss 11.5871 Epoch: 3 Global Step: 32330 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:32:09,453-Speed 119.23 samples/sec Loss 11.9528 Epoch: 3 Global Step: 32335 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:32:29,750-Speed 126.13 samples/sec Loss 11.7837 Epoch: 3 Global Step: 32340 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:32:49,981-Speed 126.55 samples/sec Loss 11.5823 Epoch: 3 Global Step: 32345 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:33:10,318-Speed 125.89 samples/sec Loss 11.7700 Epoch: 3 Global Step: 32350 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:33:30,628-Speed 126.05 samples/sec Loss 11.8063 Epoch: 3 Global Step: 32355 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:33:51,744-Speed 121.24 samples/sec Loss 11.7455 Epoch: 3 Global Step: 32360 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:34:11,902-Speed 127.00 samples/sec Loss 11.6641 Epoch: 3 Global Step: 32365 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:34:32,197-Speed 126.14 samples/sec Loss 11.5812 Epoch: 3 Global Step: 32370 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:34:52,872-Speed 123.83 samples/sec Loss 11.9202 Epoch: 3 Global Step: 32375 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:35:13,123-Speed 126.42 samples/sec Loss 11.7625 Epoch: 3 Global Step: 32380 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:35:33,342-Speed 126.62 samples/sec Loss 11.7927 Epoch: 3 Global Step: 32385 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:35:53,549-Speed 126.69 samples/sec Loss 11.5908 Epoch: 3 Global Step: 32390 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:36:13,869-Speed 125.99 samples/sec Loss 11.6945 Epoch: 3 Global Step: 32395 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:36:33,954-Speed 127.46 samples/sec Loss 11.9038 Epoch: 3 Global Step: 32400 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:36:54,812-Speed 122.74 samples/sec Loss 11.8555 Epoch: 3 Global Step: 32405 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:37:14,875-Speed 127.60 samples/sec Loss 12.0521 Epoch: 3 Global Step: 32410 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:37:35,139-Speed 126.34 samples/sec Loss 11.9197 Epoch: 3 Global Step: 32415 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:37:55,197-Speed 127.63 samples/sec Loss 11.9093 Epoch: 3 Global Step: 32420 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:38:15,475-Speed 126.26 samples/sec Loss 11.7351 Epoch: 3 Global Step: 32425 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:38:35,894-Speed 125.38 samples/sec Loss 11.8747 Epoch: 3 Global Step: 32430 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:38:56,360-Speed 125.09 samples/sec Loss 11.9902 Epoch: 3 Global Step: 32435 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:39:16,615-Speed 126.39 samples/sec Loss 11.8162 Epoch: 3 Global Step: 32440 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:39:36,975-Speed 125.75 samples/sec Loss 11.9454 Epoch: 3 Global Step: 32445 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:39:58,057-Speed 121.44 samples/sec Loss 11.7296 Epoch: 3 Global Step: 32450 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:40:18,182-Speed 127.21 samples/sec Loss 11.9255 Epoch: 3 Global Step: 32455 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:40:38,511-Speed 125.93 samples/sec Loss 11.6318 Epoch: 3 Global Step: 32460 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:40:58,731-Speed 126.61 samples/sec Loss 11.8894 Epoch: 3 Global Step: 32465 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:41:18,862-Speed 127.17 samples/sec Loss 11.7584 Epoch: 3 Global Step: 32470 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:41:39,005-Speed 127.10 samples/sec Loss 11.7992 Epoch: 3 Global Step: 32475 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:41:58,962-Speed 128.28 samples/sec Loss 11.4920 Epoch: 3 Global Step: 32480 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:42:19,267-Speed 126.09 samples/sec Loss 11.8822 Epoch: 3 Global Step: 32485 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:42:39,519-Speed 126.42 samples/sec Loss 11.8922 Epoch: 3 Global Step: 32490 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:43:00,546-Speed 121.75 samples/sec Loss 11.7388 Epoch: 3 Global Step: 32495 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:43:20,614-Speed 127.57 samples/sec Loss 11.7998 Epoch: 3 Global Step: 32500 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:43:40,573-Speed 128.27 samples/sec Loss 11.7027 Epoch: 3 Global Step: 32505 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:44:00,827-Speed 126.40 samples/sec Loss 11.7663 Epoch: 3 Global Step: 32510 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:44:21,095-Speed 126.31 samples/sec Loss 11.5344 Epoch: 3 Global Step: 32515 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:44:41,330-Speed 126.52 samples/sec Loss 11.8052 Epoch: 3 Global Step: 32520 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:45:01,721-Speed 125.56 samples/sec Loss 12.1504 Epoch: 3 Global Step: 32525 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:45:22,015-Speed 126.15 samples/sec Loss 11.6725 Epoch: 3 Global Step: 32530 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:45:42,306-Speed 126.18 samples/sec Loss 12.0356 Epoch: 3 Global Step: 32535 Fp16 Grad Scale: 8192 Required: 258 hours Training: 2021-06-08 09:46:03,242-Speed 122.29 samples/sec Loss 12.1258 Epoch: 3 Global Step: 32540 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:46:23,555-Speed 126.03 samples/sec Loss 11.7510 Epoch: 3 Global Step: 32545 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:46:43,808-Speed 126.40 samples/sec Loss 11.5194 Epoch: 3 Global Step: 32550 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:47:04,015-Speed 126.70 samples/sec Loss 11.6838 Epoch: 3 Global Step: 32555 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:47:24,498-Speed 124.99 samples/sec Loss 11.5207 Epoch: 3 Global Step: 32560 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:47:44,777-Speed 126.25 samples/sec Loss 11.3321 Epoch: 3 Global Step: 32565 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:48:05,270-Speed 124.93 samples/sec Loss 12.1408 Epoch: 3 Global Step: 32570 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:48:25,431-Speed 126.99 samples/sec Loss 11.7474 Epoch: 3 Global Step: 32575 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:48:45,793-Speed 125.73 samples/sec Loss 11.8638 Epoch: 3 Global Step: 32580 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:49:06,697-Speed 122.47 samples/sec Loss 11.6722 Epoch: 3 Global Step: 32585 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:49:27,636-Speed 122.27 samples/sec Loss 11.8565 Epoch: 3 Global Step: 32590 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:49:47,681-Speed 127.71 samples/sec Loss 11.7127 Epoch: 3 Global Step: 32595 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:50:08,260-Speed 124.41 samples/sec Loss 11.7971 Epoch: 3 Global Step: 32600 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:50:28,557-Speed 126.14 samples/sec Loss 12.1823 Epoch: 3 Global Step: 32605 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:50:48,793-Speed 126.51 samples/sec Loss 11.6758 Epoch: 3 Global Step: 32610 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:51:09,053-Speed 126.37 samples/sec Loss 11.8640 Epoch: 3 Global Step: 32615 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:51:29,272-Speed 126.62 samples/sec Loss 11.8276 Epoch: 3 Global Step: 32620 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:51:49,453-Speed 126.86 samples/sec Loss 11.8619 Epoch: 3 Global Step: 32625 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:52:10,422-Speed 122.09 samples/sec Loss 12.0663 Epoch: 3 Global Step: 32630 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:52:30,726-Speed 126.09 samples/sec Loss 12.2377 Epoch: 3 Global Step: 32635 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:52:50,955-Speed 126.56 samples/sec Loss 12.0135 Epoch: 3 Global Step: 32640 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:53:11,232-Speed 126.26 samples/sec Loss 11.6365 Epoch: 3 Global Step: 32645 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:53:31,466-Speed 126.53 samples/sec Loss 11.9638 Epoch: 3 Global Step: 32650 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:53:51,763-Speed 126.13 samples/sec Loss 12.2023 Epoch: 3 Global Step: 32655 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:54:12,060-Speed 126.14 samples/sec Loss 11.9629 Epoch: 3 Global Step: 32660 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:54:32,311-Speed 126.42 samples/sec Loss 11.4762 Epoch: 3 Global Step: 32665 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:54:52,550-Speed 126.49 samples/sec Loss 11.8612 Epoch: 3 Global Step: 32670 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:55:13,246-Speed 123.70 samples/sec Loss 11.8857 Epoch: 3 Global Step: 32675 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:55:33,757-Speed 124.82 samples/sec Loss 11.7379 Epoch: 3 Global Step: 32680 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:55:53,779-Speed 127.87 samples/sec Loss 11.7538 Epoch: 3 Global Step: 32685 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:56:13,909-Speed 127.18 samples/sec Loss 11.7579 Epoch: 3 Global Step: 32690 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:56:33,979-Speed 127.56 samples/sec Loss 11.7326 Epoch: 3 Global Step: 32695 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:56:54,070-Speed 127.43 samples/sec Loss 11.8892 Epoch: 3 Global Step: 32700 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:57:14,310-Speed 126.49 samples/sec Loss 12.0165 Epoch: 3 Global Step: 32705 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:57:34,368-Speed 127.63 samples/sec Loss 11.6836 Epoch: 3 Global Step: 32710 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:57:54,778-Speed 125.44 samples/sec Loss 11.9224 Epoch: 3 Global Step: 32715 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:58:15,997-Speed 120.65 samples/sec Loss 11.2125 Epoch: 3 Global Step: 32720 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:58:36,248-Speed 126.42 samples/sec Loss 11.9114 Epoch: 3 Global Step: 32725 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:58:56,393-Speed 127.09 samples/sec Loss 11.4689 Epoch: 3 Global Step: 32730 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:59:16,621-Speed 126.56 samples/sec Loss 11.5519 Epoch: 3 Global Step: 32735 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:59:36,967-Speed 125.83 samples/sec Loss 12.0049 Epoch: 3 Global Step: 32740 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 09:59:57,198-Speed 126.55 samples/sec Loss 11.9956 Epoch: 3 Global Step: 32745 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:00:18,499-Speed 120.19 samples/sec Loss 11.9966 Epoch: 3 Global Step: 32750 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:00:38,777-Speed 126.25 samples/sec Loss 11.7903 Epoch: 3 Global Step: 32755 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:00:59,083-Speed 126.07 samples/sec Loss 11.6777 Epoch: 3 Global Step: 32760 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:01:20,378-Speed 120.23 samples/sec Loss 11.5984 Epoch: 3 Global Step: 32765 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:01:40,686-Speed 126.06 samples/sec Loss 11.8365 Epoch: 3 Global Step: 32770 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:02:00,797-Speed 127.30 samples/sec Loss 11.7758 Epoch: 3 Global Step: 32775 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:02:21,116-Speed 126.00 samples/sec Loss 11.5821 Epoch: 3 Global Step: 32780 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:02:41,443-Speed 125.95 samples/sec Loss 12.1937 Epoch: 3 Global Step: 32785 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:03:01,718-Speed 126.27 samples/sec Loss 12.0343 Epoch: 3 Global Step: 32790 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:03:22,006-Speed 126.20 samples/sec Loss 11.9332 Epoch: 3 Global Step: 32795 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:03:42,339-Speed 125.91 samples/sec Loss 11.6168 Epoch: 3 Global Step: 32800 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:04:02,553-Speed 126.66 samples/sec Loss 12.0144 Epoch: 3 Global Step: 32805 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:04:23,274-Speed 123.55 samples/sec Loss 12.1153 Epoch: 3 Global Step: 32810 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:04:43,485-Speed 126.67 samples/sec Loss 11.9082 Epoch: 3 Global Step: 32815 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:05:03,714-Speed 126.56 samples/sec Loss 11.1102 Epoch: 3 Global Step: 32820 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:05:24,005-Speed 126.17 samples/sec Loss 11.7765 Epoch: 3 Global Step: 32825 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:05:44,364-Speed 125.75 samples/sec Loss 11.6858 Epoch: 3 Global Step: 32830 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:06:04,668-Speed 126.09 samples/sec Loss 11.7708 Epoch: 3 Global Step: 32835 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:06:25,151-Speed 124.99 samples/sec Loss 11.6924 Epoch: 3 Global Step: 32840 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:06:45,518-Speed 125.70 samples/sec Loss 11.8297 Epoch: 3 Global Step: 32845 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:07:05,795-Speed 126.26 samples/sec Loss 11.7687 Epoch: 3 Global Step: 32850 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:07:26,725-Speed 122.36 samples/sec Loss 11.8678 Epoch: 3 Global Step: 32855 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:07:47,216-Speed 124.94 samples/sec Loss 11.7852 Epoch: 3 Global Step: 32860 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:08:07,507-Speed 126.17 samples/sec Loss 11.7531 Epoch: 3 Global Step: 32865 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:08:27,714-Speed 126.70 samples/sec Loss 11.9558 Epoch: 3 Global Step: 32870 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:08:47,944-Speed 126.56 samples/sec Loss 11.8769 Epoch: 3 Global Step: 32875 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:09:08,018-Speed 127.53 samples/sec Loss 11.9312 Epoch: 3 Global Step: 32880 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:09:28,267-Speed 126.43 samples/sec Loss 11.8963 Epoch: 3 Global Step: 32885 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:09:48,499-Speed 126.54 samples/sec Loss 12.0636 Epoch: 3 Global Step: 32890 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:10:08,582-Speed 127.48 samples/sec Loss 11.5293 Epoch: 3 Global Step: 32895 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:10:29,501-Speed 122.38 samples/sec Loss 12.0736 Epoch: 3 Global Step: 32900 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:10:49,654-Speed 127.03 samples/sec Loss 11.9229 Epoch: 3 Global Step: 32905 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:11:09,714-Speed 127.62 samples/sec Loss 12.0318 Epoch: 3 Global Step: 32910 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:11:29,933-Speed 126.62 samples/sec Loss 12.0309 Epoch: 3 Global Step: 32915 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:11:50,594-Speed 123.91 samples/sec Loss 11.9567 Epoch: 3 Global Step: 32920 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:12:10,828-Speed 126.53 samples/sec Loss 11.8201 Epoch: 3 Global Step: 32925 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:12:30,949-Speed 127.24 samples/sec Loss 11.3498 Epoch: 3 Global Step: 32930 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:12:51,275-Speed 125.95 samples/sec Loss 12.1731 Epoch: 3 Global Step: 32935 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:13:11,400-Speed 127.21 samples/sec Loss 11.9996 Epoch: 3 Global Step: 32940 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:13:32,158-Speed 123.34 samples/sec Loss 11.8143 Epoch: 3 Global Step: 32945 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:13:52,502-Speed 125.84 samples/sec Loss 11.7315 Epoch: 3 Global Step: 32950 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:14:12,834-Speed 125.91 samples/sec Loss 11.9915 Epoch: 3 Global Step: 32955 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:14:33,133-Speed 126.12 samples/sec Loss 11.8183 Epoch: 3 Global Step: 32960 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:14:53,350-Speed 126.63 samples/sec Loss 11.8898 Epoch: 3 Global Step: 32965 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:15:13,660-Speed 126.05 samples/sec Loss 11.8788 Epoch: 3 Global Step: 32970 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:15:33,859-Speed 126.75 samples/sec Loss 11.9469 Epoch: 3 Global Step: 32975 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:15:54,009-Speed 127.06 samples/sec Loss 11.9568 Epoch: 3 Global Step: 32980 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:16:14,140-Speed 127.17 samples/sec Loss 12.0581 Epoch: 3 Global Step: 32985 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:16:34,802-Speed 123.91 samples/sec Loss 11.8249 Epoch: 3 Global Step: 32990 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:16:55,072-Speed 126.30 samples/sec Loss 12.0969 Epoch: 3 Global Step: 32995 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:17:15,538-Speed 125.09 samples/sec Loss 11.7773 Epoch: 3 Global Step: 33000 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:17:35,801-Speed 126.35 samples/sec Loss 11.8570 Epoch: 3 Global Step: 33005 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:17:55,989-Speed 126.81 samples/sec Loss 11.7197 Epoch: 3 Global Step: 33010 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:18:16,326-Speed 125.89 samples/sec Loss 11.9222 Epoch: 3 Global Step: 33015 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:18:36,612-Speed 126.20 samples/sec Loss 11.7721 Epoch: 3 Global Step: 33020 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:18:56,960-Speed 125.82 samples/sec Loss 11.8570 Epoch: 3 Global Step: 33025 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:19:17,396-Speed 125.28 samples/sec Loss 11.6407 Epoch: 3 Global Step: 33030 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:19:38,274-Speed 122.63 samples/sec Loss 11.7456 Epoch: 3 Global Step: 33035 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:19:58,491-Speed 126.63 samples/sec Loss 12.1939 Epoch: 3 Global Step: 33040 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:20:18,747-Speed 126.39 samples/sec Loss 12.0561 Epoch: 3 Global Step: 33045 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:20:39,025-Speed 126.25 samples/sec Loss 11.7773 Epoch: 3 Global Step: 33050 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:20:59,304-Speed 126.25 samples/sec Loss 11.4045 Epoch: 3 Global Step: 33055 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:21:19,552-Speed 126.44 samples/sec Loss 12.1112 Epoch: 3 Global Step: 33060 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:21:40,661-Speed 121.28 samples/sec Loss 11.5649 Epoch: 3 Global Step: 33065 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:22:00,906-Speed 126.46 samples/sec Loss 11.6576 Epoch: 3 Global Step: 33070 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:22:21,120-Speed 126.65 samples/sec Loss 12.0456 Epoch: 3 Global Step: 33075 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:22:42,402-Speed 120.29 samples/sec Loss 12.1920 Epoch: 3 Global Step: 33080 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:23:02,526-Speed 127.22 samples/sec Loss 11.5808 Epoch: 3 Global Step: 33085 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:23:22,857-Speed 125.92 samples/sec Loss 11.5412 Epoch: 3 Global Step: 33090 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:23:43,145-Speed 126.19 samples/sec Loss 12.1293 Epoch: 3 Global Step: 33095 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:24:03,383-Speed 126.50 samples/sec Loss 11.7651 Epoch: 3 Global Step: 33100 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:24:23,596-Speed 126.66 samples/sec Loss 11.6311 Epoch: 3 Global Step: 33105 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:24:43,897-Speed 126.11 samples/sec Loss 11.9850 Epoch: 3 Global Step: 33110 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:25:04,253-Speed 125.77 samples/sec Loss 11.5837 Epoch: 3 Global Step: 33115 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:25:24,603-Speed 125.81 samples/sec Loss 12.5119 Epoch: 3 Global Step: 33120 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:25:45,825-Speed 120.63 samples/sec Loss 12.3577 Epoch: 3 Global Step: 33125 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:26:06,395-Speed 124.46 samples/sec Loss 11.6917 Epoch: 3 Global Step: 33130 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:26:26,573-Speed 126.88 samples/sec Loss 11.6355 Epoch: 3 Global Step: 33135 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:26:46,887-Speed 126.03 samples/sec Loss 11.9657 Epoch: 3 Global Step: 33140 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:27:07,079-Speed 126.79 samples/sec Loss 11.6804 Epoch: 3 Global Step: 33145 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:27:27,280-Speed 126.73 samples/sec Loss 11.5838 Epoch: 3 Global Step: 33150 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:27:47,509-Speed 126.56 samples/sec Loss 11.8385 Epoch: 3 Global Step: 33155 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:28:07,763-Speed 126.40 samples/sec Loss 11.8603 Epoch: 3 Global Step: 33160 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:28:28,064-Speed 126.11 samples/sec Loss 11.7670 Epoch: 3 Global Step: 33165 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:28:48,954-Speed 122.55 samples/sec Loss 12.0378 Epoch: 3 Global Step: 33170 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:29:09,227-Speed 126.28 samples/sec Loss 12.0539 Epoch: 3 Global Step: 33175 Fp16 Grad Scale: 8192 Required: 257 hours Training: 2021-06-08 10:29:29,547-Speed 125.99 samples/sec Loss 11.8614 Epoch: 3 Global Step: 33180 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:29:49,771-Speed 126.59 samples/sec Loss 11.8953 Epoch: 3 Global Step: 33185 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:30:10,092-Speed 125.98 samples/sec Loss 11.8208 Epoch: 3 Global Step: 33190 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:30:30,374-Speed 126.23 samples/sec Loss 11.4822 Epoch: 3 Global Step: 33195 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:30:50,972-Speed 124.29 samples/sec Loss 11.4248 Epoch: 3 Global Step: 33200 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:31:11,138-Speed 126.95 samples/sec Loss 11.7903 Epoch: 3 Global Step: 33205 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:31:31,349-Speed 126.67 samples/sec Loss 11.7353 Epoch: 3 Global Step: 33210 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:31:52,226-Speed 122.63 samples/sec Loss 11.8445 Epoch: 3 Global Step: 33215 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:32:12,474-Speed 126.44 samples/sec Loss 11.6940 Epoch: 3 Global Step: 33220 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:32:32,804-Speed 125.92 samples/sec Loss 11.4231 Epoch: 3 Global Step: 33225 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:32:53,078-Speed 126.28 samples/sec Loss 11.5239 Epoch: 3 Global Step: 33230 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:33:13,380-Speed 126.11 samples/sec Loss 12.4938 Epoch: 3 Global Step: 33235 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:33:33,688-Speed 126.07 samples/sec Loss 12.0081 Epoch: 3 Global Step: 33240 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:33:53,925-Speed 126.51 samples/sec Loss 11.9664 Epoch: 3 Global Step: 33245 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:34:14,187-Speed 126.35 samples/sec Loss 11.9719 Epoch: 3 Global Step: 33250 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:34:34,448-Speed 126.36 samples/sec Loss 11.9643 Epoch: 3 Global Step: 33255 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:34:55,369-Speed 122.37 samples/sec Loss 12.1870 Epoch: 3 Global Step: 33260 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:35:15,382-Speed 127.92 samples/sec Loss 11.7742 Epoch: 3 Global Step: 33265 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:35:35,617-Speed 126.52 samples/sec Loss 11.8423 Epoch: 3 Global Step: 33270 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:35:55,922-Speed 126.08 samples/sec Loss 11.9962 Epoch: 3 Global Step: 33275 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:36:16,157-Speed 126.52 samples/sec Loss 11.3437 Epoch: 3 Global Step: 33280 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:36:36,411-Speed 126.41 samples/sec Loss 12.0050 Epoch: 3 Global Step: 33285 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:36:56,676-Speed 126.33 samples/sec Loss 11.6717 Epoch: 3 Global Step: 33290 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:37:16,807-Speed 127.18 samples/sec Loss 11.7526 Epoch: 3 Global Step: 33295 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:37:37,052-Speed 126.46 samples/sec Loss 11.7985 Epoch: 3 Global Step: 33300 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:37:58,059-Speed 121.87 samples/sec Loss 11.6083 Epoch: 3 Global Step: 33305 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:38:18,355-Speed 126.15 samples/sec Loss 11.8719 Epoch: 3 Global Step: 33310 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:38:38,460-Speed 127.34 samples/sec Loss 11.3783 Epoch: 3 Global Step: 33315 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:38:58,564-Speed 127.34 samples/sec Loss 12.1032 Epoch: 3 Global Step: 33320 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:39:18,781-Speed 126.63 samples/sec Loss 11.6402 Epoch: 3 Global Step: 33325 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:39:38,983-Speed 126.73 samples/sec Loss 11.8427 Epoch: 3 Global Step: 33330 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:39:59,192-Speed 126.68 samples/sec Loss 11.7118 Epoch: 3 Global Step: 33335 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:40:19,404-Speed 126.67 samples/sec Loss 11.7888 Epoch: 3 Global Step: 33340 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:40:39,713-Speed 126.06 samples/sec Loss 11.4998 Epoch: 3 Global Step: 33345 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:41:00,676-Speed 122.12 samples/sec Loss 11.6927 Epoch: 3 Global Step: 33350 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:41:21,075-Speed 125.50 samples/sec Loss 11.7180 Epoch: 3 Global Step: 33355 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:41:41,479-Speed 125.47 samples/sec Loss 11.7517 Epoch: 3 Global Step: 33360 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:42:01,813-Speed 125.91 samples/sec Loss 11.7224 Epoch: 3 Global Step: 33365 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:42:22,103-Speed 126.18 samples/sec Loss 11.9103 Epoch: 3 Global Step: 33370 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:42:42,249-Speed 127.08 samples/sec Loss 11.5967 Epoch: 3 Global Step: 33375 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:43:02,469-Speed 126.61 samples/sec Loss 11.7110 Epoch: 3 Global Step: 33380 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:43:22,580-Speed 127.30 samples/sec Loss 11.9462 Epoch: 3 Global Step: 33385 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:43:42,818-Speed 126.51 samples/sec Loss 11.8709 Epoch: 3 Global Step: 33390 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:44:04,138-Speed 120.08 samples/sec Loss 11.7722 Epoch: 3 Global Step: 33395 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:44:24,515-Speed 125.64 samples/sec Loss 11.8931 Epoch: 3 Global Step: 33400 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:44:44,725-Speed 126.67 samples/sec Loss 11.5995 Epoch: 3 Global Step: 33405 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:45:04,739-Speed 127.92 samples/sec Loss 12.2001 Epoch: 3 Global Step: 33410 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:45:25,000-Speed 126.36 samples/sec Loss 11.5716 Epoch: 3 Global Step: 33415 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:45:45,307-Speed 126.07 samples/sec Loss 11.7230 Epoch: 3 Global Step: 33420 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:46:05,613-Speed 126.08 samples/sec Loss 12.1285 Epoch: 3 Global Step: 33425 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:46:25,886-Speed 126.28 samples/sec Loss 11.8739 Epoch: 3 Global Step: 33430 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:46:46,141-Speed 126.40 samples/sec Loss 12.2771 Epoch: 3 Global Step: 33435 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:47:06,845-Speed 123.66 samples/sec Loss 11.9908 Epoch: 3 Global Step: 33440 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:47:27,122-Speed 126.25 samples/sec Loss 12.0219 Epoch: 3 Global Step: 33445 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:47:47,161-Speed 127.76 samples/sec Loss 11.7631 Epoch: 3 Global Step: 33450 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:48:07,374-Speed 126.65 samples/sec Loss 11.5135 Epoch: 3 Global Step: 33455 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:48:27,581-Speed 126.70 samples/sec Loss 12.0091 Epoch: 3 Global Step: 33460 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:48:47,959-Speed 125.63 samples/sec Loss 11.9128 Epoch: 3 Global Step: 33465 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:49:08,239-Speed 126.24 samples/sec Loss 11.8009 Epoch: 3 Global Step: 33470 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:49:28,470-Speed 126.55 samples/sec Loss 11.7739 Epoch: 3 Global Step: 33475 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:49:48,751-Speed 126.24 samples/sec Loss 11.8045 Epoch: 3 Global Step: 33480 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:50:09,728-Speed 122.04 samples/sec Loss 11.6997 Epoch: 3 Global Step: 33485 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:50:30,043-Speed 126.02 samples/sec Loss 11.9680 Epoch: 3 Global Step: 33490 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:50:50,277-Speed 126.53 samples/sec Loss 11.4641 Epoch: 3 Global Step: 33495 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:51:10,528-Speed 126.42 samples/sec Loss 11.6623 Epoch: 3 Global Step: 33500 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:51:30,747-Speed 126.62 samples/sec Loss 11.7141 Epoch: 3 Global Step: 33505 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:51:51,019-Speed 126.30 samples/sec Loss 11.9940 Epoch: 3 Global Step: 33510 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:52:11,060-Speed 127.74 samples/sec Loss 12.1282 Epoch: 3 Global Step: 33515 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:52:31,385-Speed 125.96 samples/sec Loss 12.0782 Epoch: 3 Global Step: 33520 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:52:51,759-Speed 125.66 samples/sec Loss 11.7901 Epoch: 3 Global Step: 33525 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:53:12,669-Speed 122.44 samples/sec Loss 11.7239 Epoch: 3 Global Step: 33530 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:53:32,911-Speed 126.47 samples/sec Loss 11.8140 Epoch: 3 Global Step: 33535 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:53:53,143-Speed 126.54 samples/sec Loss 11.3615 Epoch: 3 Global Step: 33540 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:54:13,501-Speed 125.75 samples/sec Loss 11.7641 Epoch: 3 Global Step: 33545 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:54:33,763-Speed 126.36 samples/sec Loss 12.0960 Epoch: 3 Global Step: 33550 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:54:53,952-Speed 126.81 samples/sec Loss 11.5291 Epoch: 3 Global Step: 33555 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:55:14,064-Speed 127.29 samples/sec Loss 11.8161 Epoch: 3 Global Step: 33560 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:55:34,328-Speed 126.35 samples/sec Loss 11.8400 Epoch: 3 Global Step: 33565 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:55:54,426-Speed 127.38 samples/sec Loss 11.8115 Epoch: 3 Global Step: 33570 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:56:15,434-Speed 121.86 samples/sec Loss 11.7052 Epoch: 3 Global Step: 33575 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:56:35,658-Speed 126.59 samples/sec Loss 11.9434 Epoch: 3 Global Step: 33580 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:56:55,952-Speed 126.15 samples/sec Loss 12.1601 Epoch: 3 Global Step: 33585 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:57:16,117-Speed 126.96 samples/sec Loss 11.9026 Epoch: 3 Global Step: 33590 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:57:36,275-Speed 127.01 samples/sec Loss 12.1575 Epoch: 3 Global Step: 33595 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:57:56,542-Speed 126.32 samples/sec Loss 11.9242 Epoch: 3 Global Step: 33600 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:58:16,840-Speed 126.13 samples/sec Loss 11.9829 Epoch: 3 Global Step: 33605 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:58:37,192-Speed 125.79 samples/sec Loss 12.0086 Epoch: 3 Global Step: 33610 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:58:57,336-Speed 127.09 samples/sec Loss 12.0800 Epoch: 3 Global Step: 33615 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:59:18,447-Speed 121.27 samples/sec Loss 12.1522 Epoch: 3 Global Step: 33620 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:59:38,678-Speed 126.55 samples/sec Loss 11.6041 Epoch: 3 Global Step: 33625 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 10:59:58,940-Speed 126.35 samples/sec Loss 11.8295 Epoch: 3 Global Step: 33630 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:00:19,313-Speed 125.67 samples/sec Loss 11.7123 Epoch: 3 Global Step: 33635 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:00:39,643-Speed 125.93 samples/sec Loss 11.6666 Epoch: 3 Global Step: 33640 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:01:00,041-Speed 125.51 samples/sec Loss 11.7971 Epoch: 3 Global Step: 33645 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:01:20,357-Speed 126.02 samples/sec Loss 11.5821 Epoch: 3 Global Step: 33650 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:01:40,559-Speed 126.73 samples/sec Loss 11.5108 Epoch: 3 Global Step: 33655 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:02:00,722-Speed 126.97 samples/sec Loss 11.7616 Epoch: 3 Global Step: 33660 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:02:21,667-Speed 122.23 samples/sec Loss 11.7592 Epoch: 3 Global Step: 33665 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:02:41,869-Speed 126.74 samples/sec Loss 11.6615 Epoch: 3 Global Step: 33670 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:03:02,118-Speed 126.43 samples/sec Loss 11.8808 Epoch: 3 Global Step: 33675 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:03:22,395-Speed 126.26 samples/sec Loss 11.7095 Epoch: 3 Global Step: 33680 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:03:42,640-Speed 126.46 samples/sec Loss 11.5400 Epoch: 3 Global Step: 33685 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:04:02,772-Speed 127.17 samples/sec Loss 11.6710 Epoch: 3 Global Step: 33690 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:04:22,812-Speed 127.75 samples/sec Loss 11.5786 Epoch: 3 Global Step: 33695 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:04:43,064-Speed 126.41 samples/sec Loss 11.9097 Epoch: 3 Global Step: 33700 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:05:03,503-Speed 125.26 samples/sec Loss 12.2810 Epoch: 3 Global Step: 33705 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:05:24,575-Speed 121.49 samples/sec Loss 11.8107 Epoch: 3 Global Step: 33710 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:05:45,419-Speed 122.83 samples/sec Loss 11.6638 Epoch: 3 Global Step: 33715 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:06:05,910-Speed 124.93 samples/sec Loss 11.7859 Epoch: 3 Global Step: 33720 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:06:26,276-Speed 125.70 samples/sec Loss 12.3133 Epoch: 3 Global Step: 33725 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:06:46,491-Speed 126.65 samples/sec Loss 11.8558 Epoch: 3 Global Step: 33730 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:07:06,736-Speed 126.46 samples/sec Loss 11.8434 Epoch: 3 Global Step: 33735 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:07:27,013-Speed 126.26 samples/sec Loss 11.6864 Epoch: 3 Global Step: 33740 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:07:47,219-Speed 126.70 samples/sec Loss 11.9130 Epoch: 3 Global Step: 33745 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:08:07,453-Speed 126.53 samples/sec Loss 11.9002 Epoch: 3 Global Step: 33750 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:08:28,163-Speed 123.62 samples/sec Loss 11.8839 Epoch: 3 Global Step: 33755 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:08:48,450-Speed 126.19 samples/sec Loss 11.8925 Epoch: 3 Global Step: 33760 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:09:08,729-Speed 126.24 samples/sec Loss 11.5608 Epoch: 3 Global Step: 33765 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:09:28,940-Speed 126.67 samples/sec Loss 11.9133 Epoch: 3 Global Step: 33770 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:09:48,980-Speed 127.75 samples/sec Loss 11.7367 Epoch: 3 Global Step: 33775 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:10:09,188-Speed 126.69 samples/sec Loss 11.6184 Epoch: 3 Global Step: 33780 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:10:29,274-Speed 127.46 samples/sec Loss 11.5092 Epoch: 3 Global Step: 33785 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:10:49,487-Speed 126.66 samples/sec Loss 11.7336 Epoch: 3 Global Step: 33790 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:11:09,503-Speed 127.91 samples/sec Loss 11.5650 Epoch: 3 Global Step: 33795 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:11:30,331-Speed 122.91 samples/sec Loss 11.8303 Epoch: 3 Global Step: 33800 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:11:50,683-Speed 125.79 samples/sec Loss 11.6718 Epoch: 3 Global Step: 33805 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:12:10,920-Speed 126.51 samples/sec Loss 11.6983 Epoch: 3 Global Step: 33810 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:12:31,038-Speed 127.26 samples/sec Loss 11.6386 Epoch: 3 Global Step: 33815 Fp16 Grad Scale: 8192 Required: 256 hours Training: 2021-06-08 11:12:51,213-Speed 126.90 samples/sec Loss 11.7883 Epoch: 3 Global Step: 33820 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:13:11,442-Speed 126.56 samples/sec Loss 12.0010 Epoch: 3 Global Step: 33825 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:13:31,706-Speed 126.34 samples/sec Loss 11.9947 Epoch: 3 Global Step: 33830 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:13:51,920-Speed 126.65 samples/sec Loss 11.7812 Epoch: 3 Global Step: 33835 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:14:12,203-Speed 126.23 samples/sec Loss 11.6045 Epoch: 3 Global Step: 33840 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:14:33,144-Speed 122.25 samples/sec Loss 11.6110 Epoch: 3 Global Step: 33845 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:14:53,385-Speed 126.48 samples/sec Loss 11.8377 Epoch: 3 Global Step: 33850 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:15:13,641-Speed 126.39 samples/sec Loss 12.1503 Epoch: 3 Global Step: 33855 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:15:33,750-Speed 127.31 samples/sec Loss 11.5360 Epoch: 3 Global Step: 33860 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:15:53,884-Speed 127.16 samples/sec Loss 11.4465 Epoch: 3 Global Step: 33865 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:16:14,227-Speed 125.85 samples/sec Loss 12.0803 Epoch: 3 Global Step: 33870 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:16:34,538-Speed 126.05 samples/sec Loss 11.7420 Epoch: 3 Global Step: 33875 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:16:54,713-Speed 126.89 samples/sec Loss 12.2152 Epoch: 3 Global Step: 33880 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:17:14,996-Speed 126.22 samples/sec Loss 11.9879 Epoch: 3 Global Step: 33885 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:17:35,862-Speed 122.70 samples/sec Loss 11.5989 Epoch: 3 Global Step: 33890 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:17:56,115-Speed 126.40 samples/sec Loss 11.5943 Epoch: 3 Global Step: 33895 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:18:16,285-Speed 126.93 samples/sec Loss 11.6688 Epoch: 3 Global Step: 33900 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:18:36,603-Speed 126.00 samples/sec Loss 11.8092 Epoch: 3 Global Step: 33905 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:18:56,877-Speed 126.28 samples/sec Loss 11.7385 Epoch: 3 Global Step: 33910 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:19:17,150-Speed 126.29 samples/sec Loss 11.8324 Epoch: 3 Global Step: 33915 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:19:37,475-Speed 125.96 samples/sec Loss 11.6013 Epoch: 3 Global Step: 33920 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:19:57,736-Speed 126.36 samples/sec Loss 11.9903 Epoch: 3 Global Step: 33925 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:20:18,005-Speed 126.31 samples/sec Loss 12.4315 Epoch: 3 Global Step: 33930 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:20:38,816-Speed 123.02 samples/sec Loss 11.4664 Epoch: 3 Global Step: 33935 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:20:59,104-Speed 126.19 samples/sec Loss 11.6247 Epoch: 3 Global Step: 33940 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:21:19,648-Speed 124.62 samples/sec Loss 12.1077 Epoch: 3 Global Step: 33945 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:21:40,591-Speed 122.24 samples/sec Loss 11.8831 Epoch: 3 Global Step: 33950 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:22:00,849-Speed 126.38 samples/sec Loss 11.6050 Epoch: 3 Global Step: 33955 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:22:21,229-Speed 125.63 samples/sec Loss 11.7620 Epoch: 3 Global Step: 33960 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:22:41,488-Speed 126.37 samples/sec Loss 11.8992 Epoch: 3 Global Step: 33965 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:23:01,678-Speed 126.80 samples/sec Loss 11.6020 Epoch: 3 Global Step: 33970 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:23:21,990-Speed 126.04 samples/sec Loss 11.8383 Epoch: 3 Global Step: 33975 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:23:42,999-Speed 121.86 samples/sec Loss 11.7496 Epoch: 3 Global Step: 33980 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:24:03,261-Speed 126.35 samples/sec Loss 11.8273 Epoch: 3 Global Step: 33985 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:24:23,421-Speed 127.00 samples/sec Loss 11.5114 Epoch: 3 Global Step: 33990 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:24:43,659-Speed 126.50 samples/sec Loss 11.7368 Epoch: 3 Global Step: 33995 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:25:03,949-Speed 126.18 samples/sec Loss 12.0289 Epoch: 3 Global Step: 34000 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:25:24,291-Speed 125.86 samples/sec Loss 12.1917 Epoch: 3 Global Step: 34005 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:25:44,676-Speed 125.59 samples/sec Loss 12.0372 Epoch: 3 Global Step: 34010 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:26:04,891-Speed 126.64 samples/sec Loss 11.8592 Epoch: 3 Global Step: 34015 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:26:25,270-Speed 125.63 samples/sec Loss 11.4252 Epoch: 3 Global Step: 34020 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:26:46,314-Speed 121.66 samples/sec Loss 11.8131 Epoch: 3 Global Step: 34025 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:27:06,453-Speed 127.12 samples/sec Loss 11.5084 Epoch: 3 Global Step: 34030 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:27:26,647-Speed 126.77 samples/sec Loss 11.5742 Epoch: 3 Global Step: 34035 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:27:46,886-Speed 126.50 samples/sec Loss 11.9635 Epoch: 3 Global Step: 34040 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:28:07,146-Speed 126.37 samples/sec Loss 12.0049 Epoch: 3 Global Step: 34045 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:28:27,412-Speed 126.32 samples/sec Loss 11.9227 Epoch: 3 Global Step: 34050 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:28:47,685-Speed 126.28 samples/sec Loss 11.5919 Epoch: 3 Global Step: 34055 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:29:07,954-Speed 126.31 samples/sec Loss 11.5807 Epoch: 3 Global Step: 34060 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:29:28,172-Speed 126.63 samples/sec Loss 11.4779 Epoch: 3 Global Step: 34065 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:29:49,039-Speed 122.69 samples/sec Loss 12.0582 Epoch: 3 Global Step: 34070 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:30:09,932-Speed 122.53 samples/sec Loss 11.5447 Epoch: 3 Global Step: 34075 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:30:30,245-Speed 126.04 samples/sec Loss 11.5911 Epoch: 3 Global Step: 34080 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:30:50,501-Speed 126.39 samples/sec Loss 11.9320 Epoch: 3 Global Step: 34085 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:31:10,822-Speed 125.99 samples/sec Loss 11.7450 Epoch: 3 Global Step: 34090 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:31:31,089-Speed 126.32 samples/sec Loss 11.8867 Epoch: 3 Global Step: 34095 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:31:51,904-Speed 122.99 samples/sec Loss 12.0704 Epoch: 3 Global Step: 34100 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:32:12,211-Speed 126.07 samples/sec Loss 12.1277 Epoch: 3 Global Step: 34105 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:32:32,976-Speed 123.29 samples/sec Loss 11.5791 Epoch: 3 Global Step: 34110 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:32:53,915-Speed 122.27 samples/sec Loss 11.9664 Epoch: 3 Global Step: 34115 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:33:14,222-Speed 126.07 samples/sec Loss 11.9743 Epoch: 3 Global Step: 34120 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:33:34,435-Speed 126.66 samples/sec Loss 11.7423 Epoch: 3 Global Step: 34125 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:33:55,148-Speed 123.61 samples/sec Loss 11.8645 Epoch: 3 Global Step: 34130 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:34:15,596-Speed 125.20 samples/sec Loss 11.9479 Epoch: 3 Global Step: 34135 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:34:36,060-Speed 125.11 samples/sec Loss 12.0108 Epoch: 3 Global Step: 34140 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:34:56,257-Speed 126.75 samples/sec Loss 11.6986 Epoch: 3 Global Step: 34145 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:35:16,820-Speed 124.50 samples/sec Loss 11.8032 Epoch: 3 Global Step: 34150 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:35:37,180-Speed 125.75 samples/sec Loss 11.9348 Epoch: 3 Global Step: 34155 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:35:58,215-Speed 121.71 samples/sec Loss 11.7400 Epoch: 3 Global Step: 34160 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:36:18,443-Speed 126.56 samples/sec Loss 11.4885 Epoch: 3 Global Step: 34165 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:36:38,643-Speed 126.74 samples/sec Loss 11.3649 Epoch: 3 Global Step: 34170 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:36:58,912-Speed 126.31 samples/sec Loss 11.5286 Epoch: 3 Global Step: 34175 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:37:19,219-Speed 126.07 samples/sec Loss 11.4528 Epoch: 3 Global Step: 34180 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:37:39,435-Speed 126.64 samples/sec Loss 11.8749 Epoch: 3 Global Step: 34185 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:37:59,689-Speed 126.40 samples/sec Loss 11.2864 Epoch: 3 Global Step: 34190 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:38:19,925-Speed 126.51 samples/sec Loss 11.6480 Epoch: 3 Global Step: 34195 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:38:40,108-Speed 126.85 samples/sec Loss 11.8646 Epoch: 3 Global Step: 34200 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:39:00,848-Speed 123.44 samples/sec Loss 11.9928 Epoch: 3 Global Step: 34205 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:39:21,200-Speed 125.79 samples/sec Loss 11.7512 Epoch: 3 Global Step: 34210 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:39:41,453-Speed 126.42 samples/sec Loss 11.6781 Epoch: 3 Global Step: 34215 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:40:01,447-Speed 128.05 samples/sec Loss 11.6398 Epoch: 3 Global Step: 34220 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:40:21,668-Speed 126.61 samples/sec Loss 11.7717 Epoch: 3 Global Step: 34225 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:40:41,809-Speed 127.11 samples/sec Loss 11.6012 Epoch: 3 Global Step: 34230 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:41:01,977-Speed 126.94 samples/sec Loss 11.6008 Epoch: 3 Global Step: 34235 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:41:22,236-Speed 126.38 samples/sec Loss 11.5944 Epoch: 3 Global Step: 34240 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:41:42,508-Speed 126.29 samples/sec Loss 11.6039 Epoch: 3 Global Step: 34245 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:42:03,611-Speed 121.32 samples/sec Loss 11.7665 Epoch: 3 Global Step: 34250 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:42:23,844-Speed 126.53 samples/sec Loss 11.9183 Epoch: 3 Global Step: 34255 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:42:44,123-Speed 126.25 samples/sec Loss 12.1131 Epoch: 3 Global Step: 34260 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:43:04,241-Speed 127.25 samples/sec Loss 12.0869 Epoch: 3 Global Step: 34265 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:43:24,364-Speed 127.23 samples/sec Loss 11.5793 Epoch: 3 Global Step: 34270 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:43:44,773-Speed 125.44 samples/sec Loss 11.7158 Epoch: 3 Global Step: 34275 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:44:05,348-Speed 124.43 samples/sec Loss 11.9375 Epoch: 3 Global Step: 34280 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:44:25,636-Speed 126.19 samples/sec Loss 11.8176 Epoch: 3 Global Step: 34285 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:44:45,894-Speed 126.38 samples/sec Loss 11.4724 Epoch: 3 Global Step: 34290 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:45:06,691-Speed 123.10 samples/sec Loss 11.7313 Epoch: 3 Global Step: 34295 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:45:26,864-Speed 126.91 samples/sec Loss 11.8073 Epoch: 3 Global Step: 34300 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:45:47,269-Speed 125.46 samples/sec Loss 11.4729 Epoch: 3 Global Step: 34305 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:46:07,526-Speed 126.38 samples/sec Loss 11.7147 Epoch: 3 Global Step: 34310 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:46:27,747-Speed 126.61 samples/sec Loss 11.8623 Epoch: 3 Global Step: 34315 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:46:48,004-Speed 126.38 samples/sec Loss 11.7359 Epoch: 3 Global Step: 34320 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:47:08,245-Speed 126.48 samples/sec Loss 11.6109 Epoch: 3 Global Step: 34325 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:47:28,532-Speed 126.20 samples/sec Loss 11.7967 Epoch: 3 Global Step: 34330 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:47:48,775-Speed 126.47 samples/sec Loss 11.8544 Epoch: 3 Global Step: 34335 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:48:09,804-Speed 121.75 samples/sec Loss 11.4630 Epoch: 3 Global Step: 34340 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:48:30,120-Speed 126.01 samples/sec Loss 11.9186 Epoch: 3 Global Step: 34345 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:48:50,366-Speed 126.45 samples/sec Loss 11.8215 Epoch: 3 Global Step: 34350 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:49:10,538-Speed 126.91 samples/sec Loss 11.7466 Epoch: 3 Global Step: 34355 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:49:30,731-Speed 126.79 samples/sec Loss 11.7905 Epoch: 3 Global Step: 34360 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:49:50,927-Speed 126.76 samples/sec Loss 11.7275 Epoch: 3 Global Step: 34365 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:50:10,862-Speed 128.42 samples/sec Loss 11.6396 Epoch: 3 Global Step: 34370 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:50:31,269-Speed 125.45 samples/sec Loss 11.6100 Epoch: 3 Global Step: 34375 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:50:51,515-Speed 126.45 samples/sec Loss 11.7599 Epoch: 3 Global Step: 34380 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:51:12,198-Speed 123.78 samples/sec Loss 11.9038 Epoch: 3 Global Step: 34385 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:51:32,201-Speed 127.99 samples/sec Loss 11.4915 Epoch: 3 Global Step: 34390 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:51:52,403-Speed 126.72 samples/sec Loss 11.7853 Epoch: 3 Global Step: 34395 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:52:12,693-Speed 126.18 samples/sec Loss 11.8968 Epoch: 3 Global Step: 34400 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:52:32,966-Speed 126.28 samples/sec Loss 11.4694 Epoch: 3 Global Step: 34405 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:52:53,226-Speed 126.37 samples/sec Loss 11.9316 Epoch: 3 Global Step: 34410 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:53:13,361-Speed 127.15 samples/sec Loss 11.4333 Epoch: 3 Global Step: 34415 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:53:33,554-Speed 126.78 samples/sec Loss 11.6221 Epoch: 3 Global Step: 34420 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:53:53,867-Speed 126.04 samples/sec Loss 11.7997 Epoch: 3 Global Step: 34425 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:54:14,750-Speed 122.59 samples/sec Loss 11.8774 Epoch: 3 Global Step: 34430 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:54:34,999-Speed 126.43 samples/sec Loss 11.5549 Epoch: 3 Global Step: 34435 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:54:55,035-Speed 127.78 samples/sec Loss 11.6971 Epoch: 3 Global Step: 34440 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:55:15,248-Speed 126.65 samples/sec Loss 11.4307 Epoch: 3 Global Step: 34445 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:55:35,635-Speed 125.58 samples/sec Loss 11.7351 Epoch: 3 Global Step: 34450 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:55:56,077-Speed 125.24 samples/sec Loss 11.7453 Epoch: 3 Global Step: 34455 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:56:16,515-Speed 125.26 samples/sec Loss 12.1366 Epoch: 3 Global Step: 34460 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:56:36,865-Speed 125.81 samples/sec Loss 11.4820 Epoch: 3 Global Step: 34465 Fp16 Grad Scale: 8192 Required: 255 hours Training: 2021-06-08 11:56:56,979-Speed 127.28 samples/sec Loss 11.8179 Epoch: 3 Global Step: 34470 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 11:57:17,844-Speed 122.70 samples/sec Loss 12.1092 Epoch: 3 Global Step: 34475 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 11:57:37,913-Speed 127.57 samples/sec Loss 11.4930 Epoch: 3 Global Step: 34480 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 11:57:58,221-Speed 126.06 samples/sec Loss 11.5840 Epoch: 3 Global Step: 34485 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 11:58:18,319-Speed 127.39 samples/sec Loss 11.4614 Epoch: 3 Global Step: 34490 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 11:58:38,562-Speed 126.47 samples/sec Loss 11.9370 Epoch: 3 Global Step: 34495 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 11:58:58,806-Speed 126.47 samples/sec Loss 11.8031 Epoch: 3 Global Step: 34500 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 11:59:19,063-Speed 126.38 samples/sec Loss 11.6857 Epoch: 3 Global Step: 34505 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 11:59:39,848-Speed 123.18 samples/sec Loss 11.9974 Epoch: 3 Global Step: 34510 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:00:00,142-Speed 126.15 samples/sec Loss 11.9280 Epoch: 3 Global Step: 34515 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:00:21,132-Speed 121.97 samples/sec Loss 11.6145 Epoch: 3 Global Step: 34520 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:00:41,276-Speed 127.09 samples/sec Loss 11.6116 Epoch: 3 Global Step: 34525 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:01:01,570-Speed 126.15 samples/sec Loss 11.5348 Epoch: 3 Global Step: 34530 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:01:21,677-Speed 127.32 samples/sec Loss 11.6726 Epoch: 3 Global Step: 34535 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:01:42,202-Speed 124.74 samples/sec Loss 11.8603 Epoch: 3 Global Step: 34540 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:02:02,493-Speed 126.18 samples/sec Loss 11.5609 Epoch: 3 Global Step: 34545 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:02:22,767-Speed 126.28 samples/sec Loss 11.6698 Epoch: 3 Global Step: 34550 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:02:42,995-Speed 126.56 samples/sec Loss 11.6257 Epoch: 3 Global Step: 34555 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:03:03,339-Speed 125.85 samples/sec Loss 11.6912 Epoch: 3 Global Step: 34560 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:03:24,282-Speed 122.24 samples/sec Loss 11.5772 Epoch: 3 Global Step: 34565 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:03:44,571-Speed 126.18 samples/sec Loss 12.3442 Epoch: 3 Global Step: 34570 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:04:04,618-Speed 127.70 samples/sec Loss 11.8493 Epoch: 3 Global Step: 34575 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:04:24,726-Speed 127.32 samples/sec Loss 11.6327 Epoch: 3 Global Step: 34580 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:04:44,913-Speed 126.82 samples/sec Loss 11.2890 Epoch: 3 Global Step: 34585 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:05:05,046-Speed 127.17 samples/sec Loss 11.4063 Epoch: 3 Global Step: 34590 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:05:25,154-Speed 127.32 samples/sec Loss 11.6311 Epoch: 3 Global Step: 34595 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:05:45,342-Speed 126.81 samples/sec Loss 11.3897 Epoch: 3 Global Step: 34600 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:06:05,562-Speed 126.62 samples/sec Loss 11.6829 Epoch: 3 Global Step: 34605 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:06:26,306-Speed 123.42 samples/sec Loss 12.0384 Epoch: 3 Global Step: 34610 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:06:46,598-Speed 126.16 samples/sec Loss 11.8579 Epoch: 3 Global Step: 34615 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:07:06,976-Speed 125.64 samples/sec Loss 11.9264 Epoch: 3 Global Step: 34620 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:07:27,407-Speed 125.30 samples/sec Loss 12.4003 Epoch: 3 Global Step: 34625 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:07:47,717-Speed 126.05 samples/sec Loss 11.6563 Epoch: 3 Global Step: 34630 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:08:07,959-Speed 126.48 samples/sec Loss 11.9771 Epoch: 3 Global Step: 34635 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:08:28,319-Speed 125.75 samples/sec Loss 11.7522 Epoch: 3 Global Step: 34640 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:08:48,583-Speed 126.34 samples/sec Loss 11.6885 Epoch: 3 Global Step: 34645 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:09:08,975-Speed 125.54 samples/sec Loss 12.0043 Epoch: 3 Global Step: 34650 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:09:29,793-Speed 122.98 samples/sec Loss 11.5143 Epoch: 3 Global Step: 34655 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:09:50,041-Speed 126.44 samples/sec Loss 11.7313 Epoch: 3 Global Step: 34660 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:10:10,196-Speed 127.02 samples/sec Loss 11.4078 Epoch: 3 Global Step: 34665 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:10:30,413-Speed 126.63 samples/sec Loss 11.7922 Epoch: 3 Global Step: 34670 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:10:50,716-Speed 126.09 samples/sec Loss 11.7184 Epoch: 3 Global Step: 34675 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:11:10,745-Speed 127.83 samples/sec Loss 11.5705 Epoch: 3 Global Step: 34680 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:11:30,942-Speed 126.76 samples/sec Loss 11.7942 Epoch: 3 Global Step: 34685 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:11:50,958-Speed 127.90 samples/sec Loss 11.6629 Epoch: 3 Global Step: 34690 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:12:11,199-Speed 126.48 samples/sec Loss 11.3638 Epoch: 3 Global Step: 34695 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:12:32,039-Speed 122.85 samples/sec Loss 11.6781 Epoch: 3 Global Step: 34700 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:12:52,405-Speed 125.72 samples/sec Loss 11.7145 Epoch: 3 Global Step: 34705 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:13:12,579-Speed 126.90 samples/sec Loss 11.7226 Epoch: 3 Global Step: 34710 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:13:32,760-Speed 126.86 samples/sec Loss 11.9837 Epoch: 3 Global Step: 34715 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:13:53,799-Speed 121.68 samples/sec Loss 11.8497 Epoch: 3 Global Step: 34720 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:14:13,838-Speed 127.76 samples/sec Loss 12.0324 Epoch: 3 Global Step: 34725 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:14:34,033-Speed 126.77 samples/sec Loss 11.7943 Epoch: 3 Global Step: 34730 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:14:54,241-Speed 126.69 samples/sec Loss 11.9510 Epoch: 3 Global Step: 34735 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:15:14,438-Speed 126.76 samples/sec Loss 12.0025 Epoch: 3 Global Step: 34740 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:15:35,586-Speed 121.06 samples/sec Loss 11.7271 Epoch: 3 Global Step: 34745 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:15:56,023-Speed 125.27 samples/sec Loss 12.0437 Epoch: 3 Global Step: 34750 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:16:16,264-Speed 126.48 samples/sec Loss 11.7854 Epoch: 3 Global Step: 34755 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:16:36,481-Speed 126.63 samples/sec Loss 11.8607 Epoch: 3 Global Step: 34760 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:16:57,163-Speed 123.79 samples/sec Loss 11.8274 Epoch: 3 Global Step: 34765 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:17:17,360-Speed 126.76 samples/sec Loss 11.6912 Epoch: 3 Global Step: 34770 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:17:37,596-Speed 126.51 samples/sec Loss 11.7774 Epoch: 3 Global Step: 34775 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:17:57,663-Speed 127.58 samples/sec Loss 11.6245 Epoch: 3 Global Step: 34780 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:18:17,722-Speed 127.63 samples/sec Loss 11.7727 Epoch: 3 Global Step: 34785 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:18:38,725-Speed 121.89 samples/sec Loss 11.7531 Epoch: 3 Global Step: 34790 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:18:59,058-Speed 125.91 samples/sec Loss 11.5034 Epoch: 3 Global Step: 34795 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:19:19,042-Speed 128.11 samples/sec Loss 11.6114 Epoch: 3 Global Step: 34800 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:19:39,228-Speed 126.83 samples/sec Loss 11.4242 Epoch: 3 Global Step: 34805 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:19:59,503-Speed 126.27 samples/sec Loss 11.4879 Epoch: 3 Global Step: 34810 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:20:19,684-Speed 126.86 samples/sec Loss 11.8658 Epoch: 3 Global Step: 34815 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:20:39,976-Speed 126.17 samples/sec Loss 11.9063 Epoch: 3 Global Step: 34820 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:21:00,278-Speed 126.11 samples/sec Loss 11.8920 Epoch: 3 Global Step: 34825 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:21:20,406-Speed 127.19 samples/sec Loss 11.8253 Epoch: 3 Global Step: 34830 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:21:41,177-Speed 123.26 samples/sec Loss 11.9262 Epoch: 3 Global Step: 34835 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:22:01,412-Speed 126.52 samples/sec Loss 11.8232 Epoch: 3 Global Step: 34840 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:22:21,645-Speed 126.53 samples/sec Loss 11.4597 Epoch: 3 Global Step: 34845 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:22:41,889-Speed 126.46 samples/sec Loss 11.7434 Epoch: 3 Global Step: 34850 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:23:02,194-Speed 126.09 samples/sec Loss 11.6364 Epoch: 3 Global Step: 34855 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:23:22,838-Speed 124.01 samples/sec Loss 11.2957 Epoch: 3 Global Step: 34860 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:23:43,119-Speed 126.24 samples/sec Loss 11.5069 Epoch: 3 Global Step: 34865 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:24:03,371-Speed 126.42 samples/sec Loss 11.6258 Epoch: 3 Global Step: 34870 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:24:23,630-Speed 126.37 samples/sec Loss 11.7899 Epoch: 3 Global Step: 34875 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:24:44,556-Speed 122.34 samples/sec Loss 11.9530 Epoch: 3 Global Step: 34880 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:25:04,770-Speed 126.65 samples/sec Loss 11.7126 Epoch: 3 Global Step: 34885 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:25:25,011-Speed 126.48 samples/sec Loss 11.8238 Epoch: 3 Global Step: 34890 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:25:45,239-Speed 126.56 samples/sec Loss 11.7252 Epoch: 3 Global Step: 34895 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:26:05,954-Speed 123.59 samples/sec Loss 11.7169 Epoch: 3 Global Step: 34900 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:26:26,253-Speed 126.13 samples/sec Loss 11.7440 Epoch: 3 Global Step: 34905 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:26:46,303-Speed 127.69 samples/sec Loss 11.7489 Epoch: 3 Global Step: 34910 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:27:07,087-Speed 123.18 samples/sec Loss 12.0920 Epoch: 3 Global Step: 34915 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:27:27,311-Speed 126.59 samples/sec Loss 12.0787 Epoch: 3 Global Step: 34920 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:27:48,355-Speed 121.65 samples/sec Loss 11.5365 Epoch: 3 Global Step: 34925 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:28:08,591-Speed 126.51 samples/sec Loss 11.7050 Epoch: 3 Global Step: 34930 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:28:28,886-Speed 126.14 samples/sec Loss 11.3055 Epoch: 3 Global Step: 34935 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:28:49,113-Speed 126.57 samples/sec Loss 11.6812 Epoch: 3 Global Step: 34940 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:29:09,534-Speed 125.37 samples/sec Loss 11.8863 Epoch: 3 Global Step: 34945 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:29:29,767-Speed 126.53 samples/sec Loss 12.0909 Epoch: 3 Global Step: 34950 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:29:49,969-Speed 126.73 samples/sec Loss 11.4209 Epoch: 3 Global Step: 34955 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:30:10,329-Speed 125.75 samples/sec Loss 11.6967 Epoch: 3 Global Step: 34960 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:30:30,592-Speed 126.34 samples/sec Loss 11.4322 Epoch: 3 Global Step: 34965 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:30:51,549-Speed 122.16 samples/sec Loss 11.4567 Epoch: 3 Global Step: 34970 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:31:11,645-Speed 127.40 samples/sec Loss 11.8273 Epoch: 3 Global Step: 34975 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:31:31,929-Speed 126.21 samples/sec Loss 11.5987 Epoch: 3 Global Step: 34980 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:31:52,163-Speed 126.53 samples/sec Loss 11.4865 Epoch: 3 Global Step: 34985 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:32:12,407-Speed 126.46 samples/sec Loss 11.8168 Epoch: 3 Global Step: 34990 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:32:32,727-Speed 126.00 samples/sec Loss 11.8513 Epoch: 3 Global Step: 34995 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:32:52,972-Speed 126.46 samples/sec Loss 11.7092 Epoch: 3 Global Step: 35000 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:33:13,248-Speed 126.26 samples/sec Loss 11.9574 Epoch: 3 Global Step: 35005 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:33:33,321-Speed 127.54 samples/sec Loss 11.6016 Epoch: 3 Global Step: 35010 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:33:54,231-Speed 122.43 samples/sec Loss 12.1250 Epoch: 3 Global Step: 35015 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:34:14,333-Speed 127.36 samples/sec Loss 11.5966 Epoch: 3 Global Step: 35020 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:34:34,412-Speed 127.50 samples/sec Loss 11.7404 Epoch: 3 Global Step: 35025 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:34:54,484-Speed 127.55 samples/sec Loss 11.8017 Epoch: 3 Global Step: 35030 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:35:14,758-Speed 126.28 samples/sec Loss 11.7214 Epoch: 3 Global Step: 35035 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:35:34,996-Speed 126.51 samples/sec Loss 11.7366 Epoch: 3 Global Step: 35040 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:35:55,230-Speed 126.53 samples/sec Loss 11.8794 Epoch: 3 Global Step: 35045 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:36:15,460-Speed 126.55 samples/sec Loss 11.5841 Epoch: 3 Global Step: 35050 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:36:35,710-Speed 126.43 samples/sec Loss 11.8152 Epoch: 3 Global Step: 35055 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:36:56,621-Speed 122.43 samples/sec Loss 12.0670 Epoch: 3 Global Step: 35060 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:37:17,026-Speed 125.47 samples/sec Loss 11.6867 Epoch: 3 Global Step: 35065 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:37:37,603-Speed 124.42 samples/sec Loss 11.7084 Epoch: 3 Global Step: 35070 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:37:57,987-Speed 125.59 samples/sec Loss 11.6217 Epoch: 3 Global Step: 35075 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:38:18,225-Speed 126.50 samples/sec Loss 12.1102 Epoch: 3 Global Step: 35080 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:38:38,442-Speed 126.63 samples/sec Loss 11.5429 Epoch: 3 Global Step: 35085 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:38:58,768-Speed 125.96 samples/sec Loss 11.5385 Epoch: 3 Global Step: 35090 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:39:19,024-Speed 126.39 samples/sec Loss 11.6222 Epoch: 3 Global Step: 35095 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:39:39,293-Speed 126.31 samples/sec Loss 11.5684 Epoch: 3 Global Step: 35100 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:39:59,974-Speed 123.79 samples/sec Loss 11.9206 Epoch: 3 Global Step: 35105 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:40:20,457-Speed 124.99 samples/sec Loss 11.6785 Epoch: 3 Global Step: 35110 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:40:40,521-Speed 127.59 samples/sec Loss 11.5285 Epoch: 3 Global Step: 35115 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:41:00,707-Speed 126.83 samples/sec Loss 12.0706 Epoch: 3 Global Step: 35120 Fp16 Grad Scale: 8192 Required: 254 hours Training: 2021-06-08 12:41:20,835-Speed 127.19 samples/sec Loss 11.2624 Epoch: 3 Global Step: 35125 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:41:41,054-Speed 126.63 samples/sec Loss 11.9005 Epoch: 3 Global Step: 35130 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:42:01,127-Speed 127.54 samples/sec Loss 11.8459 Epoch: 3 Global Step: 35135 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:42:21,336-Speed 126.68 samples/sec Loss 11.6842 Epoch: 3 Global Step: 35140 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:42:41,481-Speed 127.09 samples/sec Loss 11.6578 Epoch: 3 Global Step: 35145 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:43:02,447-Speed 122.11 samples/sec Loss 11.7807 Epoch: 3 Global Step: 35150 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:43:22,589-Speed 127.11 samples/sec Loss 11.2456 Epoch: 3 Global Step: 35155 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:43:42,931-Speed 125.86 samples/sec Loss 11.7273 Epoch: 3 Global Step: 35160 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:44:03,154-Speed 126.60 samples/sec Loss 11.7357 Epoch: 3 Global Step: 35165 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:44:23,413-Speed 126.37 samples/sec Loss 11.6306 Epoch: 3 Global Step: 35170 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:44:43,694-Speed 126.24 samples/sec Loss 12.1769 Epoch: 3 Global Step: 35175 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:45:04,001-Speed 126.07 samples/sec Loss 12.0184 Epoch: 3 Global Step: 35180 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:45:24,423-Speed 125.36 samples/sec Loss 11.6435 Epoch: 3 Global Step: 35185 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:45:44,650-Speed 126.57 samples/sec Loss 11.7266 Epoch: 3 Global Step: 35190 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:46:05,505-Speed 122.76 samples/sec Loss 11.7788 Epoch: 3 Global Step: 35195 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:46:25,713-Speed 126.69 samples/sec Loss 11.9110 Epoch: 3 Global Step: 35200 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:46:45,814-Speed 127.36 samples/sec Loss 11.6276 Epoch: 3 Global Step: 35205 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:47:06,077-Speed 126.35 samples/sec Loss 11.6576 Epoch: 3 Global Step: 35210 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:47:26,006-Speed 128.46 samples/sec Loss 12.0562 Epoch: 3 Global Step: 35215 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:47:46,176-Speed 126.93 samples/sec Loss 11.9142 Epoch: 3 Global Step: 35220 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:48:06,329-Speed 127.04 samples/sec Loss 11.7638 Epoch: 3 Global Step: 35225 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:48:26,819-Speed 124.94 samples/sec Loss 11.8661 Epoch: 3 Global Step: 35230 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:48:47,064-Speed 126.46 samples/sec Loss 12.2021 Epoch: 3 Global Step: 35235 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:49:07,864-Speed 123.08 samples/sec Loss 11.6072 Epoch: 3 Global Step: 35240 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:49:28,157-Speed 126.16 samples/sec Loss 11.9736 Epoch: 3 Global Step: 35245 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:49:48,361-Speed 126.71 samples/sec Loss 11.8980 Epoch: 3 Global Step: 35250 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:50:08,402-Speed 127.75 samples/sec Loss 11.7072 Epoch: 3 Global Step: 35255 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:50:28,671-Speed 126.31 samples/sec Loss 12.0706 Epoch: 3 Global Step: 35260 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:50:48,728-Speed 127.65 samples/sec Loss 11.8593 Epoch: 3 Global Step: 35265 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:51:08,915-Speed 126.83 samples/sec Loss 11.7356 Epoch: 3 Global Step: 35270 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:51:29,163-Speed 126.44 samples/sec Loss 11.6526 Epoch: 3 Global Step: 35275 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:51:49,426-Speed 126.35 samples/sec Loss 11.7029 Epoch: 3 Global Step: 35280 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:52:10,220-Speed 123.12 samples/sec Loss 11.2647 Epoch: 3 Global Step: 35285 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:52:30,433-Speed 126.66 samples/sec Loss 11.6279 Epoch: 3 Global Step: 35290 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:52:50,565-Speed 127.16 samples/sec Loss 11.8377 Epoch: 3 Global Step: 35295 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:53:10,777-Speed 126.67 samples/sec Loss 11.7051 Epoch: 3 Global Step: 35300 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:53:31,077-Speed 126.11 samples/sec Loss 11.5682 Epoch: 3 Global Step: 35305 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:53:51,212-Speed 127.15 samples/sec Loss 11.8306 Epoch: 3 Global Step: 35310 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:54:11,414-Speed 126.73 samples/sec Loss 11.6815 Epoch: 3 Global Step: 35315 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:54:31,742-Speed 125.94 samples/sec Loss 11.8051 Epoch: 3 Global Step: 35320 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:54:51,809-Speed 127.58 samples/sec Loss 11.5160 Epoch: 3 Global Step: 35325 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:55:12,641-Speed 122.90 samples/sec Loss 11.5472 Epoch: 3 Global Step: 35330 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:55:32,888-Speed 126.44 samples/sec Loss 11.9340 Epoch: 3 Global Step: 35335 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:55:53,125-Speed 126.51 samples/sec Loss 12.1999 Epoch: 3 Global Step: 35340 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:56:13,217-Speed 127.42 samples/sec Loss 12.1356 Epoch: 3 Global Step: 35345 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:56:33,402-Speed 126.83 samples/sec Loss 11.5884 Epoch: 3 Global Step: 35350 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:56:53,734-Speed 125.92 samples/sec Loss 11.6099 Epoch: 3 Global Step: 35355 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:57:13,974-Speed 126.49 samples/sec Loss 11.7193 Epoch: 3 Global Step: 35360 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:57:34,211-Speed 126.51 samples/sec Loss 11.8586 Epoch: 3 Global Step: 35365 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:57:54,403-Speed 126.80 samples/sec Loss 12.0960 Epoch: 3 Global Step: 35370 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:58:15,351-Speed 122.21 samples/sec Loss 11.8374 Epoch: 3 Global Step: 35375 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:58:35,501-Speed 127.05 samples/sec Loss 11.6233 Epoch: 3 Global Step: 35380 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:58:55,734-Speed 126.53 samples/sec Loss 11.8644 Epoch: 3 Global Step: 35385 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:59:15,922-Speed 126.82 samples/sec Loss 11.5073 Epoch: 3 Global Step: 35390 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:59:36,265-Speed 125.85 samples/sec Loss 11.3599 Epoch: 3 Global Step: 35395 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 12:59:56,532-Speed 126.32 samples/sec Loss 11.8067 Epoch: 3 Global Step: 35400 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:00:16,669-Speed 127.14 samples/sec Loss 11.6127 Epoch: 3 Global Step: 35405 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:00:36,841-Speed 126.92 samples/sec Loss 11.7204 Epoch: 3 Global Step: 35410 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:00:57,084-Speed 126.47 samples/sec Loss 11.9073 Epoch: 3 Global Step: 35415 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:01:17,710-Speed 124.12 samples/sec Loss 11.8531 Epoch: 3 Global Step: 35420 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:01:38,143-Speed 125.29 samples/sec Loss 11.5960 Epoch: 3 Global Step: 35425 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:01:58,531-Speed 125.57 samples/sec Loss 11.7616 Epoch: 3 Global Step: 35430 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:02:19,653-Speed 121.20 samples/sec Loss 11.5723 Epoch: 3 Global Step: 35435 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:02:39,810-Speed 127.01 samples/sec Loss 11.9812 Epoch: 3 Global Step: 35440 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:02:59,888-Speed 127.51 samples/sec Loss 11.4707 Epoch: 3 Global Step: 35445 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:03:20,175-Speed 126.19 samples/sec Loss 11.4989 Epoch: 3 Global Step: 35450 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:03:40,300-Speed 127.21 samples/sec Loss 12.0258 Epoch: 3 Global Step: 35455 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:04:00,516-Speed 126.64 samples/sec Loss 11.6023 Epoch: 3 Global Step: 35460 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:04:21,256-Speed 123.44 samples/sec Loss 11.6270 Epoch: 3 Global Step: 35465 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:04:41,552-Speed 126.13 samples/sec Loss 11.9398 Epoch: 3 Global Step: 35470 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:05:01,572-Speed 127.88 samples/sec Loss 11.6578 Epoch: 3 Global Step: 35475 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:05:21,763-Speed 126.79 samples/sec Loss 11.5838 Epoch: 3 Global Step: 35480 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:05:42,095-Speed 125.92 samples/sec Loss 11.5465 Epoch: 3 Global Step: 35485 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:06:02,301-Speed 126.70 samples/sec Loss 11.9022 Epoch: 3 Global Step: 35490 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:06:22,412-Speed 127.30 samples/sec Loss 11.8376 Epoch: 3 Global Step: 35495 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:06:42,707-Speed 126.15 samples/sec Loss 11.5028 Epoch: 3 Global Step: 35500 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:07:02,922-Speed 126.65 samples/sec Loss 11.7347 Epoch: 3 Global Step: 35505 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:07:23,794-Speed 122.66 samples/sec Loss 11.8382 Epoch: 3 Global Step: 35510 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:07:43,987-Speed 126.78 samples/sec Loss 12.1958 Epoch: 3 Global Step: 35515 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:08:04,183-Speed 126.76 samples/sec Loss 11.4392 Epoch: 3 Global Step: 35520 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:08:24,469-Speed 126.20 samples/sec Loss 11.8181 Epoch: 3 Global Step: 35525 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:08:44,671-Speed 126.73 samples/sec Loss 11.6270 Epoch: 3 Global Step: 35530 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:09:04,762-Speed 127.43 samples/sec Loss 12.2449 Epoch: 3 Global Step: 35535 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:09:25,078-Speed 126.02 samples/sec Loss 11.3991 Epoch: 3 Global Step: 35540 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:09:45,422-Speed 125.84 samples/sec Loss 11.8655 Epoch: 3 Global Step: 35545 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:10:05,693-Speed 126.30 samples/sec Loss 11.9022 Epoch: 3 Global Step: 35550 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:10:26,372-Speed 123.80 samples/sec Loss 11.8950 Epoch: 3 Global Step: 35555 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:10:46,622-Speed 126.43 samples/sec Loss 11.8235 Epoch: 3 Global Step: 35560 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:11:06,918-Speed 126.15 samples/sec Loss 12.0352 Epoch: 3 Global Step: 35565 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:11:27,156-Speed 126.50 samples/sec Loss 12.0858 Epoch: 3 Global Step: 35570 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:11:47,412-Speed 126.39 samples/sec Loss 11.8915 Epoch: 3 Global Step: 35575 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:12:07,614-Speed 126.73 samples/sec Loss 11.6848 Epoch: 3 Global Step: 35580 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:12:27,905-Speed 126.17 samples/sec Loss 11.6562 Epoch: 3 Global Step: 35585 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:12:48,140-Speed 126.52 samples/sec Loss 11.9036 Epoch: 3 Global Step: 35590 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:13:08,418-Speed 126.25 samples/sec Loss 11.7229 Epoch: 3 Global Step: 35595 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:13:29,320-Speed 122.48 samples/sec Loss 11.7709 Epoch: 3 Global Step: 35600 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:13:49,605-Speed 126.21 samples/sec Loss 11.4699 Epoch: 3 Global Step: 35605 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:14:09,769-Speed 126.97 samples/sec Loss 11.7347 Epoch: 3 Global Step: 35610 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:14:30,012-Speed 126.48 samples/sec Loss 11.7876 Epoch: 3 Global Step: 35615 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:14:50,332-Speed 125.99 samples/sec Loss 11.7509 Epoch: 3 Global Step: 35620 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:15:10,530-Speed 126.76 samples/sec Loss 11.7332 Epoch: 3 Global Step: 35625 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:15:30,770-Speed 126.49 samples/sec Loss 11.7571 Epoch: 3 Global Step: 35630 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:15:51,001-Speed 126.55 samples/sec Loss 12.0935 Epoch: 3 Global Step: 35635 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:16:11,205-Speed 126.72 samples/sec Loss 11.5220 Epoch: 3 Global Step: 35640 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:16:32,214-Speed 121.86 samples/sec Loss 11.5133 Epoch: 3 Global Step: 35645 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:16:52,460-Speed 126.45 samples/sec Loss 11.7319 Epoch: 3 Global Step: 35650 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:17:12,643-Speed 126.84 samples/sec Loss 11.6501 Epoch: 3 Global Step: 35655 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:17:32,938-Speed 126.15 samples/sec Loss 11.6945 Epoch: 3 Global Step: 35660 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:17:53,208-Speed 126.30 samples/sec Loss 11.8120 Epoch: 3 Global Step: 35665 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:18:13,493-Speed 126.21 samples/sec Loss 11.6452 Epoch: 3 Global Step: 35670 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:18:33,690-Speed 126.76 samples/sec Loss 11.6676 Epoch: 3 Global Step: 35675 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:18:53,924-Speed 126.53 samples/sec Loss 11.6323 Epoch: 3 Global Step: 35680 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:19:14,196-Speed 126.29 samples/sec Loss 11.7499 Epoch: 3 Global Step: 35685 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:19:35,114-Speed 122.39 samples/sec Loss 11.5693 Epoch: 3 Global Step: 35690 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:19:55,228-Speed 127.28 samples/sec Loss 11.5827 Epoch: 3 Global Step: 35695 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:20:15,441-Speed 126.66 samples/sec Loss 11.2433 Epoch: 3 Global Step: 35700 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:20:35,648-Speed 126.70 samples/sec Loss 11.7065 Epoch: 3 Global Step: 35705 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:20:55,754-Speed 127.33 samples/sec Loss 11.6500 Epoch: 3 Global Step: 35710 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:21:16,029-Speed 126.27 samples/sec Loss 12.0101 Epoch: 3 Global Step: 35715 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:21:36,100-Speed 127.55 samples/sec Loss 11.3061 Epoch: 3 Global Step: 35720 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:21:56,464-Speed 125.72 samples/sec Loss 11.6450 Epoch: 3 Global Step: 35725 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:22:16,795-Speed 125.92 samples/sec Loss 11.5823 Epoch: 3 Global Step: 35730 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:22:37,667-Speed 122.66 samples/sec Loss 11.7412 Epoch: 3 Global Step: 35735 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:22:57,720-Speed 127.66 samples/sec Loss 11.3267 Epoch: 3 Global Step: 35740 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:23:18,229-Speed 124.83 samples/sec Loss 11.7816 Epoch: 3 Global Step: 35745 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:23:38,471-Speed 126.47 samples/sec Loss 11.5761 Epoch: 3 Global Step: 35750 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:23:59,363-Speed 122.54 samples/sec Loss 11.6325 Epoch: 3 Global Step: 35755 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:24:19,468-Speed 127.34 samples/sec Loss 11.6432 Epoch: 3 Global Step: 35760 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:24:39,564-Speed 127.39 samples/sec Loss 11.5972 Epoch: 3 Global Step: 35765 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:24:59,722-Speed 127.01 samples/sec Loss 11.4545 Epoch: 3 Global Step: 35770 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:25:19,788-Speed 127.58 samples/sec Loss 12.0422 Epoch: 3 Global Step: 35775 Fp16 Grad Scale: 8192 Required: 253 hours Training: 2021-06-08 13:25:40,722-Speed 122.30 samples/sec Loss 11.5720 Epoch: 3 Global Step: 35780 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:26:00,782-Speed 127.62 samples/sec Loss 11.6939 Epoch: 3 Global Step: 35785 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:26:20,994-Speed 126.66 samples/sec Loss 11.9657 Epoch: 3 Global Step: 35790 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:26:41,032-Speed 127.76 samples/sec Loss 11.4765 Epoch: 3 Global Step: 35795 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:27:01,072-Speed 127.75 samples/sec Loss 11.8294 Epoch: 3 Global Step: 35800 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:27:21,229-Speed 127.01 samples/sec Loss 12.0353 Epoch: 3 Global Step: 35805 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:27:41,157-Speed 128.47 samples/sec Loss 11.6726 Epoch: 3 Global Step: 35810 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:28:01,395-Speed 126.50 samples/sec Loss 12.2479 Epoch: 3 Global Step: 35815 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:28:21,453-Speed 127.64 samples/sec Loss 11.6186 Epoch: 3 Global Step: 35820 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:28:42,364-Speed 122.43 samples/sec Loss 11.5962 Epoch: 3 Global Step: 35825 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:29:02,668-Speed 126.09 samples/sec Loss 11.7835 Epoch: 3 Global Step: 35830 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:29:22,927-Speed 126.37 samples/sec Loss 11.7845 Epoch: 3 Global Step: 35835 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:29:43,169-Speed 126.47 samples/sec Loss 11.8830 Epoch: 3 Global Step: 35840 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:30:03,650-Speed 125.00 samples/sec Loss 11.7013 Epoch: 3 Global Step: 35845 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:30:23,901-Speed 126.42 samples/sec Loss 11.6646 Epoch: 3 Global Step: 35850 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:30:44,115-Speed 126.65 samples/sec Loss 11.6530 Epoch: 3 Global Step: 35855 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:31:04,343-Speed 126.56 samples/sec Loss 11.5758 Epoch: 3 Global Step: 35860 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:31:24,787-Speed 125.23 samples/sec Loss 11.7657 Epoch: 3 Global Step: 35865 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:31:45,765-Speed 122.03 samples/sec Loss 11.8039 Epoch: 3 Global Step: 35870 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:32:06,067-Speed 126.11 samples/sec Loss 11.7561 Epoch: 3 Global Step: 35875 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:32:26,346-Speed 126.24 samples/sec Loss 11.5754 Epoch: 3 Global Step: 35880 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:32:46,570-Speed 126.59 samples/sec Loss 11.6220 Epoch: 3 Global Step: 35885 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:33:06,828-Speed 126.38 samples/sec Loss 11.6594 Epoch: 3 Global Step: 35890 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:33:27,535-Speed 123.64 samples/sec Loss 11.9289 Epoch: 3 Global Step: 35895 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:33:47,732-Speed 126.76 samples/sec Loss 11.8465 Epoch: 3 Global Step: 35900 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:34:07,983-Speed 126.42 samples/sec Loss 11.5191 Epoch: 3 Global Step: 35905 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:34:28,198-Speed 126.65 samples/sec Loss 11.5711 Epoch: 3 Global Step: 35910 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:34:49,176-Speed 122.03 samples/sec Loss 11.7846 Epoch: 3 Global Step: 35915 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:35:09,487-Speed 126.04 samples/sec Loss 11.7430 Epoch: 3 Global Step: 35920 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:35:29,789-Speed 126.10 samples/sec Loss 11.5488 Epoch: 3 Global Step: 35925 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:35:50,030-Speed 126.48 samples/sec Loss 11.4718 Epoch: 3 Global Step: 35930 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:36:10,356-Speed 125.96 samples/sec Loss 11.7496 Epoch: 3 Global Step: 35935 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:36:30,632-Speed 126.27 samples/sec Loss 11.6317 Epoch: 3 Global Step: 35940 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:36:50,861-Speed 126.56 samples/sec Loss 11.5947 Epoch: 3 Global Step: 35945 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:37:11,156-Speed 126.14 samples/sec Loss 11.7240 Epoch: 3 Global Step: 35950 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:37:31,259-Speed 127.36 samples/sec Loss 11.6153 Epoch: 3 Global Step: 35955 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:37:52,181-Speed 122.36 samples/sec Loss 12.0501 Epoch: 3 Global Step: 35960 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:38:12,188-Speed 127.96 samples/sec Loss 11.7566 Epoch: 3 Global Step: 35965 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:38:32,736-Speed 124.59 samples/sec Loss 11.5657 Epoch: 3 Global Step: 35970 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:38:52,992-Speed 126.39 samples/sec Loss 11.8095 Epoch: 3 Global Step: 35975 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:39:13,185-Speed 126.78 samples/sec Loss 11.7225 Epoch: 3 Global Step: 35980 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:39:33,337-Speed 127.05 samples/sec Loss 11.5039 Epoch: 3 Global Step: 35985 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:39:53,530-Speed 126.78 samples/sec Loss 12.3085 Epoch: 3 Global Step: 35990 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:40:13,760-Speed 126.55 samples/sec Loss 11.4475 Epoch: 3 Global Step: 35995 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:40:33,991-Speed 126.55 samples/sec Loss 12.0183 Epoch: 3 Global Step: 36000 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:40:54,828-Speed 122.87 samples/sec Loss 11.3606 Epoch: 3 Global Step: 36005 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:41:15,083-Speed 126.39 samples/sec Loss 11.6148 Epoch: 3 Global Step: 36010 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:41:35,231-Speed 127.07 samples/sec Loss 11.5548 Epoch: 3 Global Step: 36015 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:41:55,473-Speed 126.48 samples/sec Loss 11.6226 Epoch: 3 Global Step: 36020 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:42:15,728-Speed 126.40 samples/sec Loss 11.7781 Epoch: 3 Global Step: 36025 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:42:35,952-Speed 126.59 samples/sec Loss 11.6471 Epoch: 3 Global Step: 36030 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:42:56,187-Speed 126.52 samples/sec Loss 11.9347 Epoch: 3 Global Step: 36035 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:43:16,369-Speed 126.86 samples/sec Loss 11.5417 Epoch: 3 Global Step: 36040 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:43:36,640-Speed 126.30 samples/sec Loss 11.8853 Epoch: 3 Global Step: 36045 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:43:57,611-Speed 122.08 samples/sec Loss 11.8506 Epoch: 3 Global Step: 36050 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:44:17,570-Speed 128.27 samples/sec Loss 11.4127 Epoch: 3 Global Step: 36055 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:44:37,744-Speed 126.90 samples/sec Loss 11.9225 Epoch: 3 Global Step: 36060 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:44:57,947-Speed 126.72 samples/sec Loss 11.5327 Epoch: 3 Global Step: 36065 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:45:18,154-Speed 126.70 samples/sec Loss 11.7238 Epoch: 3 Global Step: 36070 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:45:38,374-Speed 126.62 samples/sec Loss 11.8917 Epoch: 3 Global Step: 36075 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:45:58,794-Speed 125.37 samples/sec Loss 11.8776 Epoch: 3 Global Step: 36080 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:46:19,047-Speed 126.41 samples/sec Loss 11.7274 Epoch: 3 Global Step: 36085 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:46:39,416-Speed 125.69 samples/sec Loss 11.9038 Epoch: 3 Global Step: 36090 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:47:01,130-Speed 117.91 samples/sec Loss 11.6880 Epoch: 3 Global Step: 36095 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:47:21,348-Speed 126.62 samples/sec Loss 11.8640 Epoch: 3 Global Step: 36100 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:47:41,583-Speed 126.52 samples/sec Loss 11.5753 Epoch: 3 Global Step: 36105 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:48:01,796-Speed 126.66 samples/sec Loss 11.4485 Epoch: 3 Global Step: 36110 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:48:22,047-Speed 126.42 samples/sec Loss 11.7641 Epoch: 3 Global Step: 36115 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:48:42,447-Speed 125.50 samples/sec Loss 11.8385 Epoch: 3 Global Step: 36120 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:49:02,692-Speed 126.46 samples/sec Loss 11.8101 Epoch: 3 Global Step: 36125 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:49:22,891-Speed 126.74 samples/sec Loss 11.7282 Epoch: 3 Global Step: 36130 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:49:43,130-Speed 126.50 samples/sec Loss 11.6807 Epoch: 3 Global Step: 36135 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:50:04,275-Speed 121.07 samples/sec Loss 11.3506 Epoch: 3 Global Step: 36140 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:50:24,488-Speed 126.65 samples/sec Loss 11.1461 Epoch: 3 Global Step: 36145 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:50:44,695-Speed 126.69 samples/sec Loss 11.5922 Epoch: 3 Global Step: 36150 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:51:05,136-Speed 125.24 samples/sec Loss 11.8549 Epoch: 3 Global Step: 36155 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:51:25,309-Speed 126.91 samples/sec Loss 11.7632 Epoch: 3 Global Step: 36160 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:51:45,511-Speed 126.72 samples/sec Loss 11.7723 Epoch: 3 Global Step: 36165 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:52:06,078-Speed 124.48 samples/sec Loss 11.9326 Epoch: 3 Global Step: 36170 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:52:26,304-Speed 126.57 samples/sec Loss 11.5074 Epoch: 3 Global Step: 36175 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:52:46,531-Speed 126.57 samples/sec Loss 11.6924 Epoch: 3 Global Step: 36180 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:53:07,456-Speed 122.35 samples/sec Loss 11.7107 Epoch: 3 Global Step: 36185 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:53:27,646-Speed 126.80 samples/sec Loss 11.4870 Epoch: 3 Global Step: 36190 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:53:47,827-Speed 126.86 samples/sec Loss 12.0640 Epoch: 3 Global Step: 36195 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:54:08,117-Speed 126.18 samples/sec Loss 11.9244 Epoch: 3 Global Step: 36200 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:54:28,286-Speed 126.93 samples/sec Loss 11.9618 Epoch: 3 Global Step: 36205 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:54:48,526-Speed 126.49 samples/sec Loss 11.8580 Epoch: 3 Global Step: 36210 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:55:08,600-Speed 127.53 samples/sec Loss 11.7129 Epoch: 3 Global Step: 36215 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:55:28,861-Speed 126.36 samples/sec Loss 11.6294 Epoch: 3 Global Step: 36220 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:55:49,160-Speed 126.12 samples/sec Loss 11.4402 Epoch: 3 Global Step: 36225 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:56:10,129-Speed 122.09 samples/sec Loss 11.7798 Epoch: 3 Global Step: 36230 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:56:30,342-Speed 126.65 samples/sec Loss 11.9181 Epoch: 3 Global Step: 36235 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:56:50,573-Speed 126.54 samples/sec Loss 11.4238 Epoch: 3 Global Step: 36240 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:57:10,725-Speed 127.04 samples/sec Loss 11.7448 Epoch: 3 Global Step: 36245 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:57:31,048-Speed 125.98 samples/sec Loss 11.4234 Epoch: 3 Global Step: 36250 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:57:51,360-Speed 126.04 samples/sec Loss 11.5449 Epoch: 3 Global Step: 36255 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:58:11,570-Speed 126.68 samples/sec Loss 11.6891 Epoch: 3 Global Step: 36260 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:58:31,844-Speed 126.28 samples/sec Loss 11.5583 Epoch: 3 Global Step: 36265 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:58:52,043-Speed 126.74 samples/sec Loss 11.6131 Epoch: 3 Global Step: 36270 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:59:12,742-Speed 123.68 samples/sec Loss 11.6408 Epoch: 3 Global Step: 36275 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:59:32,954-Speed 126.66 samples/sec Loss 11.8213 Epoch: 3 Global Step: 36280 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 13:59:53,196-Speed 126.47 samples/sec Loss 11.2457 Epoch: 3 Global Step: 36285 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:00:13,473-Speed 126.26 samples/sec Loss 11.3871 Epoch: 3 Global Step: 36290 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:00:33,756-Speed 126.22 samples/sec Loss 11.9229 Epoch: 3 Global Step: 36295 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:00:53,964-Speed 126.69 samples/sec Loss 12.0404 Epoch: 3 Global Step: 36300 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:01:14,245-Speed 126.24 samples/sec Loss 11.5194 Epoch: 3 Global Step: 36305 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:01:34,481-Speed 126.52 samples/sec Loss 11.6427 Epoch: 3 Global Step: 36310 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:01:54,607-Speed 127.21 samples/sec Loss 12.0046 Epoch: 3 Global Step: 36315 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:02:15,515-Speed 122.45 samples/sec Loss 11.6229 Epoch: 3 Global Step: 36320 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:02:35,754-Speed 126.50 samples/sec Loss 11.8366 Epoch: 3 Global Step: 36325 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:02:56,002-Speed 126.44 samples/sec Loss 11.6151 Epoch: 3 Global Step: 36330 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:03:16,267-Speed 126.33 samples/sec Loss 11.7332 Epoch: 3 Global Step: 36335 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:03:36,675-Speed 125.46 samples/sec Loss 11.8496 Epoch: 3 Global Step: 36340 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:03:56,948-Speed 126.28 samples/sec Loss 11.4592 Epoch: 3 Global Step: 36345 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:04:17,187-Speed 126.50 samples/sec Loss 11.6229 Epoch: 3 Global Step: 36350 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:04:37,419-Speed 126.54 samples/sec Loss 12.0307 Epoch: 3 Global Step: 36355 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:04:57,578-Speed 127.00 samples/sec Loss 11.8053 Epoch: 3 Global Step: 36360 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:05:18,451-Speed 122.65 samples/sec Loss 11.2647 Epoch: 3 Global Step: 36365 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:05:38,680-Speed 126.56 samples/sec Loss 11.8544 Epoch: 3 Global Step: 36370 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:05:58,923-Speed 126.47 samples/sec Loss 11.6925 Epoch: 3 Global Step: 36375 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:06:19,682-Speed 123.33 samples/sec Loss 11.9760 Epoch: 3 Global Step: 36380 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:06:40,041-Speed 125.75 samples/sec Loss 12.1325 Epoch: 3 Global Step: 36385 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:07:00,227-Speed 126.82 samples/sec Loss 11.7592 Epoch: 3 Global Step: 36390 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:07:20,216-Speed 128.08 samples/sec Loss 11.9484 Epoch: 3 Global Step: 36395 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:07:40,477-Speed 126.36 samples/sec Loss 11.6879 Epoch: 3 Global Step: 36400 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:08:00,541-Speed 127.60 samples/sec Loss 11.3909 Epoch: 3 Global Step: 36405 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:08:21,429-Speed 122.56 samples/sec Loss 11.7620 Epoch: 3 Global Step: 36410 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:08:41,703-Speed 126.28 samples/sec Loss 11.8190 Epoch: 3 Global Step: 36415 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:09:01,960-Speed 126.38 samples/sec Loss 11.7709 Epoch: 3 Global Step: 36420 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:09:22,350-Speed 125.56 samples/sec Loss 11.3510 Epoch: 3 Global Step: 36425 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:09:42,516-Speed 126.95 samples/sec Loss 11.7666 Epoch: 3 Global Step: 36430 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:10:02,644-Speed 127.19 samples/sec Loss 11.8034 Epoch: 3 Global Step: 36435 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:10:22,696-Speed 127.68 samples/sec Loss 11.7097 Epoch: 3 Global Step: 36440 Fp16 Grad Scale: 8192 Required: 252 hours Training: 2021-06-08 14:10:42,994-Speed 126.13 samples/sec Loss 11.6110 Epoch: 3 Global Step: 36445 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:11:03,198-Speed 126.71 samples/sec Loss 11.5067 Epoch: 3 Global Step: 36450 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:11:24,085-Speed 122.57 samples/sec Loss 11.9266 Epoch: 3 Global Step: 36455 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:11:44,351-Speed 126.33 samples/sec Loss 11.7623 Epoch: 3 Global Step: 36460 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:12:04,526-Speed 126.89 samples/sec Loss 11.7435 Epoch: 3 Global Step: 36465 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:12:24,716-Speed 126.80 samples/sec Loss 11.5269 Epoch: 3 Global Step: 36470 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:12:44,954-Speed 126.50 samples/sec Loss 11.4912 Epoch: 3 Global Step: 36475 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:13:05,258-Speed 126.10 samples/sec Loss 12.1785 Epoch: 3 Global Step: 36480 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:13:25,500-Speed 126.48 samples/sec Loss 11.7218 Epoch: 3 Global Step: 36485 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:13:45,758-Speed 126.38 samples/sec Loss 11.8294 Epoch: 3 Global Step: 36490 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:14:06,144-Speed 125.58 samples/sec Loss 11.3830 Epoch: 3 Global Step: 36495 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:14:27,010-Speed 122.69 samples/sec Loss 11.6343 Epoch: 3 Global Step: 36500 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:14:47,279-Speed 126.30 samples/sec Loss 11.7082 Epoch: 3 Global Step: 36505 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:15:07,518-Speed 126.51 samples/sec Loss 11.2996 Epoch: 3 Global Step: 36510 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:15:27,738-Speed 126.62 samples/sec Loss 11.7165 Epoch: 3 Global Step: 36515 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:15:48,024-Speed 126.20 samples/sec Loss 11.4693 Epoch: 3 Global Step: 36520 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:16:08,302-Speed 126.25 samples/sec Loss 11.3298 Epoch: 3 Global Step: 36525 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:16:28,389-Speed 127.46 samples/sec Loss 11.6608 Epoch: 3 Global Step: 36530 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:16:48,666-Speed 126.26 samples/sec Loss 11.7018 Epoch: 3 Global Step: 36535 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:17:08,866-Speed 126.74 samples/sec Loss 11.7775 Epoch: 3 Global Step: 36540 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:17:29,890-Speed 121.77 samples/sec Loss 11.5499 Epoch: 3 Global Step: 36545 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:17:50,087-Speed 126.75 samples/sec Loss 11.9540 Epoch: 3 Global Step: 36550 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:18:10,302-Speed 126.65 samples/sec Loss 11.8218 Epoch: 3 Global Step: 36555 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:18:30,502-Speed 126.74 samples/sec Loss 11.5993 Epoch: 3 Global Step: 36560 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:18:50,471-Speed 128.20 samples/sec Loss 12.1848 Epoch: 3 Global Step: 36565 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:19:10,610-Speed 127.13 samples/sec Loss 11.9046 Epoch: 3 Global Step: 36570 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:19:30,634-Speed 127.85 samples/sec Loss 11.6879 Epoch: 3 Global Step: 36575 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:19:50,917-Speed 126.23 samples/sec Loss 11.8148 Epoch: 3 Global Step: 36580 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:20:11,275-Speed 125.75 samples/sec Loss 11.5709 Epoch: 3 Global Step: 36585 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:20:32,213-Speed 122.27 samples/sec Loss 11.9125 Epoch: 3 Global Step: 36590 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:20:52,514-Speed 126.11 samples/sec Loss 11.6729 Epoch: 3 Global Step: 36595 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:21:12,723-Speed 126.68 samples/sec Loss 11.6518 Epoch: 3 Global Step: 36600 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:21:32,933-Speed 126.68 samples/sec Loss 12.0367 Epoch: 3 Global Step: 36605 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:21:53,142-Speed 126.68 samples/sec Loss 11.8071 Epoch: 3 Global Step: 36610 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:22:13,111-Speed 128.21 samples/sec Loss 11.4079 Epoch: 3 Global Step: 36615 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:22:33,315-Speed 126.72 samples/sec Loss 11.8202 Epoch: 3 Global Step: 36620 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:22:53,578-Speed 126.34 samples/sec Loss 11.7298 Epoch: 3 Global Step: 36625 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:23:13,850-Speed 126.29 samples/sec Loss 12.1053 Epoch: 3 Global Step: 36630 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:23:34,566-Speed 123.58 samples/sec Loss 11.7797 Epoch: 3 Global Step: 36635 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:23:54,805-Speed 126.49 samples/sec Loss 11.4157 Epoch: 3 Global Step: 36640 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:24:14,989-Speed 126.84 samples/sec Loss 11.5820 Epoch: 3 Global Step: 36645 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:24:35,189-Speed 126.75 samples/sec Loss 11.8951 Epoch: 3 Global Step: 36650 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:24:55,373-Speed 126.84 samples/sec Loss 11.5183 Epoch: 3 Global Step: 36655 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:25:15,597-Speed 126.59 samples/sec Loss 11.7166 Epoch: 3 Global Step: 36660 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:25:35,653-Speed 127.65 samples/sec Loss 11.9647 Epoch: 3 Global Step: 36665 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:25:55,810-Speed 127.01 samples/sec Loss 11.7634 Epoch: 3 Global Step: 36670 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:26:16,056-Speed 126.45 samples/sec Loss 11.6421 Epoch: 3 Global Step: 36675 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:26:36,996-Speed 122.26 samples/sec Loss 11.4057 Epoch: 3 Global Step: 36680 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:26:57,217-Speed 126.61 samples/sec Loss 11.7458 Epoch: 3 Global Step: 36685 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:27:18,022-Speed 123.05 samples/sec Loss 11.6353 Epoch: 3 Global Step: 36690 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:27:38,195-Speed 126.91 samples/sec Loss 11.9957 Epoch: 3 Global Step: 36695 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:27:58,249-Speed 127.67 samples/sec Loss 11.8585 Epoch: 3 Global Step: 36700 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:28:18,784-Speed 124.68 samples/sec Loss 11.7152 Epoch: 3 Global Step: 36705 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:28:39,086-Speed 126.10 samples/sec Loss 11.6050 Epoch: 3 Global Step: 36710 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:28:59,342-Speed 126.39 samples/sec Loss 11.4187 Epoch: 3 Global Step: 36715 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:29:19,792-Speed 125.19 samples/sec Loss 11.3666 Epoch: 3 Global Step: 36720 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:29:40,761-Speed 122.09 samples/sec Loss 11.2380 Epoch: 3 Global Step: 36725 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:30:00,828-Speed 127.58 samples/sec Loss 11.6005 Epoch: 3 Global Step: 36730 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:30:21,094-Speed 126.32 samples/sec Loss 11.6126 Epoch: 3 Global Step: 36735 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:30:41,214-Speed 127.24 samples/sec Loss 11.9578 Epoch: 3 Global Step: 36740 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:31:01,515-Speed 126.11 samples/sec Loss 11.9369 Epoch: 3 Global Step: 36745 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:31:21,841-Speed 125.95 samples/sec Loss 11.4043 Epoch: 3 Global Step: 36750 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:31:42,811-Speed 122.09 samples/sec Loss 12.0816 Epoch: 3 Global Step: 36755 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:32:03,238-Speed 125.34 samples/sec Loss 12.1123 Epoch: 3 Global Step: 36760 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:32:23,300-Speed 127.61 samples/sec Loss 12.0104 Epoch: 3 Global Step: 36765 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:32:44,213-Speed 122.42 samples/sec Loss 12.0482 Epoch: 3 Global Step: 36770 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:33:04,450-Speed 126.51 samples/sec Loss 11.7407 Epoch: 3 Global Step: 36775 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:33:24,721-Speed 126.29 samples/sec Loss 11.4282 Epoch: 3 Global Step: 36780 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:33:45,213-Speed 124.93 samples/sec Loss 11.8543 Epoch: 3 Global Step: 36785 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:34:05,457-Speed 126.46 samples/sec Loss 11.4584 Epoch: 3 Global Step: 36790 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:34:25,867-Speed 125.44 samples/sec Loss 11.7948 Epoch: 3 Global Step: 36795 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:34:46,089-Speed 126.60 samples/sec Loss 11.6519 Epoch: 3 Global Step: 36800 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:35:06,507-Speed 125.39 samples/sec Loss 11.6012 Epoch: 3 Global Step: 36805 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:35:26,697-Speed 126.80 samples/sec Loss 11.6276 Epoch: 3 Global Step: 36810 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:35:47,618-Speed 122.37 samples/sec Loss 11.7105 Epoch: 3 Global Step: 36815 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:36:07,822-Speed 126.71 samples/sec Loss 11.9622 Epoch: 3 Global Step: 36820 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:36:28,026-Speed 126.71 samples/sec Loss 11.7654 Epoch: 3 Global Step: 36825 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:36:48,283-Speed 126.39 samples/sec Loss 11.6476 Epoch: 3 Global Step: 36830 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:37:08,496-Speed 126.66 samples/sec Loss 11.3813 Epoch: 3 Global Step: 36835 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:37:28,612-Speed 127.27 samples/sec Loss 11.3907 Epoch: 3 Global Step: 36840 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:37:48,815-Speed 126.72 samples/sec Loss 11.7832 Epoch: 3 Global Step: 36845 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:38:09,026-Speed 126.67 samples/sec Loss 11.8724 Epoch: 3 Global Step: 36850 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:38:29,277-Speed 126.42 samples/sec Loss 11.6922 Epoch: 3 Global Step: 36855 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:38:50,142-Speed 122.70 samples/sec Loss 12.0356 Epoch: 3 Global Step: 36860 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:39:10,383-Speed 126.48 samples/sec Loss 11.7446 Epoch: 3 Global Step: 36865 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:39:30,601-Speed 126.63 samples/sec Loss 11.5109 Epoch: 3 Global Step: 36870 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:39:50,811-Speed 126.68 samples/sec Loss 11.8085 Epoch: 3 Global Step: 36875 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:40:11,003-Speed 126.79 samples/sec Loss 11.7525 Epoch: 3 Global Step: 36880 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:40:31,260-Speed 126.38 samples/sec Loss 11.6477 Epoch: 3 Global Step: 36885 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:40:51,546-Speed 126.20 samples/sec Loss 11.8847 Epoch: 3 Global Step: 36890 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:41:11,728-Speed 126.85 samples/sec Loss 11.6508 Epoch: 3 Global Step: 36895 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:41:31,905-Speed 126.88 samples/sec Loss 11.7545 Epoch: 3 Global Step: 36900 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:41:53,458-Speed 118.78 samples/sec Loss 11.6005 Epoch: 3 Global Step: 36905 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:42:13,787-Speed 125.93 samples/sec Loss 11.6932 Epoch: 3 Global Step: 36910 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:42:33,919-Speed 127.17 samples/sec Loss 11.3856 Epoch: 3 Global Step: 36915 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:42:54,060-Speed 127.16 samples/sec Loss 11.4154 Epoch: 3 Global Step: 36920 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:43:14,262-Speed 126.73 samples/sec Loss 11.8128 Epoch: 3 Global Step: 36925 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:43:34,608-Speed 125.83 samples/sec Loss 11.7377 Epoch: 3 Global Step: 36930 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:43:55,160-Speed 124.57 samples/sec Loss 11.6428 Epoch: 3 Global Step: 36935 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:44:15,393-Speed 126.54 samples/sec Loss 11.7503 Epoch: 3 Global Step: 36940 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:44:35,549-Speed 127.02 samples/sec Loss 11.6199 Epoch: 3 Global Step: 36945 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:44:56,220-Speed 123.85 samples/sec Loss 11.8581 Epoch: 3 Global Step: 36950 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:45:16,408-Speed 126.82 samples/sec Loss 11.7218 Epoch: 3 Global Step: 36955 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:45:36,635-Speed 126.57 samples/sec Loss 11.5959 Epoch: 3 Global Step: 36960 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:45:56,703-Speed 127.57 samples/sec Loss 11.3259 Epoch: 3 Global Step: 36965 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:46:16,913-Speed 126.68 samples/sec Loss 11.3616 Epoch: 3 Global Step: 36970 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:46:37,141-Speed 126.56 samples/sec Loss 11.6156 Epoch: 3 Global Step: 36975 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:46:57,323-Speed 126.86 samples/sec Loss 11.5694 Epoch: 3 Global Step: 36980 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:47:17,560-Speed 126.51 samples/sec Loss 11.6279 Epoch: 3 Global Step: 36985 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:47:37,783-Speed 126.60 samples/sec Loss 11.7517 Epoch: 3 Global Step: 36990 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:47:58,799-Speed 121.82 samples/sec Loss 11.6435 Epoch: 3 Global Step: 36995 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:48:18,971-Speed 126.91 samples/sec Loss 11.2342 Epoch: 3 Global Step: 37000 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:48:39,166-Speed 126.77 samples/sec Loss 11.5968 Epoch: 3 Global Step: 37005 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:48:59,371-Speed 126.71 samples/sec Loss 11.9486 Epoch: 3 Global Step: 37010 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:49:19,932-Speed 124.51 samples/sec Loss 11.6249 Epoch: 3 Global Step: 37015 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:49:40,151-Speed 126.62 samples/sec Loss 11.3938 Epoch: 3 Global Step: 37020 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:50:00,317-Speed 126.95 samples/sec Loss 11.8039 Epoch: 3 Global Step: 37025 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:50:20,416-Speed 127.38 samples/sec Loss 11.8068 Epoch: 3 Global Step: 37030 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:50:40,757-Speed 125.86 samples/sec Loss 11.7497 Epoch: 3 Global Step: 37035 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:51:02,205-Speed 119.36 samples/sec Loss 11.8990 Epoch: 3 Global Step: 37040 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:51:22,466-Speed 126.36 samples/sec Loss 11.7166 Epoch: 3 Global Step: 37045 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:51:42,614-Speed 127.07 samples/sec Loss 11.7490 Epoch: 3 Global Step: 37050 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:52:02,695-Speed 127.49 samples/sec Loss 11.5603 Epoch: 3 Global Step: 37055 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:52:22,805-Speed 127.31 samples/sec Loss 11.6012 Epoch: 3 Global Step: 37060 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:52:43,041-Speed 126.51 samples/sec Loss 11.8463 Epoch: 3 Global Step: 37065 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:53:03,306-Speed 126.33 samples/sec Loss 11.6007 Epoch: 3 Global Step: 37070 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:53:23,450-Speed 127.10 samples/sec Loss 11.6882 Epoch: 3 Global Step: 37075 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:53:43,656-Speed 126.70 samples/sec Loss 11.9472 Epoch: 3 Global Step: 37080 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:54:04,546-Speed 122.55 samples/sec Loss 11.8495 Epoch: 3 Global Step: 37085 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:54:24,542-Speed 128.03 samples/sec Loss 11.8296 Epoch: 3 Global Step: 37090 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:54:44,593-Speed 127.68 samples/sec Loss 11.5622 Epoch: 3 Global Step: 37095 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:55:04,768-Speed 126.90 samples/sec Loss 11.4978 Epoch: 3 Global Step: 37100 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:55:25,017-Speed 126.43 samples/sec Loss 11.6682 Epoch: 3 Global Step: 37105 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:55:45,287-Speed 126.30 samples/sec Loss 11.5399 Epoch: 3 Global Step: 37110 Fp16 Grad Scale: 8192 Required: 251 hours Training: 2021-06-08 14:56:05,314-Speed 127.83 samples/sec Loss 11.8131 Epoch: 3 Global Step: 37115 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:56:25,493-Speed 126.87 samples/sec Loss 11.9293 Epoch: 3 Global Step: 37120 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:56:45,721-Speed 126.57 samples/sec Loss 11.5587 Epoch: 3 Global Step: 37125 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:57:06,557-Speed 122.87 samples/sec Loss 11.6901 Epoch: 3 Global Step: 37130 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:57:26,700-Speed 127.09 samples/sec Loss 11.7862 Epoch: 3 Global Step: 37135 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:57:47,088-Speed 125.57 samples/sec Loss 11.7840 Epoch: 3 Global Step: 37140 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:58:07,441-Speed 125.79 samples/sec Loss 11.9553 Epoch: 3 Global Step: 37145 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:58:27,701-Speed 126.37 samples/sec Loss 11.7268 Epoch: 3 Global Step: 37150 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:58:47,932-Speed 126.54 samples/sec Loss 11.6538 Epoch: 3 Global Step: 37155 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:59:08,207-Speed 126.27 samples/sec Loss 11.4245 Epoch: 3 Global Step: 37160 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:59:28,463-Speed 126.39 samples/sec Loss 11.5415 Epoch: 3 Global Step: 37165 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 14:59:48,867-Speed 125.47 samples/sec Loss 11.6240 Epoch: 3 Global Step: 37170 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:00:09,763-Speed 122.52 samples/sec Loss 11.8329 Epoch: 3 Global Step: 37175 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:00:29,806-Speed 127.73 samples/sec Loss 11.6215 Epoch: 3 Global Step: 37180 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:00:50,049-Speed 126.47 samples/sec Loss 11.7265 Epoch: 3 Global Step: 37185 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:01:10,299-Speed 126.43 samples/sec Loss 11.5622 Epoch: 3 Global Step: 37190 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:01:30,724-Speed 125.34 samples/sec Loss 11.3779 Epoch: 3 Global Step: 37195 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:01:51,043-Speed 126.00 samples/sec Loss 11.5730 Epoch: 3 Global Step: 37200 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:02:11,230-Speed 126.82 samples/sec Loss 11.8142 Epoch: 3 Global Step: 37205 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:02:31,558-Speed 125.95 samples/sec Loss 11.7318 Epoch: 3 Global Step: 37210 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:02:51,769-Speed 126.67 samples/sec Loss 11.6039 Epoch: 3 Global Step: 37215 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:03:12,577-Speed 123.03 samples/sec Loss 11.9826 Epoch: 3 Global Step: 37220 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:03:32,770-Speed 126.78 samples/sec Loss 11.8945 Epoch: 3 Global Step: 37225 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:03:52,834-Speed 127.59 samples/sec Loss 11.7693 Epoch: 3 Global Step: 37230 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:04:13,110-Speed 126.27 samples/sec Loss 11.7712 Epoch: 3 Global Step: 37235 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:04:33,350-Speed 126.49 samples/sec Loss 11.7257 Epoch: 3 Global Step: 37240 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:04:53,551-Speed 126.73 samples/sec Loss 11.5785 Epoch: 3 Global Step: 37245 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:05:13,843-Speed 126.17 samples/sec Loss 11.2515 Epoch: 3 Global Step: 37250 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:05:34,029-Speed 126.83 samples/sec Loss 11.6963 Epoch: 3 Global Step: 37255 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:05:54,232-Speed 126.72 samples/sec Loss 11.6294 Epoch: 3 Global Step: 37260 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:06:14,948-Speed 123.58 samples/sec Loss 11.8927 Epoch: 3 Global Step: 37265 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:06:35,174-Speed 126.57 samples/sec Loss 11.5983 Epoch: 3 Global Step: 37270 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:06:55,364-Speed 126.80 samples/sec Loss 12.0323 Epoch: 3 Global Step: 37275 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:07:15,514-Speed 127.06 samples/sec Loss 11.8010 Epoch: 3 Global Step: 37280 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:07:35,674-Speed 126.99 samples/sec Loss 11.7492 Epoch: 3 Global Step: 37285 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:07:55,945-Speed 126.29 samples/sec Loss 11.6986 Epoch: 3 Global Step: 37290 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:08:16,135-Speed 126.80 samples/sec Loss 11.5788 Epoch: 3 Global Step: 37295 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:08:36,200-Speed 127.60 samples/sec Loss 11.5969 Epoch: 3 Global Step: 37300 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:08:56,321-Speed 127.24 samples/sec Loss 11.6095 Epoch: 3 Global Step: 37305 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:09:17,559-Speed 120.54 samples/sec Loss 11.6429 Epoch: 3 Global Step: 37310 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:09:37,769-Speed 126.68 samples/sec Loss 11.6709 Epoch: 3 Global Step: 37315 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:09:57,924-Speed 127.02 samples/sec Loss 11.4508 Epoch: 3 Global Step: 37320 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:10:18,133-Speed 126.69 samples/sec Loss 11.5443 Epoch: 3 Global Step: 37325 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:10:38,292-Speed 127.00 samples/sec Loss 11.5355 Epoch: 3 Global Step: 37330 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:10:58,464-Speed 126.91 samples/sec Loss 11.7034 Epoch: 3 Global Step: 37335 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:11:18,510-Speed 127.73 samples/sec Loss 11.7355 Epoch: 3 Global Step: 37340 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:11:39,365-Speed 122.76 samples/sec Loss 11.3898 Epoch: 3 Global Step: 37345 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:11:59,731-Speed 125.71 samples/sec Loss 11.4604 Epoch: 3 Global Step: 37350 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:12:20,617-Speed 122.58 samples/sec Loss 11.5434 Epoch: 3 Global Step: 37355 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:12:40,840-Speed 126.59 samples/sec Loss 11.7976 Epoch: 3 Global Step: 37360 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:13:01,035-Speed 126.77 samples/sec Loss 11.5124 Epoch: 3 Global Step: 37365 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:13:21,272-Speed 126.51 samples/sec Loss 11.5446 Epoch: 3 Global Step: 37370 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:13:41,550-Speed 126.25 samples/sec Loss 11.6598 Epoch: 3 Global Step: 37375 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:14:01,790-Speed 126.49 samples/sec Loss 11.7482 Epoch: 3 Global Step: 37380 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:14:21,880-Speed 127.43 samples/sec Loss 11.9585 Epoch: 3 Global Step: 37385 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:14:42,075-Speed 126.77 samples/sec Loss 11.2545 Epoch: 3 Global Step: 37390 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:15:02,291-Speed 126.64 samples/sec Loss 11.4243 Epoch: 3 Global Step: 37395 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:15:23,265-Speed 122.06 samples/sec Loss 11.6275 Epoch: 3 Global Step: 37400 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:15:43,461-Speed 126.76 samples/sec Loss 11.1782 Epoch: 3 Global Step: 37405 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:16:03,701-Speed 126.49 samples/sec Loss 11.5887 Epoch: 3 Global Step: 37410 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:16:23,908-Speed 126.70 samples/sec Loss 11.5811 Epoch: 3 Global Step: 37415 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:16:43,961-Speed 127.67 samples/sec Loss 11.4019 Epoch: 3 Global Step: 37420 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:17:04,167-Speed 126.70 samples/sec Loss 11.3480 Epoch: 3 Global Step: 37425 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:17:24,391-Speed 126.59 samples/sec Loss 11.6993 Epoch: 3 Global Step: 37430 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:17:44,623-Speed 126.54 samples/sec Loss 11.7605 Epoch: 3 Global Step: 37435 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:18:04,843-Speed 126.61 samples/sec Loss 11.6988 Epoch: 3 Global Step: 37440 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:18:25,665-Speed 122.95 samples/sec Loss 11.4359 Epoch: 3 Global Step: 37445 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:18:45,650-Speed 128.10 samples/sec Loss 11.7922 Epoch: 3 Global Step: 37450 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:19:06,080-Speed 125.31 samples/sec Loss 11.6756 Epoch: 3 Global Step: 37455 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:19:26,297-Speed 126.64 samples/sec Loss 11.6138 Epoch: 3 Global Step: 37460 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:19:46,564-Speed 126.32 samples/sec Loss 11.8601 Epoch: 3 Global Step: 37465 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:20:06,787-Speed 126.59 samples/sec Loss 11.6083 Epoch: 3 Global Step: 37470 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:20:26,944-Speed 127.01 samples/sec Loss 11.9771 Epoch: 3 Global Step: 37475 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:20:46,980-Speed 127.78 samples/sec Loss 11.8514 Epoch: 3 Global Step: 37480 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:21:07,422-Speed 125.24 samples/sec Loss 11.5317 Epoch: 3 Global Step: 37485 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:21:28,434-Speed 121.84 samples/sec Loss 11.4652 Epoch: 3 Global Step: 37490 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:21:49,242-Speed 123.03 samples/sec Loss 11.6405 Epoch: 3 Global Step: 37495 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:22:09,445-Speed 126.72 samples/sec Loss 11.6498 Epoch: 3 Global Step: 37500 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:22:29,714-Speed 126.31 samples/sec Loss 11.6675 Epoch: 3 Global Step: 37505 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:22:49,994-Speed 126.24 samples/sec Loss 11.8676 Epoch: 3 Global Step: 37510 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:23:10,277-Speed 126.22 samples/sec Loss 11.5845 Epoch: 3 Global Step: 37515 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:23:31,040-Speed 123.31 samples/sec Loss 11.7315 Epoch: 3 Global Step: 37520 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:23:51,295-Speed 126.40 samples/sec Loss 12.0712 Epoch: 3 Global Step: 37525 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:24:11,705-Speed 125.44 samples/sec Loss 11.8055 Epoch: 3 Global Step: 37530 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:24:32,592-Speed 122.57 samples/sec Loss 11.8486 Epoch: 3 Global Step: 37535 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:24:52,765-Speed 126.90 samples/sec Loss 11.6053 Epoch: 3 Global Step: 37540 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:25:12,963-Speed 126.75 samples/sec Loss 11.9271 Epoch: 3 Global Step: 37545 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:25:33,163-Speed 126.73 samples/sec Loss 11.2609 Epoch: 3 Global Step: 37550 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:25:53,381-Speed 126.63 samples/sec Loss 11.5725 Epoch: 3 Global Step: 37555 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:26:13,627-Speed 126.46 samples/sec Loss 11.2995 Epoch: 3 Global Step: 37560 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:26:33,788-Speed 126.99 samples/sec Loss 11.7264 Epoch: 3 Global Step: 37565 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:26:53,934-Speed 127.08 samples/sec Loss 11.7552 Epoch: 3 Global Step: 37570 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:27:14,005-Speed 127.56 samples/sec Loss 11.7674 Epoch: 3 Global Step: 37575 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:27:34,998-Speed 121.95 samples/sec Loss 11.6726 Epoch: 3 Global Step: 37580 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:27:55,230-Speed 126.54 samples/sec Loss 11.6159 Epoch: 3 Global Step: 37585 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:28:15,452-Speed 126.60 samples/sec Loss 11.7252 Epoch: 3 Global Step: 37590 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:28:35,702-Speed 126.43 samples/sec Loss 11.6935 Epoch: 3 Global Step: 37595 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:28:55,996-Speed 126.16 samples/sec Loss 11.7926 Epoch: 3 Global Step: 37600 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:29:16,198-Speed 126.73 samples/sec Loss 11.5867 Epoch: 3 Global Step: 37605 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:29:36,543-Speed 125.83 samples/sec Loss 11.7943 Epoch: 3 Global Step: 37610 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:29:57,090-Speed 124.60 samples/sec Loss 11.1498 Epoch: 3 Global Step: 37615 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:30:17,384-Speed 126.15 samples/sec Loss 11.4924 Epoch: 3 Global Step: 37620 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:30:38,209-Speed 122.93 samples/sec Loss 11.8077 Epoch: 3 Global Step: 37625 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:30:58,411-Speed 126.72 samples/sec Loss 11.6310 Epoch: 3 Global Step: 37630 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:31:18,585-Speed 126.90 samples/sec Loss 11.3891 Epoch: 3 Global Step: 37635 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:31:38,552-Speed 128.22 samples/sec Loss 11.5054 Epoch: 3 Global Step: 37640 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:31:58,788-Speed 126.52 samples/sec Loss 11.7798 Epoch: 3 Global Step: 37645 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:32:18,992-Speed 126.72 samples/sec Loss 11.2099 Epoch: 3 Global Step: 37650 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:32:39,442-Speed 125.19 samples/sec Loss 11.5887 Epoch: 3 Global Step: 37655 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:32:59,706-Speed 126.34 samples/sec Loss 11.4326 Epoch: 3 Global Step: 37660 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:33:19,982-Speed 126.27 samples/sec Loss 11.8622 Epoch: 3 Global Step: 37665 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:33:40,820-Speed 122.86 samples/sec Loss 11.5171 Epoch: 3 Global Step: 37670 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:34:01,022-Speed 126.73 samples/sec Loss 11.6245 Epoch: 3 Global Step: 37675 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:34:21,336-Speed 126.03 samples/sec Loss 11.9382 Epoch: 3 Global Step: 37680 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:34:41,549-Speed 126.66 samples/sec Loss 11.7012 Epoch: 3 Global Step: 37685 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:35:02,144-Speed 124.31 samples/sec Loss 11.6405 Epoch: 3 Global Step: 37690 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:35:22,469-Speed 125.96 samples/sec Loss 11.4700 Epoch: 3 Global Step: 37695 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:35:42,633-Speed 126.96 samples/sec Loss 11.8391 Epoch: 3 Global Step: 37700 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:36:03,013-Speed 125.62 samples/sec Loss 11.5952 Epoch: 3 Global Step: 37705 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:36:23,054-Speed 127.74 samples/sec Loss 11.7205 Epoch: 3 Global Step: 37710 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:36:44,019-Speed 122.11 samples/sec Loss 11.3724 Epoch: 3 Global Step: 37715 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:37:04,084-Speed 127.60 samples/sec Loss 11.8333 Epoch: 3 Global Step: 37720 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:37:24,309-Speed 126.58 samples/sec Loss 11.9739 Epoch: 3 Global Step: 37725 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:37:44,538-Speed 126.56 samples/sec Loss 11.6790 Epoch: 3 Global Step: 37730 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:38:04,746-Speed 126.69 samples/sec Loss 11.4134 Epoch: 3 Global Step: 37735 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:38:25,031-Speed 126.21 samples/sec Loss 11.7902 Epoch: 3 Global Step: 37740 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:38:45,037-Speed 127.97 samples/sec Loss 11.7520 Epoch: 3 Global Step: 37745 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:39:05,343-Speed 126.08 samples/sec Loss 11.6354 Epoch: 3 Global Step: 37750 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:39:25,550-Speed 126.70 samples/sec Loss 11.8577 Epoch: 3 Global Step: 37755 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:39:46,264-Speed 123.59 samples/sec Loss 11.4683 Epoch: 3 Global Step: 37760 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:40:06,464-Speed 126.74 samples/sec Loss 11.7940 Epoch: 3 Global Step: 37765 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:40:26,679-Speed 126.64 samples/sec Loss 11.6240 Epoch: 3 Global Step: 37770 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:40:46,835-Speed 127.01 samples/sec Loss 11.5552 Epoch: 3 Global Step: 37775 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:41:07,041-Speed 126.71 samples/sec Loss 11.3621 Epoch: 3 Global Step: 37780 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:41:27,290-Speed 126.43 samples/sec Loss 11.6709 Epoch: 3 Global Step: 37785 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:41:47,521-Speed 126.55 samples/sec Loss 11.7434 Epoch: 3 Global Step: 37790 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 15:42:07,772-Speed 126.43 samples/sec Loss 11.6222 Epoch: 3 Global Step: 37795 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:42:27,855-Speed 127.48 samples/sec Loss 11.7771 Epoch: 3 Global Step: 37800 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:42:48,467-Speed 124.20 samples/sec Loss 11.7387 Epoch: 3 Global Step: 37805 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:43:08,506-Speed 127.75 samples/sec Loss 11.6080 Epoch: 3 Global Step: 37810 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:43:28,782-Speed 126.26 samples/sec Loss 11.7537 Epoch: 3 Global Step: 37815 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:43:49,021-Speed 126.50 samples/sec Loss 11.8948 Epoch: 3 Global Step: 37820 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:44:09,122-Speed 127.36 samples/sec Loss 11.9943 Epoch: 3 Global Step: 37825 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:44:29,400-Speed 126.25 samples/sec Loss 11.6568 Epoch: 3 Global Step: 37830 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:44:49,645-Speed 126.46 samples/sec Loss 11.6750 Epoch: 3 Global Step: 37835 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:45:09,882-Speed 126.51 samples/sec Loss 11.7617 Epoch: 3 Global Step: 37840 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:45:30,039-Speed 127.01 samples/sec Loss 11.5806 Epoch: 3 Global Step: 37845 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:45:50,887-Speed 122.80 samples/sec Loss 11.5119 Epoch: 3 Global Step: 37850 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:46:11,169-Speed 126.23 samples/sec Loss 11.6492 Epoch: 3 Global Step: 37855 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:46:31,336-Speed 126.94 samples/sec Loss 11.5518 Epoch: 3 Global Step: 37860 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:46:51,539-Speed 126.72 samples/sec Loss 11.9433 Epoch: 3 Global Step: 37865 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:47:11,603-Speed 127.60 samples/sec Loss 11.5678 Epoch: 3 Global Step: 37870 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:47:31,813-Speed 126.68 samples/sec Loss 11.4093 Epoch: 3 Global Step: 37875 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:47:52,046-Speed 126.53 samples/sec Loss 11.7978 Epoch: 3 Global Step: 37880 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:48:12,303-Speed 126.38 samples/sec Loss 11.6632 Epoch: 3 Global Step: 37885 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:48:32,588-Speed 126.21 samples/sec Loss 11.5101 Epoch: 3 Global Step: 37890 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:48:53,472-Speed 122.59 samples/sec Loss 11.4871 Epoch: 3 Global Step: 37895 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:49:13,526-Speed 127.66 samples/sec Loss 11.4062 Epoch: 3 Global Step: 37900 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:49:33,700-Speed 126.90 samples/sec Loss 11.7482 Epoch: 3 Global Step: 37905 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:49:53,949-Speed 126.43 samples/sec Loss 12.0361 Epoch: 3 Global Step: 37910 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:50:14,156-Speed 126.70 samples/sec Loss 11.3709 Epoch: 3 Global Step: 37915 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:50:34,334-Speed 126.87 samples/sec Loss 11.9202 Epoch: 3 Global Step: 37920 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:50:54,405-Speed 127.56 samples/sec Loss 11.5309 Epoch: 3 Global Step: 37925 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:51:14,576-Speed 126.92 samples/sec Loss 11.8957 Epoch: 3 Global Step: 37930 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:51:34,757-Speed 126.85 samples/sec Loss 11.6890 Epoch: 3 Global Step: 37935 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:51:55,539-Speed 123.19 samples/sec Loss 11.6723 Epoch: 3 Global Step: 37940 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:52:15,729-Speed 126.80 samples/sec Loss 11.8507 Epoch: 3 Global Step: 37945 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:52:35,919-Speed 126.80 samples/sec Loss 11.7239 Epoch: 3 Global Step: 37950 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:52:56,167-Speed 126.44 samples/sec Loss 11.5502 Epoch: 3 Global Step: 37955 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:53:16,366-Speed 126.75 samples/sec Loss 11.8512 Epoch: 3 Global Step: 37960 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:53:36,568-Speed 126.73 samples/sec Loss 11.3396 Epoch: 3 Global Step: 37965 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:53:56,877-Speed 126.06 samples/sec Loss 11.7934 Epoch: 3 Global Step: 37970 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:54:17,119-Speed 126.47 samples/sec Loss 11.3739 Epoch: 3 Global Step: 37975 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:54:37,190-Speed 127.55 samples/sec Loss 11.6821 Epoch: 3 Global Step: 37980 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:54:57,868-Speed 123.81 samples/sec Loss 11.5335 Epoch: 3 Global Step: 37985 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:55:18,063-Speed 126.77 samples/sec Loss 11.7699 Epoch: 3 Global Step: 37990 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:55:38,235-Speed 126.91 samples/sec Loss 11.3805 Epoch: 3 Global Step: 37995 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:55:58,463-Speed 126.57 samples/sec Loss 11.8805 Epoch: 3 Global Step: 38000 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:56:18,834-Speed 125.67 samples/sec Loss 11.8701 Epoch: 3 Global Step: 38005 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:56:39,352-Speed 124.77 samples/sec Loss 11.6958 Epoch: 3 Global Step: 38010 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:56:59,521-Speed 126.93 samples/sec Loss 11.7631 Epoch: 3 Global Step: 38015 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:57:19,768-Speed 126.45 samples/sec Loss 11.4359 Epoch: 3 Global Step: 38020 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:57:39,918-Speed 127.06 samples/sec Loss 11.5942 Epoch: 3 Global Step: 38025 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:58:00,822-Speed 122.47 samples/sec Loss 11.7003 Epoch: 3 Global Step: 38030 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:58:20,913-Speed 127.42 samples/sec Loss 11.4881 Epoch: 3 Global Step: 38035 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:58:41,241-Speed 125.95 samples/sec Loss 11.6675 Epoch: 3 Global Step: 38040 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:59:01,517-Speed 126.26 samples/sec Loss 11.8574 Epoch: 3 Global Step: 38045 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:59:21,609-Speed 127.42 samples/sec Loss 11.5333 Epoch: 3 Global Step: 38050 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 15:59:41,771-Speed 126.98 samples/sec Loss 11.7601 Epoch: 3 Global Step: 38055 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:00:02,097-Speed 125.95 samples/sec Loss 11.6413 Epoch: 3 Global Step: 38060 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:00:22,303-Speed 126.70 samples/sec Loss 11.6851 Epoch: 3 Global Step: 38065 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:00:42,578-Speed 126.27 samples/sec Loss 11.6160 Epoch: 3 Global Step: 38070 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:01:03,273-Speed 123.71 samples/sec Loss 11.6662 Epoch: 3 Global Step: 38075 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:01:23,445-Speed 126.92 samples/sec Loss 11.9534 Epoch: 3 Global Step: 38080 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:01:43,607-Speed 126.97 samples/sec Loss 11.5343 Epoch: 3 Global Step: 38085 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:02:03,669-Speed 127.61 samples/sec Loss 11.4509 Epoch: 3 Global Step: 38090 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:02:23,799-Speed 127.18 samples/sec Loss 11.6005 Epoch: 3 Global Step: 38095 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:02:43,755-Speed 128.29 samples/sec Loss 11.7011 Epoch: 3 Global Step: 38100 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:03:03,952-Speed 126.76 samples/sec Loss 11.7240 Epoch: 3 Global Step: 38105 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:03:23,993-Speed 127.74 samples/sec Loss 11.4936 Epoch: 3 Global Step: 38110 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:03:44,074-Speed 127.49 samples/sec Loss 11.3496 Epoch: 3 Global Step: 38115 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:04:04,731-Speed 123.94 samples/sec Loss 11.3299 Epoch: 3 Global Step: 38120 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:04:24,936-Speed 126.71 samples/sec Loss 11.3185 Epoch: 3 Global Step: 38125 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:04:45,106-Speed 126.92 samples/sec Loss 11.6570 Epoch: 3 Global Step: 38130 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:05:05,280-Speed 126.90 samples/sec Loss 11.4583 Epoch: 3 Global Step: 38135 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:05:25,350-Speed 127.56 samples/sec Loss 11.4756 Epoch: 3 Global Step: 38140 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:05:45,420-Speed 127.56 samples/sec Loss 11.6178 Epoch: 3 Global Step: 38145 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:06:05,557-Speed 127.14 samples/sec Loss 11.5162 Epoch: 3 Global Step: 38150 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:06:25,569-Speed 127.93 samples/sec Loss 11.5438 Epoch: 3 Global Step: 38155 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:06:45,724-Speed 127.02 samples/sec Loss 11.3494 Epoch: 3 Global Step: 38160 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:07:06,349-Speed 124.12 samples/sec Loss 11.5623 Epoch: 3 Global Step: 38165 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:07:26,362-Speed 127.92 samples/sec Loss 11.9725 Epoch: 3 Global Step: 38170 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:07:46,936-Speed 124.43 samples/sec Loss 11.4155 Epoch: 3 Global Step: 38175 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:08:07,123-Speed 126.83 samples/sec Loss 11.8860 Epoch: 3 Global Step: 38180 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:08:27,105-Speed 128.12 samples/sec Loss 11.8187 Epoch: 3 Global Step: 38185 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:08:47,299-Speed 126.78 samples/sec Loss 11.4622 Epoch: 3 Global Step: 38190 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:09:07,392-Speed 127.41 samples/sec Loss 11.4859 Epoch: 3 Global Step: 38195 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:09:27,758-Speed 125.71 samples/sec Loss 11.8886 Epoch: 3 Global Step: 38200 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:09:47,864-Speed 127.33 samples/sec Loss 11.7795 Epoch: 3 Global Step: 38205 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:10:08,499-Speed 124.06 samples/sec Loss 11.6268 Epoch: 3 Global Step: 38210 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:10:28,673-Speed 126.90 samples/sec Loss 11.4434 Epoch: 3 Global Step: 38215 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:10:48,917-Speed 126.46 samples/sec Loss 11.7897 Epoch: 3 Global Step: 38220 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:11:09,103-Speed 126.83 samples/sec Loss 11.3143 Epoch: 3 Global Step: 38225 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:11:29,394-Speed 126.17 samples/sec Loss 11.7221 Epoch: 3 Global Step: 38230 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:11:49,642-Speed 126.44 samples/sec Loss 11.7634 Epoch: 3 Global Step: 38235 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:12:09,901-Speed 126.37 samples/sec Loss 11.6178 Epoch: 3 Global Step: 38240 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:12:30,380-Speed 125.02 samples/sec Loss 11.5305 Epoch: 3 Global Step: 38245 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:12:50,610-Speed 126.56 samples/sec Loss 11.2320 Epoch: 3 Global Step: 38250 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:13:11,488-Speed 122.62 samples/sec Loss 11.8111 Epoch: 3 Global Step: 38255 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:13:31,693-Speed 126.71 samples/sec Loss 11.3847 Epoch: 3 Global Step: 38260 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:13:51,897-Speed 126.71 samples/sec Loss 11.4964 Epoch: 3 Global Step: 38265 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:14:11,919-Speed 127.87 samples/sec Loss 11.4480 Epoch: 3 Global Step: 38270 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:14:32,062-Speed 127.10 samples/sec Loss 11.7344 Epoch: 3 Global Step: 38275 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:14:52,093-Speed 127.81 samples/sec Loss 11.6264 Epoch: 3 Global Step: 38280 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:15:12,336-Speed 126.47 samples/sec Loss 11.4433 Epoch: 3 Global Step: 38285 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:15:32,344-Speed 127.95 samples/sec Loss 11.5905 Epoch: 3 Global Step: 38290 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:15:53,085-Speed 123.43 samples/sec Loss 11.2091 Epoch: 3 Global Step: 38295 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:16:13,907-Speed 122.96 samples/sec Loss 11.3649 Epoch: 3 Global Step: 38300 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:16:34,206-Speed 126.12 samples/sec Loss 11.3821 Epoch: 3 Global Step: 38305 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:16:54,592-Speed 125.58 samples/sec Loss 11.7596 Epoch: 3 Global Step: 38310 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:17:14,799-Speed 126.70 samples/sec Loss 11.6231 Epoch: 3 Global Step: 38315 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:17:35,050-Speed 126.43 samples/sec Loss 11.1757 Epoch: 3 Global Step: 38320 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:17:55,272-Speed 126.60 samples/sec Loss 11.4197 Epoch: 3 Global Step: 38325 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:18:15,435-Speed 126.97 samples/sec Loss 11.7041 Epoch: 3 Global Step: 38330 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:18:35,597-Speed 126.98 samples/sec Loss 11.4814 Epoch: 3 Global Step: 38335 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:18:55,774-Speed 126.89 samples/sec Loss 11.2605 Epoch: 3 Global Step: 38340 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:19:16,635-Speed 122.72 samples/sec Loss 11.7790 Epoch: 3 Global Step: 38345 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:19:36,765-Speed 127.18 samples/sec Loss 12.0990 Epoch: 3 Global Step: 38350 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:19:56,991-Speed 126.58 samples/sec Loss 11.7794 Epoch: 3 Global Step: 38355 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:20:17,089-Speed 127.38 samples/sec Loss 11.7523 Epoch: 3 Global Step: 38360 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:20:37,169-Speed 127.50 samples/sec Loss 11.3007 Epoch: 3 Global Step: 38365 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:20:57,375-Speed 126.70 samples/sec Loss 11.6855 Epoch: 3 Global Step: 38370 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:21:17,633-Speed 126.38 samples/sec Loss 11.6528 Epoch: 3 Global Step: 38375 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:21:37,942-Speed 126.06 samples/sec Loss 11.4997 Epoch: 3 Global Step: 38380 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:21:58,209-Speed 126.32 samples/sec Loss 11.8791 Epoch: 3 Global Step: 38385 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:22:19,220-Speed 121.85 samples/sec Loss 11.5462 Epoch: 3 Global Step: 38390 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:22:39,381-Speed 126.99 samples/sec Loss 11.8411 Epoch: 3 Global Step: 38395 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:22:59,598-Speed 126.63 samples/sec Loss 11.7737 Epoch: 3 Global Step: 38400 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:23:19,967-Speed 125.69 samples/sec Loss 11.4575 Epoch: 3 Global Step: 38405 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:23:40,190-Speed 126.60 samples/sec Loss 11.5382 Epoch: 3 Global Step: 38410 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:24:00,368-Speed 126.88 samples/sec Loss 11.5766 Epoch: 3 Global Step: 38415 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:24:20,379-Speed 127.94 samples/sec Loss 11.2902 Epoch: 3 Global Step: 38420 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:24:40,561-Speed 126.86 samples/sec Loss 11.7181 Epoch: 3 Global Step: 38425 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:25:00,741-Speed 126.87 samples/sec Loss 11.6393 Epoch: 3 Global Step: 38430 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:25:21,468-Speed 123.52 samples/sec Loss 11.1788 Epoch: 3 Global Step: 38435 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:25:41,744-Speed 126.26 samples/sec Loss 11.6578 Epoch: 3 Global Step: 38440 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:26:02,094-Speed 125.80 samples/sec Loss 11.6357 Epoch: 3 Global Step: 38445 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:26:22,301-Speed 126.70 samples/sec Loss 11.3323 Epoch: 3 Global Step: 38450 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:26:42,512-Speed 126.67 samples/sec Loss 11.8803 Epoch: 3 Global Step: 38455 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:27:02,764-Speed 126.42 samples/sec Loss 11.7746 Epoch: 3 Global Step: 38460 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 16:27:22,906-Speed 127.11 samples/sec Loss 11.2426 Epoch: 3 Global Step: 38465 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:27:43,103-Speed 126.76 samples/sec Loss 11.3126 Epoch: 3 Global Step: 38470 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:28:03,290-Speed 126.83 samples/sec Loss 11.8654 Epoch: 3 Global Step: 38475 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:28:24,312-Speed 121.78 samples/sec Loss 11.6954 Epoch: 3 Global Step: 38480 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:28:44,331-Speed 127.88 samples/sec Loss 11.2662 Epoch: 3 Global Step: 38485 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:29:04,432-Speed 127.36 samples/sec Loss 11.6924 Epoch: 3 Global Step: 38490 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:29:24,558-Speed 127.21 samples/sec Loss 11.7119 Epoch: 3 Global Step: 38495 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:29:44,627-Speed 127.56 samples/sec Loss 11.2402 Epoch: 3 Global Step: 38500 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:30:04,813-Speed 126.82 samples/sec Loss 11.6740 Epoch: 3 Global Step: 38505 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:30:24,839-Speed 127.85 samples/sec Loss 11.7394 Epoch: 3 Global Step: 38510 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:30:45,112-Speed 126.28 samples/sec Loss 11.7115 Epoch: 3 Global Step: 38515 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:31:05,402-Speed 126.18 samples/sec Loss 11.2982 Epoch: 3 Global Step: 38520 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:31:26,310-Speed 122.45 samples/sec Loss 11.7630 Epoch: 3 Global Step: 38525 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:31:46,524-Speed 126.65 samples/sec Loss 11.3752 Epoch: 3 Global Step: 38530 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:32:06,736-Speed 126.66 samples/sec Loss 11.6458 Epoch: 3 Global Step: 38535 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:32:26,947-Speed 126.67 samples/sec Loss 11.5369 Epoch: 3 Global Step: 38540 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:32:47,037-Speed 127.43 samples/sec Loss 11.7345 Epoch: 3 Global Step: 38545 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:33:07,238-Speed 126.74 samples/sec Loss 11.6875 Epoch: 3 Global Step: 38550 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:33:27,445-Speed 126.70 samples/sec Loss 11.7647 Epoch: 3 Global Step: 38555 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:33:47,617-Speed 126.92 samples/sec Loss 11.4502 Epoch: 3 Global Step: 38560 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:34:07,698-Speed 127.49 samples/sec Loss 11.6891 Epoch: 3 Global Step: 38565 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:34:28,660-Speed 122.13 samples/sec Loss 11.7853 Epoch: 3 Global Step: 38570 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:34:48,928-Speed 126.31 samples/sec Loss 11.8719 Epoch: 3 Global Step: 38575 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:35:09,175-Speed 126.44 samples/sec Loss 11.6537 Epoch: 3 Global Step: 38580 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:35:29,376-Speed 126.73 samples/sec Loss 11.4685 Epoch: 3 Global Step: 38585 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:35:49,600-Speed 126.59 samples/sec Loss 11.4722 Epoch: 3 Global Step: 38590 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:36:09,856-Speed 126.39 samples/sec Loss 11.4735 Epoch: 3 Global Step: 38595 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:36:30,065-Speed 126.69 samples/sec Loss 11.5646 Epoch: 3 Global Step: 38600 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:36:50,299-Speed 126.53 samples/sec Loss 11.5303 Epoch: 3 Global Step: 38605 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:37:10,478-Speed 126.87 samples/sec Loss 11.8344 Epoch: 3 Global Step: 38610 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:37:31,391-Speed 122.42 samples/sec Loss 11.6628 Epoch: 3 Global Step: 38615 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:37:51,594-Speed 126.72 samples/sec Loss 11.6984 Epoch: 3 Global Step: 38620 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:38:11,829-Speed 126.52 samples/sec Loss 12.0953 Epoch: 3 Global Step: 38625 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:38:32,045-Speed 126.64 samples/sec Loss 11.4624 Epoch: 3 Global Step: 38630 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:38:52,272-Speed 126.57 samples/sec Loss 11.8381 Epoch: 3 Global Step: 38635 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:39:12,567-Speed 126.15 samples/sec Loss 12.0116 Epoch: 3 Global Step: 38640 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:39:32,789-Speed 126.60 samples/sec Loss 11.6891 Epoch: 3 Global Step: 38645 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:39:53,067-Speed 126.25 samples/sec Loss 11.4059 Epoch: 3 Global Step: 38650 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:40:13,448-Speed 125.62 samples/sec Loss 11.8373 Epoch: 3 Global Step: 38655 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:40:34,373-Speed 122.35 samples/sec Loss 11.5091 Epoch: 3 Global Step: 38660 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:40:54,568-Speed 126.77 samples/sec Loss 11.5997 Epoch: 3 Global Step: 38665 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:41:14,690-Speed 127.23 samples/sec Loss 11.5534 Epoch: 3 Global Step: 38670 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:41:34,880-Speed 126.80 samples/sec Loss 11.7089 Epoch: 3 Global Step: 38675 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:41:55,347-Speed 125.09 samples/sec Loss 11.4207 Epoch: 3 Global Step: 38680 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:42:15,555-Speed 126.69 samples/sec Loss 11.7953 Epoch: 3 Global Step: 38685 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:42:35,658-Speed 127.35 samples/sec Loss 11.8182 Epoch: 3 Global Step: 38690 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:42:55,881-Speed 126.59 samples/sec Loss 11.6870 Epoch: 3 Global Step: 38695 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:43:16,090-Speed 126.69 samples/sec Loss 11.5735 Epoch: 3 Global Step: 38700 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:43:37,096-Speed 121.87 samples/sec Loss 11.7769 Epoch: 3 Global Step: 38705 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:43:57,409-Speed 126.03 samples/sec Loss 11.4514 Epoch: 3 Global Step: 38710 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:44:17,736-Speed 125.95 samples/sec Loss 11.8199 Epoch: 3 Global Step: 38715 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:44:37,957-Speed 126.61 samples/sec Loss 11.4882 Epoch: 3 Global Step: 38720 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:44:58,205-Speed 126.44 samples/sec Loss 11.6027 Epoch: 3 Global Step: 38725 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:45:18,311-Speed 127.33 samples/sec Loss 11.6642 Epoch: 3 Global Step: 38730 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:45:38,375-Speed 127.60 samples/sec Loss 11.6359 Epoch: 3 Global Step: 38735 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:45:58,550-Speed 126.90 samples/sec Loss 12.0207 Epoch: 3 Global Step: 38740 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:46:18,567-Speed 127.90 samples/sec Loss 11.7373 Epoch: 3 Global Step: 38745 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:46:39,335-Speed 123.27 samples/sec Loss 11.8828 Epoch: 3 Global Step: 38750 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:46:59,517-Speed 126.85 samples/sec Loss 11.1745 Epoch: 3 Global Step: 38755 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:47:19,502-Speed 128.10 samples/sec Loss 11.3479 Epoch: 3 Global Step: 38760 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:47:39,513-Speed 127.94 samples/sec Loss 11.3524 Epoch: 3 Global Step: 38765 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:47:59,656-Speed 127.10 samples/sec Loss 11.5736 Epoch: 3 Global Step: 38770 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:48:19,679-Speed 127.85 samples/sec Loss 11.9630 Epoch: 3 Global Step: 38775 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:48:39,847-Speed 126.95 samples/sec Loss 11.4698 Epoch: 3 Global Step: 38780 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:48:59,892-Speed 127.72 samples/sec Loss 11.6608 Epoch: 3 Global Step: 38785 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:49:20,138-Speed 126.45 samples/sec Loss 11.4639 Epoch: 3 Global Step: 38790 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:49:41,087-Speed 122.21 samples/sec Loss 11.4927 Epoch: 3 Global Step: 38795 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:50:01,325-Speed 126.50 samples/sec Loss 11.5752 Epoch: 3 Global Step: 38800 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:50:21,627-Speed 126.10 samples/sec Loss 11.4713 Epoch: 3 Global Step: 38805 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:50:41,758-Speed 127.17 samples/sec Loss 11.5901 Epoch: 3 Global Step: 38810 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:51:02,072-Speed 126.03 samples/sec Loss 11.6885 Epoch: 3 Global Step: 38815 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:51:22,323-Speed 126.43 samples/sec Loss 11.2824 Epoch: 3 Global Step: 38820 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:51:42,487-Speed 126.96 samples/sec Loss 11.7069 Epoch: 3 Global Step: 38825 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:52:02,678-Speed 126.80 samples/sec Loss 11.6521 Epoch: 3 Global Step: 38830 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:52:22,988-Speed 126.05 samples/sec Loss 11.6150 Epoch: 3 Global Step: 38835 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:52:43,921-Speed 122.30 samples/sec Loss 11.7662 Epoch: 3 Global Step: 38840 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:53:04,143-Speed 126.60 samples/sec Loss 11.6891 Epoch: 3 Global Step: 38845 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:53:24,405-Speed 126.35 samples/sec Loss 11.6371 Epoch: 3 Global Step: 38850 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:53:44,644-Speed 126.49 samples/sec Loss 12.0697 Epoch: 3 Global Step: 38855 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:54:04,994-Speed 125.81 samples/sec Loss 11.6868 Epoch: 3 Global Step: 38860 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:54:25,319-Speed 125.96 samples/sec Loss 11.6593 Epoch: 3 Global Step: 38865 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:54:45,518-Speed 126.74 samples/sec Loss 11.4303 Epoch: 3 Global Step: 38870 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:55:05,669-Speed 127.05 samples/sec Loss 11.6407 Epoch: 3 Global Step: 38875 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:55:25,881-Speed 126.66 samples/sec Loss 11.8390 Epoch: 3 Global Step: 38880 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:55:46,606-Speed 123.53 samples/sec Loss 11.5565 Epoch: 3 Global Step: 38885 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:56:06,723-Speed 127.26 samples/sec Loss 11.1996 Epoch: 3 Global Step: 38890 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:56:26,893-Speed 126.93 samples/sec Loss 11.3679 Epoch: 3 Global Step: 38895 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:56:47,207-Speed 126.03 samples/sec Loss 11.8475 Epoch: 3 Global Step: 38900 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:57:07,400-Speed 126.79 samples/sec Loss 11.6758 Epoch: 3 Global Step: 38905 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:57:27,572-Speed 126.92 samples/sec Loss 11.8632 Epoch: 3 Global Step: 38910 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:57:47,768-Speed 126.76 samples/sec Loss 11.6569 Epoch: 3 Global Step: 38915 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:58:07,862-Speed 127.41 samples/sec Loss 11.6681 Epoch: 3 Global Step: 38920 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:58:28,002-Speed 127.11 samples/sec Loss 11.6394 Epoch: 3 Global Step: 38925 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:58:48,887-Speed 122.59 samples/sec Loss 11.7881 Epoch: 3 Global Step: 38930 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:59:09,118-Speed 126.54 samples/sec Loss 11.2183 Epoch: 3 Global Step: 38935 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:59:29,289-Speed 126.92 samples/sec Loss 11.8237 Epoch: 3 Global Step: 38940 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 16:59:49,463-Speed 126.90 samples/sec Loss 11.5832 Epoch: 3 Global Step: 38945 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:00:09,920-Speed 125.15 samples/sec Loss 11.5427 Epoch: 3 Global Step: 38950 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:00:30,120-Speed 126.74 samples/sec Loss 11.8879 Epoch: 3 Global Step: 38955 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:00:50,295-Speed 126.90 samples/sec Loss 11.5643 Epoch: 3 Global Step: 38960 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:01:10,320-Speed 127.85 samples/sec Loss 11.7101 Epoch: 3 Global Step: 38965 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:01:30,431-Speed 127.30 samples/sec Loss 11.5336 Epoch: 3 Global Step: 38970 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:01:51,192-Speed 123.32 samples/sec Loss 11.6436 Epoch: 3 Global Step: 38975 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:02:11,386-Speed 126.78 samples/sec Loss 11.4118 Epoch: 3 Global Step: 38980 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:02:31,445-Speed 127.63 samples/sec Loss 11.5405 Epoch: 3 Global Step: 38985 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:02:51,621-Speed 126.89 samples/sec Loss 11.6392 Epoch: 3 Global Step: 38990 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:03:11,698-Speed 127.51 samples/sec Loss 11.5124 Epoch: 3 Global Step: 38995 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:03:31,967-Speed 126.31 samples/sec Loss 11.5768 Epoch: 3 Global Step: 39000 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:03:52,158-Speed 126.79 samples/sec Loss 11.9934 Epoch: 3 Global Step: 39005 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:04:12,252-Speed 127.41 samples/sec Loss 11.5427 Epoch: 3 Global Step: 39010 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:04:32,437-Speed 126.83 samples/sec Loss 12.0658 Epoch: 3 Global Step: 39015 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:04:53,286-Speed 122.80 samples/sec Loss 11.6601 Epoch: 3 Global Step: 39020 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:05:13,445-Speed 126.99 samples/sec Loss 11.7865 Epoch: 3 Global Step: 39025 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:05:34,233-Speed 123.15 samples/sec Loss 11.4841 Epoch: 3 Global Step: 39030 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:05:54,429-Speed 126.77 samples/sec Loss 11.7816 Epoch: 3 Global Step: 39035 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:06:14,790-Speed 125.74 samples/sec Loss 11.8364 Epoch: 3 Global Step: 39040 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:06:34,996-Speed 126.71 samples/sec Loss 11.6266 Epoch: 3 Global Step: 39045 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:06:55,025-Speed 127.82 samples/sec Loss 11.7190 Epoch: 3 Global Step: 39050 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:07:15,203-Speed 126.88 samples/sec Loss 11.8038 Epoch: 3 Global Step: 39055 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:07:35,450-Speed 126.45 samples/sec Loss 11.4469 Epoch: 3 Global Step: 39060 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:07:56,545-Speed 121.36 samples/sec Loss 11.7382 Epoch: 3 Global Step: 39065 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:08:16,718-Speed 126.91 samples/sec Loss 12.0338 Epoch: 3 Global Step: 39070 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:08:36,939-Speed 126.61 samples/sec Loss 11.8430 Epoch: 3 Global Step: 39075 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:08:57,156-Speed 126.63 samples/sec Loss 11.2226 Epoch: 3 Global Step: 39080 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:09:17,444-Speed 126.19 samples/sec Loss 11.5358 Epoch: 3 Global Step: 39085 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:09:37,657-Speed 126.66 samples/sec Loss 11.4909 Epoch: 3 Global Step: 39090 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:09:57,820-Speed 126.97 samples/sec Loss 11.6560 Epoch: 3 Global Step: 39095 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:10:17,879-Speed 127.63 samples/sec Loss 11.6358 Epoch: 3 Global Step: 39100 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:10:37,986-Speed 127.33 samples/sec Loss 11.8443 Epoch: 3 Global Step: 39105 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:10:58,719-Speed 123.48 samples/sec Loss 11.4759 Epoch: 3 Global Step: 39110 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:11:18,962-Speed 126.47 samples/sec Loss 11.8595 Epoch: 3 Global Step: 39115 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:11:39,152-Speed 126.80 samples/sec Loss 11.9432 Epoch: 3 Global Step: 39120 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:11:59,325-Speed 126.91 samples/sec Loss 11.3149 Epoch: 3 Global Step: 39125 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:12:19,509-Speed 126.84 samples/sec Loss 11.3659 Epoch: 3 Global Step: 39130 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:12:39,757-Speed 126.44 samples/sec Loss 11.6909 Epoch: 3 Global Step: 39135 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:12:59,912-Speed 127.03 samples/sec Loss 11.5805 Epoch: 3 Global Step: 39140 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:13:20,176-Speed 126.34 samples/sec Loss 11.4686 Epoch: 3 Global Step: 39145 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 17:13:40,341-Speed 126.96 samples/sec Loss 11.6540 Epoch: 3 Global Step: 39150 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:14:01,077-Speed 123.46 samples/sec Loss 11.5955 Epoch: 3 Global Step: 39155 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:14:21,467-Speed 125.56 samples/sec Loss 11.3142 Epoch: 3 Global Step: 39160 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:14:41,733-Speed 126.32 samples/sec Loss 11.4059 Epoch: 3 Global Step: 39165 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:15:01,907-Speed 126.91 samples/sec Loss 11.4380 Epoch: 3 Global Step: 39170 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:15:22,090-Speed 126.85 samples/sec Loss 11.3960 Epoch: 3 Global Step: 39175 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:15:42,283-Speed 126.78 samples/sec Loss 11.7797 Epoch: 3 Global Step: 39180 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:16:02,479-Speed 126.77 samples/sec Loss 11.6989 Epoch: 3 Global Step: 39185 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:16:22,626-Speed 127.07 samples/sec Loss 11.2411 Epoch: 3 Global Step: 39190 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:16:42,819-Speed 126.78 samples/sec Loss 11.9864 Epoch: 3 Global Step: 39195 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:17:03,785-Speed 122.11 samples/sec Loss 11.6372 Epoch: 3 Global Step: 39200 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:17:24,053-Speed 126.31 samples/sec Loss 11.6520 Epoch: 3 Global Step: 39205 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:17:44,273-Speed 126.62 samples/sec Loss 11.6312 Epoch: 3 Global Step: 39210 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:18:04,513-Speed 126.49 samples/sec Loss 11.7551 Epoch: 3 Global Step: 39215 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:18:24,706-Speed 126.79 samples/sec Loss 11.4487 Epoch: 3 Global Step: 39220 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:18:45,027-Speed 125.99 samples/sec Loss 11.5359 Epoch: 3 Global Step: 39225 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:19:05,257-Speed 126.55 samples/sec Loss 11.4741 Epoch: 3 Global Step: 39230 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:19:25,479-Speed 126.60 samples/sec Loss 11.6843 Epoch: 3 Global Step: 39235 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:19:45,677-Speed 126.75 samples/sec Loss 11.9611 Epoch: 3 Global Step: 39240 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:20:06,540-Speed 122.71 samples/sec Loss 11.5511 Epoch: 3 Global Step: 39245 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:20:26,698-Speed 127.00 samples/sec Loss 11.6545 Epoch: 3 Global Step: 39250 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:20:47,168-Speed 125.07 samples/sec Loss 11.7398 Epoch: 3 Global Step: 39255 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:21:07,359-Speed 126.81 samples/sec Loss 11.5055 Epoch: 3 Global Step: 39260 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:21:27,550-Speed 126.80 samples/sec Loss 11.4987 Epoch: 3 Global Step: 39265 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:21:47,802-Speed 126.42 samples/sec Loss 11.6022 Epoch: 3 Global Step: 39270 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:22:08,154-Speed 125.79 samples/sec Loss 11.5816 Epoch: 3 Global Step: 39275 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:22:28,384-Speed 126.55 samples/sec Loss 11.6713 Epoch: 3 Global Step: 39280 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:22:48,696-Speed 126.04 samples/sec Loss 11.5715 Epoch: 3 Global Step: 39285 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:23:09,483-Speed 123.16 samples/sec Loss 11.6831 Epoch: 3 Global Step: 39290 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:23:29,834-Speed 125.80 samples/sec Loss 11.5161 Epoch: 3 Global Step: 39295 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:23:49,891-Speed 127.65 samples/sec Loss 11.7336 Epoch: 3 Global Step: 39300 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:24:10,085-Speed 126.78 samples/sec Loss 11.7203 Epoch: 3 Global Step: 39305 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:24:30,226-Speed 127.11 samples/sec Loss 11.2467 Epoch: 3 Global Step: 39310 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:24:50,461-Speed 126.52 samples/sec Loss 11.7651 Epoch: 3 Global Step: 39315 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:25:10,584-Speed 127.22 samples/sec Loss 11.4247 Epoch: 3 Global Step: 39320 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:25:30,824-Speed 126.49 samples/sec Loss 11.8836 Epoch: 3 Global Step: 39325 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:25:50,875-Speed 127.68 samples/sec Loss 11.4387 Epoch: 3 Global Step: 39330 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:26:11,585-Speed 123.62 samples/sec Loss 11.4937 Epoch: 3 Global Step: 39335 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:26:31,745-Speed 126.99 samples/sec Loss 11.4304 Epoch: 3 Global Step: 39340 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:26:51,968-Speed 126.59 samples/sec Loss 11.6474 Epoch: 3 Global Step: 39345 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:27:12,203-Speed 126.52 samples/sec Loss 11.6110 Epoch: 3 Global Step: 39350 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:27:32,362-Speed 127.00 samples/sec Loss 11.5661 Epoch: 3 Global Step: 39355 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:27:52,525-Speed 126.97 samples/sec Loss 11.6495 Epoch: 3 Global Step: 39360 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:28:12,966-Speed 125.25 samples/sec Loss 11.8145 Epoch: 3 Global Step: 39365 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:28:33,211-Speed 126.46 samples/sec Loss 11.1349 Epoch: 3 Global Step: 39370 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:28:53,504-Speed 126.16 samples/sec Loss 11.5384 Epoch: 3 Global Step: 39375 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:29:14,451-Speed 122.22 samples/sec Loss 11.5018 Epoch: 3 Global Step: 39380 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:29:34,695-Speed 126.46 samples/sec Loss 11.4505 Epoch: 3 Global Step: 39385 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:29:54,723-Speed 127.83 samples/sec Loss 11.3540 Epoch: 3 Global Step: 39390 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:30:14,957-Speed 126.53 samples/sec Loss 12.1256 Epoch: 3 Global Step: 39395 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:30:35,133-Speed 126.89 samples/sec Loss 11.4903 Epoch: 3 Global Step: 39400 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:30:55,328-Speed 126.77 samples/sec Loss 11.4971 Epoch: 3 Global Step: 39405 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:31:15,657-Speed 125.94 samples/sec Loss 11.1484 Epoch: 3 Global Step: 39410 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:31:35,886-Speed 126.56 samples/sec Loss 11.4792 Epoch: 3 Global Step: 39415 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:31:56,089-Speed 126.72 samples/sec Loss 11.5598 Epoch: 3 Global Step: 39420 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:32:17,245-Speed 121.01 samples/sec Loss 11.4963 Epoch: 3 Global Step: 39425 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:32:37,495-Speed 126.43 samples/sec Loss 11.7946 Epoch: 3 Global Step: 39430 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:32:57,752-Speed 126.39 samples/sec Loss 12.0202 Epoch: 3 Global Step: 39435 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:33:17,918-Speed 126.96 samples/sec Loss 11.4977 Epoch: 3 Global Step: 39440 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:33:37,937-Speed 127.88 samples/sec Loss 11.6652 Epoch: 3 Global Step: 39445 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:33:58,258-Speed 125.99 samples/sec Loss 11.4842 Epoch: 3 Global Step: 39450 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:34:18,336-Speed 127.51 samples/sec Loss 11.7008 Epoch: 3 Global Step: 39455 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:34:38,471-Speed 127.15 samples/sec Loss 11.5901 Epoch: 3 Global Step: 39460 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:34:58,618-Speed 127.07 samples/sec Loss 11.5962 Epoch: 3 Global Step: 39465 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:35:19,466-Speed 122.80 samples/sec Loss 11.4435 Epoch: 3 Global Step: 39470 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:35:39,530-Speed 127.59 samples/sec Loss 11.7910 Epoch: 3 Global Step: 39475 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:35:59,677-Speed 127.07 samples/sec Loss 11.5398 Epoch: 3 Global Step: 39480 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:36:19,850-Speed 126.91 samples/sec Loss 11.4008 Epoch: 3 Global Step: 39485 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:36:40,021-Speed 126.92 samples/sec Loss 11.8947 Epoch: 3 Global Step: 39490 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:37:00,773-Speed 123.37 samples/sec Loss 11.4812 Epoch: 3 Global Step: 39495 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:37:20,924-Speed 127.05 samples/sec Loss 11.2871 Epoch: 3 Global Step: 39500 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:37:41,088-Speed 126.97 samples/sec Loss 11.4769 Epoch: 3 Global Step: 39505 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:38:01,262-Speed 126.90 samples/sec Loss 11.5779 Epoch: 3 Global Step: 39510 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:38:22,210-Speed 122.21 samples/sec Loss 11.6505 Epoch: 3 Global Step: 39515 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:38:42,384-Speed 126.90 samples/sec Loss 11.2136 Epoch: 3 Global Step: 39520 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:39:02,544-Speed 126.99 samples/sec Loss 11.4180 Epoch: 3 Global Step: 39525 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:39:23,136-Speed 124.33 samples/sec Loss 11.7101 Epoch: 3 Global Step: 39530 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:39:43,295-Speed 127.00 samples/sec Loss 11.4812 Epoch: 3 Global Step: 39535 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:40:03,433-Speed 127.13 samples/sec Loss 11.7188 Epoch: 3 Global Step: 39540 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:40:23,456-Speed 127.86 samples/sec Loss 11.4153 Epoch: 3 Global Step: 39545 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:40:43,659-Speed 126.72 samples/sec Loss 11.4702 Epoch: 3 Global Step: 39550 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:41:03,797-Speed 127.13 samples/sec Loss 11.2599 Epoch: 3 Global Step: 39555 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:41:24,797-Speed 121.91 samples/sec Loss 11.4362 Epoch: 3 Global Step: 39560 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:41:45,180-Speed 125.60 samples/sec Loss 11.6873 Epoch: 3 Global Step: 39565 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:42:05,832-Speed 123.97 samples/sec Loss 11.3044 Epoch: 3 Global Step: 39570 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:42:26,156-Speed 125.96 samples/sec Loss 11.6997 Epoch: 3 Global Step: 39575 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:42:46,211-Speed 127.66 samples/sec Loss 11.3292 Epoch: 3 Global Step: 39580 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:43:06,382-Speed 126.93 samples/sec Loss 11.5669 Epoch: 3 Global Step: 39585 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:43:26,618-Speed 126.51 samples/sec Loss 11.6500 Epoch: 3 Global Step: 39590 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:43:46,833-Speed 126.65 samples/sec Loss 11.6862 Epoch: 3 Global Step: 39595 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:44:07,067-Speed 126.53 samples/sec Loss 11.5541 Epoch: 3 Global Step: 39600 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:44:28,086-Speed 121.80 samples/sec Loss 11.4862 Epoch: 3 Global Step: 39605 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:44:48,246-Speed 126.99 samples/sec Loss 11.7508 Epoch: 3 Global Step: 39610 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:45:08,432-Speed 126.82 samples/sec Loss 11.7368 Epoch: 3 Global Step: 39615 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:45:28,627-Speed 126.77 samples/sec Loss 11.7016 Epoch: 3 Global Step: 39620 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:45:48,841-Speed 126.66 samples/sec Loss 11.4364 Epoch: 3 Global Step: 39625 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:46:09,187-Speed 125.83 samples/sec Loss 11.4689 Epoch: 3 Global Step: 39630 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:46:29,490-Speed 126.10 samples/sec Loss 11.5627 Epoch: 3 Global Step: 39635 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:46:49,656-Speed 126.95 samples/sec Loss 11.3981 Epoch: 3 Global Step: 39640 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:47:09,791-Speed 127.15 samples/sec Loss 11.2781 Epoch: 3 Global Step: 39645 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:47:30,701-Speed 122.44 samples/sec Loss 11.4941 Epoch: 3 Global Step: 39650 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:47:50,866-Speed 126.96 samples/sec Loss 11.6593 Epoch: 3 Global Step: 39655 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:48:11,058-Speed 126.78 samples/sec Loss 11.5905 Epoch: 3 Global Step: 39660 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:48:31,226-Speed 126.94 samples/sec Loss 11.7367 Epoch: 3 Global Step: 39665 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:48:51,430-Speed 126.72 samples/sec Loss 11.3923 Epoch: 3 Global Step: 39670 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:49:11,607-Speed 126.88 samples/sec Loss 11.7231 Epoch: 3 Global Step: 39675 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:49:31,792-Speed 126.84 samples/sec Loss 11.7123 Epoch: 3 Global Step: 39680 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:49:52,262-Speed 125.07 samples/sec Loss 11.5502 Epoch: 3 Global Step: 39685 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:50:12,526-Speed 126.34 samples/sec Loss 11.6295 Epoch: 3 Global Step: 39690 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:50:33,384-Speed 122.74 samples/sec Loss 11.1496 Epoch: 3 Global Step: 39695 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:50:53,374-Speed 128.07 samples/sec Loss 11.5241 Epoch: 3 Global Step: 39700 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:51:13,556-Speed 126.85 samples/sec Loss 11.4581 Epoch: 3 Global Step: 39705 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:51:33,714-Speed 127.00 samples/sec Loss 11.9241 Epoch: 3 Global Step: 39710 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:51:53,893-Speed 126.87 samples/sec Loss 11.4511 Epoch: 3 Global Step: 39715 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:52:14,396-Speed 124.87 samples/sec Loss 11.3730 Epoch: 3 Global Step: 39720 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:52:34,374-Speed 128.15 samples/sec Loss 11.8135 Epoch: 3 Global Step: 39725 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:52:54,519-Speed 127.09 samples/sec Loss 11.9277 Epoch: 3 Global Step: 39730 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:53:14,663-Speed 127.09 samples/sec Loss 11.7415 Epoch: 3 Global Step: 39735 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:53:35,353-Speed 123.73 samples/sec Loss 11.6186 Epoch: 3 Global Step: 39740 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:53:55,510-Speed 127.01 samples/sec Loss 11.4290 Epoch: 3 Global Step: 39745 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:54:15,678-Speed 126.94 samples/sec Loss 11.6389 Epoch: 3 Global Step: 39750 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:54:35,656-Speed 128.15 samples/sec Loss 11.6795 Epoch: 3 Global Step: 39755 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:54:56,017-Speed 125.74 samples/sec Loss 11.5545 Epoch: 3 Global Step: 39760 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:55:16,316-Speed 126.12 samples/sec Loss 11.3820 Epoch: 3 Global Step: 39765 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:55:36,520-Speed 126.72 samples/sec Loss 11.4763 Epoch: 3 Global Step: 39770 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:55:56,721-Speed 126.73 samples/sec Loss 11.6558 Epoch: 3 Global Step: 39775 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:56:16,918-Speed 126.76 samples/sec Loss 11.6530 Epoch: 3 Global Step: 39780 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:56:37,515-Speed 124.30 samples/sec Loss 11.9014 Epoch: 3 Global Step: 39785 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:56:57,969-Speed 125.16 samples/sec Loss 11.4739 Epoch: 3 Global Step: 39790 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:57:17,978-Speed 127.95 samples/sec Loss 11.4154 Epoch: 3 Global Step: 39795 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:57:38,180-Speed 126.73 samples/sec Loss 11.4721 Epoch: 3 Global Step: 39800 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:57:58,691-Speed 124.82 samples/sec Loss 11.5990 Epoch: 3 Global Step: 39805 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:58:19,326-Speed 124.07 samples/sec Loss 11.6400 Epoch: 3 Global Step: 39810 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:58:39,535-Speed 126.69 samples/sec Loss 11.4719 Epoch: 3 Global Step: 39815 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:58:59,758-Speed 126.60 samples/sec Loss 11.7251 Epoch: 3 Global Step: 39820 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:59:19,916-Speed 127.00 samples/sec Loss 11.4722 Epoch: 3 Global Step: 39825 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 17:59:40,917-Speed 121.91 samples/sec Loss 11.5628 Epoch: 3 Global Step: 39830 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 18:00:01,372-Speed 125.16 samples/sec Loss 11.5311 Epoch: 3 Global Step: 39835 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 18:00:21,604-Speed 126.53 samples/sec Loss 11.5848 Epoch: 3 Global Step: 39840 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:00:41,837-Speed 126.54 samples/sec Loss 11.6143 Epoch: 3 Global Step: 39845 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:01:01,973-Speed 127.14 samples/sec Loss 11.4423 Epoch: 3 Global Step: 39850 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:01:22,186-Speed 126.65 samples/sec Loss 11.5104 Epoch: 3 Global Step: 39855 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:02:02,878-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-08 18:02:26,138-Speed 40.03 samples/sec Loss 11.2522 Epoch: 4 Global Step: 39860 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 18:02:46,057-Speed 128.53 samples/sec Loss 10.6071 Epoch: 4 Global Step: 39865 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 18:03:06,646-Speed 124.34 samples/sec Loss 11.1327 Epoch: 4 Global Step: 39870 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 18:03:26,885-Speed 126.50 samples/sec Loss 10.9599 Epoch: 4 Global Step: 39875 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 18:03:46,823-Speed 128.40 samples/sec Loss 11.1697 Epoch: 4 Global Step: 39880 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 18:04:07,530-Speed 123.64 samples/sec Loss 10.9578 Epoch: 4 Global Step: 39885 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:04:27,439-Speed 128.59 samples/sec Loss 11.0487 Epoch: 4 Global Step: 39890 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:04:47,755-Speed 126.02 samples/sec Loss 11.1465 Epoch: 4 Global Step: 39895 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:05:07,768-Speed 127.92 samples/sec Loss 10.5676 Epoch: 4 Global Step: 39900 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:05:27,822-Speed 127.66 samples/sec Loss 11.0959 Epoch: 4 Global Step: 39905 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:05:47,795-Speed 128.18 samples/sec Loss 10.7433 Epoch: 4 Global Step: 39910 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:06:08,312-Speed 124.78 samples/sec Loss 11.0016 Epoch: 4 Global Step: 39915 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:06:28,774-Speed 125.11 samples/sec Loss 11.3939 Epoch: 4 Global Step: 39920 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:06:48,902-Speed 127.19 samples/sec Loss 11.0133 Epoch: 4 Global Step: 39925 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:07:09,273-Speed 125.68 samples/sec Loss 11.2176 Epoch: 4 Global Step: 39930 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:07:29,246-Speed 128.18 samples/sec Loss 10.7672 Epoch: 4 Global Step: 39935 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:07:49,560-Speed 126.03 samples/sec Loss 10.9310 Epoch: 4 Global Step: 39940 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:08:09,632-Speed 127.55 samples/sec Loss 10.8971 Epoch: 4 Global Step: 39945 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:08:29,862-Speed 126.56 samples/sec Loss 10.9835 Epoch: 4 Global Step: 39950 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:08:49,732-Speed 128.84 samples/sec Loss 11.1893 Epoch: 4 Global Step: 39955 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:09:10,379-Speed 124.00 samples/sec Loss 10.9692 Epoch: 4 Global Step: 39960 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:09:30,474-Speed 127.40 samples/sec Loss 11.2875 Epoch: 4 Global Step: 39965 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:09:50,328-Speed 128.95 samples/sec Loss 11.0407 Epoch: 4 Global Step: 39970 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:10:10,449-Speed 127.24 samples/sec Loss 11.0221 Epoch: 4 Global Step: 39975 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:10:30,285-Speed 129.07 samples/sec Loss 10.9001 Epoch: 4 Global Step: 39980 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:10:50,600-Speed 126.02 samples/sec Loss 10.7119 Epoch: 4 Global Step: 39985 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:11:10,591-Speed 128.06 samples/sec Loss 10.8293 Epoch: 4 Global Step: 39990 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:11:30,951-Speed 125.75 samples/sec Loss 10.9996 Epoch: 4 Global Step: 39995 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:11:51,095-Speed 127.09 samples/sec Loss 11.0967 Epoch: 4 Global Step: 40000 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 18:26:37,601-[lfw][40000]XNorm: 23.523673 Training: 2021-06-08 18:26:37,601-[lfw][40000]Accuracy-Flip: 0.99617+-0.00325 Training: 2021-06-08 18:26:37,601-[lfw][40000]Accuracy-Highest: 0.99700 Training: 2021-06-08 18:43:53,467-[cfp_fp][40000]XNorm: 21.855540 Training: 2021-06-08 18:43:53,467-[cfp_fp][40000]Accuracy-Flip: 0.96814+-0.00793 Training: 2021-06-08 18:43:53,467-[cfp_fp][40000]Accuracy-Highest: 0.96886 Training: 2021-06-08 18:58:42,474-[agedb_30][40000]XNorm: 23.376783 Training: 2021-06-08 18:58:42,475-[agedb_30][40000]Accuracy-Flip: 0.97000+-0.00632 Training: 2021-06-08 18:58:42,475-[agedb_30][40000]Accuracy-Highest: 0.97000 Training: 2021-06-08 18:59:02,406-Speed 0.90 samples/sec Loss 10.8877 Epoch: 4 Global Step: 40005 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 18:59:23,615-Speed 120.71 samples/sec Loss 11.1805 Epoch: 4 Global Step: 40010 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 18:59:43,835-Speed 126.61 samples/sec Loss 10.7177 Epoch: 4 Global Step: 40015 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:00:04,104-Speed 126.31 samples/sec Loss 10.7485 Epoch: 4 Global Step: 40020 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:00:24,390-Speed 126.20 samples/sec Loss 11.2170 Epoch: 4 Global Step: 40025 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:00:44,613-Speed 126.59 samples/sec Loss 11.1493 Epoch: 4 Global Step: 40030 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:01:04,884-Speed 126.30 samples/sec Loss 11.0906 Epoch: 4 Global Step: 40035 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:01:25,233-Speed 125.81 samples/sec Loss 10.7822 Epoch: 4 Global Step: 40040 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:01:45,466-Speed 126.54 samples/sec Loss 11.3063 Epoch: 4 Global Step: 40045 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:02:05,561-Speed 127.40 samples/sec Loss 10.6685 Epoch: 4 Global Step: 40050 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:02:26,609-Speed 121.63 samples/sec Loss 10.8680 Epoch: 4 Global Step: 40055 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:02:46,990-Speed 125.61 samples/sec Loss 11.3912 Epoch: 4 Global Step: 40060 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:03:07,250-Speed 126.37 samples/sec Loss 10.8013 Epoch: 4 Global Step: 40065 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:03:27,382-Speed 127.17 samples/sec Loss 11.0311 Epoch: 4 Global Step: 40070 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:03:47,605-Speed 126.60 samples/sec Loss 10.7667 Epoch: 4 Global Step: 40075 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:04:07,832-Speed 126.57 samples/sec Loss 11.2155 Epoch: 4 Global Step: 40080 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:04:28,081-Speed 126.43 samples/sec Loss 10.9798 Epoch: 4 Global Step: 40085 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:04:48,227-Speed 127.08 samples/sec Loss 11.0332 Epoch: 4 Global Step: 40090 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:05:08,497-Speed 126.30 samples/sec Loss 11.1179 Epoch: 4 Global Step: 40095 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:05:29,673-Speed 120.90 samples/sec Loss 11.1152 Epoch: 4 Global Step: 40100 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:05:49,913-Speed 126.49 samples/sec Loss 10.7587 Epoch: 4 Global Step: 40105 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:06:09,979-Speed 127.58 samples/sec Loss 10.8996 Epoch: 4 Global Step: 40110 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:06:30,253-Speed 126.28 samples/sec Loss 10.9257 Epoch: 4 Global Step: 40115 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:06:50,604-Speed 125.80 samples/sec Loss 11.0391 Epoch: 4 Global Step: 40120 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:07:10,931-Speed 125.94 samples/sec Loss 10.9037 Epoch: 4 Global Step: 40125 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:07:31,170-Speed 126.50 samples/sec Loss 10.8983 Epoch: 4 Global Step: 40130 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:07:51,391-Speed 126.61 samples/sec Loss 10.7716 Epoch: 4 Global Step: 40135 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:08:11,637-Speed 126.46 samples/sec Loss 11.2760 Epoch: 4 Global Step: 40140 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:08:32,579-Speed 122.24 samples/sec Loss 11.2306 Epoch: 4 Global Step: 40145 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:08:52,861-Speed 126.23 samples/sec Loss 11.2239 Epoch: 4 Global Step: 40150 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:09:13,118-Speed 126.38 samples/sec Loss 11.0178 Epoch: 4 Global Step: 40155 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:09:33,476-Speed 125.76 samples/sec Loss 11.2821 Epoch: 4 Global Step: 40160 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:09:53,854-Speed 125.63 samples/sec Loss 11.0248 Epoch: 4 Global Step: 40165 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:10:14,048-Speed 126.78 samples/sec Loss 11.4163 Epoch: 4 Global Step: 40170 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:10:34,344-Speed 126.14 samples/sec Loss 11.2141 Epoch: 4 Global Step: 40175 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:10:54,592-Speed 126.45 samples/sec Loss 11.3091 Epoch: 4 Global Step: 40180 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:11:14,814-Speed 126.60 samples/sec Loss 11.2217 Epoch: 4 Global Step: 40185 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:11:35,745-Speed 122.31 samples/sec Loss 11.2090 Epoch: 4 Global Step: 40190 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:11:56,071-Speed 125.95 samples/sec Loss 11.3167 Epoch: 4 Global Step: 40195 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:12:16,163-Speed 127.42 samples/sec Loss 10.9286 Epoch: 4 Global Step: 40200 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:12:36,453-Speed 126.18 samples/sec Loss 11.0832 Epoch: 4 Global Step: 40205 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:12:56,771-Speed 126.00 samples/sec Loss 11.3531 Epoch: 4 Global Step: 40210 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:13:17,015-Speed 126.46 samples/sec Loss 11.3299 Epoch: 4 Global Step: 40215 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:13:37,433-Speed 125.39 samples/sec Loss 11.4185 Epoch: 4 Global Step: 40220 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:13:57,794-Speed 125.74 samples/sec Loss 11.4517 Epoch: 4 Global Step: 40225 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:14:17,822-Speed 127.83 samples/sec Loss 11.5891 Epoch: 4 Global Step: 40230 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:14:38,851-Speed 121.75 samples/sec Loss 11.0085 Epoch: 4 Global Step: 40235 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:14:59,127-Speed 126.26 samples/sec Loss 10.9724 Epoch: 4 Global Step: 40240 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:15:19,304-Speed 126.88 samples/sec Loss 11.4461 Epoch: 4 Global Step: 40245 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:15:39,609-Speed 126.08 samples/sec Loss 11.2740 Epoch: 4 Global Step: 40250 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:15:59,781-Speed 126.91 samples/sec Loss 11.0464 Epoch: 4 Global Step: 40255 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:16:20,064-Speed 126.22 samples/sec Loss 11.3781 Epoch: 4 Global Step: 40260 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:16:40,369-Speed 126.08 samples/sec Loss 11.6175 Epoch: 4 Global Step: 40265 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:17:00,616-Speed 126.45 samples/sec Loss 11.1702 Epoch: 4 Global Step: 40270 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:17:20,962-Speed 125.83 samples/sec Loss 11.1240 Epoch: 4 Global Step: 40275 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:17:41,962-Speed 121.91 samples/sec Loss 11.2018 Epoch: 4 Global Step: 40280 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:18:02,289-Speed 125.95 samples/sec Loss 11.3775 Epoch: 4 Global Step: 40285 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:18:22,522-Speed 126.53 samples/sec Loss 11.4019 Epoch: 4 Global Step: 40290 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:18:42,851-Speed 125.94 samples/sec Loss 11.1699 Epoch: 4 Global Step: 40295 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:19:03,103-Speed 126.42 samples/sec Loss 11.4000 Epoch: 4 Global Step: 40300 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:19:23,328-Speed 126.58 samples/sec Loss 11.0421 Epoch: 4 Global Step: 40305 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:19:43,316-Speed 128.08 samples/sec Loss 11.3106 Epoch: 4 Global Step: 40310 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:20:03,507-Speed 126.80 samples/sec Loss 11.4565 Epoch: 4 Global Step: 40315 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:20:23,550-Speed 127.74 samples/sec Loss 11.1925 Epoch: 4 Global Step: 40320 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:20:44,460-Speed 122.43 samples/sec Loss 11.1722 Epoch: 4 Global Step: 40325 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:21:04,736-Speed 126.26 samples/sec Loss 11.3354 Epoch: 4 Global Step: 40330 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:21:25,012-Speed 126.27 samples/sec Loss 11.4473 Epoch: 4 Global Step: 40335 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:21:45,262-Speed 126.43 samples/sec Loss 11.0809 Epoch: 4 Global Step: 40340 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:22:05,354-Speed 127.42 samples/sec Loss 11.1921 Epoch: 4 Global Step: 40345 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:22:25,542-Speed 126.82 samples/sec Loss 11.2196 Epoch: 4 Global Step: 40350 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:22:45,778-Speed 126.52 samples/sec Loss 11.3599 Epoch: 4 Global Step: 40355 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:23:06,014-Speed 126.51 samples/sec Loss 11.2225 Epoch: 4 Global Step: 40360 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:23:26,286-Speed 126.29 samples/sec Loss 11.3519 Epoch: 4 Global Step: 40365 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:23:47,223-Speed 122.28 samples/sec Loss 11.2264 Epoch: 4 Global Step: 40370 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:24:07,506-Speed 126.22 samples/sec Loss 11.9083 Epoch: 4 Global Step: 40375 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:24:27,515-Speed 127.95 samples/sec Loss 11.2964 Epoch: 4 Global Step: 40380 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:24:47,838-Speed 125.97 samples/sec Loss 11.5341 Epoch: 4 Global Step: 40385 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:25:08,058-Speed 126.62 samples/sec Loss 11.6092 Epoch: 4 Global Step: 40390 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:25:28,517-Speed 125.13 samples/sec Loss 11.3384 Epoch: 4 Global Step: 40395 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:25:49,685-Speed 120.95 samples/sec Loss 10.9902 Epoch: 4 Global Step: 40400 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:26:09,893-Speed 126.69 samples/sec Loss 11.0583 Epoch: 4 Global Step: 40405 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:26:30,145-Speed 126.42 samples/sec Loss 11.0862 Epoch: 4 Global Step: 40410 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:26:50,885-Speed 123.44 samples/sec Loss 11.4457 Epoch: 4 Global Step: 40415 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:27:10,884-Speed 128.01 samples/sec Loss 11.3156 Epoch: 4 Global Step: 40420 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:27:31,075-Speed 126.80 samples/sec Loss 11.2675 Epoch: 4 Global Step: 40425 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:27:51,114-Speed 127.76 samples/sec Loss 10.8875 Epoch: 4 Global Step: 40430 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:28:11,362-Speed 126.44 samples/sec Loss 11.2744 Epoch: 4 Global Step: 40435 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:28:31,707-Speed 125.84 samples/sec Loss 11.3697 Epoch: 4 Global Step: 40440 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:28:51,875-Speed 126.94 samples/sec Loss 10.9201 Epoch: 4 Global Step: 40445 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:29:12,042-Speed 126.95 samples/sec Loss 11.4984 Epoch: 4 Global Step: 40450 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:29:32,117-Speed 127.53 samples/sec Loss 11.2838 Epoch: 4 Global Step: 40455 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:29:53,331-Speed 120.68 samples/sec Loss 11.2721 Epoch: 4 Global Step: 40460 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:30:13,554-Speed 126.60 samples/sec Loss 11.2261 Epoch: 4 Global Step: 40465 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:30:33,823-Speed 126.30 samples/sec Loss 11.3789 Epoch: 4 Global Step: 40470 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:30:53,894-Speed 127.56 samples/sec Loss 11.6160 Epoch: 4 Global Step: 40475 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:31:13,943-Speed 127.69 samples/sec Loss 11.2507 Epoch: 4 Global Step: 40480 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:31:34,134-Speed 126.80 samples/sec Loss 11.4277 Epoch: 4 Global Step: 40485 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:31:54,174-Speed 127.76 samples/sec Loss 11.2520 Epoch: 4 Global Step: 40490 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:32:14,545-Speed 125.68 samples/sec Loss 11.4261 Epoch: 4 Global Step: 40495 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:32:34,773-Speed 126.57 samples/sec Loss 11.1575 Epoch: 4 Global Step: 40500 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:32:55,652-Speed 122.61 samples/sec Loss 11.3520 Epoch: 4 Global Step: 40505 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:33:16,231-Speed 124.41 samples/sec Loss 11.2044 Epoch: 4 Global Step: 40510 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:33:36,528-Speed 126.13 samples/sec Loss 11.5388 Epoch: 4 Global Step: 40515 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:33:56,808-Speed 126.24 samples/sec Loss 11.3289 Epoch: 4 Global Step: 40520 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:34:17,081-Speed 126.29 samples/sec Loss 11.4144 Epoch: 4 Global Step: 40525 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:34:37,328-Speed 126.45 samples/sec Loss 11.1134 Epoch: 4 Global Step: 40530 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:34:57,629-Speed 126.11 samples/sec Loss 11.3231 Epoch: 4 Global Step: 40535 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:35:18,236-Speed 124.24 samples/sec Loss 11.3603 Epoch: 4 Global Step: 40540 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:35:38,481-Speed 126.46 samples/sec Loss 11.3932 Epoch: 4 Global Step: 40545 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:35:59,358-Speed 122.62 samples/sec Loss 11.2319 Epoch: 4 Global Step: 40550 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:36:19,744-Speed 125.58 samples/sec Loss 10.9872 Epoch: 4 Global Step: 40555 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:36:40,011-Speed 126.32 samples/sec Loss 11.3679 Epoch: 4 Global Step: 40560 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:37:00,076-Speed 127.60 samples/sec Loss 11.3279 Epoch: 4 Global Step: 40565 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:37:20,380-Speed 126.09 samples/sec Loss 11.6487 Epoch: 4 Global Step: 40570 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:37:40,682-Speed 126.10 samples/sec Loss 11.3430 Epoch: 4 Global Step: 40575 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:38:00,907-Speed 126.58 samples/sec Loss 11.4717 Epoch: 4 Global Step: 40580 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:38:20,980-Speed 127.54 samples/sec Loss 11.0115 Epoch: 4 Global Step: 40585 Fp16 Grad Scale: 8192 Required: 250 hours Training: 2021-06-08 19:38:41,146-Speed 126.96 samples/sec Loss 11.2398 Epoch: 4 Global Step: 40590 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:39:01,858-Speed 123.61 samples/sec Loss 11.2031 Epoch: 4 Global Step: 40595 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:39:22,150-Speed 126.16 samples/sec Loss 11.4472 Epoch: 4 Global Step: 40600 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:39:42,399-Speed 126.44 samples/sec Loss 11.3417 Epoch: 4 Global Step: 40605 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:40:02,538-Speed 127.12 samples/sec Loss 11.4401 Epoch: 4 Global Step: 40610 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:40:22,626-Speed 127.45 samples/sec Loss 11.7362 Epoch: 4 Global Step: 40615 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:40:42,898-Speed 126.29 samples/sec Loss 11.4254 Epoch: 4 Global Step: 40620 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:41:03,711-Speed 123.00 samples/sec Loss 11.2624 Epoch: 4 Global Step: 40625 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:41:24,042-Speed 125.93 samples/sec Loss 11.2739 Epoch: 4 Global Step: 40630 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:41:44,570-Speed 124.71 samples/sec Loss 11.3728 Epoch: 4 Global Step: 40635 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:42:05,612-Speed 121.67 samples/sec Loss 10.9924 Epoch: 4 Global Step: 40640 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:42:26,049-Speed 125.27 samples/sec Loss 11.6830 Epoch: 4 Global Step: 40645 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:42:46,367-Speed 126.00 samples/sec Loss 11.3701 Epoch: 4 Global Step: 40650 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:43:06,627-Speed 126.37 samples/sec Loss 11.1866 Epoch: 4 Global Step: 40655 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:43:27,037-Speed 125.44 samples/sec Loss 11.5419 Epoch: 4 Global Step: 40660 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:43:47,254-Speed 126.63 samples/sec Loss 11.2439 Epoch: 4 Global Step: 40665 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:44:07,269-Speed 127.91 samples/sec Loss 11.5427 Epoch: 4 Global Step: 40670 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:44:27,502-Speed 126.53 samples/sec Loss 11.5858 Epoch: 4 Global Step: 40675 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:44:47,714-Speed 126.66 samples/sec Loss 11.4812 Epoch: 4 Global Step: 40680 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:45:08,546-Speed 122.89 samples/sec Loss 11.0966 Epoch: 4 Global Step: 40685 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:45:29,278-Speed 123.49 samples/sec Loss 11.5326 Epoch: 4 Global Step: 40690 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:45:49,598-Speed 125.98 samples/sec Loss 11.3461 Epoch: 4 Global Step: 40695 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:46:09,790-Speed 126.79 samples/sec Loss 11.3730 Epoch: 4 Global Step: 40700 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:46:30,007-Speed 126.63 samples/sec Loss 11.2646 Epoch: 4 Global Step: 40705 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:46:50,309-Speed 126.11 samples/sec Loss 11.3049 Epoch: 4 Global Step: 40710 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:47:10,530-Speed 126.61 samples/sec Loss 11.3826 Epoch: 4 Global Step: 40715 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:47:30,733-Speed 126.72 samples/sec Loss 11.2359 Epoch: 4 Global Step: 40720 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:47:50,966-Speed 126.53 samples/sec Loss 11.3896 Epoch: 4 Global Step: 40725 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:48:12,037-Speed 121.50 samples/sec Loss 10.9181 Epoch: 4 Global Step: 40730 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:48:32,353-Speed 126.02 samples/sec Loss 11.3712 Epoch: 4 Global Step: 40735 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:48:52,754-Speed 125.49 samples/sec Loss 11.5285 Epoch: 4 Global Step: 40740 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:49:12,994-Speed 126.49 samples/sec Loss 11.3920 Epoch: 4 Global Step: 40745 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:49:33,139-Speed 127.09 samples/sec Loss 11.4709 Epoch: 4 Global Step: 40750 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:49:53,388-Speed 126.43 samples/sec Loss 11.3743 Epoch: 4 Global Step: 40755 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:50:13,758-Speed 125.68 samples/sec Loss 11.6282 Epoch: 4 Global Step: 40760 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:50:34,148-Speed 125.56 samples/sec Loss 11.6196 Epoch: 4 Global Step: 40765 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:50:54,433-Speed 126.21 samples/sec Loss 11.5260 Epoch: 4 Global Step: 40770 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:51:15,553-Speed 121.22 samples/sec Loss 11.5825 Epoch: 4 Global Step: 40775 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:51:35,770-Speed 126.63 samples/sec Loss 11.5155 Epoch: 4 Global Step: 40780 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:51:56,488-Speed 123.57 samples/sec Loss 11.5822 Epoch: 4 Global Step: 40785 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:52:16,969-Speed 125.00 samples/sec Loss 11.2763 Epoch: 4 Global Step: 40790 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:52:37,113-Speed 127.09 samples/sec Loss 11.3094 Epoch: 4 Global Step: 40795 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:52:57,370-Speed 126.39 samples/sec Loss 11.5444 Epoch: 4 Global Step: 40800 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:53:17,695-Speed 125.96 samples/sec Loss 11.6467 Epoch: 4 Global Step: 40805 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:53:38,126-Speed 125.31 samples/sec Loss 11.2898 Epoch: 4 Global Step: 40810 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:53:58,356-Speed 126.55 samples/sec Loss 11.3063 Epoch: 4 Global Step: 40815 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:54:19,420-Speed 121.54 samples/sec Loss 11.5140 Epoch: 4 Global Step: 40820 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:54:39,645-Speed 126.58 samples/sec Loss 11.5385 Epoch: 4 Global Step: 40825 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:54:59,874-Speed 126.56 samples/sec Loss 11.4397 Epoch: 4 Global Step: 40830 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:55:20,130-Speed 126.39 samples/sec Loss 11.5265 Epoch: 4 Global Step: 40835 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:55:40,393-Speed 126.35 samples/sec Loss 11.2464 Epoch: 4 Global Step: 40840 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:56:00,453-Speed 127.62 samples/sec Loss 11.4453 Epoch: 4 Global Step: 40845 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:56:20,754-Speed 126.11 samples/sec Loss 10.9231 Epoch: 4 Global Step: 40850 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:56:41,048-Speed 126.15 samples/sec Loss 11.6396 Epoch: 4 Global Step: 40855 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:57:01,346-Speed 126.13 samples/sec Loss 11.5423 Epoch: 4 Global Step: 40860 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:57:22,357-Speed 121.85 samples/sec Loss 11.4307 Epoch: 4 Global Step: 40865 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:57:42,545-Speed 126.81 samples/sec Loss 11.3425 Epoch: 4 Global Step: 40870 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:58:02,677-Speed 127.16 samples/sec Loss 11.2712 Epoch: 4 Global Step: 40875 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:58:22,957-Speed 126.24 samples/sec Loss 11.6059 Epoch: 4 Global Step: 40880 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:58:43,192-Speed 126.53 samples/sec Loss 11.2320 Epoch: 4 Global Step: 40885 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:59:03,891-Speed 123.68 samples/sec Loss 11.5677 Epoch: 4 Global Step: 40890 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:59:24,127-Speed 126.52 samples/sec Loss 11.4874 Epoch: 4 Global Step: 40895 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 19:59:44,348-Speed 126.61 samples/sec Loss 11.5890 Epoch: 4 Global Step: 40900 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:00:04,746-Speed 125.51 samples/sec Loss 11.2941 Epoch: 4 Global Step: 40905 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:00:25,661-Speed 122.41 samples/sec Loss 11.3158 Epoch: 4 Global Step: 40910 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:00:45,907-Speed 126.45 samples/sec Loss 11.3518 Epoch: 4 Global Step: 40915 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:01:06,103-Speed 126.76 samples/sec Loss 11.4670 Epoch: 4 Global Step: 40920 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:01:26,296-Speed 126.79 samples/sec Loss 11.4901 Epoch: 4 Global Step: 40925 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:01:46,469-Speed 126.91 samples/sec Loss 11.5649 Epoch: 4 Global Step: 40930 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:02:06,714-Speed 126.46 samples/sec Loss 11.3580 Epoch: 4 Global Step: 40935 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:02:27,032-Speed 126.01 samples/sec Loss 11.7110 Epoch: 4 Global Step: 40940 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:02:47,329-Speed 126.14 samples/sec Loss 11.2460 Epoch: 4 Global Step: 40945 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:03:07,536-Speed 126.70 samples/sec Loss 11.3567 Epoch: 4 Global Step: 40950 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:03:28,385-Speed 122.80 samples/sec Loss 11.7823 Epoch: 4 Global Step: 40955 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:03:48,466-Speed 127.48 samples/sec Loss 11.6860 Epoch: 4 Global Step: 40960 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:04:08,559-Speed 127.42 samples/sec Loss 11.1250 Epoch: 4 Global Step: 40965 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:04:28,795-Speed 126.51 samples/sec Loss 11.5615 Epoch: 4 Global Step: 40970 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:04:49,008-Speed 126.66 samples/sec Loss 11.1205 Epoch: 4 Global Step: 40975 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:05:09,287-Speed 126.24 samples/sec Loss 11.4740 Epoch: 4 Global Step: 40980 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:05:29,403-Speed 127.27 samples/sec Loss 11.5292 Epoch: 4 Global Step: 40985 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:05:50,856-Speed 119.34 samples/sec Loss 11.4489 Epoch: 4 Global Step: 40990 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:06:11,063-Speed 126.70 samples/sec Loss 11.6835 Epoch: 4 Global Step: 40995 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:06:32,047-Speed 122.00 samples/sec Loss 11.6214 Epoch: 4 Global Step: 41000 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:06:52,134-Speed 127.45 samples/sec Loss 11.2819 Epoch: 4 Global Step: 41005 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:07:12,382-Speed 126.44 samples/sec Loss 11.4549 Epoch: 4 Global Step: 41010 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:07:32,701-Speed 126.00 samples/sec Loss 11.2135 Epoch: 4 Global Step: 41015 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:07:53,055-Speed 125.78 samples/sec Loss 11.4135 Epoch: 4 Global Step: 41020 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:08:13,767-Speed 123.62 samples/sec Loss 11.1276 Epoch: 4 Global Step: 41025 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:08:33,862-Speed 127.40 samples/sec Loss 11.5836 Epoch: 4 Global Step: 41030 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:08:54,079-Speed 126.63 samples/sec Loss 11.5367 Epoch: 4 Global Step: 41035 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:09:14,104-Speed 127.84 samples/sec Loss 11.3811 Epoch: 4 Global Step: 41040 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:09:34,972-Speed 122.69 samples/sec Loss 11.4118 Epoch: 4 Global Step: 41045 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:09:55,138-Speed 126.95 samples/sec Loss 11.5965 Epoch: 4 Global Step: 41050 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:10:15,215-Speed 127.52 samples/sec Loss 11.5066 Epoch: 4 Global Step: 41055 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:10:35,347-Speed 127.17 samples/sec Loss 11.4588 Epoch: 4 Global Step: 41060 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:10:55,576-Speed 126.56 samples/sec Loss 11.8289 Epoch: 4 Global Step: 41065 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:11:15,820-Speed 126.46 samples/sec Loss 11.8633 Epoch: 4 Global Step: 41070 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:11:35,976-Speed 127.02 samples/sec Loss 11.7286 Epoch: 4 Global Step: 41075 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:11:56,115-Speed 127.13 samples/sec Loss 11.7014 Epoch: 4 Global Step: 41080 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:12:16,235-Speed 127.25 samples/sec Loss 11.4535 Epoch: 4 Global Step: 41085 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:12:37,226-Speed 121.96 samples/sec Loss 11.2986 Epoch: 4 Global Step: 41090 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:12:57,442-Speed 126.64 samples/sec Loss 11.2142 Epoch: 4 Global Step: 41095 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:13:17,733-Speed 126.17 samples/sec Loss 11.6150 Epoch: 4 Global Step: 41100 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:13:38,061-Speed 125.94 samples/sec Loss 11.5563 Epoch: 4 Global Step: 41105 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:13:58,289-Speed 126.57 samples/sec Loss 11.0591 Epoch: 4 Global Step: 41110 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:14:18,637-Speed 125.81 samples/sec Loss 11.5158 Epoch: 4 Global Step: 41115 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:14:38,735-Speed 127.38 samples/sec Loss 11.6058 Epoch: 4 Global Step: 41120 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:14:58,968-Speed 126.54 samples/sec Loss 11.3399 Epoch: 4 Global Step: 41125 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:15:19,126-Speed 127.00 samples/sec Loss 11.4639 Epoch: 4 Global Step: 41130 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:15:40,129-Speed 121.90 samples/sec Loss 11.4391 Epoch: 4 Global Step: 41135 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:16:00,344-Speed 126.64 samples/sec Loss 11.5205 Epoch: 4 Global Step: 41140 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:16:20,499-Speed 127.02 samples/sec Loss 11.4119 Epoch: 4 Global Step: 41145 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:16:40,718-Speed 126.62 samples/sec Loss 11.9412 Epoch: 4 Global Step: 41150 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:17:00,996-Speed 126.25 samples/sec Loss 11.7271 Epoch: 4 Global Step: 41155 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:17:21,096-Speed 127.37 samples/sec Loss 12.0652 Epoch: 4 Global Step: 41160 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:17:41,348-Speed 126.41 samples/sec Loss 11.5618 Epoch: 4 Global Step: 41165 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:18:01,885-Speed 124.66 samples/sec Loss 11.4719 Epoch: 4 Global Step: 41170 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:18:22,159-Speed 126.28 samples/sec Loss 11.3692 Epoch: 4 Global Step: 41175 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:18:43,013-Speed 122.77 samples/sec Loss 11.7838 Epoch: 4 Global Step: 41180 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:19:03,243-Speed 126.55 samples/sec Loss 11.5545 Epoch: 4 Global Step: 41185 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:19:23,509-Speed 126.33 samples/sec Loss 11.5123 Epoch: 4 Global Step: 41190 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:19:43,755-Speed 126.45 samples/sec Loss 11.6313 Epoch: 4 Global Step: 41195 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:20:03,970-Speed 126.65 samples/sec Loss 11.3693 Epoch: 4 Global Step: 41200 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:20:24,307-Speed 125.88 samples/sec Loss 11.5881 Epoch: 4 Global Step: 41205 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:20:44,491-Speed 126.84 samples/sec Loss 11.7683 Epoch: 4 Global Step: 41210 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:21:04,856-Speed 125.71 samples/sec Loss 11.7525 Epoch: 4 Global Step: 41215 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:21:25,112-Speed 126.39 samples/sec Loss 11.5094 Epoch: 4 Global Step: 41220 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:21:45,853-Speed 123.43 samples/sec Loss 11.6583 Epoch: 4 Global Step: 41225 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:22:06,130-Speed 126.26 samples/sec Loss 11.2323 Epoch: 4 Global Step: 41230 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:22:26,400-Speed 126.30 samples/sec Loss 11.4564 Epoch: 4 Global Step: 41235 Fp16 Grad Scale: 8192 Required: 249 hours Training: 2021-06-08 20:22:46,568-Speed 126.94 samples/sec Loss 12.0009 Epoch: 4 Global Step: 41240 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:23:06,747-Speed 126.87 samples/sec Loss 11.6621 Epoch: 4 Global Step: 41245 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:23:26,714-Speed 128.22 samples/sec Loss 11.7884 Epoch: 4 Global Step: 41250 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:23:46,871-Speed 127.01 samples/sec Loss 11.8480 Epoch: 4 Global Step: 41255 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:24:06,954-Speed 127.48 samples/sec Loss 11.5265 Epoch: 4 Global Step: 41260 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:24:27,071-Speed 127.26 samples/sec Loss 11.5798 Epoch: 4 Global Step: 41265 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:24:47,935-Speed 122.70 samples/sec Loss 11.5871 Epoch: 4 Global Step: 41270 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:25:08,108-Speed 126.90 samples/sec Loss 11.4850 Epoch: 4 Global Step: 41275 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:25:28,386-Speed 126.25 samples/sec Loss 11.5987 Epoch: 4 Global Step: 41280 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:25:48,571-Speed 126.83 samples/sec Loss 11.5094 Epoch: 4 Global Step: 41285 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:26:08,767-Speed 126.76 samples/sec Loss 11.3512 Epoch: 4 Global Step: 41290 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:26:28,797-Speed 127.82 samples/sec Loss 11.4378 Epoch: 4 Global Step: 41295 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:26:48,975-Speed 126.88 samples/sec Loss 11.1297 Epoch: 4 Global Step: 41300 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:27:09,229-Speed 126.40 samples/sec Loss 11.3033 Epoch: 4 Global Step: 41305 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:27:29,522-Speed 126.16 samples/sec Loss 11.2413 Epoch: 4 Global Step: 41310 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:27:50,543-Speed 121.79 samples/sec Loss 11.5233 Epoch: 4 Global Step: 41315 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:28:10,789-Speed 126.45 samples/sec Loss 11.6437 Epoch: 4 Global Step: 41320 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:28:31,045-Speed 126.39 samples/sec Loss 11.3361 Epoch: 4 Global Step: 41325 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:28:51,209-Speed 126.97 samples/sec Loss 11.4074 Epoch: 4 Global Step: 41330 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:29:11,272-Speed 127.60 samples/sec Loss 11.3239 Epoch: 4 Global Step: 41335 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:29:31,501-Speed 126.56 samples/sec Loss 11.5065 Epoch: 4 Global Step: 41340 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:29:51,648-Speed 127.08 samples/sec Loss 11.2945 Epoch: 4 Global Step: 41345 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:30:11,878-Speed 126.55 samples/sec Loss 11.5669 Epoch: 4 Global Step: 41350 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:30:32,244-Speed 125.71 samples/sec Loss 11.3526 Epoch: 4 Global Step: 41355 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:30:53,238-Speed 121.95 samples/sec Loss 11.0213 Epoch: 4 Global Step: 41360 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:31:13,483-Speed 126.46 samples/sec Loss 11.2348 Epoch: 4 Global Step: 41365 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:31:33,932-Speed 125.19 samples/sec Loss 11.3523 Epoch: 4 Global Step: 41370 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:31:54,874-Speed 122.25 samples/sec Loss 11.3645 Epoch: 4 Global Step: 41375 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:32:15,156-Speed 126.23 samples/sec Loss 11.5770 Epoch: 4 Global Step: 41380 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:32:35,428-Speed 126.29 samples/sec Loss 11.7557 Epoch: 4 Global Step: 41385 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:32:56,149-Speed 123.55 samples/sec Loss 11.3217 Epoch: 4 Global Step: 41390 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:33:16,391-Speed 126.48 samples/sec Loss 11.4533 Epoch: 4 Global Step: 41395 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:33:36,694-Speed 126.10 samples/sec Loss 11.6166 Epoch: 4 Global Step: 41400 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:33:57,653-Speed 122.15 samples/sec Loss 11.0657 Epoch: 4 Global Step: 41405 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:34:17,852-Speed 126.75 samples/sec Loss 11.5889 Epoch: 4 Global Step: 41410 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:34:38,293-Speed 125.24 samples/sec Loss 11.5840 Epoch: 4 Global Step: 41415 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:34:58,579-Speed 126.20 samples/sec Loss 11.5926 Epoch: 4 Global Step: 41420 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:35:18,858-Speed 126.25 samples/sec Loss 11.8820 Epoch: 4 Global Step: 41425 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:35:39,123-Speed 126.33 samples/sec Loss 11.6212 Epoch: 4 Global Step: 41430 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:35:59,344-Speed 126.62 samples/sec Loss 11.3713 Epoch: 4 Global Step: 41435 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:36:19,909-Speed 124.49 samples/sec Loss 11.4514 Epoch: 4 Global Step: 41440 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:36:40,126-Speed 126.64 samples/sec Loss 11.4975 Epoch: 4 Global Step: 41445 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:37:01,039-Speed 122.41 samples/sec Loss 11.1101 Epoch: 4 Global Step: 41450 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:37:21,282-Speed 126.47 samples/sec Loss 11.3742 Epoch: 4 Global Step: 41455 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:37:41,314-Speed 127.80 samples/sec Loss 11.2291 Epoch: 4 Global Step: 41460 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:38:01,531-Speed 126.63 samples/sec Loss 11.4161 Epoch: 4 Global Step: 41465 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:38:21,692-Speed 126.99 samples/sec Loss 11.6192 Epoch: 4 Global Step: 41470 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:38:42,150-Speed 125.14 samples/sec Loss 11.1440 Epoch: 4 Global Step: 41475 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:39:02,230-Speed 127.49 samples/sec Loss 11.4769 Epoch: 4 Global Step: 41480 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:39:22,391-Speed 126.98 samples/sec Loss 11.9535 Epoch: 4 Global Step: 41485 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:39:42,685-Speed 126.15 samples/sec Loss 11.4284 Epoch: 4 Global Step: 41490 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:40:03,414-Speed 123.50 samples/sec Loss 11.6063 Epoch: 4 Global Step: 41495 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:40:23,561-Speed 127.07 samples/sec Loss 11.3935 Epoch: 4 Global Step: 41500 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:40:43,715-Speed 127.03 samples/sec Loss 11.4795 Epoch: 4 Global Step: 41505 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:41:03,875-Speed 126.99 samples/sec Loss 11.2234 Epoch: 4 Global Step: 41510 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:41:24,069-Speed 126.78 samples/sec Loss 11.2663 Epoch: 4 Global Step: 41515 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:41:44,251-Speed 126.85 samples/sec Loss 11.7122 Epoch: 4 Global Step: 41520 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:42:04,579-Speed 125.95 samples/sec Loss 10.9819 Epoch: 4 Global Step: 41525 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:42:24,959-Speed 125.62 samples/sec Loss 11.2860 Epoch: 4 Global Step: 41530 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:42:45,140-Speed 126.86 samples/sec Loss 11.4034 Epoch: 4 Global Step: 41535 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:43:06,076-Speed 122.28 samples/sec Loss 11.7072 Epoch: 4 Global Step: 41540 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:43:26,250-Speed 126.90 samples/sec Loss 11.3794 Epoch: 4 Global Step: 41545 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:43:46,459-Speed 126.68 samples/sec Loss 11.1923 Epoch: 4 Global Step: 41550 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:44:06,474-Speed 127.91 samples/sec Loss 11.2405 Epoch: 4 Global Step: 41555 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:44:26,647-Speed 126.91 samples/sec Loss 11.3616 Epoch: 4 Global Step: 41560 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:44:46,690-Speed 127.73 samples/sec Loss 11.8987 Epoch: 4 Global Step: 41565 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:45:06,891-Speed 126.73 samples/sec Loss 11.7131 Epoch: 4 Global Step: 41570 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:45:26,901-Speed 127.94 samples/sec Loss 11.5021 Epoch: 4 Global Step: 41575 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:45:47,147-Speed 126.45 samples/sec Loss 11.5812 Epoch: 4 Global Step: 41580 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:46:08,241-Speed 121.37 samples/sec Loss 11.4404 Epoch: 4 Global Step: 41585 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:46:28,447-Speed 126.70 samples/sec Loss 11.2909 Epoch: 4 Global Step: 41590 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:46:48,747-Speed 126.11 samples/sec Loss 11.6961 Epoch: 4 Global Step: 41595 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:47:08,942-Speed 126.78 samples/sec Loss 11.3608 Epoch: 4 Global Step: 41600 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:47:29,191-Speed 126.43 samples/sec Loss 11.4628 Epoch: 4 Global Step: 41605 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:47:49,513-Speed 125.98 samples/sec Loss 11.3390 Epoch: 4 Global Step: 41610 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:48:10,639-Speed 121.18 samples/sec Loss 11.9548 Epoch: 4 Global Step: 41615 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:48:30,853-Speed 126.65 samples/sec Loss 11.2520 Epoch: 4 Global Step: 41620 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:48:50,899-Speed 127.72 samples/sec Loss 11.5797 Epoch: 4 Global Step: 41625 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:49:11,996-Speed 121.35 samples/sec Loss 12.0230 Epoch: 4 Global Step: 41630 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:49:32,181-Speed 126.83 samples/sec Loss 11.8291 Epoch: 4 Global Step: 41635 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:49:52,437-Speed 126.39 samples/sec Loss 11.4977 Epoch: 4 Global Step: 41640 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:50:12,685-Speed 126.44 samples/sec Loss 11.4016 Epoch: 4 Global Step: 41645 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:50:32,815-Speed 127.18 samples/sec Loss 11.5891 Epoch: 4 Global Step: 41650 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:50:53,057-Speed 126.47 samples/sec Loss 11.4478 Epoch: 4 Global Step: 41655 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:51:13,244-Speed 126.83 samples/sec Loss 11.4411 Epoch: 4 Global Step: 41660 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:51:33,302-Speed 127.63 samples/sec Loss 11.5563 Epoch: 4 Global Step: 41665 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:51:53,610-Speed 126.06 samples/sec Loss 11.3482 Epoch: 4 Global Step: 41670 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:52:14,570-Speed 122.14 samples/sec Loss 11.5256 Epoch: 4 Global Step: 41675 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:52:34,750-Speed 126.87 samples/sec Loss 11.6123 Epoch: 4 Global Step: 41680 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:52:54,984-Speed 126.53 samples/sec Loss 11.7496 Epoch: 4 Global Step: 41685 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:53:15,308-Speed 125.97 samples/sec Loss 11.6213 Epoch: 4 Global Step: 41690 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:53:35,490-Speed 126.85 samples/sec Loss 11.7156 Epoch: 4 Global Step: 41695 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:53:55,677-Speed 126.82 samples/sec Loss 11.1379 Epoch: 4 Global Step: 41700 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:54:16,175-Speed 124.90 samples/sec Loss 11.6149 Epoch: 4 Global Step: 41705 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:54:36,278-Speed 127.35 samples/sec Loss 11.3863 Epoch: 4 Global Step: 41710 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:54:56,470-Speed 126.79 samples/sec Loss 11.1387 Epoch: 4 Global Step: 41715 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:55:17,476-Speed 121.87 samples/sec Loss 11.3220 Epoch: 4 Global Step: 41720 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:55:37,763-Speed 126.20 samples/sec Loss 11.3553 Epoch: 4 Global Step: 41725 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:55:58,045-Speed 126.22 samples/sec Loss 11.8739 Epoch: 4 Global Step: 41730 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:56:18,298-Speed 126.41 samples/sec Loss 11.4229 Epoch: 4 Global Step: 41735 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:56:38,313-Speed 127.91 samples/sec Loss 11.6796 Epoch: 4 Global Step: 41740 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:56:58,516-Speed 126.72 samples/sec Loss 11.3722 Epoch: 4 Global Step: 41745 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:57:18,771-Speed 126.40 samples/sec Loss 11.4589 Epoch: 4 Global Step: 41750 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:57:39,043-Speed 126.29 samples/sec Loss 11.3350 Epoch: 4 Global Step: 41755 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:57:59,353-Speed 126.05 samples/sec Loss 11.2222 Epoch: 4 Global Step: 41760 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:58:20,403-Speed 121.62 samples/sec Loss 11.5568 Epoch: 4 Global Step: 41765 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:58:40,682-Speed 126.24 samples/sec Loss 11.3090 Epoch: 4 Global Step: 41770 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:59:00,973-Speed 126.18 samples/sec Loss 11.5954 Epoch: 4 Global Step: 41775 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:59:21,196-Speed 126.59 samples/sec Loss 11.4771 Epoch: 4 Global Step: 41780 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 20:59:41,432-Speed 126.52 samples/sec Loss 11.3821 Epoch: 4 Global Step: 41785 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:00:01,416-Speed 128.11 samples/sec Loss 11.7192 Epoch: 4 Global Step: 41790 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:00:21,678-Speed 126.36 samples/sec Loss 11.6318 Epoch: 4 Global Step: 41795 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:00:41,898-Speed 126.61 samples/sec Loss 11.0134 Epoch: 4 Global Step: 41800 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:01:02,091-Speed 126.79 samples/sec Loss 11.3818 Epoch: 4 Global Step: 41805 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:01:22,647-Speed 124.54 samples/sec Loss 11.0547 Epoch: 4 Global Step: 41810 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:01:43,228-Speed 124.39 samples/sec Loss 11.4628 Epoch: 4 Global Step: 41815 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:02:03,512-Speed 126.22 samples/sec Loss 11.3269 Epoch: 4 Global Step: 41820 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:02:23,808-Speed 126.14 samples/sec Loss 11.2387 Epoch: 4 Global Step: 41825 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:02:43,972-Speed 126.97 samples/sec Loss 10.9616 Epoch: 4 Global Step: 41830 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:03:04,288-Speed 126.02 samples/sec Loss 11.6598 Epoch: 4 Global Step: 41835 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:03:24,598-Speed 126.05 samples/sec Loss 11.3258 Epoch: 4 Global Step: 41840 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:03:44,870-Speed 126.29 samples/sec Loss 11.4703 Epoch: 4 Global Step: 41845 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:04:05,102-Speed 126.54 samples/sec Loss 11.4281 Epoch: 4 Global Step: 41850 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:04:26,152-Speed 121.62 samples/sec Loss 11.4165 Epoch: 4 Global Step: 41855 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:04:46,316-Speed 126.96 samples/sec Loss 11.2936 Epoch: 4 Global Step: 41860 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:05:06,552-Speed 126.51 samples/sec Loss 11.5745 Epoch: 4 Global Step: 41865 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:05:26,665-Speed 127.29 samples/sec Loss 11.4786 Epoch: 4 Global Step: 41870 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:05:46,742-Speed 127.51 samples/sec Loss 11.6648 Epoch: 4 Global Step: 41875 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:06:06,945-Speed 126.72 samples/sec Loss 11.7212 Epoch: 4 Global Step: 41880 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:06:26,957-Speed 127.93 samples/sec Loss 11.3018 Epoch: 4 Global Step: 41885 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:06:47,042-Speed 127.46 samples/sec Loss 11.1789 Epoch: 4 Global Step: 41890 Fp16 Grad Scale: 8192 Required: 248 hours Training: 2021-06-08 21:07:07,233-Speed 126.80 samples/sec Loss 11.6636 Epoch: 4 Global Step: 41895 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:07:28,023-Speed 123.14 samples/sec Loss 11.2900 Epoch: 4 Global Step: 41900 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:07:48,102-Speed 127.50 samples/sec Loss 11.4028 Epoch: 4 Global Step: 41905 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:08:08,191-Speed 127.44 samples/sec Loss 11.2129 Epoch: 4 Global Step: 41910 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:08:28,780-Speed 124.35 samples/sec Loss 11.6879 Epoch: 4 Global Step: 41915 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:08:49,082-Speed 126.10 samples/sec Loss 11.4086 Epoch: 4 Global Step: 41920 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:09:09,383-Speed 126.10 samples/sec Loss 11.7617 Epoch: 4 Global Step: 41925 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:09:29,847-Speed 125.11 samples/sec Loss 11.3023 Epoch: 4 Global Step: 41930 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:09:50,062-Speed 126.65 samples/sec Loss 11.4493 Epoch: 4 Global Step: 41935 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:10:10,270-Speed 126.70 samples/sec Loss 11.7407 Epoch: 4 Global Step: 41940 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:10:31,313-Speed 121.66 samples/sec Loss 11.4182 Epoch: 4 Global Step: 41945 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:10:51,463-Speed 127.06 samples/sec Loss 11.5093 Epoch: 4 Global Step: 41950 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:11:11,629-Speed 126.95 samples/sec Loss 11.7876 Epoch: 4 Global Step: 41955 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:11:31,853-Speed 126.60 samples/sec Loss 11.5212 Epoch: 4 Global Step: 41960 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:11:52,029-Speed 126.89 samples/sec Loss 11.3053 Epoch: 4 Global Step: 41965 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:12:12,187-Speed 127.00 samples/sec Loss 11.6355 Epoch: 4 Global Step: 41970 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:12:32,548-Speed 125.74 samples/sec Loss 11.4453 Epoch: 4 Global Step: 41975 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:12:52,914-Speed 125.70 samples/sec Loss 11.5370 Epoch: 4 Global Step: 41980 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:13:13,259-Speed 125.84 samples/sec Loss 11.3199 Epoch: 4 Global Step: 41985 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:13:34,249-Speed 121.97 samples/sec Loss 11.4865 Epoch: 4 Global Step: 41990 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:13:54,481-Speed 126.54 samples/sec Loss 11.5482 Epoch: 4 Global Step: 41995 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:14:14,659-Speed 126.88 samples/sec Loss 11.5838 Epoch: 4 Global Step: 42000 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:14:34,820-Speed 126.99 samples/sec Loss 11.3510 Epoch: 4 Global Step: 42005 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:14:55,105-Speed 126.20 samples/sec Loss 11.8322 Epoch: 4 Global Step: 42010 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:15:15,473-Speed 125.70 samples/sec Loss 11.7914 Epoch: 4 Global Step: 42015 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:15:35,709-Speed 126.52 samples/sec Loss 11.4506 Epoch: 4 Global Step: 42020 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:15:55,956-Speed 126.45 samples/sec Loss 11.4573 Epoch: 4 Global Step: 42025 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:16:16,160-Speed 126.72 samples/sec Loss 11.4049 Epoch: 4 Global Step: 42030 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:16:37,060-Speed 122.49 samples/sec Loss 11.5267 Epoch: 4 Global Step: 42035 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:16:57,128-Speed 127.57 samples/sec Loss 11.4805 Epoch: 4 Global Step: 42040 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:17:17,252-Speed 127.22 samples/sec Loss 11.4194 Epoch: 4 Global Step: 42045 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:17:37,328-Speed 127.52 samples/sec Loss 11.2678 Epoch: 4 Global Step: 42050 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:17:57,864-Speed 124.67 samples/sec Loss 11.2927 Epoch: 4 Global Step: 42055 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:18:17,893-Speed 127.82 samples/sec Loss 11.3782 Epoch: 4 Global Step: 42060 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:18:38,106-Speed 126.66 samples/sec Loss 11.2791 Epoch: 4 Global Step: 42065 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:18:58,068-Speed 128.25 samples/sec Loss 11.2301 Epoch: 4 Global Step: 42070 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:19:18,078-Speed 127.94 samples/sec Loss 11.3429 Epoch: 4 Global Step: 42075 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:19:38,706-Speed 124.11 samples/sec Loss 11.5220 Epoch: 4 Global Step: 42080 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:19:58,943-Speed 126.50 samples/sec Loss 11.5755 Epoch: 4 Global Step: 42085 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:20:18,903-Speed 128.27 samples/sec Loss 11.6120 Epoch: 4 Global Step: 42090 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:20:39,210-Speed 126.07 samples/sec Loss 11.3807 Epoch: 4 Global Step: 42095 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:20:59,297-Speed 127.45 samples/sec Loss 11.6949 Epoch: 4 Global Step: 42100 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:21:19,485-Speed 126.82 samples/sec Loss 11.5094 Epoch: 4 Global Step: 42105 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:21:39,738-Speed 126.41 samples/sec Loss 11.7528 Epoch: 4 Global Step: 42110 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:22:00,031-Speed 126.16 samples/sec Loss 11.6228 Epoch: 4 Global Step: 42115 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:22:20,281-Speed 126.45 samples/sec Loss 11.2701 Epoch: 4 Global Step: 42120 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:22:41,235-Speed 122.18 samples/sec Loss 11.4807 Epoch: 4 Global Step: 42125 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:23:01,456-Speed 126.61 samples/sec Loss 11.3380 Epoch: 4 Global Step: 42130 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:23:21,681-Speed 126.58 samples/sec Loss 11.4922 Epoch: 4 Global Step: 42135 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:23:41,960-Speed 126.25 samples/sec Loss 11.5956 Epoch: 4 Global Step: 42140 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:24:02,168-Speed 126.69 samples/sec Loss 11.7651 Epoch: 4 Global Step: 42145 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:24:22,422-Speed 126.40 samples/sec Loss 11.4489 Epoch: 4 Global Step: 42150 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:24:42,727-Speed 126.08 samples/sec Loss 11.6914 Epoch: 4 Global Step: 42155 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:25:02,943-Speed 126.64 samples/sec Loss 11.4840 Epoch: 4 Global Step: 42160 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:25:23,139-Speed 126.77 samples/sec Loss 11.5725 Epoch: 4 Global Step: 42165 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:25:44,000-Speed 122.72 samples/sec Loss 11.4873 Epoch: 4 Global Step: 42170 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:26:04,450-Speed 125.19 samples/sec Loss 11.5142 Epoch: 4 Global Step: 42175 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:26:24,851-Speed 125.49 samples/sec Loss 11.3406 Epoch: 4 Global Step: 42180 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:26:45,074-Speed 126.59 samples/sec Loss 11.2842 Epoch: 4 Global Step: 42185 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:27:05,286-Speed 126.66 samples/sec Loss 11.2663 Epoch: 4 Global Step: 42190 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:27:25,505-Speed 126.62 samples/sec Loss 11.7798 Epoch: 4 Global Step: 42195 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:27:45,714-Speed 126.68 samples/sec Loss 11.5322 Epoch: 4 Global Step: 42200 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:28:06,010-Speed 126.14 samples/sec Loss 11.3774 Epoch: 4 Global Step: 42205 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:28:26,055-Speed 127.72 samples/sec Loss 11.3194 Epoch: 4 Global Step: 42210 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:28:46,917-Speed 122.72 samples/sec Loss 11.5083 Epoch: 4 Global Step: 42215 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:29:07,107-Speed 126.80 samples/sec Loss 11.4859 Epoch: 4 Global Step: 42220 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:29:27,261-Speed 127.03 samples/sec Loss 11.5501 Epoch: 4 Global Step: 42225 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:29:47,212-Speed 128.32 samples/sec Loss 11.9554 Epoch: 4 Global Step: 42230 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:30:07,627-Speed 125.40 samples/sec Loss 11.4675 Epoch: 4 Global Step: 42235 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:30:28,099-Speed 125.06 samples/sec Loss 11.4724 Epoch: 4 Global Step: 42240 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:30:48,151-Speed 127.68 samples/sec Loss 11.4843 Epoch: 4 Global Step: 42245 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:31:08,304-Speed 127.04 samples/sec Loss 11.4462 Epoch: 4 Global Step: 42250 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:31:28,506-Speed 126.73 samples/sec Loss 11.2741 Epoch: 4 Global Step: 42255 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:31:49,457-Speed 122.19 samples/sec Loss 11.4307 Epoch: 4 Global Step: 42260 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:32:09,649-Speed 126.79 samples/sec Loss 11.3052 Epoch: 4 Global Step: 42265 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:32:29,811-Speed 126.98 samples/sec Loss 11.6877 Epoch: 4 Global Step: 42270 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:32:50,087-Speed 126.26 samples/sec Loss 11.7712 Epoch: 4 Global Step: 42275 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:33:10,352-Speed 126.34 samples/sec Loss 11.3461 Epoch: 4 Global Step: 42280 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:33:30,505-Speed 127.04 samples/sec Loss 11.5815 Epoch: 4 Global Step: 42285 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:33:50,766-Speed 126.36 samples/sec Loss 11.3190 Epoch: 4 Global Step: 42290 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:34:11,042-Speed 126.27 samples/sec Loss 11.3567 Epoch: 4 Global Step: 42295 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:34:31,401-Speed 125.75 samples/sec Loss 11.9642 Epoch: 4 Global Step: 42300 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:34:52,441-Speed 121.68 samples/sec Loss 11.7511 Epoch: 4 Global Step: 42305 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:35:12,614-Speed 126.91 samples/sec Loss 11.6112 Epoch: 4 Global Step: 42310 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:35:32,768-Speed 127.03 samples/sec Loss 11.6506 Epoch: 4 Global Step: 42315 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:35:53,082-Speed 126.03 samples/sec Loss 11.3016 Epoch: 4 Global Step: 42320 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:36:13,322-Speed 126.49 samples/sec Loss 11.6958 Epoch: 4 Global Step: 42325 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:36:33,493-Speed 126.92 samples/sec Loss 11.6916 Epoch: 4 Global Step: 42330 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:36:53,682-Speed 126.81 samples/sec Loss 11.7498 Epoch: 4 Global Step: 42335 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:37:13,913-Speed 126.56 samples/sec Loss 11.2478 Epoch: 4 Global Step: 42340 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:37:34,135-Speed 126.60 samples/sec Loss 11.6410 Epoch: 4 Global Step: 42345 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:37:54,821-Speed 123.76 samples/sec Loss 11.5552 Epoch: 4 Global Step: 42350 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:38:14,999-Speed 126.88 samples/sec Loss 11.6192 Epoch: 4 Global Step: 42355 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:38:35,300-Speed 126.11 samples/sec Loss 11.9852 Epoch: 4 Global Step: 42360 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:38:55,460-Speed 126.99 samples/sec Loss 11.4563 Epoch: 4 Global Step: 42365 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:39:15,763-Speed 126.09 samples/sec Loss 11.4598 Epoch: 4 Global Step: 42370 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:39:35,941-Speed 126.88 samples/sec Loss 11.3412 Epoch: 4 Global Step: 42375 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:39:56,098-Speed 127.01 samples/sec Loss 11.2175 Epoch: 4 Global Step: 42380 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:40:16,326-Speed 126.57 samples/sec Loss 11.1863 Epoch: 4 Global Step: 42385 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:40:36,657-Speed 125.92 samples/sec Loss 11.5330 Epoch: 4 Global Step: 42390 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:40:57,706-Speed 121.63 samples/sec Loss 11.7713 Epoch: 4 Global Step: 42395 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:41:18,099-Speed 125.54 samples/sec Loss 11.6521 Epoch: 4 Global Step: 42400 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:41:38,077-Speed 128.15 samples/sec Loss 11.7223 Epoch: 4 Global Step: 42405 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:41:58,262-Speed 126.83 samples/sec Loss 11.5646 Epoch: 4 Global Step: 42410 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:42:18,469-Speed 126.69 samples/sec Loss 11.5912 Epoch: 4 Global Step: 42415 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:42:38,760-Speed 126.17 samples/sec Loss 11.5340 Epoch: 4 Global Step: 42420 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:42:58,989-Speed 126.56 samples/sec Loss 11.5277 Epoch: 4 Global Step: 42425 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:43:19,186-Speed 126.76 samples/sec Loss 11.5183 Epoch: 4 Global Step: 42430 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:43:40,027-Speed 122.84 samples/sec Loss 11.4155 Epoch: 4 Global Step: 42435 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:44:01,042-Speed 121.82 samples/sec Loss 11.4711 Epoch: 4 Global Step: 42440 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:44:21,350-Speed 126.06 samples/sec Loss 11.5602 Epoch: 4 Global Step: 42445 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:44:41,612-Speed 126.35 samples/sec Loss 11.1472 Epoch: 4 Global Step: 42450 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:45:01,981-Speed 125.69 samples/sec Loss 11.5570 Epoch: 4 Global Step: 42455 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:45:22,283-Speed 126.10 samples/sec Loss 11.7785 Epoch: 4 Global Step: 42460 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:45:42,617-Speed 125.91 samples/sec Loss 11.5494 Epoch: 4 Global Step: 42465 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:46:02,854-Speed 126.51 samples/sec Loss 11.5014 Epoch: 4 Global Step: 42470 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:46:23,069-Speed 126.64 samples/sec Loss 11.3520 Epoch: 4 Global Step: 42475 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:46:43,247-Speed 126.88 samples/sec Loss 11.3702 Epoch: 4 Global Step: 42480 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:47:03,948-Speed 123.67 samples/sec Loss 11.3639 Epoch: 4 Global Step: 42485 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:47:24,150-Speed 126.73 samples/sec Loss 11.1436 Epoch: 4 Global Step: 42490 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:47:44,269-Speed 127.24 samples/sec Loss 11.3737 Epoch: 4 Global Step: 42495 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:48:04,554-Speed 126.21 samples/sec Loss 11.3825 Epoch: 4 Global Step: 42500 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:48:24,936-Speed 125.61 samples/sec Loss 11.6460 Epoch: 4 Global Step: 42505 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:48:45,072-Speed 127.14 samples/sec Loss 11.6328 Epoch: 4 Global Step: 42510 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:49:05,394-Speed 125.98 samples/sec Loss 11.4435 Epoch: 4 Global Step: 42515 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:49:25,605-Speed 126.67 samples/sec Loss 11.1983 Epoch: 4 Global Step: 42520 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:49:45,662-Speed 127.64 samples/sec Loss 11.3493 Epoch: 4 Global Step: 42525 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:50:06,577-Speed 122.41 samples/sec Loss 11.5556 Epoch: 4 Global Step: 42530 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:50:26,857-Speed 126.24 samples/sec Loss 11.0702 Epoch: 4 Global Step: 42535 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:50:47,141-Speed 126.21 samples/sec Loss 11.1970 Epoch: 4 Global Step: 42540 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:51:07,320-Speed 126.87 samples/sec Loss 11.3735 Epoch: 4 Global Step: 42545 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:51:27,488-Speed 126.94 samples/sec Loss 11.2375 Epoch: 4 Global Step: 42550 Fp16 Grad Scale: 8192 Required: 247 hours Training: 2021-06-08 21:51:47,721-Speed 126.53 samples/sec Loss 11.7148 Epoch: 4 Global Step: 42555 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:52:07,978-Speed 126.39 samples/sec Loss 11.3826 Epoch: 4 Global Step: 42560 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:52:28,216-Speed 126.50 samples/sec Loss 11.5480 Epoch: 4 Global Step: 42565 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:52:48,423-Speed 126.72 samples/sec Loss 11.5228 Epoch: 4 Global Step: 42570 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:53:09,410-Speed 121.99 samples/sec Loss 11.2661 Epoch: 4 Global Step: 42575 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:53:29,609-Speed 126.74 samples/sec Loss 11.2429 Epoch: 4 Global Step: 42580 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:53:49,571-Speed 128.25 samples/sec Loss 11.6122 Epoch: 4 Global Step: 42585 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:54:09,902-Speed 125.92 samples/sec Loss 11.6505 Epoch: 4 Global Step: 42590 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:54:30,164-Speed 126.35 samples/sec Loss 11.5864 Epoch: 4 Global Step: 42595 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:54:50,212-Speed 127.70 samples/sec Loss 11.3409 Epoch: 4 Global Step: 42600 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:55:10,423-Speed 126.67 samples/sec Loss 11.5720 Epoch: 4 Global Step: 42605 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:55:30,590-Speed 126.95 samples/sec Loss 10.7880 Epoch: 4 Global Step: 42610 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:55:51,349-Speed 123.32 samples/sec Loss 11.3078 Epoch: 4 Global Step: 42615 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:56:12,568-Speed 120.65 samples/sec Loss 11.4528 Epoch: 4 Global Step: 42620 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:56:32,793-Speed 126.58 samples/sec Loss 11.7545 Epoch: 4 Global Step: 42625 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:56:52,801-Speed 127.96 samples/sec Loss 11.6825 Epoch: 4 Global Step: 42630 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:57:12,950-Speed 127.06 samples/sec Loss 11.4854 Epoch: 4 Global Step: 42635 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:57:33,122-Speed 126.92 samples/sec Loss 11.4670 Epoch: 4 Global Step: 42640 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:57:53,855-Speed 123.49 samples/sec Loss 11.5150 Epoch: 4 Global Step: 42645 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:58:14,024-Speed 126.93 samples/sec Loss 11.4971 Epoch: 4 Global Step: 42650 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:58:34,039-Speed 127.91 samples/sec Loss 11.5230 Epoch: 4 Global Step: 42655 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:58:54,847-Speed 123.04 samples/sec Loss 11.4111 Epoch: 4 Global Step: 42660 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:59:15,503-Speed 123.94 samples/sec Loss 11.4792 Epoch: 4 Global Step: 42665 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:59:35,687-Speed 126.84 samples/sec Loss 11.4502 Epoch: 4 Global Step: 42670 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 21:59:55,982-Speed 126.14 samples/sec Loss 11.4357 Epoch: 4 Global Step: 42675 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:00:16,069-Speed 127.46 samples/sec Loss 11.1893 Epoch: 4 Global Step: 42680 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:00:36,237-Speed 126.94 samples/sec Loss 12.0652 Epoch: 4 Global Step: 42685 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:00:56,382-Speed 127.09 samples/sec Loss 11.3708 Epoch: 4 Global Step: 42690 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:01:16,646-Speed 126.34 samples/sec Loss 11.5490 Epoch: 4 Global Step: 42695 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:01:36,888-Speed 126.48 samples/sec Loss 11.5490 Epoch: 4 Global Step: 42700 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:01:57,071-Speed 126.87 samples/sec Loss 11.4612 Epoch: 4 Global Step: 42705 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:02:17,798-Speed 123.52 samples/sec Loss 11.8852 Epoch: 4 Global Step: 42710 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:02:38,062-Speed 126.34 samples/sec Loss 11.4364 Epoch: 4 Global Step: 42715 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:02:58,270-Speed 126.69 samples/sec Loss 11.4755 Epoch: 4 Global Step: 42720 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:03:18,504-Speed 126.53 samples/sec Loss 11.1160 Epoch: 4 Global Step: 42725 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:03:38,758-Speed 126.40 samples/sec Loss 11.5970 Epoch: 4 Global Step: 42730 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:03:58,943-Speed 126.84 samples/sec Loss 11.3590 Epoch: 4 Global Step: 42735 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:04:19,166-Speed 126.60 samples/sec Loss 11.3937 Epoch: 4 Global Step: 42740 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:04:39,360-Speed 126.78 samples/sec Loss 11.3392 Epoch: 4 Global Step: 42745 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:04:59,518-Speed 127.01 samples/sec Loss 11.5133 Epoch: 4 Global Step: 42750 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:05:20,409-Speed 122.55 samples/sec Loss 11.0893 Epoch: 4 Global Step: 42755 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:05:40,607-Speed 126.75 samples/sec Loss 11.6307 Epoch: 4 Global Step: 42760 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:06:00,848-Speed 126.48 samples/sec Loss 11.5663 Epoch: 4 Global Step: 42765 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:06:21,057-Speed 126.69 samples/sec Loss 11.1336 Epoch: 4 Global Step: 42770 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:06:41,327-Speed 126.30 samples/sec Loss 11.6784 Epoch: 4 Global Step: 42775 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:07:01,550-Speed 126.60 samples/sec Loss 11.3413 Epoch: 4 Global Step: 42780 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:07:21,712-Speed 126.98 samples/sec Loss 11.5688 Epoch: 4 Global Step: 42785 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:07:41,872-Speed 126.99 samples/sec Loss 11.8233 Epoch: 4 Global Step: 42790 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:08:01,865-Speed 128.05 samples/sec Loss 11.6528 Epoch: 4 Global Step: 42795 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:08:22,751-Speed 122.58 samples/sec Loss 11.3923 Epoch: 4 Global Step: 42800 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:08:42,967-Speed 126.63 samples/sec Loss 11.1383 Epoch: 4 Global Step: 42805 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:09:03,145-Speed 126.88 samples/sec Loss 11.5594 Epoch: 4 Global Step: 42810 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:09:23,367-Speed 126.60 samples/sec Loss 11.8456 Epoch: 4 Global Step: 42815 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:09:43,569-Speed 126.73 samples/sec Loss 11.4345 Epoch: 4 Global Step: 42820 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:10:03,773-Speed 126.71 samples/sec Loss 11.6151 Epoch: 4 Global Step: 42825 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:10:24,080-Speed 126.07 samples/sec Loss 11.3702 Epoch: 4 Global Step: 42830 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:10:44,322-Speed 126.48 samples/sec Loss 11.5691 Epoch: 4 Global Step: 42835 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:11:04,554-Speed 126.54 samples/sec Loss 11.6162 Epoch: 4 Global Step: 42840 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:11:25,543-Speed 121.97 samples/sec Loss 11.5793 Epoch: 4 Global Step: 42845 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:11:45,830-Speed 126.19 samples/sec Loss 11.5388 Epoch: 4 Global Step: 42850 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:12:06,077-Speed 126.45 samples/sec Loss 11.4227 Epoch: 4 Global Step: 42855 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:12:26,140-Speed 127.60 samples/sec Loss 11.1388 Epoch: 4 Global Step: 42860 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:12:46,254-Speed 127.28 samples/sec Loss 11.6056 Epoch: 4 Global Step: 42865 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:13:06,412-Speed 127.01 samples/sec Loss 11.4246 Epoch: 4 Global Step: 42870 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:13:26,449-Speed 127.77 samples/sec Loss 11.4026 Epoch: 4 Global Step: 42875 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:13:46,626-Speed 126.88 samples/sec Loss 11.4158 Epoch: 4 Global Step: 42880 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:14:06,861-Speed 126.52 samples/sec Loss 11.3715 Epoch: 4 Global Step: 42885 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:14:27,881-Speed 121.79 samples/sec Loss 11.4741 Epoch: 4 Global Step: 42890 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:14:48,025-Speed 127.10 samples/sec Loss 11.6098 Epoch: 4 Global Step: 42895 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:15:08,264-Speed 126.50 samples/sec Loss 11.6645 Epoch: 4 Global Step: 42900 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:15:28,542-Speed 126.26 samples/sec Loss 11.6528 Epoch: 4 Global Step: 42905 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:15:48,519-Speed 128.15 samples/sec Loss 11.1304 Epoch: 4 Global Step: 42910 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:16:08,649-Speed 127.18 samples/sec Loss 11.4113 Epoch: 4 Global Step: 42915 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:16:28,844-Speed 126.77 samples/sec Loss 11.2668 Epoch: 4 Global Step: 42920 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:16:49,502-Speed 123.93 samples/sec Loss 11.6135 Epoch: 4 Global Step: 42925 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:17:09,491-Speed 128.08 samples/sec Loss 11.4469 Epoch: 4 Global Step: 42930 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:17:30,395-Speed 122.47 samples/sec Loss 11.3338 Epoch: 4 Global Step: 42935 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:17:51,249-Speed 122.77 samples/sec Loss 11.4816 Epoch: 4 Global Step: 42940 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:18:11,375-Speed 127.20 samples/sec Loss 11.1503 Epoch: 4 Global Step: 42945 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:18:31,661-Speed 126.20 samples/sec Loss 11.3587 Epoch: 4 Global Step: 42950 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:18:52,027-Speed 125.71 samples/sec Loss 11.3519 Epoch: 4 Global Step: 42955 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:19:12,482-Speed 125.16 samples/sec Loss 11.1521 Epoch: 4 Global Step: 42960 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:19:32,644-Speed 126.98 samples/sec Loss 11.1909 Epoch: 4 Global Step: 42965 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:19:53,497-Speed 122.78 samples/sec Loss 11.6207 Epoch: 4 Global Step: 42970 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:20:13,717-Speed 126.61 samples/sec Loss 11.5026 Epoch: 4 Global Step: 42975 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:20:34,570-Speed 122.77 samples/sec Loss 11.2973 Epoch: 4 Global Step: 42980 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:20:54,824-Speed 126.40 samples/sec Loss 11.5914 Epoch: 4 Global Step: 42985 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:21:15,272-Speed 125.20 samples/sec Loss 11.4003 Epoch: 4 Global Step: 42990 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:21:35,799-Speed 124.72 samples/sec Loss 11.4599 Epoch: 4 Global Step: 42995 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:21:55,990-Speed 126.80 samples/sec Loss 11.3562 Epoch: 4 Global Step: 43000 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:22:16,153-Speed 126.98 samples/sec Loss 11.7267 Epoch: 4 Global Step: 43005 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:22:36,421-Speed 126.31 samples/sec Loss 11.3777 Epoch: 4 Global Step: 43010 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:22:56,699-Speed 126.26 samples/sec Loss 11.2391 Epoch: 4 Global Step: 43015 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:23:17,075-Speed 125.64 samples/sec Loss 11.3967 Epoch: 4 Global Step: 43020 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:23:37,976-Speed 122.49 samples/sec Loss 11.5291 Epoch: 4 Global Step: 43025 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:23:58,152-Speed 126.89 samples/sec Loss 11.5755 Epoch: 4 Global Step: 43030 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:24:18,448-Speed 126.14 samples/sec Loss 11.4005 Epoch: 4 Global Step: 43035 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:24:38,464-Speed 127.90 samples/sec Loss 11.3875 Epoch: 4 Global Step: 43040 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:24:58,593-Speed 127.18 samples/sec Loss 11.6243 Epoch: 4 Global Step: 43045 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:25:18,709-Speed 127.27 samples/sec Loss 11.3470 Epoch: 4 Global Step: 43050 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:25:38,946-Speed 126.51 samples/sec Loss 11.4517 Epoch: 4 Global Step: 43055 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:25:59,207-Speed 126.36 samples/sec Loss 11.5386 Epoch: 4 Global Step: 43060 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:26:19,456-Speed 126.43 samples/sec Loss 11.3568 Epoch: 4 Global Step: 43065 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:26:40,423-Speed 122.11 samples/sec Loss 11.6243 Epoch: 4 Global Step: 43070 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:27:00,659-Speed 126.51 samples/sec Loss 11.5335 Epoch: 4 Global Step: 43075 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:27:20,837-Speed 126.88 samples/sec Loss 11.5530 Epoch: 4 Global Step: 43080 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:27:40,829-Speed 128.06 samples/sec Loss 11.4082 Epoch: 4 Global Step: 43085 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:28:00,994-Speed 126.96 samples/sec Loss 11.5067 Epoch: 4 Global Step: 43090 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:28:21,264-Speed 126.31 samples/sec Loss 11.3604 Epoch: 4 Global Step: 43095 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:28:41,265-Speed 128.00 samples/sec Loss 11.3141 Epoch: 4 Global Step: 43100 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:29:01,458-Speed 126.79 samples/sec Loss 11.3834 Epoch: 4 Global Step: 43105 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:29:21,708-Speed 126.43 samples/sec Loss 11.9028 Epoch: 4 Global Step: 43110 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:29:42,429-Speed 123.55 samples/sec Loss 11.5201 Epoch: 4 Global Step: 43115 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:30:02,578-Speed 127.05 samples/sec Loss 11.4957 Epoch: 4 Global Step: 43120 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:30:22,783-Speed 126.71 samples/sec Loss 11.3315 Epoch: 4 Global Step: 43125 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:30:42,911-Speed 127.19 samples/sec Loss 11.5319 Epoch: 4 Global Step: 43130 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:31:03,234-Speed 125.97 samples/sec Loss 11.2045 Epoch: 4 Global Step: 43135 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:31:23,469-Speed 126.52 samples/sec Loss 11.6311 Epoch: 4 Global Step: 43140 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:31:43,696-Speed 126.57 samples/sec Loss 11.6815 Epoch: 4 Global Step: 43145 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:32:03,938-Speed 126.48 samples/sec Loss 11.1406 Epoch: 4 Global Step: 43150 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:32:24,134-Speed 126.77 samples/sec Loss 11.4206 Epoch: 4 Global Step: 43155 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:32:45,171-Speed 121.69 samples/sec Loss 11.4711 Epoch: 4 Global Step: 43160 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:33:05,360-Speed 126.81 samples/sec Loss 11.5260 Epoch: 4 Global Step: 43165 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:33:25,815-Speed 125.16 samples/sec Loss 11.5653 Epoch: 4 Global Step: 43170 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:33:46,297-Speed 124.99 samples/sec Loss 11.1683 Epoch: 4 Global Step: 43175 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:34:06,805-Speed 124.83 samples/sec Loss 11.6648 Epoch: 4 Global Step: 43180 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:34:27,000-Speed 126.78 samples/sec Loss 11.4496 Epoch: 4 Global Step: 43185 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:34:47,156-Speed 127.01 samples/sec Loss 11.7295 Epoch: 4 Global Step: 43190 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:35:07,294-Speed 127.13 samples/sec Loss 11.3903 Epoch: 4 Global Step: 43195 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:35:27,501-Speed 126.70 samples/sec Loss 11.4185 Epoch: 4 Global Step: 43200 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:35:48,455-Speed 122.18 samples/sec Loss 11.1878 Epoch: 4 Global Step: 43205 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:36:08,612-Speed 127.01 samples/sec Loss 11.3252 Epoch: 4 Global Step: 43210 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:36:28,752-Speed 127.12 samples/sec Loss 11.7038 Epoch: 4 Global Step: 43215 Fp16 Grad Scale: 8192 Required: 246 hours Training: 2021-06-08 22:36:48,901-Speed 127.06 samples/sec Loss 11.3890 Epoch: 4 Global Step: 43220 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:37:09,199-Speed 126.13 samples/sec Loss 11.3167 Epoch: 4 Global Step: 43225 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:37:29,440-Speed 126.49 samples/sec Loss 11.5607 Epoch: 4 Global Step: 43230 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:37:49,601-Speed 126.98 samples/sec Loss 11.5406 Epoch: 4 Global Step: 43235 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:38:09,976-Speed 125.66 samples/sec Loss 11.3951 Epoch: 4 Global Step: 43240 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:38:30,043-Speed 127.58 samples/sec Loss 11.1759 Epoch: 4 Global Step: 43245 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:38:50,728-Speed 123.76 samples/sec Loss 11.8236 Epoch: 4 Global Step: 43250 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:39:10,877-Speed 127.06 samples/sec Loss 11.4261 Epoch: 4 Global Step: 43255 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:39:30,964-Speed 127.45 samples/sec Loss 11.5900 Epoch: 4 Global Step: 43260 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:39:50,947-Speed 128.12 samples/sec Loss 11.2570 Epoch: 4 Global Step: 43265 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:40:11,085-Speed 127.13 samples/sec Loss 11.6622 Epoch: 4 Global Step: 43270 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:40:31,277-Speed 126.79 samples/sec Loss 11.7523 Epoch: 4 Global Step: 43275 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:40:51,540-Speed 126.34 samples/sec Loss 11.4124 Epoch: 4 Global Step: 43280 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:41:11,714-Speed 126.91 samples/sec Loss 11.5527 Epoch: 4 Global Step: 43285 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:41:31,568-Speed 128.95 samples/sec Loss 11.2648 Epoch: 4 Global Step: 43290 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:41:52,329-Speed 123.31 samples/sec Loss 11.4982 Epoch: 4 Global Step: 43295 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:42:12,472-Speed 127.09 samples/sec Loss 11.4631 Epoch: 4 Global Step: 43300 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:42:32,750-Speed 126.25 samples/sec Loss 11.6817 Epoch: 4 Global Step: 43305 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:42:52,992-Speed 126.48 samples/sec Loss 11.4946 Epoch: 4 Global Step: 43310 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:43:13,006-Speed 127.92 samples/sec Loss 11.6112 Epoch: 4 Global Step: 43315 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:43:33,139-Speed 127.16 samples/sec Loss 11.1043 Epoch: 4 Global Step: 43320 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:43:53,263-Speed 127.22 samples/sec Loss 11.2488 Epoch: 4 Global Step: 43325 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:44:13,499-Speed 126.52 samples/sec Loss 11.7270 Epoch: 4 Global Step: 43330 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:44:34,407-Speed 122.45 samples/sec Loss 11.5889 Epoch: 4 Global Step: 43335 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:44:55,389-Speed 122.01 samples/sec Loss 11.6083 Epoch: 4 Global Step: 43340 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:45:15,700-Speed 126.05 samples/sec Loss 11.1891 Epoch: 4 Global Step: 43345 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:45:36,046-Speed 125.83 samples/sec Loss 11.5452 Epoch: 4 Global Step: 43350 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:45:56,173-Speed 127.20 samples/sec Loss 11.3769 Epoch: 4 Global Step: 43355 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:46:16,098-Speed 128.49 samples/sec Loss 11.6617 Epoch: 4 Global Step: 43360 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:46:36,233-Speed 127.15 samples/sec Loss 11.5348 Epoch: 4 Global Step: 43365 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:46:56,405-Speed 126.92 samples/sec Loss 11.5835 Epoch: 4 Global Step: 43370 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:47:16,550-Speed 127.08 samples/sec Loss 11.7384 Epoch: 4 Global Step: 43375 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:47:36,605-Speed 127.66 samples/sec Loss 11.4350 Epoch: 4 Global Step: 43380 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:47:57,441-Speed 122.88 samples/sec Loss 11.5201 Epoch: 4 Global Step: 43385 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:48:17,425-Speed 128.10 samples/sec Loss 11.5289 Epoch: 4 Global Step: 43390 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:48:37,632-Speed 126.70 samples/sec Loss 11.5361 Epoch: 4 Global Step: 43395 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:48:57,812-Speed 126.86 samples/sec Loss 11.6688 Epoch: 4 Global Step: 43400 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:49:18,085-Speed 126.29 samples/sec Loss 11.3764 Epoch: 4 Global Step: 43405 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:49:38,301-Speed 126.64 samples/sec Loss 11.2161 Epoch: 4 Global Step: 43410 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:49:58,485-Speed 126.84 samples/sec Loss 11.4542 Epoch: 4 Global Step: 43415 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:50:18,667-Speed 126.86 samples/sec Loss 11.2824 Epoch: 4 Global Step: 43420 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:50:38,819-Speed 127.04 samples/sec Loss 11.6619 Epoch: 4 Global Step: 43425 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:50:59,436-Speed 124.18 samples/sec Loss 11.3751 Epoch: 4 Global Step: 43430 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:51:19,554-Speed 127.26 samples/sec Loss 11.8030 Epoch: 4 Global Step: 43435 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:51:39,690-Speed 127.14 samples/sec Loss 11.1393 Epoch: 4 Global Step: 43440 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:51:59,713-Speed 127.86 samples/sec Loss 11.5584 Epoch: 4 Global Step: 43445 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:52:19,802-Speed 127.44 samples/sec Loss 11.4223 Epoch: 4 Global Step: 43450 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:52:40,019-Speed 126.63 samples/sec Loss 11.4347 Epoch: 4 Global Step: 43455 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:53:00,177-Speed 127.01 samples/sec Loss 11.4944 Epoch: 4 Global Step: 43460 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:53:20,352-Speed 126.90 samples/sec Loss 11.1665 Epoch: 4 Global Step: 43465 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:53:40,567-Speed 126.65 samples/sec Loss 11.2539 Epoch: 4 Global Step: 43470 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:54:01,250-Speed 123.78 samples/sec Loss 11.8631 Epoch: 4 Global Step: 43475 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:54:21,645-Speed 125.52 samples/sec Loss 11.5175 Epoch: 4 Global Step: 43480 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:54:41,849-Speed 126.72 samples/sec Loss 11.5346 Epoch: 4 Global Step: 43485 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:55:02,022-Speed 126.90 samples/sec Loss 11.7613 Epoch: 4 Global Step: 43490 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:55:22,255-Speed 126.54 samples/sec Loss 11.4147 Epoch: 4 Global Step: 43495 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:55:42,474-Speed 126.62 samples/sec Loss 11.3207 Epoch: 4 Global Step: 43500 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:56:02,694-Speed 126.61 samples/sec Loss 11.0395 Epoch: 4 Global Step: 43505 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:56:22,886-Speed 126.79 samples/sec Loss 11.3264 Epoch: 4 Global Step: 43510 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:56:43,136-Speed 126.43 samples/sec Loss 11.4165 Epoch: 4 Global Step: 43515 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:57:03,848-Speed 123.60 samples/sec Loss 11.4478 Epoch: 4 Global Step: 43520 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:57:24,115-Speed 126.32 samples/sec Loss 11.4310 Epoch: 4 Global Step: 43525 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:57:44,328-Speed 126.66 samples/sec Loss 11.1104 Epoch: 4 Global Step: 43530 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:58:04,572-Speed 126.47 samples/sec Loss 11.3528 Epoch: 4 Global Step: 43535 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:58:24,602-Speed 127.81 samples/sec Loss 11.4048 Epoch: 4 Global Step: 43540 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:58:44,807-Speed 126.70 samples/sec Loss 11.4332 Epoch: 4 Global Step: 43545 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:59:04,752-Speed 128.36 samples/sec Loss 11.0186 Epoch: 4 Global Step: 43550 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:59:24,889-Speed 127.14 samples/sec Loss 11.5577 Epoch: 4 Global Step: 43555 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 22:59:44,828-Speed 128.40 samples/sec Loss 11.2609 Epoch: 4 Global Step: 43560 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:00:05,632-Speed 123.06 samples/sec Loss 11.4101 Epoch: 4 Global Step: 43565 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:00:25,847-Speed 126.64 samples/sec Loss 11.5800 Epoch: 4 Global Step: 43570 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:00:46,057-Speed 126.68 samples/sec Loss 11.7139 Epoch: 4 Global Step: 43575 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:01:06,501-Speed 125.23 samples/sec Loss 11.4349 Epoch: 4 Global Step: 43580 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:01:26,687-Speed 126.83 samples/sec Loss 11.3165 Epoch: 4 Global Step: 43585 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:01:46,805-Speed 127.25 samples/sec Loss 11.5080 Epoch: 4 Global Step: 43590 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:02:06,883-Speed 127.52 samples/sec Loss 11.3303 Epoch: 4 Global Step: 43595 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:02:27,050-Speed 126.94 samples/sec Loss 11.4415 Epoch: 4 Global Step: 43600 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:02:47,240-Speed 126.80 samples/sec Loss 11.0178 Epoch: 4 Global Step: 43605 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:03:08,585-Speed 119.94 samples/sec Loss 11.4612 Epoch: 4 Global Step: 43610 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:03:29,292-Speed 123.63 samples/sec Loss 11.4139 Epoch: 4 Global Step: 43615 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:03:49,264-Speed 128.18 samples/sec Loss 11.3053 Epoch: 4 Global Step: 43620 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:04:09,425-Speed 126.99 samples/sec Loss 11.3092 Epoch: 4 Global Step: 43625 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:04:29,450-Speed 127.84 samples/sec Loss 11.0446 Epoch: 4 Global Step: 43630 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:04:49,547-Speed 127.39 samples/sec Loss 11.2887 Epoch: 4 Global Step: 43635 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:05:09,610-Speed 127.60 samples/sec Loss 11.5155 Epoch: 4 Global Step: 43640 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:05:29,710-Speed 127.37 samples/sec Loss 11.6154 Epoch: 4 Global Step: 43645 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:05:49,814-Speed 127.35 samples/sec Loss 11.3897 Epoch: 4 Global Step: 43650 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:06:10,566-Speed 123.37 samples/sec Loss 11.7497 Epoch: 4 Global Step: 43655 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:06:30,739-Speed 126.91 samples/sec Loss 11.4901 Epoch: 4 Global Step: 43660 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:06:50,822-Speed 127.48 samples/sec Loss 11.3585 Epoch: 4 Global Step: 43665 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:07:10,978-Speed 127.02 samples/sec Loss 11.3948 Epoch: 4 Global Step: 43670 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:07:31,399-Speed 125.37 samples/sec Loss 10.9757 Epoch: 4 Global Step: 43675 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:07:51,577-Speed 126.88 samples/sec Loss 11.1236 Epoch: 4 Global Step: 43680 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:08:11,808-Speed 126.55 samples/sec Loss 11.4217 Epoch: 4 Global Step: 43685 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:08:31,977-Speed 126.94 samples/sec Loss 11.3211 Epoch: 4 Global Step: 43690 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:08:52,207-Speed 126.55 samples/sec Loss 11.6237 Epoch: 4 Global Step: 43695 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:09:13,125-Speed 122.39 samples/sec Loss 11.6072 Epoch: 4 Global Step: 43700 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:09:33,321-Speed 126.76 samples/sec Loss 11.7625 Epoch: 4 Global Step: 43705 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:09:53,785-Speed 125.10 samples/sec Loss 10.9931 Epoch: 4 Global Step: 43710 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:10:14,010-Speed 126.58 samples/sec Loss 11.5677 Epoch: 4 Global Step: 43715 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:10:34,316-Speed 126.08 samples/sec Loss 11.2602 Epoch: 4 Global Step: 43720 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:10:54,550-Speed 126.53 samples/sec Loss 11.4667 Epoch: 4 Global Step: 43725 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:11:14,791-Speed 126.48 samples/sec Loss 11.4327 Epoch: 4 Global Step: 43730 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:11:34,874-Speed 127.48 samples/sec Loss 11.4375 Epoch: 4 Global Step: 43735 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:11:55,050-Speed 126.89 samples/sec Loss 11.7095 Epoch: 4 Global Step: 43740 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:12:15,951-Speed 122.49 samples/sec Loss 11.3120 Epoch: 4 Global Step: 43745 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:12:35,833-Speed 128.78 samples/sec Loss 11.2700 Epoch: 4 Global Step: 43750 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:12:56,029-Speed 126.76 samples/sec Loss 11.4105 Epoch: 4 Global Step: 43755 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:13:16,305-Speed 126.26 samples/sec Loss 11.5640 Epoch: 4 Global Step: 43760 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:13:36,689-Speed 125.61 samples/sec Loss 11.2885 Epoch: 4 Global Step: 43765 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:13:56,875-Speed 126.83 samples/sec Loss 11.2626 Epoch: 4 Global Step: 43770 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:14:17,075-Speed 126.74 samples/sec Loss 11.6115 Epoch: 4 Global Step: 43775 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:14:37,237-Speed 126.98 samples/sec Loss 11.1206 Epoch: 4 Global Step: 43780 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:14:57,480-Speed 126.47 samples/sec Loss 11.5868 Epoch: 4 Global Step: 43785 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:15:18,304-Speed 122.94 samples/sec Loss 11.4056 Epoch: 4 Global Step: 43790 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:15:38,431-Speed 127.20 samples/sec Loss 11.3359 Epoch: 4 Global Step: 43795 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:15:58,656-Speed 126.58 samples/sec Loss 11.2536 Epoch: 4 Global Step: 43800 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:16:18,896-Speed 126.49 samples/sec Loss 11.1508 Epoch: 4 Global Step: 43805 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:16:39,092-Speed 126.77 samples/sec Loss 11.5948 Epoch: 4 Global Step: 43810 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:16:59,064-Speed 128.19 samples/sec Loss 11.6416 Epoch: 4 Global Step: 43815 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:17:19,138-Speed 127.53 samples/sec Loss 11.3503 Epoch: 4 Global Step: 43820 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:17:39,256-Speed 127.26 samples/sec Loss 11.6006 Epoch: 4 Global Step: 43825 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:17:59,200-Speed 128.36 samples/sec Loss 11.5155 Epoch: 4 Global Step: 43830 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:18:19,907-Speed 123.64 samples/sec Loss 11.6504 Epoch: 4 Global Step: 43835 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:18:40,178-Speed 126.30 samples/sec Loss 11.5762 Epoch: 4 Global Step: 43840 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:19:00,318-Speed 127.11 samples/sec Loss 11.6646 Epoch: 4 Global Step: 43845 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:19:20,812-Speed 124.92 samples/sec Loss 11.4181 Epoch: 4 Global Step: 43850 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:19:40,756-Speed 128.36 samples/sec Loss 11.1832 Epoch: 4 Global Step: 43855 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:20:00,748-Speed 128.06 samples/sec Loss 11.1811 Epoch: 4 Global Step: 43860 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:20:20,838-Speed 127.43 samples/sec Loss 11.2757 Epoch: 4 Global Step: 43865 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:20:41,222-Speed 125.60 samples/sec Loss 11.6965 Epoch: 4 Global Step: 43870 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:21:01,593-Speed 125.67 samples/sec Loss 11.5954 Epoch: 4 Global Step: 43875 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:21:22,166-Speed 124.44 samples/sec Loss 11.6461 Epoch: 4 Global Step: 43880 Fp16 Grad Scale: 8192 Required: 245 hours Training: 2021-06-08 23:21:42,245-Speed 127.50 samples/sec Loss 11.3296 Epoch: 4 Global Step: 43885 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:22:02,314-Speed 127.57 samples/sec Loss 11.3655 Epoch: 4 Global Step: 43890 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:22:22,780-Speed 125.10 samples/sec Loss 11.4450 Epoch: 4 Global Step: 43895 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:22:43,281-Speed 124.87 samples/sec Loss 11.4913 Epoch: 4 Global Step: 43900 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:23:03,564-Speed 126.22 samples/sec Loss 11.4808 Epoch: 4 Global Step: 43905 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:23:23,735-Speed 126.93 samples/sec Loss 11.4660 Epoch: 4 Global Step: 43910 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:23:43,895-Speed 126.99 samples/sec Loss 11.3140 Epoch: 4 Global Step: 43915 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:24:04,152-Speed 126.38 samples/sec Loss 11.7012 Epoch: 4 Global Step: 43920 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:24:24,844-Speed 123.73 samples/sec Loss 11.1583 Epoch: 4 Global Step: 43925 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:24:44,961-Speed 127.26 samples/sec Loss 11.4989 Epoch: 4 Global Step: 43930 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:25:05,234-Speed 126.28 samples/sec Loss 11.6191 Epoch: 4 Global Step: 43935 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:25:25,289-Speed 127.65 samples/sec Loss 11.6755 Epoch: 4 Global Step: 43940 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:25:45,505-Speed 126.64 samples/sec Loss 11.6400 Epoch: 4 Global Step: 43945 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:26:05,504-Speed 128.02 samples/sec Loss 11.3487 Epoch: 4 Global Step: 43950 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:26:25,637-Speed 127.16 samples/sec Loss 11.1360 Epoch: 4 Global Step: 43955 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:26:45,605-Speed 128.21 samples/sec Loss 11.4895 Epoch: 4 Global Step: 43960 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:27:05,893-Speed 126.19 samples/sec Loss 11.5557 Epoch: 4 Global Step: 43965 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:27:26,425-Speed 124.69 samples/sec Loss 11.7191 Epoch: 4 Global Step: 43970 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:27:46,650-Speed 126.58 samples/sec Loss 11.6995 Epoch: 4 Global Step: 43975 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:28:06,830-Speed 126.86 samples/sec Loss 11.6602 Epoch: 4 Global Step: 43980 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:28:27,003-Speed 126.91 samples/sec Loss 11.2839 Epoch: 4 Global Step: 43985 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:28:47,119-Speed 127.27 samples/sec Loss 11.3195 Epoch: 4 Global Step: 43990 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:29:07,346-Speed 126.57 samples/sec Loss 11.5085 Epoch: 4 Global Step: 43995 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:29:27,535-Speed 126.81 samples/sec Loss 11.5878 Epoch: 4 Global Step: 44000 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:29:47,709-Speed 126.92 samples/sec Loss 11.1216 Epoch: 4 Global Step: 44005 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:30:07,849-Speed 127.11 samples/sec Loss 11.0397 Epoch: 4 Global Step: 44010 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:30:28,495-Speed 124.01 samples/sec Loss 11.4257 Epoch: 4 Global Step: 44015 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:30:48,793-Speed 126.13 samples/sec Loss 11.4732 Epoch: 4 Global Step: 44020 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:31:08,972-Speed 126.87 samples/sec Loss 11.4856 Epoch: 4 Global Step: 44025 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:31:29,322-Speed 125.80 samples/sec Loss 11.7763 Epoch: 4 Global Step: 44030 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:31:49,392-Speed 127.56 samples/sec Loss 11.0894 Epoch: 4 Global Step: 44035 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:32:09,638-Speed 126.45 samples/sec Loss 11.1377 Epoch: 4 Global Step: 44040 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:32:29,769-Speed 127.18 samples/sec Loss 11.5413 Epoch: 4 Global Step: 44045 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:32:49,920-Speed 127.05 samples/sec Loss 11.4659 Epoch: 4 Global Step: 44050 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:33:10,034-Speed 127.28 samples/sec Loss 11.3855 Epoch: 4 Global Step: 44055 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:33:30,661-Speed 124.11 samples/sec Loss 11.6940 Epoch: 4 Global Step: 44060 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:33:50,584-Speed 128.50 samples/sec Loss 11.2357 Epoch: 4 Global Step: 44065 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:34:10,703-Speed 127.25 samples/sec Loss 11.1856 Epoch: 4 Global Step: 44070 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:34:30,819-Speed 127.27 samples/sec Loss 11.6331 Epoch: 4 Global Step: 44075 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:34:50,805-Speed 128.09 samples/sec Loss 11.4885 Epoch: 4 Global Step: 44080 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:35:10,907-Speed 127.36 samples/sec Loss 11.6432 Epoch: 4 Global Step: 44085 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:35:30,965-Speed 127.64 samples/sec Loss 11.6197 Epoch: 4 Global Step: 44090 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:35:51,126-Speed 126.99 samples/sec Loss 11.8762 Epoch: 4 Global Step: 44095 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:36:11,300-Speed 126.90 samples/sec Loss 11.2889 Epoch: 4 Global Step: 44100 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:36:32,131-Speed 122.90 samples/sec Loss 11.4301 Epoch: 4 Global Step: 44105 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:36:52,286-Speed 127.03 samples/sec Loss 11.4137 Epoch: 4 Global Step: 44110 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:37:12,426-Speed 127.12 samples/sec Loss 10.9488 Epoch: 4 Global Step: 44115 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:37:32,444-Speed 127.89 samples/sec Loss 11.6322 Epoch: 4 Global Step: 44120 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:37:52,832-Speed 125.58 samples/sec Loss 11.2871 Epoch: 4 Global Step: 44125 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:38:12,999-Speed 126.95 samples/sec Loss 11.5575 Epoch: 4 Global Step: 44130 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:38:33,219-Speed 126.62 samples/sec Loss 11.2819 Epoch: 4 Global Step: 44135 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:38:53,517-Speed 126.12 samples/sec Loss 11.4999 Epoch: 4 Global Step: 44140 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:39:13,777-Speed 126.37 samples/sec Loss 11.1878 Epoch: 4 Global Step: 44145 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:39:34,579-Speed 123.07 samples/sec Loss 11.0195 Epoch: 4 Global Step: 44150 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:39:54,740-Speed 126.98 samples/sec Loss 11.2321 Epoch: 4 Global Step: 44155 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:40:14,999-Speed 126.37 samples/sec Loss 11.5580 Epoch: 4 Global Step: 44160 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:40:34,942-Speed 128.37 samples/sec Loss 11.1843 Epoch: 4 Global Step: 44165 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:40:55,196-Speed 126.40 samples/sec Loss 11.3823 Epoch: 4 Global Step: 44170 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:41:15,151-Speed 128.29 samples/sec Loss 11.3224 Epoch: 4 Global Step: 44175 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:41:35,350-Speed 126.75 samples/sec Loss 11.4496 Epoch: 4 Global Step: 44180 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:41:55,655-Speed 126.08 samples/sec Loss 11.7572 Epoch: 4 Global Step: 44185 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:42:15,753-Speed 127.45 samples/sec Loss 11.3048 Epoch: 4 Global Step: 44190 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:42:36,598-Speed 122.82 samples/sec Loss 11.6086 Epoch: 4 Global Step: 44195 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:42:56,916-Speed 126.00 samples/sec Loss 11.5298 Epoch: 4 Global Step: 44200 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:43:17,208-Speed 126.16 samples/sec Loss 11.2648 Epoch: 4 Global Step: 44205 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:43:37,369-Speed 126.99 samples/sec Loss 11.1320 Epoch: 4 Global Step: 44210 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:43:57,660-Speed 126.17 samples/sec Loss 11.5378 Epoch: 4 Global Step: 44215 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:44:18,204-Speed 124.61 samples/sec Loss 11.2491 Epoch: 4 Global Step: 44220 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:44:38,461-Speed 126.39 samples/sec Loss 11.2668 Epoch: 4 Global Step: 44225 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:44:58,626-Speed 126.96 samples/sec Loss 11.1977 Epoch: 4 Global Step: 44230 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:45:19,136-Speed 124.82 samples/sec Loss 11.2699 Epoch: 4 Global Step: 44235 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:45:39,789-Speed 123.96 samples/sec Loss 11.6088 Epoch: 4 Global Step: 44240 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:45:59,906-Speed 127.27 samples/sec Loss 11.2722 Epoch: 4 Global Step: 44245 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:46:20,106-Speed 126.73 samples/sec Loss 11.2489 Epoch: 4 Global Step: 44250 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:46:40,285-Speed 126.88 samples/sec Loss 11.3986 Epoch: 4 Global Step: 44255 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:47:00,393-Speed 127.31 samples/sec Loss 11.3001 Epoch: 4 Global Step: 44260 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:47:20,360-Speed 128.22 samples/sec Loss 11.5136 Epoch: 4 Global Step: 44265 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:47:40,355-Speed 128.04 samples/sec Loss 11.2518 Epoch: 4 Global Step: 44270 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:48:00,452-Speed 127.39 samples/sec Loss 11.3583 Epoch: 4 Global Step: 44275 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:48:20,579-Speed 127.20 samples/sec Loss 11.3021 Epoch: 4 Global Step: 44280 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:48:41,476-Speed 122.51 samples/sec Loss 11.6483 Epoch: 4 Global Step: 44285 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:49:01,627-Speed 127.04 samples/sec Loss 11.3175 Epoch: 4 Global Step: 44290 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:49:21,559-Speed 128.44 samples/sec Loss 11.4174 Epoch: 4 Global Step: 44295 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:49:41,779-Speed 126.61 samples/sec Loss 11.3292 Epoch: 4 Global Step: 44300 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:50:01,984-Speed 126.71 samples/sec Loss 11.2373 Epoch: 4 Global Step: 44305 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:50:22,130-Speed 127.08 samples/sec Loss 11.5152 Epoch: 4 Global Step: 44310 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:50:42,086-Speed 128.29 samples/sec Loss 11.5279 Epoch: 4 Global Step: 44315 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:51:02,302-Speed 126.64 samples/sec Loss 11.8233 Epoch: 4 Global Step: 44320 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:51:22,214-Speed 128.58 samples/sec Loss 11.4070 Epoch: 4 Global Step: 44325 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:51:42,842-Speed 124.11 samples/sec Loss 11.4702 Epoch: 4 Global Step: 44330 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:52:02,994-Speed 127.04 samples/sec Loss 11.2771 Epoch: 4 Global Step: 44335 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:52:23,073-Speed 127.50 samples/sec Loss 11.3324 Epoch: 4 Global Step: 44340 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:52:43,311-Speed 126.50 samples/sec Loss 11.4249 Epoch: 4 Global Step: 44345 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:53:03,470-Speed 127.00 samples/sec Loss 11.3059 Epoch: 4 Global Step: 44350 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:53:23,638-Speed 126.94 samples/sec Loss 11.4462 Epoch: 4 Global Step: 44355 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:53:43,900-Speed 126.35 samples/sec Loss 11.3444 Epoch: 4 Global Step: 44360 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:54:04,101-Speed 126.74 samples/sec Loss 11.3296 Epoch: 4 Global Step: 44365 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:54:24,240-Speed 127.12 samples/sec Loss 11.6231 Epoch: 4 Global Step: 44370 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:54:44,985-Speed 123.41 samples/sec Loss 11.2855 Epoch: 4 Global Step: 44375 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:55:05,046-Speed 127.62 samples/sec Loss 11.0788 Epoch: 4 Global Step: 44380 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:55:25,171-Speed 127.21 samples/sec Loss 11.5457 Epoch: 4 Global Step: 44385 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:55:45,390-Speed 126.62 samples/sec Loss 11.5495 Epoch: 4 Global Step: 44390 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:56:05,492-Speed 127.36 samples/sec Loss 11.2442 Epoch: 4 Global Step: 44395 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:56:25,627-Speed 127.15 samples/sec Loss 11.1955 Epoch: 4 Global Step: 44400 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:56:45,601-Speed 128.17 samples/sec Loss 11.6816 Epoch: 4 Global Step: 44405 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:57:05,893-Speed 126.17 samples/sec Loss 11.2055 Epoch: 4 Global Step: 44410 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:57:25,872-Speed 128.14 samples/sec Loss 11.3640 Epoch: 4 Global Step: 44415 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:57:46,595-Speed 123.54 samples/sec Loss 11.5133 Epoch: 4 Global Step: 44420 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:58:06,794-Speed 126.74 samples/sec Loss 11.2646 Epoch: 4 Global Step: 44425 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:58:26,911-Speed 127.26 samples/sec Loss 11.1609 Epoch: 4 Global Step: 44430 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:58:47,045-Speed 127.16 samples/sec Loss 11.4302 Epoch: 4 Global Step: 44435 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:59:07,320-Speed 126.27 samples/sec Loss 11.4466 Epoch: 4 Global Step: 44440 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:59:27,474-Speed 127.02 samples/sec Loss 11.5415 Epoch: 4 Global Step: 44445 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-08 23:59:47,900-Speed 125.34 samples/sec Loss 11.1233 Epoch: 4 Global Step: 44450 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:00:08,031-Speed 127.18 samples/sec Loss 11.2078 Epoch: 4 Global Step: 44455 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:00:27,969-Speed 128.40 samples/sec Loss 11.3619 Epoch: 4 Global Step: 44460 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:00:48,832-Speed 122.71 samples/sec Loss 11.5382 Epoch: 4 Global Step: 44465 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:01:09,274-Speed 125.24 samples/sec Loss 11.3265 Epoch: 4 Global Step: 44470 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:01:29,188-Speed 128.56 samples/sec Loss 11.5480 Epoch: 4 Global Step: 44475 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:01:49,441-Speed 126.41 samples/sec Loss 11.3478 Epoch: 4 Global Step: 44480 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:02:09,712-Speed 126.30 samples/sec Loss 11.3375 Epoch: 4 Global Step: 44485 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:02:29,860-Speed 127.06 samples/sec Loss 11.3441 Epoch: 4 Global Step: 44490 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:02:49,831-Speed 128.20 samples/sec Loss 11.3749 Epoch: 4 Global Step: 44495 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:03:09,980-Speed 127.06 samples/sec Loss 10.9954 Epoch: 4 Global Step: 44500 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:03:30,123-Speed 127.10 samples/sec Loss 11.2219 Epoch: 4 Global Step: 44505 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:03:50,987-Speed 122.70 samples/sec Loss 11.6334 Epoch: 4 Global Step: 44510 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:04:11,112-Speed 127.21 samples/sec Loss 11.4438 Epoch: 4 Global Step: 44515 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:04:31,473-Speed 125.73 samples/sec Loss 11.1929 Epoch: 4 Global Step: 44520 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:04:51,615-Speed 127.11 samples/sec Loss 11.2669 Epoch: 4 Global Step: 44525 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:05:11,761-Speed 127.08 samples/sec Loss 11.2656 Epoch: 4 Global Step: 44530 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:05:32,002-Speed 126.48 samples/sec Loss 11.4745 Epoch: 4 Global Step: 44535 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:05:52,266-Speed 126.34 samples/sec Loss 11.3360 Epoch: 4 Global Step: 44540 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:06:12,406-Speed 127.12 samples/sec Loss 11.6854 Epoch: 4 Global Step: 44545 Fp16 Grad Scale: 8192 Required: 244 hours Training: 2021-06-09 00:06:32,911-Speed 124.86 samples/sec Loss 11.6168 Epoch: 4 Global Step: 44550 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:06:53,515-Speed 124.25 samples/sec Loss 11.5124 Epoch: 4 Global Step: 44555 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:07:13,649-Speed 127.15 samples/sec Loss 11.6970 Epoch: 4 Global Step: 44560 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:07:33,753-Speed 127.35 samples/sec Loss 11.3361 Epoch: 4 Global Step: 44565 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:07:53,871-Speed 127.26 samples/sec Loss 11.3722 Epoch: 4 Global Step: 44570 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:08:14,095-Speed 126.59 samples/sec Loss 11.2704 Epoch: 4 Global Step: 44575 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:08:34,247-Speed 127.04 samples/sec Loss 11.1546 Epoch: 4 Global Step: 44580 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:08:54,395-Speed 127.07 samples/sec Loss 11.4304 Epoch: 4 Global Step: 44585 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:09:14,440-Speed 127.72 samples/sec Loss 11.1975 Epoch: 4 Global Step: 44590 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:09:34,519-Speed 127.51 samples/sec Loss 11.5114 Epoch: 4 Global Step: 44595 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:09:55,235-Speed 123.58 samples/sec Loss 11.4365 Epoch: 4 Global Step: 44600 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:10:15,240-Speed 127.97 samples/sec Loss 11.3139 Epoch: 4 Global Step: 44605 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:10:35,389-Speed 127.06 samples/sec Loss 11.0026 Epoch: 4 Global Step: 44610 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:10:55,323-Speed 128.43 samples/sec Loss 11.7384 Epoch: 4 Global Step: 44615 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:11:15,406-Speed 127.48 samples/sec Loss 11.4048 Epoch: 4 Global Step: 44620 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:11:35,454-Speed 127.70 samples/sec Loss 11.2366 Epoch: 4 Global Step: 44625 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:11:55,580-Speed 127.20 samples/sec Loss 11.5865 Epoch: 4 Global Step: 44630 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:12:15,724-Speed 127.09 samples/sec Loss 11.5681 Epoch: 4 Global Step: 44635 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:12:35,926-Speed 126.73 samples/sec Loss 11.2728 Epoch: 4 Global Step: 44640 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:12:57,254-Speed 120.03 samples/sec Loss 11.3282 Epoch: 4 Global Step: 44645 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:13:17,434-Speed 126.86 samples/sec Loss 11.5357 Epoch: 4 Global Step: 44650 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:13:37,368-Speed 128.43 samples/sec Loss 11.5973 Epoch: 4 Global Step: 44655 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:13:57,539-Speed 126.92 samples/sec Loss 11.1336 Epoch: 4 Global Step: 44660 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:14:17,733-Speed 126.78 samples/sec Loss 11.1832 Epoch: 4 Global Step: 44665 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:14:37,710-Speed 128.16 samples/sec Loss 11.4303 Epoch: 4 Global Step: 44670 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:14:57,776-Speed 127.59 samples/sec Loss 11.5613 Epoch: 4 Global Step: 44675 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:15:18,285-Speed 124.83 samples/sec Loss 11.4750 Epoch: 4 Global Step: 44680 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:15:38,524-Speed 126.50 samples/sec Loss 11.1910 Epoch: 4 Global Step: 44685 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:15:59,472-Speed 122.22 samples/sec Loss 11.4464 Epoch: 4 Global Step: 44690 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:16:19,590-Speed 127.25 samples/sec Loss 11.3096 Epoch: 4 Global Step: 44695 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:16:39,845-Speed 126.39 samples/sec Loss 11.3277 Epoch: 4 Global Step: 44700 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:16:59,977-Speed 127.16 samples/sec Loss 11.5442 Epoch: 4 Global Step: 44705 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:17:20,066-Speed 127.44 samples/sec Loss 11.5016 Epoch: 4 Global Step: 44710 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:17:40,143-Speed 127.52 samples/sec Loss 11.5937 Epoch: 4 Global Step: 44715 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:18:00,293-Speed 127.06 samples/sec Loss 11.1988 Epoch: 4 Global Step: 44720 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:18:20,447-Speed 127.03 samples/sec Loss 11.0574 Epoch: 4 Global Step: 44725 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:18:40,987-Speed 124.64 samples/sec Loss 11.2387 Epoch: 4 Global Step: 44730 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:19:01,872-Speed 122.58 samples/sec Loss 11.1076 Epoch: 4 Global Step: 44735 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:19:21,993-Speed 127.24 samples/sec Loss 11.4227 Epoch: 4 Global Step: 44740 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:19:42,119-Speed 127.20 samples/sec Loss 11.4399 Epoch: 4 Global Step: 44745 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:20:02,238-Speed 127.25 samples/sec Loss 11.4980 Epoch: 4 Global Step: 44750 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:20:22,281-Speed 127.74 samples/sec Loss 11.4501 Epoch: 4 Global Step: 44755 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:20:42,415-Speed 127.16 samples/sec Loss 11.6432 Epoch: 4 Global Step: 44760 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:21:02,565-Speed 127.05 samples/sec Loss 11.2787 Epoch: 4 Global Step: 44765 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:21:22,827-Speed 126.35 samples/sec Loss 11.5041 Epoch: 4 Global Step: 44770 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:21:42,981-Speed 127.03 samples/sec Loss 11.4984 Epoch: 4 Global Step: 44775 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:22:03,775-Speed 123.12 samples/sec Loss 11.7116 Epoch: 4 Global Step: 44780 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:22:23,851-Speed 127.52 samples/sec Loss 11.4206 Epoch: 4 Global Step: 44785 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:22:43,905-Speed 127.66 samples/sec Loss 11.4529 Epoch: 4 Global Step: 44790 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:23:04,092-Speed 126.83 samples/sec Loss 11.3676 Epoch: 4 Global Step: 44795 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:23:24,229-Speed 127.14 samples/sec Loss 11.2698 Epoch: 4 Global Step: 44800 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:23:44,368-Speed 127.12 samples/sec Loss 11.1173 Epoch: 4 Global Step: 44805 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:24:04,550-Speed 126.86 samples/sec Loss 11.3910 Epoch: 4 Global Step: 44810 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:24:24,835-Speed 126.20 samples/sec Loss 11.1876 Epoch: 4 Global Step: 44815 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:24:45,087-Speed 126.42 samples/sec Loss 11.4516 Epoch: 4 Global Step: 44820 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:25:05,891-Speed 123.06 samples/sec Loss 11.3412 Epoch: 4 Global Step: 44825 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:25:26,489-Speed 124.28 samples/sec Loss 11.5334 Epoch: 4 Global Step: 44830 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:25:46,647-Speed 127.01 samples/sec Loss 11.1850 Epoch: 4 Global Step: 44835 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:26:06,604-Speed 128.28 samples/sec Loss 11.4903 Epoch: 4 Global Step: 44840 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:26:26,741-Speed 127.14 samples/sec Loss 11.3886 Epoch: 4 Global Step: 44845 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:26:46,671-Speed 128.46 samples/sec Loss 11.2768 Epoch: 4 Global Step: 44850 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:27:06,804-Speed 127.16 samples/sec Loss 11.3720 Epoch: 4 Global Step: 44855 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:27:27,063-Speed 126.37 samples/sec Loss 11.7350 Epoch: 4 Global Step: 44860 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:27:47,447-Speed 125.59 samples/sec Loss 11.5300 Epoch: 4 Global Step: 44865 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:28:08,573-Speed 121.18 samples/sec Loss 11.1553 Epoch: 4 Global Step: 44870 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:28:28,681-Speed 127.32 samples/sec Loss 11.6924 Epoch: 4 Global Step: 44875 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:28:48,847-Speed 126.95 samples/sec Loss 11.7502 Epoch: 4 Global Step: 44880 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:29:09,040-Speed 126.79 samples/sec Loss 11.3731 Epoch: 4 Global Step: 44885 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:29:29,163-Speed 127.22 samples/sec Loss 11.5828 Epoch: 4 Global Step: 44890 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:29:49,346-Speed 126.85 samples/sec Loss 11.5390 Epoch: 4 Global Step: 44895 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:30:09,490-Speed 127.10 samples/sec Loss 11.4605 Epoch: 4 Global Step: 44900 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:30:29,713-Speed 126.59 samples/sec Loss 11.8324 Epoch: 4 Global Step: 44905 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:30:49,846-Speed 127.17 samples/sec Loss 11.4887 Epoch: 4 Global Step: 44910 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:31:10,647-Speed 123.07 samples/sec Loss 11.4954 Epoch: 4 Global Step: 44915 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:31:30,793-Speed 127.08 samples/sec Loss 11.3837 Epoch: 4 Global Step: 44920 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:31:51,052-Speed 126.36 samples/sec Loss 11.3782 Epoch: 4 Global Step: 44925 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:32:11,467-Speed 125.41 samples/sec Loss 11.2247 Epoch: 4 Global Step: 44930 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:32:31,591-Speed 127.22 samples/sec Loss 11.4221 Epoch: 4 Global Step: 44935 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:32:51,775-Speed 126.84 samples/sec Loss 11.5355 Epoch: 4 Global Step: 44940 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:33:11,924-Speed 127.06 samples/sec Loss 11.7206 Epoch: 4 Global Step: 44945 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:33:32,112-Speed 126.82 samples/sec Loss 11.1356 Epoch: 4 Global Step: 44950 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:33:52,290-Speed 126.88 samples/sec Loss 11.3927 Epoch: 4 Global Step: 44955 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:34:12,880-Speed 124.34 samples/sec Loss 11.4142 Epoch: 4 Global Step: 44960 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:34:32,987-Speed 127.33 samples/sec Loss 11.5016 Epoch: 4 Global Step: 44965 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:34:53,103-Speed 127.27 samples/sec Loss 11.5342 Epoch: 4 Global Step: 44970 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:35:13,075-Speed 128.19 samples/sec Loss 11.1819 Epoch: 4 Global Step: 44975 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:35:33,259-Speed 126.84 samples/sec Loss 11.4249 Epoch: 4 Global Step: 44980 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:35:53,433-Speed 126.90 samples/sec Loss 11.1513 Epoch: 4 Global Step: 44985 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:36:13,610-Speed 126.88 samples/sec Loss 11.4323 Epoch: 4 Global Step: 44990 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:36:33,780-Speed 126.93 samples/sec Loss 11.2709 Epoch: 4 Global Step: 44995 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:36:53,936-Speed 127.01 samples/sec Loss 11.2628 Epoch: 4 Global Step: 45000 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:37:14,574-Speed 124.05 samples/sec Loss 11.1136 Epoch: 4 Global Step: 45005 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:37:34,688-Speed 127.28 samples/sec Loss 11.6473 Epoch: 4 Global Step: 45010 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:37:54,748-Speed 127.62 samples/sec Loss 11.4917 Epoch: 4 Global Step: 45015 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:38:14,880-Speed 127.17 samples/sec Loss 11.2309 Epoch: 4 Global Step: 45020 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:38:35,044-Speed 126.96 samples/sec Loss 11.4183 Epoch: 4 Global Step: 45025 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:38:54,941-Speed 128.67 samples/sec Loss 11.6466 Epoch: 4 Global Step: 45030 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:39:14,786-Speed 129.01 samples/sec Loss 11.4912 Epoch: 4 Global Step: 45035 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:39:34,929-Speed 127.10 samples/sec Loss 11.3131 Epoch: 4 Global Step: 45040 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:39:54,881-Speed 128.31 samples/sec Loss 11.4010 Epoch: 4 Global Step: 45045 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:40:15,619-Speed 123.45 samples/sec Loss 11.6647 Epoch: 4 Global Step: 45050 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:40:35,599-Speed 128.14 samples/sec Loss 11.3980 Epoch: 4 Global Step: 45055 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:40:55,693-Speed 127.40 samples/sec Loss 11.2694 Epoch: 4 Global Step: 45060 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:41:15,728-Speed 127.79 samples/sec Loss 11.4153 Epoch: 4 Global Step: 45065 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:41:35,831-Speed 127.35 samples/sec Loss 11.4813 Epoch: 4 Global Step: 45070 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:41:55,991-Speed 126.99 samples/sec Loss 11.6319 Epoch: 4 Global Step: 45075 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:42:16,148-Speed 127.01 samples/sec Loss 11.2628 Epoch: 4 Global Step: 45080 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:42:36,301-Speed 127.03 samples/sec Loss 11.6668 Epoch: 4 Global Step: 45085 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:42:56,561-Speed 126.37 samples/sec Loss 11.4963 Epoch: 4 Global Step: 45090 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:43:17,740-Speed 120.88 samples/sec Loss 11.3953 Epoch: 4 Global Step: 45095 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:43:38,033-Speed 126.16 samples/sec Loss 11.1504 Epoch: 4 Global Step: 45100 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:43:58,211-Speed 126.87 samples/sec Loss 11.2464 Epoch: 4 Global Step: 45105 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:44:18,344-Speed 127.16 samples/sec Loss 11.3781 Epoch: 4 Global Step: 45110 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:44:38,283-Speed 128.40 samples/sec Loss 11.2864 Epoch: 4 Global Step: 45115 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:44:58,413-Speed 127.17 samples/sec Loss 11.1362 Epoch: 4 Global Step: 45120 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:45:18,628-Speed 126.65 samples/sec Loss 11.4754 Epoch: 4 Global Step: 45125 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:45:38,919-Speed 126.17 samples/sec Loss 11.3829 Epoch: 4 Global Step: 45130 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:45:59,068-Speed 127.06 samples/sec Loss 11.2636 Epoch: 4 Global Step: 45135 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:46:20,177-Speed 121.28 samples/sec Loss 11.4199 Epoch: 4 Global Step: 45140 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:46:40,256-Speed 127.50 samples/sec Loss 11.4160 Epoch: 4 Global Step: 45145 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:47:00,302-Speed 127.71 samples/sec Loss 11.6813 Epoch: 4 Global Step: 45150 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:47:20,312-Speed 127.94 samples/sec Loss 11.0818 Epoch: 4 Global Step: 45155 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:47:40,475-Speed 126.98 samples/sec Loss 11.4327 Epoch: 4 Global Step: 45160 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:48:00,483-Speed 127.95 samples/sec Loss 11.3332 Epoch: 4 Global Step: 45165 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:48:20,605-Speed 127.23 samples/sec Loss 11.1020 Epoch: 4 Global Step: 45170 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:48:40,534-Speed 128.46 samples/sec Loss 11.4698 Epoch: 4 Global Step: 45175 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:49:00,633-Speed 127.38 samples/sec Loss 11.4105 Epoch: 4 Global Step: 45180 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:49:21,145-Speed 124.81 samples/sec Loss 11.4872 Epoch: 4 Global Step: 45185 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:49:41,285-Speed 127.12 samples/sec Loss 11.6615 Epoch: 4 Global Step: 45190 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:50:01,230-Speed 128.35 samples/sec Loss 11.2792 Epoch: 4 Global Step: 45195 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:50:21,397-Speed 126.95 samples/sec Loss 11.6160 Epoch: 4 Global Step: 45200 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:50:41,748-Speed 125.80 samples/sec Loss 11.4040 Epoch: 4 Global Step: 45205 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:51:02,029-Speed 126.24 samples/sec Loss 11.8564 Epoch: 4 Global Step: 45210 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:51:22,136-Speed 127.32 samples/sec Loss 11.4450 Epoch: 4 Global Step: 45215 Fp16 Grad Scale: 8192 Required: 243 hours Training: 2021-06-09 00:51:42,260-Speed 127.22 samples/sec Loss 11.1067 Epoch: 4 Global Step: 45220 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:52:02,458-Speed 126.76 samples/sec Loss 11.4332 Epoch: 4 Global Step: 45225 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:52:23,300-Speed 122.83 samples/sec Loss 11.4525 Epoch: 4 Global Step: 45230 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:52:43,540-Speed 126.49 samples/sec Loss 11.3269 Epoch: 4 Global Step: 45235 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:53:03,618-Speed 127.51 samples/sec Loss 11.5512 Epoch: 4 Global Step: 45240 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:53:23,835-Speed 126.63 samples/sec Loss 11.5469 Epoch: 4 Global Step: 45245 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:53:44,318-Speed 124.99 samples/sec Loss 11.6907 Epoch: 4 Global Step: 45250 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:54:04,351-Speed 127.80 samples/sec Loss 11.2348 Epoch: 4 Global Step: 45255 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:54:25,204-Speed 122.77 samples/sec Loss 11.5961 Epoch: 4 Global Step: 45260 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:54:45,393-Speed 126.81 samples/sec Loss 11.0715 Epoch: 4 Global Step: 45265 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:55:05,509-Speed 127.27 samples/sec Loss 11.0934 Epoch: 4 Global Step: 45270 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:55:26,147-Speed 124.05 samples/sec Loss 11.7490 Epoch: 4 Global Step: 45275 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:55:46,041-Speed 128.69 samples/sec Loss 11.5740 Epoch: 4 Global Step: 45280 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:56:06,136-Speed 127.40 samples/sec Loss 11.1243 Epoch: 4 Global Step: 45285 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:56:26,154-Speed 127.89 samples/sec Loss 11.4404 Epoch: 4 Global Step: 45290 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:56:46,098-Speed 128.37 samples/sec Loss 11.1673 Epoch: 4 Global Step: 45295 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:57:06,181-Speed 127.48 samples/sec Loss 11.6021 Epoch: 4 Global Step: 45300 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:57:26,170-Speed 128.07 samples/sec Loss 11.5117 Epoch: 4 Global Step: 45305 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:57:46,279-Speed 127.32 samples/sec Loss 11.6423 Epoch: 4 Global Step: 45310 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:58:06,473-Speed 126.78 samples/sec Loss 11.3902 Epoch: 4 Global Step: 45315 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:58:27,162-Speed 123.74 samples/sec Loss 11.5942 Epoch: 4 Global Step: 45320 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:58:47,231-Speed 127.57 samples/sec Loss 11.9024 Epoch: 4 Global Step: 45325 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:59:07,365-Speed 127.15 samples/sec Loss 11.3879 Epoch: 4 Global Step: 45330 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:59:27,312-Speed 128.35 samples/sec Loss 11.4517 Epoch: 4 Global Step: 45335 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 00:59:47,400-Speed 127.45 samples/sec Loss 11.6116 Epoch: 4 Global Step: 45340 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:00:07,474-Speed 127.54 samples/sec Loss 11.5767 Epoch: 4 Global Step: 45345 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:00:27,949-Speed 125.04 samples/sec Loss 11.7865 Epoch: 4 Global Step: 45350 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:00:48,136-Speed 126.82 samples/sec Loss 11.5269 Epoch: 4 Global Step: 45355 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:01:08,097-Speed 128.25 samples/sec Loss 11.5468 Epoch: 4 Global Step: 45360 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:01:29,421-Speed 120.06 samples/sec Loss 11.4319 Epoch: 4 Global Step: 45365 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:01:49,590-Speed 126.93 samples/sec Loss 11.6433 Epoch: 4 Global Step: 45370 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:02:09,837-Speed 126.44 samples/sec Loss 11.5730 Epoch: 4 Global Step: 45375 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:02:30,093-Speed 126.39 samples/sec Loss 11.1908 Epoch: 4 Global Step: 45380 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:02:50,237-Speed 127.09 samples/sec Loss 11.3670 Epoch: 4 Global Step: 45385 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:03:10,210-Speed 128.18 samples/sec Loss 11.4450 Epoch: 4 Global Step: 45390 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:03:30,466-Speed 126.39 samples/sec Loss 11.4504 Epoch: 4 Global Step: 45395 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:03:50,589-Speed 127.23 samples/sec Loss 11.4846 Epoch: 4 Global Step: 45400 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:04:10,646-Speed 127.64 samples/sec Loss 11.7508 Epoch: 4 Global Step: 45405 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:04:31,290-Speed 124.01 samples/sec Loss 11.4116 Epoch: 4 Global Step: 45410 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:04:51,392-Speed 127.36 samples/sec Loss 11.7325 Epoch: 4 Global Step: 45415 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:05:11,694-Speed 126.10 samples/sec Loss 11.4569 Epoch: 4 Global Step: 45420 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:05:31,677-Speed 128.11 samples/sec Loss 11.5438 Epoch: 4 Global Step: 45425 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:05:52,013-Speed 125.89 samples/sec Loss 11.3811 Epoch: 4 Global Step: 45430 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:06:12,728-Speed 123.59 samples/sec Loss 11.4753 Epoch: 4 Global Step: 45435 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:06:32,858-Speed 127.18 samples/sec Loss 11.4625 Epoch: 4 Global Step: 45440 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:06:53,163-Speed 126.08 samples/sec Loss 11.1879 Epoch: 4 Global Step: 45445 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:07:13,110-Speed 128.35 samples/sec Loss 11.3824 Epoch: 4 Global Step: 45450 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:07:33,693-Speed 124.38 samples/sec Loss 11.3877 Epoch: 4 Global Step: 45455 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:07:53,868-Speed 126.89 samples/sec Loss 11.3998 Epoch: 4 Global Step: 45460 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:08:13,907-Speed 127.76 samples/sec Loss 11.5964 Epoch: 4 Global Step: 45465 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:08:34,029-Speed 127.23 samples/sec Loss 11.6147 Epoch: 4 Global Step: 45470 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:08:54,224-Speed 126.77 samples/sec Loss 11.3815 Epoch: 4 Global Step: 45475 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:09:14,461-Speed 126.51 samples/sec Loss 11.6726 Epoch: 4 Global Step: 45480 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:09:34,608-Speed 127.08 samples/sec Loss 11.5499 Epoch: 4 Global Step: 45485 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:09:54,777-Speed 126.94 samples/sec Loss 11.3334 Epoch: 4 Global Step: 45490 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:10:15,089-Speed 126.04 samples/sec Loss 11.5376 Epoch: 4 Global Step: 45495 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:10:35,925-Speed 122.87 samples/sec Loss 11.2939 Epoch: 4 Global Step: 45500 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:10:56,010-Speed 127.46 samples/sec Loss 11.4064 Epoch: 4 Global Step: 45505 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:11:16,097-Speed 127.46 samples/sec Loss 11.1942 Epoch: 4 Global Step: 45510 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:11:36,445-Speed 125.82 samples/sec Loss 11.5732 Epoch: 4 Global Step: 45515 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:11:56,539-Speed 127.40 samples/sec Loss 11.3378 Epoch: 4 Global Step: 45520 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:12:16,715-Speed 126.90 samples/sec Loss 11.5030 Epoch: 4 Global Step: 45525 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:12:36,732-Speed 127.90 samples/sec Loss 11.1984 Epoch: 4 Global Step: 45530 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:12:57,018-Speed 126.20 samples/sec Loss 11.3178 Epoch: 4 Global Step: 45535 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:13:17,185-Speed 126.95 samples/sec Loss 11.2474 Epoch: 4 Global Step: 45540 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:13:37,971-Speed 123.16 samples/sec Loss 11.0511 Epoch: 4 Global Step: 45545 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:13:58,105-Speed 127.15 samples/sec Loss 11.3246 Epoch: 4 Global Step: 45550 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:14:18,283-Speed 126.88 samples/sec Loss 11.7260 Epoch: 4 Global Step: 45555 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:14:38,410-Speed 127.20 samples/sec Loss 11.1224 Epoch: 4 Global Step: 45560 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:14:58,591-Speed 126.86 samples/sec Loss 11.2833 Epoch: 4 Global Step: 45565 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:15:18,795-Speed 126.72 samples/sec Loss 11.4089 Epoch: 4 Global Step: 45570 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:15:39,515-Speed 123.56 samples/sec Loss 11.4139 Epoch: 4 Global Step: 45575 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:15:59,729-Speed 126.65 samples/sec Loss 11.5557 Epoch: 4 Global Step: 45580 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:16:19,950-Speed 126.61 samples/sec Loss 11.6457 Epoch: 4 Global Step: 45585 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:16:40,832-Speed 122.60 samples/sec Loss 11.2311 Epoch: 4 Global Step: 45590 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:17:01,100-Speed 126.32 samples/sec Loss 11.2212 Epoch: 4 Global Step: 45595 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:17:21,229-Speed 127.20 samples/sec Loss 11.3228 Epoch: 4 Global Step: 45600 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:17:41,487-Speed 126.38 samples/sec Loss 11.1820 Epoch: 4 Global Step: 45605 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:18:01,633-Speed 127.08 samples/sec Loss 11.1558 Epoch: 4 Global Step: 45610 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:18:21,816-Speed 126.85 samples/sec Loss 11.6165 Epoch: 4 Global Step: 45615 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:18:42,085-Speed 126.31 samples/sec Loss 11.2833 Epoch: 4 Global Step: 45620 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:19:02,221-Speed 127.15 samples/sec Loss 11.6127 Epoch: 4 Global Step: 45625 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:19:22,165-Speed 128.37 samples/sec Loss 11.2485 Epoch: 4 Global Step: 45630 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:19:42,776-Speed 124.21 samples/sec Loss 11.2588 Epoch: 4 Global Step: 45635 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:20:03,112-Speed 125.89 samples/sec Loss 11.1436 Epoch: 4 Global Step: 45640 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:20:24,003-Speed 122.55 samples/sec Loss 11.2496 Epoch: 4 Global Step: 45645 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:20:44,209-Speed 126.71 samples/sec Loss 11.2487 Epoch: 4 Global Step: 45650 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:21:04,335-Speed 127.20 samples/sec Loss 11.4470 Epoch: 4 Global Step: 45655 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:21:24,488-Speed 127.04 samples/sec Loss 11.4491 Epoch: 4 Global Step: 45660 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:21:44,419-Speed 128.45 samples/sec Loss 11.5635 Epoch: 4 Global Step: 45665 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:22:04,698-Speed 126.25 samples/sec Loss 11.4286 Epoch: 4 Global Step: 45670 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:22:24,980-Speed 126.23 samples/sec Loss 11.5978 Epoch: 4 Global Step: 45675 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:22:45,924-Speed 122.23 samples/sec Loss 11.3267 Epoch: 4 Global Step: 45680 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:23:06,135-Speed 126.67 samples/sec Loss 11.0296 Epoch: 4 Global Step: 45685 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:23:26,313-Speed 126.88 samples/sec Loss 11.1844 Epoch: 4 Global Step: 45690 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:23:46,422-Speed 127.31 samples/sec Loss 11.3961 Epoch: 4 Global Step: 45695 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:24:06,574-Speed 127.04 samples/sec Loss 11.8055 Epoch: 4 Global Step: 45700 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:24:26,688-Speed 127.29 samples/sec Loss 11.4529 Epoch: 4 Global Step: 45705 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:24:46,850-Speed 126.98 samples/sec Loss 11.1348 Epoch: 4 Global Step: 45710 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:25:06,990-Speed 127.12 samples/sec Loss 11.0179 Epoch: 4 Global Step: 45715 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:25:27,156-Speed 126.95 samples/sec Loss 11.6330 Epoch: 4 Global Step: 45720 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:25:48,237-Speed 121.44 samples/sec Loss 11.4809 Epoch: 4 Global Step: 45725 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:26:08,466-Speed 126.56 samples/sec Loss 11.2377 Epoch: 4 Global Step: 45730 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:26:28,638-Speed 126.92 samples/sec Loss 11.2856 Epoch: 4 Global Step: 45735 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:26:48,840-Speed 126.73 samples/sec Loss 11.1980 Epoch: 4 Global Step: 45740 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:27:09,023-Speed 126.85 samples/sec Loss 11.3463 Epoch: 4 Global Step: 45745 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:27:29,213-Speed 126.80 samples/sec Loss 11.3697 Epoch: 4 Global Step: 45750 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:27:49,223-Speed 127.94 samples/sec Loss 11.1966 Epoch: 4 Global Step: 45755 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:28:09,398-Speed 126.90 samples/sec Loss 11.3142 Epoch: 4 Global Step: 45760 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:28:29,556-Speed 127.00 samples/sec Loss 11.1522 Epoch: 4 Global Step: 45765 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:28:50,426-Speed 122.67 samples/sec Loss 11.7200 Epoch: 4 Global Step: 45770 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:29:10,435-Speed 127.95 samples/sec Loss 11.5233 Epoch: 4 Global Step: 45775 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:29:30,546-Speed 127.30 samples/sec Loss 11.4085 Epoch: 4 Global Step: 45780 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:29:50,636-Speed 127.43 samples/sec Loss 11.1560 Epoch: 4 Global Step: 45785 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:30:10,638-Speed 127.99 samples/sec Loss 11.3636 Epoch: 4 Global Step: 45790 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:30:30,746-Speed 127.32 samples/sec Loss 10.9944 Epoch: 4 Global Step: 45795 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:30:50,938-Speed 126.79 samples/sec Loss 11.8490 Epoch: 4 Global Step: 45800 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:31:11,101-Speed 126.97 samples/sec Loss 11.3240 Epoch: 4 Global Step: 45805 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:31:31,520-Speed 125.38 samples/sec Loss 11.5967 Epoch: 4 Global Step: 45810 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:31:52,345-Speed 122.94 samples/sec Loss 11.6000 Epoch: 4 Global Step: 45815 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:32:12,596-Speed 126.42 samples/sec Loss 11.3291 Epoch: 4 Global Step: 45820 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:32:32,936-Speed 125.86 samples/sec Loss 11.2638 Epoch: 4 Global Step: 45825 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:32:53,049-Speed 127.29 samples/sec Loss 11.3897 Epoch: 4 Global Step: 45830 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:33:12,966-Speed 128.54 samples/sec Loss 11.4250 Epoch: 4 Global Step: 45835 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:33:32,957-Speed 128.06 samples/sec Loss 11.0229 Epoch: 4 Global Step: 45840 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:33:53,097-Speed 127.12 samples/sec Loss 11.3299 Epoch: 4 Global Step: 45845 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:34:13,343-Speed 126.46 samples/sec Loss 11.3273 Epoch: 4 Global Step: 45850 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:34:33,577-Speed 126.52 samples/sec Loss 11.4980 Epoch: 4 Global Step: 45855 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:34:54,272-Speed 123.71 samples/sec Loss 11.4615 Epoch: 4 Global Step: 45860 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:35:14,704-Speed 125.30 samples/sec Loss 11.4898 Epoch: 4 Global Step: 45865 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:35:34,684-Speed 128.14 samples/sec Loss 11.0488 Epoch: 4 Global Step: 45870 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:35:54,898-Speed 126.65 samples/sec Loss 11.4958 Epoch: 4 Global Step: 45875 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:36:15,093-Speed 126.77 samples/sec Loss 11.0916 Epoch: 4 Global Step: 45880 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:36:35,216-Speed 127.23 samples/sec Loss 11.3029 Epoch: 4 Global Step: 45885 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:36:55,439-Speed 126.59 samples/sec Loss 11.0615 Epoch: 4 Global Step: 45890 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:37:16,137-Speed 123.69 samples/sec Loss 11.7814 Epoch: 4 Global Step: 45895 Fp16 Grad Scale: 8192 Required: 242 hours Training: 2021-06-09 01:37:36,342-Speed 126.71 samples/sec Loss 11.3603 Epoch: 4 Global Step: 45900 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:37:57,228-Speed 122.58 samples/sec Loss 11.6049 Epoch: 4 Global Step: 45905 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:38:17,344-Speed 127.27 samples/sec Loss 11.4764 Epoch: 4 Global Step: 45910 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:38:37,629-Speed 126.20 samples/sec Loss 11.6123 Epoch: 4 Global Step: 45915 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:38:57,902-Speed 126.28 samples/sec Loss 11.2922 Epoch: 4 Global Step: 45920 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:39:18,206-Speed 126.09 samples/sec Loss 11.8299 Epoch: 4 Global Step: 45925 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:39:38,396-Speed 126.81 samples/sec Loss 11.4987 Epoch: 4 Global Step: 45930 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:39:58,527-Speed 127.17 samples/sec Loss 11.2453 Epoch: 4 Global Step: 45935 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:40:18,768-Speed 126.48 samples/sec Loss 11.3351 Epoch: 4 Global Step: 45940 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:40:38,870-Speed 127.36 samples/sec Loss 11.5208 Epoch: 4 Global Step: 45945 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:40:59,576-Speed 123.64 samples/sec Loss 11.1670 Epoch: 4 Global Step: 45950 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:41:19,831-Speed 126.40 samples/sec Loss 11.5381 Epoch: 4 Global Step: 45955 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:41:40,039-Speed 126.69 samples/sec Loss 11.1022 Epoch: 4 Global Step: 45960 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:42:00,185-Speed 127.08 samples/sec Loss 11.6330 Epoch: 4 Global Step: 45965 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:42:20,219-Speed 127.79 samples/sec Loss 11.0768 Epoch: 4 Global Step: 45970 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:42:40,379-Speed 126.99 samples/sec Loss 11.4489 Epoch: 4 Global Step: 45975 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:43:00,549-Speed 126.93 samples/sec Loss 11.4107 Epoch: 4 Global Step: 45980 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:43:20,862-Speed 126.03 samples/sec Loss 11.5664 Epoch: 4 Global Step: 45985 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:43:41,024-Speed 126.98 samples/sec Loss 11.7319 Epoch: 4 Global Step: 45990 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:44:01,839-Speed 123.00 samples/sec Loss 11.2351 Epoch: 4 Global Step: 45995 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:44:21,926-Speed 127.45 samples/sec Loss 11.4044 Epoch: 4 Global Step: 46000 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:44:42,113-Speed 126.82 samples/sec Loss 11.6506 Epoch: 4 Global Step: 46005 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:45:02,218-Speed 127.34 samples/sec Loss 11.3610 Epoch: 4 Global Step: 46010 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:45:22,295-Speed 127.52 samples/sec Loss 11.2965 Epoch: 4 Global Step: 46015 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:45:42,441-Speed 127.08 samples/sec Loss 11.3898 Epoch: 4 Global Step: 46020 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:46:02,633-Speed 126.79 samples/sec Loss 11.4980 Epoch: 4 Global Step: 46025 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:46:22,802-Speed 126.93 samples/sec Loss 11.4041 Epoch: 4 Global Step: 46030 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:46:43,041-Speed 126.50 samples/sec Loss 11.5298 Epoch: 4 Global Step: 46035 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:47:04,174-Speed 121.14 samples/sec Loss 11.5275 Epoch: 4 Global Step: 46040 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:47:24,568-Speed 125.53 samples/sec Loss 11.3624 Epoch: 4 Global Step: 46045 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:47:44,877-Speed 126.06 samples/sec Loss 11.5217 Epoch: 4 Global Step: 46050 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:48:05,048-Speed 126.93 samples/sec Loss 11.2811 Epoch: 4 Global Step: 46055 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:48:25,189-Speed 127.11 samples/sec Loss 11.3917 Epoch: 4 Global Step: 46060 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:48:45,328-Speed 127.13 samples/sec Loss 11.4312 Epoch: 4 Global Step: 46065 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:49:05,486-Speed 127.01 samples/sec Loss 11.1429 Epoch: 4 Global Step: 46070 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:49:25,997-Speed 124.82 samples/sec Loss 11.5358 Epoch: 4 Global Step: 46075 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:49:46,227-Speed 126.55 samples/sec Loss 11.6733 Epoch: 4 Global Step: 46080 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:50:07,060-Speed 122.88 samples/sec Loss 11.3219 Epoch: 4 Global Step: 46085 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:50:27,214-Speed 127.03 samples/sec Loss 11.5041 Epoch: 4 Global Step: 46090 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:50:47,233-Speed 127.89 samples/sec Loss 11.5335 Epoch: 4 Global Step: 46095 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:51:07,294-Speed 127.62 samples/sec Loss 11.4320 Epoch: 4 Global Step: 46100 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:51:27,540-Speed 126.45 samples/sec Loss 11.2249 Epoch: 4 Global Step: 46105 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:51:47,699-Speed 127.00 samples/sec Loss 10.9850 Epoch: 4 Global Step: 46110 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:52:07,908-Speed 126.68 samples/sec Loss 11.5689 Epoch: 4 Global Step: 46115 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:52:28,070-Speed 126.98 samples/sec Loss 11.3605 Epoch: 4 Global Step: 46120 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:52:48,214-Speed 127.10 samples/sec Loss 11.3653 Epoch: 4 Global Step: 46125 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:53:09,030-Speed 122.98 samples/sec Loss 11.3547 Epoch: 4 Global Step: 46130 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:53:29,231-Speed 126.73 samples/sec Loss 11.1549 Epoch: 4 Global Step: 46135 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:53:49,339-Speed 127.32 samples/sec Loss 11.2544 Epoch: 4 Global Step: 46140 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:54:09,690-Speed 125.80 samples/sec Loss 11.2762 Epoch: 4 Global Step: 46145 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:54:29,780-Speed 127.43 samples/sec Loss 11.2327 Epoch: 4 Global Step: 46150 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:54:49,689-Speed 128.59 samples/sec Loss 11.6679 Epoch: 4 Global Step: 46155 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:55:09,680-Speed 128.06 samples/sec Loss 11.4225 Epoch: 4 Global Step: 46160 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:55:29,838-Speed 127.00 samples/sec Loss 11.3572 Epoch: 4 Global Step: 46165 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:55:49,796-Speed 128.28 samples/sec Loss 11.1491 Epoch: 4 Global Step: 46170 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:56:10,382-Speed 124.36 samples/sec Loss 11.2850 Epoch: 4 Global Step: 46175 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:56:30,490-Speed 127.32 samples/sec Loss 11.5508 Epoch: 4 Global Step: 46180 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:56:50,644-Speed 127.02 samples/sec Loss 11.1891 Epoch: 4 Global Step: 46185 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:57:10,614-Speed 128.20 samples/sec Loss 11.2814 Epoch: 4 Global Step: 46190 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:57:30,850-Speed 126.51 samples/sec Loss 11.5867 Epoch: 4 Global Step: 46195 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:57:50,777-Speed 128.48 samples/sec Loss 11.6763 Epoch: 4 Global Step: 46200 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:58:10,891-Speed 127.28 samples/sec Loss 11.4213 Epoch: 4 Global Step: 46205 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:58:30,862-Speed 128.20 samples/sec Loss 11.5787 Epoch: 4 Global Step: 46210 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:58:51,031-Speed 126.94 samples/sec Loss 11.3713 Epoch: 4 Global Step: 46215 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:59:12,253-Speed 120.64 samples/sec Loss 11.4088 Epoch: 4 Global Step: 46220 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:59:32,446-Speed 126.78 samples/sec Loss 11.3783 Epoch: 4 Global Step: 46225 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 01:59:52,569-Speed 127.22 samples/sec Loss 11.5019 Epoch: 4 Global Step: 46230 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:00:12,727-Speed 127.00 samples/sec Loss 11.1854 Epoch: 4 Global Step: 46235 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:00:33,153-Speed 125.34 samples/sec Loss 11.6953 Epoch: 4 Global Step: 46240 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:00:53,330-Speed 126.88 samples/sec Loss 11.5327 Epoch: 4 Global Step: 46245 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:01:13,452-Speed 127.23 samples/sec Loss 11.6700 Epoch: 4 Global Step: 46250 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:01:33,558-Speed 127.34 samples/sec Loss 11.5737 Epoch: 4 Global Step: 46255 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:01:53,814-Speed 126.39 samples/sec Loss 11.4027 Epoch: 4 Global Step: 46260 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:02:14,998-Speed 120.85 samples/sec Loss 11.6414 Epoch: 4 Global Step: 46265 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:02:35,234-Speed 126.51 samples/sec Loss 11.2274 Epoch: 4 Global Step: 46270 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:02:55,441-Speed 126.69 samples/sec Loss 11.6541 Epoch: 4 Global Step: 46275 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:03:15,572-Speed 127.17 samples/sec Loss 11.2882 Epoch: 4 Global Step: 46280 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:03:35,689-Speed 127.26 samples/sec Loss 11.2455 Epoch: 4 Global Step: 46285 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:03:56,091-Speed 125.49 samples/sec Loss 11.5117 Epoch: 4 Global Step: 46290 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:04:16,216-Speed 127.21 samples/sec Loss 11.2007 Epoch: 4 Global Step: 46295 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:04:36,463-Speed 126.45 samples/sec Loss 11.4170 Epoch: 4 Global Step: 46300 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:04:56,732-Speed 126.31 samples/sec Loss 11.6541 Epoch: 4 Global Step: 46305 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:05:17,693-Speed 122.14 samples/sec Loss 11.1251 Epoch: 4 Global Step: 46310 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:05:37,810-Speed 127.26 samples/sec Loss 11.2707 Epoch: 4 Global Step: 46315 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:05:58,034-Speed 126.59 samples/sec Loss 11.9827 Epoch: 4 Global Step: 46320 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:06:18,385-Speed 125.80 samples/sec Loss 11.4433 Epoch: 4 Global Step: 46325 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:06:38,329-Speed 128.37 samples/sec Loss 11.3127 Epoch: 4 Global Step: 46330 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:06:58,499-Speed 126.93 samples/sec Loss 11.5345 Epoch: 4 Global Step: 46335 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:07:18,653-Speed 127.03 samples/sec Loss 11.3824 Epoch: 4 Global Step: 46340 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:07:38,775-Speed 127.23 samples/sec Loss 11.2693 Epoch: 4 Global Step: 46345 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:07:58,722-Speed 128.35 samples/sec Loss 11.1761 Epoch: 4 Global Step: 46350 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:08:19,640-Speed 122.39 samples/sec Loss 11.2197 Epoch: 4 Global Step: 46355 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:08:39,787-Speed 127.08 samples/sec Loss 11.4490 Epoch: 4 Global Step: 46360 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:08:59,780-Speed 128.05 samples/sec Loss 11.2736 Epoch: 4 Global Step: 46365 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:09:19,888-Speed 127.32 samples/sec Loss 11.3780 Epoch: 4 Global Step: 46370 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:09:40,270-Speed 125.61 samples/sec Loss 11.4707 Epoch: 4 Global Step: 46375 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:10:00,451-Speed 126.86 samples/sec Loss 11.4428 Epoch: 4 Global Step: 46380 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:10:20,738-Speed 126.20 samples/sec Loss 11.4907 Epoch: 4 Global Step: 46385 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:10:40,861-Speed 127.23 samples/sec Loss 11.0879 Epoch: 4 Global Step: 46390 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:11:00,865-Speed 127.98 samples/sec Loss 11.4342 Epoch: 4 Global Step: 46395 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:11:21,739-Speed 122.65 samples/sec Loss 11.7037 Epoch: 4 Global Step: 46400 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:11:42,034-Speed 126.15 samples/sec Loss 11.5341 Epoch: 4 Global Step: 46405 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:12:02,165-Speed 127.17 samples/sec Loss 11.4050 Epoch: 4 Global Step: 46410 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:12:22,373-Speed 126.69 samples/sec Loss 11.0988 Epoch: 4 Global Step: 46415 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:12:42,487-Speed 127.28 samples/sec Loss 11.2338 Epoch: 4 Global Step: 46420 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:13:02,649-Speed 126.98 samples/sec Loss 11.5003 Epoch: 4 Global Step: 46425 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:13:22,783-Speed 127.15 samples/sec Loss 11.1675 Epoch: 4 Global Step: 46430 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:13:42,902-Speed 127.25 samples/sec Loss 11.4454 Epoch: 4 Global Step: 46435 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:14:02,983-Speed 127.49 samples/sec Loss 11.0975 Epoch: 4 Global Step: 46440 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:14:23,893-Speed 122.43 samples/sec Loss 11.7513 Epoch: 4 Global Step: 46445 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:14:44,128-Speed 126.52 samples/sec Loss 11.3208 Epoch: 4 Global Step: 46450 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:15:04,331-Speed 126.72 samples/sec Loss 11.3211 Epoch: 4 Global Step: 46455 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:15:24,538-Speed 126.70 samples/sec Loss 10.8987 Epoch: 4 Global Step: 46460 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:15:44,817-Speed 126.25 samples/sec Loss 11.0375 Epoch: 4 Global Step: 46465 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:16:04,965-Speed 127.07 samples/sec Loss 11.5229 Epoch: 4 Global Step: 46470 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:16:24,899-Speed 128.43 samples/sec Loss 11.1922 Epoch: 4 Global Step: 46475 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:16:45,051-Speed 127.05 samples/sec Loss 10.9845 Epoch: 4 Global Step: 46480 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:17:05,239-Speed 126.81 samples/sec Loss 11.5394 Epoch: 4 Global Step: 46485 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:17:26,124-Speed 122.59 samples/sec Loss 11.4055 Epoch: 4 Global Step: 46490 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:17:46,255-Speed 127.17 samples/sec Loss 11.2522 Epoch: 4 Global Step: 46495 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:18:06,397-Speed 127.11 samples/sec Loss 11.3809 Epoch: 4 Global Step: 46500 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:18:26,339-Speed 128.38 samples/sec Loss 11.3291 Epoch: 4 Global Step: 46505 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:18:46,413-Speed 127.54 samples/sec Loss 11.2979 Epoch: 4 Global Step: 46510 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:19:06,325-Speed 128.57 samples/sec Loss 11.3907 Epoch: 4 Global Step: 46515 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:19:26,516-Speed 126.80 samples/sec Loss 11.4118 Epoch: 4 Global Step: 46520 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:19:46,686-Speed 126.93 samples/sec Loss 11.3857 Epoch: 4 Global Step: 46525 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:20:06,872-Speed 126.83 samples/sec Loss 11.5042 Epoch: 4 Global Step: 46530 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:20:27,560-Speed 123.75 samples/sec Loss 11.2922 Epoch: 4 Global Step: 46535 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:20:47,520-Speed 128.26 samples/sec Loss 11.5535 Epoch: 4 Global Step: 46540 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:21:07,415-Speed 128.68 samples/sec Loss 11.2284 Epoch: 4 Global Step: 46545 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:21:27,511-Speed 127.40 samples/sec Loss 11.3890 Epoch: 4 Global Step: 46550 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:21:47,646-Speed 127.15 samples/sec Loss 11.4221 Epoch: 4 Global Step: 46555 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:22:07,787-Speed 127.11 samples/sec Loss 11.4274 Epoch: 4 Global Step: 46560 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:22:27,774-Speed 128.09 samples/sec Loss 11.4099 Epoch: 4 Global Step: 46565 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:22:47,903-Speed 127.19 samples/sec Loss 11.8406 Epoch: 4 Global Step: 46570 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:23:08,017-Speed 127.28 samples/sec Loss 11.8302 Epoch: 4 Global Step: 46575 Fp16 Grad Scale: 8192 Required: 241 hours Training: 2021-06-09 02:23:28,723-Speed 123.64 samples/sec Loss 11.6406 Epoch: 4 Global Step: 46580 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:23:48,929-Speed 126.70 samples/sec Loss 11.3270 Epoch: 4 Global Step: 46585 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:24:09,063-Speed 127.15 samples/sec Loss 11.4929 Epoch: 4 Global Step: 46590 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:24:29,233-Speed 126.93 samples/sec Loss 11.7113 Epoch: 4 Global Step: 46595 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:24:49,336-Speed 127.35 samples/sec Loss 11.5432 Epoch: 4 Global Step: 46600 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:25:09,436-Speed 127.37 samples/sec Loss 11.3355 Epoch: 4 Global Step: 46605 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:25:29,586-Speed 127.06 samples/sec Loss 11.4594 Epoch: 4 Global Step: 46610 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:25:49,570-Speed 128.11 samples/sec Loss 11.4535 Epoch: 4 Global Step: 46615 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:26:10,715-Speed 121.07 samples/sec Loss 11.2440 Epoch: 4 Global Step: 46620 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:26:31,467-Speed 123.37 samples/sec Loss 11.3081 Epoch: 4 Global Step: 46625 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:26:51,418-Speed 128.32 samples/sec Loss 11.6332 Epoch: 4 Global Step: 46630 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:27:11,523-Speed 127.33 samples/sec Loss 11.4900 Epoch: 4 Global Step: 46635 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:27:31,466-Speed 128.37 samples/sec Loss 11.2659 Epoch: 4 Global Step: 46640 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:27:51,509-Speed 127.74 samples/sec Loss 11.2476 Epoch: 4 Global Step: 46645 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:28:11,593-Speed 127.47 samples/sec Loss 11.5319 Epoch: 4 Global Step: 46650 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:28:31,516-Speed 128.50 samples/sec Loss 11.1837 Epoch: 4 Global Step: 46655 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:28:51,595-Speed 127.50 samples/sec Loss 11.4444 Epoch: 4 Global Step: 46660 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:29:11,509-Speed 128.56 samples/sec Loss 11.1433 Epoch: 4 Global Step: 46665 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:29:32,102-Speed 124.32 samples/sec Loss 11.1619 Epoch: 4 Global Step: 46670 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:29:52,076-Speed 128.17 samples/sec Loss 11.5297 Epoch: 4 Global Step: 46675 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:30:12,139-Speed 127.60 samples/sec Loss 11.3248 Epoch: 4 Global Step: 46680 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:30:32,621-Speed 124.99 samples/sec Loss 11.4425 Epoch: 4 Global Step: 46685 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:30:52,752-Speed 127.18 samples/sec Loss 11.3054 Epoch: 4 Global Step: 46690 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:31:12,871-Speed 127.25 samples/sec Loss 11.6171 Epoch: 4 Global Step: 46695 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:31:33,093-Speed 126.60 samples/sec Loss 11.7660 Epoch: 4 Global Step: 46700 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:31:53,355-Speed 126.36 samples/sec Loss 11.3088 Epoch: 4 Global Step: 46705 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:32:13,675-Speed 125.99 samples/sec Loss 11.3468 Epoch: 4 Global Step: 46710 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:32:34,405-Speed 123.50 samples/sec Loss 11.5547 Epoch: 4 Global Step: 46715 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:32:54,486-Speed 127.49 samples/sec Loss 11.0935 Epoch: 4 Global Step: 46720 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:33:14,585-Speed 127.38 samples/sec Loss 11.3150 Epoch: 4 Global Step: 46725 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:33:34,529-Speed 128.37 samples/sec Loss 11.4148 Epoch: 4 Global Step: 46730 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:33:55,214-Speed 123.77 samples/sec Loss 11.6343 Epoch: 4 Global Step: 46735 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:34:15,289-Speed 127.53 samples/sec Loss 11.5514 Epoch: 4 Global Step: 46740 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:34:35,386-Speed 127.39 samples/sec Loss 11.1105 Epoch: 4 Global Step: 46745 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:34:55,506-Speed 127.24 samples/sec Loss 11.0402 Epoch: 4 Global Step: 46750 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:35:15,499-Speed 128.05 samples/sec Loss 11.3526 Epoch: 4 Global Step: 46755 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:35:36,305-Speed 123.05 samples/sec Loss 11.2084 Epoch: 4 Global Step: 46760 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:35:56,440-Speed 127.15 samples/sec Loss 11.6678 Epoch: 4 Global Step: 46765 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:36:16,435-Speed 128.04 samples/sec Loss 11.2673 Epoch: 4 Global Step: 46770 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:36:36,613-Speed 126.87 samples/sec Loss 11.2238 Epoch: 4 Global Step: 46775 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:36:56,548-Speed 128.43 samples/sec Loss 11.0330 Epoch: 4 Global Step: 46780 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:37:16,742-Speed 126.78 samples/sec Loss 11.1161 Epoch: 4 Global Step: 46785 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:37:36,926-Speed 126.84 samples/sec Loss 11.5810 Epoch: 4 Global Step: 46790 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:37:57,091-Speed 126.96 samples/sec Loss 11.3613 Epoch: 4 Global Step: 46795 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:38:17,397-Speed 126.08 samples/sec Loss 11.1572 Epoch: 4 Global Step: 46800 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:38:38,367-Speed 122.09 samples/sec Loss 11.6165 Epoch: 4 Global Step: 46805 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:38:58,477-Speed 127.31 samples/sec Loss 11.3368 Epoch: 4 Global Step: 46810 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:39:18,859-Speed 125.61 samples/sec Loss 11.2951 Epoch: 4 Global Step: 46815 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:39:38,967-Speed 127.32 samples/sec Loss 11.4182 Epoch: 4 Global Step: 46820 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:39:58,929-Speed 128.25 samples/sec Loss 11.1843 Epoch: 4 Global Step: 46825 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:40:19,300-Speed 125.68 samples/sec Loss 11.0901 Epoch: 4 Global Step: 46830 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:40:39,224-Speed 128.50 samples/sec Loss 11.4058 Epoch: 4 Global Step: 46835 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:40:59,375-Speed 127.05 samples/sec Loss 11.3405 Epoch: 4 Global Step: 46840 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:41:19,482-Speed 127.33 samples/sec Loss 11.4108 Epoch: 4 Global Step: 46845 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:41:40,290-Speed 123.04 samples/sec Loss 11.4714 Epoch: 4 Global Step: 46850 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:42:00,488-Speed 126.75 samples/sec Loss 11.2651 Epoch: 4 Global Step: 46855 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:42:20,624-Speed 127.14 samples/sec Loss 11.2761 Epoch: 4 Global Step: 46860 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:42:40,709-Speed 127.46 samples/sec Loss 11.4575 Epoch: 4 Global Step: 46865 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:43:00,836-Speed 127.20 samples/sec Loss 11.4776 Epoch: 4 Global Step: 46870 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:43:20,756-Speed 128.52 samples/sec Loss 11.3997 Epoch: 4 Global Step: 46875 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:43:40,939-Speed 126.85 samples/sec Loss 11.5990 Epoch: 4 Global Step: 46880 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:44:01,027-Speed 127.45 samples/sec Loss 11.2755 Epoch: 4 Global Step: 46885 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:44:21,484-Speed 125.15 samples/sec Loss 11.6800 Epoch: 4 Global Step: 46890 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:44:42,135-Speed 123.97 samples/sec Loss 11.2417 Epoch: 4 Global Step: 46895 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:45:02,868-Speed 123.48 samples/sec Loss 11.3494 Epoch: 4 Global Step: 46900 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:45:23,097-Speed 126.56 samples/sec Loss 11.5343 Epoch: 4 Global Step: 46905 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:45:43,242-Speed 127.09 samples/sec Loss 11.5403 Epoch: 4 Global Step: 46910 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:46:03,357-Speed 127.28 samples/sec Loss 10.9341 Epoch: 4 Global Step: 46915 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:46:23,575-Speed 126.63 samples/sec Loss 11.0899 Epoch: 4 Global Step: 46920 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:46:43,793-Speed 126.63 samples/sec Loss 11.1507 Epoch: 4 Global Step: 46925 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:47:04,025-Speed 126.54 samples/sec Loss 11.2146 Epoch: 4 Global Step: 46930 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:47:24,146-Speed 127.24 samples/sec Loss 11.1923 Epoch: 4 Global Step: 46935 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:47:44,882-Speed 123.46 samples/sec Loss 11.2284 Epoch: 4 Global Step: 46940 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:48:05,197-Speed 126.02 samples/sec Loss 11.5670 Epoch: 4 Global Step: 46945 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:48:25,314-Speed 127.26 samples/sec Loss 11.2795 Epoch: 4 Global Step: 46950 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:48:45,561-Speed 126.44 samples/sec Loss 11.6931 Epoch: 4 Global Step: 46955 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:49:06,103-Speed 124.63 samples/sec Loss 11.1705 Epoch: 4 Global Step: 46960 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:49:26,411-Speed 126.07 samples/sec Loss 11.4482 Epoch: 4 Global Step: 46965 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:49:46,566-Speed 127.02 samples/sec Loss 11.5919 Epoch: 4 Global Step: 46970 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:50:06,873-Speed 126.07 samples/sec Loss 11.1902 Epoch: 4 Global Step: 46975 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:50:27,263-Speed 125.56 samples/sec Loss 11.2050 Epoch: 4 Global Step: 46980 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:50:48,207-Speed 122.24 samples/sec Loss 11.2830 Epoch: 4 Global Step: 46985 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:51:08,346-Speed 127.13 samples/sec Loss 11.4993 Epoch: 4 Global Step: 46990 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:51:28,639-Speed 126.16 samples/sec Loss 11.5438 Epoch: 4 Global Step: 46995 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:51:48,790-Speed 127.05 samples/sec Loss 10.9509 Epoch: 4 Global Step: 47000 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:52:08,797-Speed 127.96 samples/sec Loss 11.2630 Epoch: 4 Global Step: 47005 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:52:29,053-Speed 126.39 samples/sec Loss 11.1132 Epoch: 4 Global Step: 47010 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:52:49,208-Speed 127.02 samples/sec Loss 11.1868 Epoch: 4 Global Step: 47015 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:53:09,427-Speed 126.62 samples/sec Loss 10.8166 Epoch: 4 Global Step: 47020 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:53:29,550-Speed 127.22 samples/sec Loss 11.2013 Epoch: 4 Global Step: 47025 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:53:50,130-Speed 124.40 samples/sec Loss 11.4813 Epoch: 4 Global Step: 47030 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:54:10,319-Speed 126.81 samples/sec Loss 11.8307 Epoch: 4 Global Step: 47035 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:54:30,445-Speed 127.20 samples/sec Loss 11.4831 Epoch: 4 Global Step: 47040 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:54:50,700-Speed 126.40 samples/sec Loss 11.1337 Epoch: 4 Global Step: 47045 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:55:10,901-Speed 126.74 samples/sec Loss 11.3051 Epoch: 4 Global Step: 47050 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:55:31,109-Speed 126.69 samples/sec Loss 11.2653 Epoch: 4 Global Step: 47055 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:55:51,597-Speed 124.96 samples/sec Loss 11.3856 Epoch: 4 Global Step: 47060 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:56:11,688-Speed 127.43 samples/sec Loss 11.4916 Epoch: 4 Global Step: 47065 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:56:31,832-Speed 127.09 samples/sec Loss 11.4872 Epoch: 4 Global Step: 47070 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:56:52,683-Speed 122.78 samples/sec Loss 11.3224 Epoch: 4 Global Step: 47075 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:57:12,858-Speed 126.89 samples/sec Loss 11.3256 Epoch: 4 Global Step: 47080 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:57:33,014-Speed 127.01 samples/sec Loss 11.4827 Epoch: 4 Global Step: 47085 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:57:52,925-Speed 128.58 samples/sec Loss 11.6399 Epoch: 4 Global Step: 47090 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:58:13,023-Speed 127.38 samples/sec Loss 11.2220 Epoch: 4 Global Step: 47095 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:58:32,949-Speed 128.48 samples/sec Loss 11.6632 Epoch: 4 Global Step: 47100 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:58:52,940-Speed 128.06 samples/sec Loss 11.2253 Epoch: 4 Global Step: 47105 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:59:13,073-Speed 127.17 samples/sec Loss 11.1979 Epoch: 4 Global Step: 47110 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:59:33,112-Speed 127.76 samples/sec Loss 11.2689 Epoch: 4 Global Step: 47115 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 02:59:53,873-Speed 123.32 samples/sec Loss 11.0014 Epoch: 4 Global Step: 47120 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:00:13,963-Speed 127.43 samples/sec Loss 11.2006 Epoch: 4 Global Step: 47125 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:00:34,101-Speed 127.13 samples/sec Loss 11.1442 Epoch: 4 Global Step: 47130 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:00:54,010-Speed 128.59 samples/sec Loss 11.8143 Epoch: 4 Global Step: 47135 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:01:14,090-Speed 127.50 samples/sec Loss 11.2293 Epoch: 4 Global Step: 47140 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:01:34,020-Speed 128.45 samples/sec Loss 11.2638 Epoch: 4 Global Step: 47145 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:01:54,102-Speed 127.49 samples/sec Loss 11.2723 Epoch: 4 Global Step: 47150 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:02:14,378-Speed 126.26 samples/sec Loss 11.3987 Epoch: 4 Global Step: 47155 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:02:34,565-Speed 126.82 samples/sec Loss 11.3745 Epoch: 4 Global Step: 47160 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:02:55,238-Speed 123.84 samples/sec Loss 11.3158 Epoch: 4 Global Step: 47165 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:03:15,337-Speed 127.38 samples/sec Loss 11.2231 Epoch: 4 Global Step: 47170 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:03:35,327-Speed 128.07 samples/sec Loss 11.6623 Epoch: 4 Global Step: 47175 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:03:55,488-Speed 126.98 samples/sec Loss 11.2081 Epoch: 4 Global Step: 47180 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:04:15,525-Speed 127.77 samples/sec Loss 11.3735 Epoch: 4 Global Step: 47185 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:04:35,703-Speed 126.88 samples/sec Loss 11.2178 Epoch: 4 Global Step: 47190 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:04:55,708-Speed 127.97 samples/sec Loss 11.3702 Epoch: 4 Global Step: 47195 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:05:15,879-Speed 126.93 samples/sec Loss 11.6951 Epoch: 4 Global Step: 47200 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:05:36,005-Speed 127.21 samples/sec Loss 11.3326 Epoch: 4 Global Step: 47205 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:05:56,831-Speed 122.93 samples/sec Loss 11.4538 Epoch: 4 Global Step: 47210 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:06:16,893-Speed 127.61 samples/sec Loss 11.4581 Epoch: 4 Global Step: 47215 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:06:37,023-Speed 127.18 samples/sec Loss 11.1381 Epoch: 4 Global Step: 47220 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:06:56,979-Speed 128.29 samples/sec Loss 11.4756 Epoch: 4 Global Step: 47225 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:07:17,134-Speed 127.03 samples/sec Loss 11.4002 Epoch: 4 Global Step: 47230 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:07:37,060-Speed 128.48 samples/sec Loss 11.7488 Epoch: 4 Global Step: 47235 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:07:57,038-Speed 128.14 samples/sec Loss 10.8408 Epoch: 4 Global Step: 47240 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:08:17,497-Speed 125.14 samples/sec Loss 11.3302 Epoch: 4 Global Step: 47245 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:08:37,471-Speed 128.17 samples/sec Loss 11.5213 Epoch: 4 Global Step: 47250 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:08:58,322-Speed 122.78 samples/sec Loss 11.4231 Epoch: 4 Global Step: 47255 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:09:18,436-Speed 127.28 samples/sec Loss 11.9421 Epoch: 4 Global Step: 47260 Fp16 Grad Scale: 8192 Required: 240 hours Training: 2021-06-09 03:09:38,491-Speed 127.66 samples/sec Loss 11.5656 Epoch: 4 Global Step: 47265 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:09:58,681-Speed 126.80 samples/sec Loss 11.4684 Epoch: 4 Global Step: 47270 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:10:18,881-Speed 126.74 samples/sec Loss 11.1909 Epoch: 4 Global Step: 47275 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:10:38,989-Speed 127.32 samples/sec Loss 11.2399 Epoch: 4 Global Step: 47280 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:10:59,217-Speed 126.56 samples/sec Loss 11.4910 Epoch: 4 Global Step: 47285 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:11:19,446-Speed 126.56 samples/sec Loss 11.0114 Epoch: 4 Global Step: 47290 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:11:39,800-Speed 125.79 samples/sec Loss 11.4766 Epoch: 4 Global Step: 47295 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:12:00,605-Speed 123.05 samples/sec Loss 11.5877 Epoch: 4 Global Step: 47300 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:12:20,803-Speed 126.75 samples/sec Loss 11.7371 Epoch: 4 Global Step: 47305 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:12:41,095-Speed 126.16 samples/sec Loss 11.2434 Epoch: 4 Global Step: 47310 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:13:01,187-Speed 127.42 samples/sec Loss 11.4523 Epoch: 4 Global Step: 47315 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:13:21,522-Speed 125.90 samples/sec Loss 11.3537 Epoch: 4 Global Step: 47320 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:13:41,789-Speed 126.32 samples/sec Loss 11.6348 Epoch: 4 Global Step: 47325 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:14:01,942-Speed 127.03 samples/sec Loss 11.6046 Epoch: 4 Global Step: 47330 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:14:21,863-Speed 128.52 samples/sec Loss 11.3955 Epoch: 4 Global Step: 47335 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:14:42,026-Speed 126.97 samples/sec Loss 11.2887 Epoch: 4 Global Step: 47340 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:15:03,016-Speed 121.97 samples/sec Loss 11.2153 Epoch: 4 Global Step: 47345 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:15:23,125-Speed 127.31 samples/sec Loss 11.1130 Epoch: 4 Global Step: 47350 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:15:43,047-Speed 128.51 samples/sec Loss 11.3550 Epoch: 4 Global Step: 47355 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:16:03,184-Speed 127.14 samples/sec Loss 11.5427 Epoch: 4 Global Step: 47360 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:16:23,325-Speed 127.12 samples/sec Loss 11.4997 Epoch: 4 Global Step: 47365 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:16:43,486-Speed 126.98 samples/sec Loss 11.2721 Epoch: 4 Global Step: 47370 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:17:03,584-Speed 127.38 samples/sec Loss 11.6239 Epoch: 4 Global Step: 47375 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:17:23,741-Speed 127.01 samples/sec Loss 11.3509 Epoch: 4 Global Step: 47380 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:17:43,871-Speed 127.18 samples/sec Loss 11.5347 Epoch: 4 Global Step: 47385 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:18:04,847-Speed 122.05 samples/sec Loss 11.3272 Epoch: 4 Global Step: 47390 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:18:25,077-Speed 126.55 samples/sec Loss 11.4760 Epoch: 4 Global Step: 47395 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:18:45,202-Speed 127.21 samples/sec Loss 11.1720 Epoch: 4 Global Step: 47400 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:19:05,353-Speed 127.05 samples/sec Loss 11.3720 Epoch: 4 Global Step: 47405 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:19:25,489-Speed 127.14 samples/sec Loss 11.5159 Epoch: 4 Global Step: 47410 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:19:45,598-Speed 127.31 samples/sec Loss 11.1778 Epoch: 4 Global Step: 47415 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:20:05,745-Speed 127.07 samples/sec Loss 10.8209 Epoch: 4 Global Step: 47420 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:20:25,933-Speed 126.82 samples/sec Loss 10.9222 Epoch: 4 Global Step: 47425 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:20:46,059-Speed 127.20 samples/sec Loss 11.2823 Epoch: 4 Global Step: 47430 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:21:06,743-Speed 123.77 samples/sec Loss 11.2713 Epoch: 4 Global Step: 47435 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:21:26,847-Speed 127.34 samples/sec Loss 11.4321 Epoch: 4 Global Step: 47440 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:21:46,968-Speed 127.24 samples/sec Loss 11.3168 Epoch: 4 Global Step: 47445 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:22:07,192-Speed 126.59 samples/sec Loss 11.2166 Epoch: 4 Global Step: 47450 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:22:27,197-Speed 127.98 samples/sec Loss 11.4283 Epoch: 4 Global Step: 47455 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:22:47,306-Speed 127.32 samples/sec Loss 11.1682 Epoch: 4 Global Step: 47460 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:23:07,516-Speed 126.67 samples/sec Loss 10.9859 Epoch: 4 Global Step: 47465 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:23:27,443-Speed 128.48 samples/sec Loss 11.2592 Epoch: 4 Global Step: 47470 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:23:47,645-Speed 126.73 samples/sec Loss 11.3014 Epoch: 4 Global Step: 47475 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:24:08,635-Speed 121.97 samples/sec Loss 11.0922 Epoch: 4 Global Step: 47480 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:24:28,733-Speed 127.38 samples/sec Loss 11.3992 Epoch: 4 Global Step: 47485 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:24:48,633-Speed 128.65 samples/sec Loss 11.1382 Epoch: 4 Global Step: 47490 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:25:08,730-Speed 127.39 samples/sec Loss 11.4612 Epoch: 4 Global Step: 47495 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:25:28,909-Speed 126.87 samples/sec Loss 11.2369 Epoch: 4 Global Step: 47500 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:25:49,053-Speed 127.09 samples/sec Loss 11.2152 Epoch: 4 Global Step: 47505 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:26:09,235-Speed 126.86 samples/sec Loss 11.0843 Epoch: 4 Global Step: 47510 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:26:29,231-Speed 128.03 samples/sec Loss 11.4342 Epoch: 4 Global Step: 47515 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:26:49,822-Speed 124.34 samples/sec Loss 11.4417 Epoch: 4 Global Step: 47520 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:27:10,421-Speed 124.28 samples/sec Loss 11.3532 Epoch: 4 Global Step: 47525 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:27:30,525-Speed 127.35 samples/sec Loss 10.9512 Epoch: 4 Global Step: 47530 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:27:50,470-Speed 128.36 samples/sec Loss 11.2165 Epoch: 4 Global Step: 47535 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:28:10,696-Speed 126.58 samples/sec Loss 11.2024 Epoch: 4 Global Step: 47540 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:28:30,650-Speed 128.30 samples/sec Loss 11.2847 Epoch: 4 Global Step: 47545 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:28:50,743-Speed 127.42 samples/sec Loss 11.2599 Epoch: 4 Global Step: 47550 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:29:10,666-Speed 128.50 samples/sec Loss 11.4579 Epoch: 4 Global Step: 47555 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:29:30,680-Speed 127.92 samples/sec Loss 11.4838 Epoch: 4 Global Step: 47560 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:29:50,785-Speed 127.33 samples/sec Loss 11.3847 Epoch: 4 Global Step: 47565 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:30:11,401-Speed 124.18 samples/sec Loss 11.5635 Epoch: 4 Global Step: 47570 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:30:31,592-Speed 126.79 samples/sec Loss 11.3120 Epoch: 4 Global Step: 47575 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:30:51,827-Speed 126.52 samples/sec Loss 11.1791 Epoch: 4 Global Step: 47580 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:31:11,940-Speed 127.29 samples/sec Loss 11.1115 Epoch: 4 Global Step: 47585 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:31:31,986-Speed 127.71 samples/sec Loss 11.3550 Epoch: 4 Global Step: 47590 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:31:52,002-Speed 127.92 samples/sec Loss 11.2499 Epoch: 4 Global Step: 47595 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:32:12,160-Speed 127.00 samples/sec Loss 11.3725 Epoch: 4 Global Step: 47600 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:32:32,362-Speed 126.73 samples/sec Loss 11.0177 Epoch: 4 Global Step: 47605 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:32:52,470-Speed 127.32 samples/sec Loss 11.4303 Epoch: 4 Global Step: 47610 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:33:13,321-Speed 122.78 samples/sec Loss 10.9537 Epoch: 4 Global Step: 47615 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:33:33,998-Speed 123.81 samples/sec Loss 11.6481 Epoch: 4 Global Step: 47620 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:33:54,151-Speed 127.03 samples/sec Loss 11.4816 Epoch: 4 Global Step: 47625 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:34:14,069-Speed 128.54 samples/sec Loss 11.1910 Epoch: 4 Global Step: 47630 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:34:34,236-Speed 126.95 samples/sec Loss 11.4185 Epoch: 4 Global Step: 47635 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:34:54,164-Speed 128.47 samples/sec Loss 11.3562 Epoch: 4 Global Step: 47640 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:35:14,289-Speed 127.21 samples/sec Loss 11.0782 Epoch: 4 Global Step: 47645 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:35:34,389-Speed 127.37 samples/sec Loss 11.4039 Epoch: 4 Global Step: 47650 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:35:54,626-Speed 126.51 samples/sec Loss 11.1612 Epoch: 4 Global Step: 47655 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:36:15,380-Speed 123.36 samples/sec Loss 11.2541 Epoch: 4 Global Step: 47660 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:36:35,592-Speed 126.66 samples/sec Loss 11.2164 Epoch: 4 Global Step: 47665 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:36:55,595-Speed 127.99 samples/sec Loss 11.5024 Epoch: 4 Global Step: 47670 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:37:15,707-Speed 127.30 samples/sec Loss 11.3702 Epoch: 4 Global Step: 47675 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:37:35,707-Speed 128.01 samples/sec Loss 11.4562 Epoch: 4 Global Step: 47680 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:37:55,819-Speed 127.29 samples/sec Loss 11.4137 Epoch: 4 Global Step: 47685 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:38:15,945-Speed 127.21 samples/sec Loss 11.3329 Epoch: 4 Global Step: 47690 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:38:36,087-Speed 127.10 samples/sec Loss 11.4550 Epoch: 4 Global Step: 47695 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:38:56,188-Speed 127.37 samples/sec Loss 11.3361 Epoch: 4 Global Step: 47700 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:39:17,032-Speed 122.82 samples/sec Loss 11.0508 Epoch: 4 Global Step: 47705 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:39:37,388-Speed 125.77 samples/sec Loss 11.1800 Epoch: 4 Global Step: 47710 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:39:57,481-Speed 127.41 samples/sec Loss 11.5885 Epoch: 4 Global Step: 47715 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:40:17,607-Speed 127.21 samples/sec Loss 11.0677 Epoch: 4 Global Step: 47720 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:40:37,570-Speed 128.25 samples/sec Loss 11.3493 Epoch: 4 Global Step: 47725 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:40:57,654-Speed 127.47 samples/sec Loss 11.4719 Epoch: 4 Global Step: 47730 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:41:17,798-Speed 127.10 samples/sec Loss 11.2670 Epoch: 4 Global Step: 47735 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:41:37,992-Speed 126.77 samples/sec Loss 11.2419 Epoch: 4 Global Step: 47740 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:41:57,913-Speed 128.52 samples/sec Loss 11.5154 Epoch: 4 Global Step: 47745 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:42:18,500-Speed 124.36 samples/sec Loss 11.6980 Epoch: 4 Global Step: 47750 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:42:38,599-Speed 127.37 samples/sec Loss 11.3068 Epoch: 4 Global Step: 47755 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:42:58,573-Speed 128.17 samples/sec Loss 11.1222 Epoch: 4 Global Step: 47760 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:43:18,519-Speed 128.35 samples/sec Loss 11.1937 Epoch: 4 Global Step: 47765 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:43:38,593-Speed 127.54 samples/sec Loss 11.2504 Epoch: 4 Global Step: 47770 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:43:58,735-Speed 127.11 samples/sec Loss 11.3942 Epoch: 4 Global Step: 47775 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:44:18,844-Speed 127.31 samples/sec Loss 11.3744 Epoch: 4 Global Step: 47780 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:44:38,950-Speed 127.33 samples/sec Loss 11.2643 Epoch: 4 Global Step: 47785 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:44:59,256-Speed 126.08 samples/sec Loss 11.0815 Epoch: 4 Global Step: 47790 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:45:20,258-Speed 121.90 samples/sec Loss 11.3276 Epoch: 4 Global Step: 47795 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:45:40,536-Speed 126.25 samples/sec Loss 11.4390 Epoch: 4 Global Step: 47800 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:46:00,725-Speed 126.80 samples/sec Loss 11.3670 Epoch: 4 Global Step: 47805 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:46:20,836-Speed 127.30 samples/sec Loss 11.2920 Epoch: 4 Global Step: 47810 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:46:40,874-Speed 127.76 samples/sec Loss 11.3085 Epoch: 4 Global Step: 47815 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:47:01,003-Speed 127.19 samples/sec Loss 11.3966 Epoch: 4 Global Step: 47820 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:47:21,106-Speed 127.35 samples/sec Loss 11.0456 Epoch: 4 Global Step: 47825 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:47:41,370-Speed 126.34 samples/sec Loss 11.3165 Epoch: 4 Global Step: 47830 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:48:01,651-Speed 126.24 samples/sec Loss 11.7150 Epoch: 4 Global Step: 47835 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:48:22,620-Speed 122.09 samples/sec Loss 11.3330 Epoch: 4 Global Step: 47840 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:48:42,719-Speed 127.38 samples/sec Loss 11.2378 Epoch: 4 Global Step: 47845 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:49:02,861-Speed 127.11 samples/sec Loss 11.6807 Epoch: 4 Global Step: 47850 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:49:23,061-Speed 126.74 samples/sec Loss 11.4160 Epoch: 4 Global Step: 47855 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:49:43,243-Speed 126.86 samples/sec Loss 11.6266 Epoch: 4 Global Step: 47860 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:50:03,659-Speed 125.40 samples/sec Loss 11.4142 Epoch: 4 Global Step: 47865 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:50:23,776-Speed 127.26 samples/sec Loss 11.3410 Epoch: 4 Global Step: 47870 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:50:43,893-Speed 127.26 samples/sec Loss 11.5986 Epoch: 4 Global Step: 47875 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:51:04,026-Speed 127.17 samples/sec Loss 11.4636 Epoch: 4 Global Step: 47880 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:51:24,836-Speed 123.02 samples/sec Loss 11.2277 Epoch: 4 Global Step: 47885 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:51:44,987-Speed 127.05 samples/sec Loss 11.2841 Epoch: 4 Global Step: 47890 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:52:05,069-Speed 127.48 samples/sec Loss 11.3753 Epoch: 4 Global Step: 47895 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:52:25,068-Speed 128.02 samples/sec Loss 11.1624 Epoch: 4 Global Step: 47900 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:52:45,120-Speed 127.68 samples/sec Loss 11.4682 Epoch: 4 Global Step: 47905 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:53:05,022-Speed 128.64 samples/sec Loss 11.6408 Epoch: 4 Global Step: 47910 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:53:25,113-Speed 127.43 samples/sec Loss 11.2963 Epoch: 4 Global Step: 47915 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:53:45,342-Speed 126.56 samples/sec Loss 10.9658 Epoch: 4 Global Step: 47920 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:54:05,518-Speed 126.89 samples/sec Loss 11.4395 Epoch: 4 Global Step: 47925 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:54:26,292-Speed 123.24 samples/sec Loss 11.0801 Epoch: 4 Global Step: 47930 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:54:46,392-Speed 127.37 samples/sec Loss 11.2181 Epoch: 4 Global Step: 47935 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:55:06,514-Speed 127.23 samples/sec Loss 11.3424 Epoch: 4 Global Step: 47940 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:55:26,712-Speed 126.75 samples/sec Loss 11.3859 Epoch: 4 Global Step: 47945 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 03:55:46,825-Speed 127.29 samples/sec Loss 11.1779 Epoch: 4 Global Step: 47950 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:56:06,777-Speed 128.32 samples/sec Loss 11.6149 Epoch: 4 Global Step: 47955 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:56:26,920-Speed 127.10 samples/sec Loss 11.7231 Epoch: 4 Global Step: 47960 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:56:47,161-Speed 126.49 samples/sec Loss 11.4004 Epoch: 4 Global Step: 47965 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:57:07,267-Speed 127.33 samples/sec Loss 11.1435 Epoch: 4 Global Step: 47970 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:57:28,259-Speed 121.96 samples/sec Loss 11.4208 Epoch: 4 Global Step: 47975 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:57:48,397-Speed 127.13 samples/sec Loss 11.5232 Epoch: 4 Global Step: 47980 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:58:08,550-Speed 127.03 samples/sec Loss 11.2638 Epoch: 4 Global Step: 47985 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:58:28,747-Speed 126.76 samples/sec Loss 11.3055 Epoch: 4 Global Step: 47990 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:58:48,768-Speed 127.87 samples/sec Loss 11.4641 Epoch: 4 Global Step: 47995 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:59:09,003-Speed 126.52 samples/sec Loss 11.1771 Epoch: 4 Global Step: 48000 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:59:29,056-Speed 127.67 samples/sec Loss 11.5086 Epoch: 4 Global Step: 48005 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 03:59:49,211-Speed 127.02 samples/sec Loss 11.3092 Epoch: 4 Global Step: 48010 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:00:09,333-Speed 127.23 samples/sec Loss 11.0663 Epoch: 4 Global Step: 48015 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:00:30,151-Speed 122.98 samples/sec Loss 11.1236 Epoch: 4 Global Step: 48020 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:00:50,454-Speed 126.09 samples/sec Loss 11.0888 Epoch: 4 Global Step: 48025 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:01:10,581-Speed 127.20 samples/sec Loss 11.2449 Epoch: 4 Global Step: 48030 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:01:30,666-Speed 127.47 samples/sec Loss 11.4116 Epoch: 4 Global Step: 48035 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:01:50,637-Speed 128.19 samples/sec Loss 11.3384 Epoch: 4 Global Step: 48040 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:02:10,703-Speed 127.59 samples/sec Loss 11.0984 Epoch: 4 Global Step: 48045 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:02:31,104-Speed 125.49 samples/sec Loss 11.6342 Epoch: 4 Global Step: 48050 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:02:51,209-Speed 127.34 samples/sec Loss 11.2765 Epoch: 4 Global Step: 48055 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:03:11,366-Speed 127.01 samples/sec Loss 11.1241 Epoch: 4 Global Step: 48060 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:03:32,250-Speed 122.59 samples/sec Loss 11.5242 Epoch: 4 Global Step: 48065 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:03:52,411-Speed 126.99 samples/sec Loss 11.3613 Epoch: 4 Global Step: 48070 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:04:12,506-Speed 127.40 samples/sec Loss 11.1236 Epoch: 4 Global Step: 48075 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:04:32,778-Speed 126.29 samples/sec Loss 11.6814 Epoch: 4 Global Step: 48080 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:04:52,925-Speed 127.07 samples/sec Loss 11.5171 Epoch: 4 Global Step: 48085 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:05:13,080-Speed 127.02 samples/sec Loss 11.3989 Epoch: 4 Global Step: 48090 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:05:33,207-Speed 127.20 samples/sec Loss 11.3675 Epoch: 4 Global Step: 48095 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:05:53,299-Speed 127.42 samples/sec Loss 11.1640 Epoch: 4 Global Step: 48100 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:06:13,312-Speed 127.92 samples/sec Loss 11.4716 Epoch: 4 Global Step: 48105 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:06:34,026-Speed 123.60 samples/sec Loss 11.1210 Epoch: 4 Global Step: 48110 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:06:54,157-Speed 127.17 samples/sec Loss 11.5741 Epoch: 4 Global Step: 48115 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:07:14,297-Speed 127.11 samples/sec Loss 11.4124 Epoch: 4 Global Step: 48120 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:07:34,600-Speed 126.10 samples/sec Loss 11.2184 Epoch: 4 Global Step: 48125 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:07:54,834-Speed 126.52 samples/sec Loss 11.0925 Epoch: 4 Global Step: 48130 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:08:14,954-Speed 127.25 samples/sec Loss 11.5788 Epoch: 4 Global Step: 48135 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:08:35,098-Speed 127.09 samples/sec Loss 11.2085 Epoch: 4 Global Step: 48140 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:08:55,303-Speed 126.71 samples/sec Loss 11.4593 Epoch: 4 Global Step: 48145 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:09:15,419-Speed 127.27 samples/sec Loss 11.7102 Epoch: 4 Global Step: 48150 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:09:36,136-Speed 123.57 samples/sec Loss 11.1868 Epoch: 4 Global Step: 48155 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:09:56,204-Speed 127.57 samples/sec Loss 11.4033 Epoch: 4 Global Step: 48160 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:10:16,234-Speed 127.81 samples/sec Loss 11.2960 Epoch: 4 Global Step: 48165 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:10:36,404-Speed 126.93 samples/sec Loss 11.4044 Epoch: 4 Global Step: 48170 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:10:56,467-Speed 127.61 samples/sec Loss 11.3510 Epoch: 4 Global Step: 48175 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:11:16,606-Speed 127.12 samples/sec Loss 11.6164 Epoch: 4 Global Step: 48180 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:11:36,721-Speed 127.28 samples/sec Loss 11.3410 Epoch: 4 Global Step: 48185 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:11:57,244-Speed 124.74 samples/sec Loss 11.1417 Epoch: 4 Global Step: 48190 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:12:17,217-Speed 128.18 samples/sec Loss 11.5385 Epoch: 4 Global Step: 48195 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:12:38,219-Speed 121.90 samples/sec Loss 11.2698 Epoch: 4 Global Step: 48200 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:12:58,324-Speed 127.34 samples/sec Loss 10.9813 Epoch: 4 Global Step: 48205 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:13:18,385-Speed 127.61 samples/sec Loss 11.1051 Epoch: 4 Global Step: 48210 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:13:38,995-Speed 124.22 samples/sec Loss 11.3007 Epoch: 4 Global Step: 48215 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:13:59,086-Speed 127.43 samples/sec Loss 11.2503 Epoch: 4 Global Step: 48220 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:14:19,227-Speed 127.11 samples/sec Loss 11.5380 Epoch: 4 Global Step: 48225 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:14:39,267-Speed 127.75 samples/sec Loss 11.1904 Epoch: 4 Global Step: 48230 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:14:59,417-Speed 127.06 samples/sec Loss 11.2288 Epoch: 4 Global Step: 48235 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:15:19,549-Speed 127.16 samples/sec Loss 11.3493 Epoch: 4 Global Step: 48240 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:15:40,513-Speed 122.12 samples/sec Loss 11.4442 Epoch: 4 Global Step: 48245 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:16:00,990-Speed 125.02 samples/sec Loss 11.0450 Epoch: 4 Global Step: 48250 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:16:21,163-Speed 126.91 samples/sec Loss 11.2484 Epoch: 4 Global Step: 48255 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:16:41,296-Speed 127.16 samples/sec Loss 11.6131 Epoch: 4 Global Step: 48260 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:17:01,447-Speed 127.05 samples/sec Loss 11.2536 Epoch: 4 Global Step: 48265 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:17:21,607-Speed 126.99 samples/sec Loss 11.2118 Epoch: 4 Global Step: 48270 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:17:41,848-Speed 126.49 samples/sec Loss 11.4560 Epoch: 4 Global Step: 48275 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:18:02,051-Speed 126.72 samples/sec Loss 11.3033 Epoch: 4 Global Step: 48280 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:18:22,133-Speed 127.49 samples/sec Loss 11.4535 Epoch: 4 Global Step: 48285 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:18:43,090-Speed 122.16 samples/sec Loss 11.0339 Epoch: 4 Global Step: 48290 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:19:03,047-Speed 128.28 samples/sec Loss 11.3875 Epoch: 4 Global Step: 48295 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:19:23,307-Speed 126.36 samples/sec Loss 11.3176 Epoch: 4 Global Step: 48300 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:19:43,545-Speed 126.50 samples/sec Loss 11.7578 Epoch: 4 Global Step: 48305 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:20:03,626-Speed 127.49 samples/sec Loss 11.2463 Epoch: 4 Global Step: 48310 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:20:23,780-Speed 127.03 samples/sec Loss 11.3605 Epoch: 4 Global Step: 48315 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:20:43,934-Speed 127.03 samples/sec Loss 11.3053 Epoch: 4 Global Step: 48320 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:21:04,098-Speed 126.97 samples/sec Loss 11.5731 Epoch: 4 Global Step: 48325 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:21:24,065-Speed 128.22 samples/sec Loss 11.2860 Epoch: 4 Global Step: 48330 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:21:44,694-Speed 124.11 samples/sec Loss 11.3479 Epoch: 4 Global Step: 48335 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:22:04,943-Speed 126.43 samples/sec Loss 11.3300 Epoch: 4 Global Step: 48340 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:22:24,845-Speed 128.64 samples/sec Loss 11.4335 Epoch: 4 Global Step: 48345 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:22:44,932-Speed 127.45 samples/sec Loss 11.4180 Epoch: 4 Global Step: 48350 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:23:04,948-Speed 127.90 samples/sec Loss 10.9987 Epoch: 4 Global Step: 48355 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:23:25,838-Speed 122.55 samples/sec Loss 11.1916 Epoch: 4 Global Step: 48360 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:23:45,918-Speed 127.50 samples/sec Loss 11.5948 Epoch: 4 Global Step: 48365 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:24:06,016-Speed 127.38 samples/sec Loss 11.5106 Epoch: 4 Global Step: 48370 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:24:26,404-Speed 125.57 samples/sec Loss 10.9873 Epoch: 4 Global Step: 48375 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:24:47,207-Speed 123.07 samples/sec Loss 11.5620 Epoch: 4 Global Step: 48380 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:25:07,626-Speed 125.38 samples/sec Loss 11.7562 Epoch: 4 Global Step: 48385 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:25:27,803-Speed 126.88 samples/sec Loss 11.3195 Epoch: 4 Global Step: 48390 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:25:48,219-Speed 125.40 samples/sec Loss 11.0179 Epoch: 4 Global Step: 48395 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:26:08,370-Speed 127.05 samples/sec Loss 11.6754 Epoch: 4 Global Step: 48400 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:26:28,490-Speed 127.24 samples/sec Loss 11.1132 Epoch: 4 Global Step: 48405 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:26:48,584-Speed 127.41 samples/sec Loss 11.1400 Epoch: 4 Global Step: 48410 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:27:08,679-Speed 127.40 samples/sec Loss 11.1347 Epoch: 4 Global Step: 48415 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:27:28,638-Speed 128.27 samples/sec Loss 11.2437 Epoch: 4 Global Step: 48420 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:27:49,244-Speed 124.24 samples/sec Loss 11.4921 Epoch: 4 Global Step: 48425 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:28:09,252-Speed 127.95 samples/sec Loss 11.1945 Epoch: 4 Global Step: 48430 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:28:29,284-Speed 127.80 samples/sec Loss 11.2781 Epoch: 4 Global Step: 48435 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:28:49,381-Speed 127.39 samples/sec Loss 11.2863 Epoch: 4 Global Step: 48440 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:29:09,502-Speed 127.24 samples/sec Loss 11.6060 Epoch: 4 Global Step: 48445 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:29:29,624-Speed 127.24 samples/sec Loss 11.6474 Epoch: 4 Global Step: 48450 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:29:49,813-Speed 126.81 samples/sec Loss 11.4620 Epoch: 4 Global Step: 48455 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:30:09,982-Speed 126.94 samples/sec Loss 11.4317 Epoch: 4 Global Step: 48460 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:30:30,395-Speed 125.42 samples/sec Loss 11.7163 Epoch: 4 Global Step: 48465 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:30:51,208-Speed 123.01 samples/sec Loss 11.4629 Epoch: 4 Global Step: 48470 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:31:11,282-Speed 127.53 samples/sec Loss 11.4661 Epoch: 4 Global Step: 48475 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:31:31,406-Speed 127.21 samples/sec Loss 11.4456 Epoch: 4 Global Step: 48480 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:31:51,498-Speed 127.42 samples/sec Loss 11.5000 Epoch: 4 Global Step: 48485 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:32:11,649-Speed 127.05 samples/sec Loss 11.4682 Epoch: 4 Global Step: 48490 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:32:31,761-Speed 127.29 samples/sec Loss 11.3011 Epoch: 4 Global Step: 48495 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:32:51,978-Speed 126.63 samples/sec Loss 11.2708 Epoch: 4 Global Step: 48500 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:33:12,094-Speed 127.27 samples/sec Loss 11.3548 Epoch: 4 Global Step: 48505 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:33:32,319-Speed 126.58 samples/sec Loss 11.2712 Epoch: 4 Global Step: 48510 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:33:53,152-Speed 122.89 samples/sec Loss 11.2882 Epoch: 4 Global Step: 48515 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:34:13,296-Speed 127.09 samples/sec Loss 11.3174 Epoch: 4 Global Step: 48520 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:34:33,223-Speed 128.48 samples/sec Loss 11.4548 Epoch: 4 Global Step: 48525 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:34:53,300-Speed 127.51 samples/sec Loss 11.3446 Epoch: 4 Global Step: 48530 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:35:13,186-Speed 128.74 samples/sec Loss 11.3059 Epoch: 4 Global Step: 48535 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:35:33,345-Speed 126.99 samples/sec Loss 11.3887 Epoch: 4 Global Step: 48540 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:35:53,253-Speed 128.60 samples/sec Loss 11.0054 Epoch: 4 Global Step: 48545 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:36:13,200-Speed 128.34 samples/sec Loss 11.1464 Epoch: 4 Global Step: 48550 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:36:33,271-Speed 127.55 samples/sec Loss 11.3991 Epoch: 4 Global Step: 48555 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:36:53,853-Speed 124.39 samples/sec Loss 11.7942 Epoch: 4 Global Step: 48560 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:37:13,940-Speed 127.45 samples/sec Loss 11.3133 Epoch: 4 Global Step: 48565 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:37:34,055-Speed 127.28 samples/sec Loss 11.3473 Epoch: 4 Global Step: 48570 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:37:54,175-Speed 127.24 samples/sec Loss 11.5061 Epoch: 4 Global Step: 48575 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:38:14,354-Speed 126.88 samples/sec Loss 11.3956 Epoch: 4 Global Step: 48580 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:38:34,568-Speed 126.65 samples/sec Loss 11.3245 Epoch: 4 Global Step: 48585 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:38:54,707-Speed 127.12 samples/sec Loss 11.6023 Epoch: 4 Global Step: 48590 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:39:14,678-Speed 128.20 samples/sec Loss 11.0925 Epoch: 4 Global Step: 48595 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:39:34,766-Speed 127.45 samples/sec Loss 11.1429 Epoch: 4 Global Step: 48600 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:39:55,616-Speed 122.79 samples/sec Loss 11.4057 Epoch: 4 Global Step: 48605 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:40:15,743-Speed 127.20 samples/sec Loss 11.4078 Epoch: 4 Global Step: 48610 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:40:35,838-Speed 127.40 samples/sec Loss 11.3768 Epoch: 4 Global Step: 48615 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:40:55,954-Speed 127.27 samples/sec Loss 10.9233 Epoch: 4 Global Step: 48620 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:41:16,116-Speed 126.98 samples/sec Loss 11.2785 Epoch: 4 Global Step: 48625 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:41:36,185-Speed 127.57 samples/sec Loss 11.2873 Epoch: 4 Global Step: 48630 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:41:56,341-Speed 127.02 samples/sec Loss 11.3170 Epoch: 4 Global Step: 48635 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:42:16,488-Speed 127.07 samples/sec Loss 11.4132 Epoch: 4 Global Step: 48640 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 04:42:36,739-Speed 126.42 samples/sec Loss 11.3434 Epoch: 4 Global Step: 48645 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:42:57,675-Speed 122.29 samples/sec Loss 11.1503 Epoch: 4 Global Step: 48650 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:43:17,837-Speed 126.98 samples/sec Loss 11.4215 Epoch: 4 Global Step: 48655 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:43:37,932-Speed 127.40 samples/sec Loss 11.4806 Epoch: 4 Global Step: 48660 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:43:58,067-Speed 127.15 samples/sec Loss 11.2852 Epoch: 4 Global Step: 48665 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:44:18,096-Speed 127.83 samples/sec Loss 11.3029 Epoch: 4 Global Step: 48670 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:44:38,168-Speed 127.55 samples/sec Loss 11.5314 Epoch: 4 Global Step: 48675 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:44:58,158-Speed 128.07 samples/sec Loss 11.3878 Epoch: 4 Global Step: 48680 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:45:18,437-Speed 126.24 samples/sec Loss 11.2622 Epoch: 4 Global Step: 48685 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:45:38,532-Speed 127.41 samples/sec Loss 11.2985 Epoch: 4 Global Step: 48690 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:45:59,175-Speed 124.02 samples/sec Loss 11.2938 Epoch: 4 Global Step: 48695 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:46:19,220-Speed 127.72 samples/sec Loss 11.2259 Epoch: 4 Global Step: 48700 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:46:39,121-Speed 128.64 samples/sec Loss 11.2760 Epoch: 4 Global Step: 48705 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:46:59,217-Speed 127.40 samples/sec Loss 11.2948 Epoch: 4 Global Step: 48710 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:47:19,356-Speed 127.12 samples/sec Loss 11.4383 Epoch: 4 Global Step: 48715 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:47:39,507-Speed 127.05 samples/sec Loss 11.3395 Epoch: 4 Global Step: 48720 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:47:59,712-Speed 126.71 samples/sec Loss 11.0603 Epoch: 4 Global Step: 48725 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:48:19,796-Speed 127.47 samples/sec Loss 11.6192 Epoch: 4 Global Step: 48730 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:48:39,987-Speed 126.79 samples/sec Loss 11.2519 Epoch: 4 Global Step: 48735 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:49:01,018-Speed 121.73 samples/sec Loss 11.5681 Epoch: 4 Global Step: 48740 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:49:21,174-Speed 127.02 samples/sec Loss 11.2049 Epoch: 4 Global Step: 48745 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:49:41,179-Speed 127.97 samples/sec Loss 11.4703 Epoch: 4 Global Step: 48750 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:50:01,281-Speed 127.36 samples/sec Loss 11.0019 Epoch: 4 Global Step: 48755 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:50:21,378-Speed 127.39 samples/sec Loss 11.2680 Epoch: 4 Global Step: 48760 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:50:41,506-Speed 127.19 samples/sec Loss 11.2145 Epoch: 4 Global Step: 48765 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:51:01,601-Speed 127.41 samples/sec Loss 11.2012 Epoch: 4 Global Step: 48770 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:51:21,825-Speed 126.59 samples/sec Loss 11.2866 Epoch: 4 Global Step: 48775 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:51:42,199-Speed 125.66 samples/sec Loss 11.4844 Epoch: 4 Global Step: 48780 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:52:03,119-Speed 122.38 samples/sec Loss 11.2998 Epoch: 4 Global Step: 48785 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:52:23,347-Speed 126.56 samples/sec Loss 11.0578 Epoch: 4 Global Step: 48790 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:52:43,547-Speed 126.74 samples/sec Loss 11.6375 Epoch: 4 Global Step: 48795 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:53:03,652-Speed 127.34 samples/sec Loss 11.3977 Epoch: 4 Global Step: 48800 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:53:23,746-Speed 127.41 samples/sec Loss 11.5519 Epoch: 4 Global Step: 48805 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:53:43,918-Speed 126.91 samples/sec Loss 11.0513 Epoch: 4 Global Step: 48810 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:54:04,137-Speed 126.62 samples/sec Loss 11.2331 Epoch: 4 Global Step: 48815 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:54:24,354-Speed 126.63 samples/sec Loss 11.9704 Epoch: 4 Global Step: 48820 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:54:44,411-Speed 127.64 samples/sec Loss 10.9730 Epoch: 4 Global Step: 48825 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:55:05,258-Speed 122.81 samples/sec Loss 11.0665 Epoch: 4 Global Step: 48830 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:55:25,295-Speed 127.77 samples/sec Loss 11.5094 Epoch: 4 Global Step: 48835 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:55:45,407-Speed 127.30 samples/sec Loss 11.0685 Epoch: 4 Global Step: 48840 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:56:05,549-Speed 127.11 samples/sec Loss 11.0596 Epoch: 4 Global Step: 48845 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:56:25,693-Speed 127.09 samples/sec Loss 11.0165 Epoch: 4 Global Step: 48850 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:56:45,866-Speed 126.92 samples/sec Loss 11.2124 Epoch: 4 Global Step: 48855 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:57:05,989-Speed 127.22 samples/sec Loss 11.2880 Epoch: 4 Global Step: 48860 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:57:26,096-Speed 127.33 samples/sec Loss 10.9979 Epoch: 4 Global Step: 48865 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:57:46,200-Speed 127.35 samples/sec Loss 11.5958 Epoch: 4 Global Step: 48870 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:58:06,960-Speed 123.32 samples/sec Loss 11.3114 Epoch: 4 Global Step: 48875 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:58:27,108-Speed 127.06 samples/sec Loss 11.2340 Epoch: 4 Global Step: 48880 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:58:47,213-Speed 127.34 samples/sec Loss 10.9919 Epoch: 4 Global Step: 48885 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:59:07,410-Speed 126.76 samples/sec Loss 11.4031 Epoch: 4 Global Step: 48890 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:59:27,548-Speed 127.13 samples/sec Loss 11.6416 Epoch: 4 Global Step: 48895 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 04:59:47,836-Speed 126.19 samples/sec Loss 11.0394 Epoch: 4 Global Step: 48900 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:00:08,107-Speed 126.29 samples/sec Loss 11.4150 Epoch: 4 Global Step: 48905 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:00:28,160-Speed 127.67 samples/sec Loss 11.0821 Epoch: 4 Global Step: 48910 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:00:48,300-Speed 127.12 samples/sec Loss 11.2837 Epoch: 4 Global Step: 48915 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:01:09,092-Speed 123.13 samples/sec Loss 11.2468 Epoch: 4 Global Step: 48920 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:01:29,231-Speed 127.12 samples/sec Loss 11.0963 Epoch: 4 Global Step: 48925 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:01:49,628-Speed 125.52 samples/sec Loss 11.1881 Epoch: 4 Global Step: 48930 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:02:09,848-Speed 126.62 samples/sec Loss 11.4432 Epoch: 4 Global Step: 48935 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:02:29,979-Speed 127.17 samples/sec Loss 11.0185 Epoch: 4 Global Step: 48940 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:02:50,120-Speed 127.11 samples/sec Loss 11.0306 Epoch: 4 Global Step: 48945 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:03:10,262-Speed 127.11 samples/sec Loss 11.1851 Epoch: 4 Global Step: 48950 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:03:30,229-Speed 128.22 samples/sec Loss 10.9367 Epoch: 4 Global Step: 48955 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:03:50,372-Speed 127.10 samples/sec Loss 11.6530 Epoch: 4 Global Step: 48960 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:04:11,412-Speed 121.68 samples/sec Loss 11.1615 Epoch: 4 Global Step: 48965 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:04:31,503-Speed 127.42 samples/sec Loss 11.0068 Epoch: 4 Global Step: 48970 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:04:51,984-Speed 125.00 samples/sec Loss 10.5984 Epoch: 4 Global Step: 48975 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:05:12,182-Speed 126.75 samples/sec Loss 11.7821 Epoch: 4 Global Step: 48980 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:05:32,499-Speed 126.01 samples/sec Loss 11.6847 Epoch: 4 Global Step: 48985 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:05:52,727-Speed 126.56 samples/sec Loss 11.4019 Epoch: 4 Global Step: 48990 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:06:13,599-Speed 122.66 samples/sec Loss 11.2255 Epoch: 4 Global Step: 48995 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:06:33,816-Speed 126.63 samples/sec Loss 11.0504 Epoch: 4 Global Step: 49000 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:06:54,243-Speed 125.33 samples/sec Loss 11.2008 Epoch: 4 Global Step: 49005 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:07:14,968-Speed 123.53 samples/sec Loss 11.0136 Epoch: 4 Global Step: 49010 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:07:34,907-Speed 128.39 samples/sec Loss 11.1017 Epoch: 4 Global Step: 49015 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:07:55,404-Speed 124.90 samples/sec Loss 11.5463 Epoch: 4 Global Step: 49020 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:08:15,500-Speed 127.40 samples/sec Loss 11.3001 Epoch: 4 Global Step: 49025 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:08:35,737-Speed 126.51 samples/sec Loss 11.2409 Epoch: 4 Global Step: 49030 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:08:55,945-Speed 126.69 samples/sec Loss 11.2506 Epoch: 4 Global Step: 49035 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:09:16,270-Speed 125.97 samples/sec Loss 11.1669 Epoch: 4 Global Step: 49040 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:09:36,468-Speed 126.76 samples/sec Loss 11.6144 Epoch: 4 Global Step: 49045 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:09:56,669-Speed 126.73 samples/sec Loss 11.1570 Epoch: 4 Global Step: 49050 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:10:17,485-Speed 122.99 samples/sec Loss 11.4086 Epoch: 4 Global Step: 49055 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:10:37,589-Speed 127.34 samples/sec Loss 11.3586 Epoch: 4 Global Step: 49060 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:10:57,747-Speed 127.00 samples/sec Loss 11.3187 Epoch: 4 Global Step: 49065 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:11:17,737-Speed 128.07 samples/sec Loss 11.0550 Epoch: 4 Global Step: 49070 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:11:37,774-Speed 127.77 samples/sec Loss 11.5748 Epoch: 4 Global Step: 49075 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:11:57,728-Speed 128.30 samples/sec Loss 11.3635 Epoch: 4 Global Step: 49080 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:12:17,875-Speed 127.07 samples/sec Loss 11.0676 Epoch: 4 Global Step: 49085 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:12:37,846-Speed 128.19 samples/sec Loss 11.1301 Epoch: 4 Global Step: 49090 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:12:57,958-Speed 127.30 samples/sec Loss 11.3774 Epoch: 4 Global Step: 49095 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:13:18,522-Speed 124.49 samples/sec Loss 10.9976 Epoch: 4 Global Step: 49100 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:13:38,694-Speed 126.92 samples/sec Loss 10.9227 Epoch: 4 Global Step: 49105 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:13:58,816-Speed 127.23 samples/sec Loss 11.6052 Epoch: 4 Global Step: 49110 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:14:18,981-Speed 126.96 samples/sec Loss 11.3143 Epoch: 4 Global Step: 49115 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:14:39,096-Speed 127.28 samples/sec Loss 11.1800 Epoch: 4 Global Step: 49120 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:14:59,263-Speed 126.95 samples/sec Loss 11.6113 Epoch: 4 Global Step: 49125 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:15:19,436-Speed 126.91 samples/sec Loss 11.1097 Epoch: 4 Global Step: 49130 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:15:39,607-Speed 126.92 samples/sec Loss 11.5859 Epoch: 4 Global Step: 49135 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:15:59,873-Speed 126.33 samples/sec Loss 11.4341 Epoch: 4 Global Step: 49140 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:16:20,557-Speed 123.77 samples/sec Loss 11.2382 Epoch: 4 Global Step: 49145 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:16:40,717-Speed 127.00 samples/sec Loss 11.2601 Epoch: 4 Global Step: 49150 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:17:00,676-Speed 128.26 samples/sec Loss 11.3537 Epoch: 4 Global Step: 49155 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:17:20,893-Speed 126.64 samples/sec Loss 11.4247 Epoch: 4 Global Step: 49160 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:17:41,118-Speed 126.58 samples/sec Loss 11.5166 Epoch: 4 Global Step: 49165 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:18:01,360-Speed 126.48 samples/sec Loss 11.3237 Epoch: 4 Global Step: 49170 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:18:21,604-Speed 126.46 samples/sec Loss 11.5299 Epoch: 4 Global Step: 49175 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:18:41,863-Speed 126.38 samples/sec Loss 11.0591 Epoch: 4 Global Step: 49180 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:19:02,094-Speed 126.54 samples/sec Loss 11.3180 Epoch: 4 Global Step: 49185 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:19:23,120-Speed 121.76 samples/sec Loss 11.2017 Epoch: 4 Global Step: 49190 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:19:43,106-Speed 128.09 samples/sec Loss 11.4804 Epoch: 4 Global Step: 49195 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:20:03,300-Speed 126.77 samples/sec Loss 10.7898 Epoch: 4 Global Step: 49200 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:20:23,484-Speed 126.84 samples/sec Loss 11.3972 Epoch: 4 Global Step: 49205 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:20:43,695-Speed 126.67 samples/sec Loss 11.1548 Epoch: 4 Global Step: 49210 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:21:03,822-Speed 127.20 samples/sec Loss 11.1151 Epoch: 4 Global Step: 49215 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:21:24,003-Speed 126.86 samples/sec Loss 11.3511 Epoch: 4 Global Step: 49220 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:21:44,237-Speed 126.52 samples/sec Loss 11.3403 Epoch: 4 Global Step: 49225 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:22:04,357-Speed 127.24 samples/sec Loss 11.1924 Epoch: 4 Global Step: 49230 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:22:24,979-Speed 124.15 samples/sec Loss 11.3160 Epoch: 4 Global Step: 49235 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:22:44,938-Speed 128.27 samples/sec Loss 11.0436 Epoch: 4 Global Step: 49240 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:23:04,960-Speed 127.86 samples/sec Loss 11.1662 Epoch: 4 Global Step: 49245 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:23:24,845-Speed 128.75 samples/sec Loss 10.9938 Epoch: 4 Global Step: 49250 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:23:45,283-Speed 125.27 samples/sec Loss 11.4503 Epoch: 4 Global Step: 49255 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:24:05,472-Speed 126.83 samples/sec Loss 11.0976 Epoch: 4 Global Step: 49260 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:24:25,667-Speed 126.77 samples/sec Loss 11.4882 Epoch: 4 Global Step: 49265 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:24:45,870-Speed 126.73 samples/sec Loss 11.3033 Epoch: 4 Global Step: 49270 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:25:05,944-Speed 127.54 samples/sec Loss 11.4083 Epoch: 4 Global Step: 49275 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:25:26,743-Speed 123.09 samples/sec Loss 11.5519 Epoch: 4 Global Step: 49280 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:25:46,858-Speed 127.27 samples/sec Loss 10.9228 Epoch: 4 Global Step: 49285 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:26:07,095-Speed 126.51 samples/sec Loss 11.2684 Epoch: 4 Global Step: 49290 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:26:27,228-Speed 127.16 samples/sec Loss 11.2882 Epoch: 4 Global Step: 49295 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:26:47,416-Speed 126.82 samples/sec Loss 11.6288 Epoch: 4 Global Step: 49300 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:27:07,546-Speed 127.18 samples/sec Loss 11.3932 Epoch: 4 Global Step: 49305 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:27:27,699-Speed 127.04 samples/sec Loss 11.2275 Epoch: 4 Global Step: 49310 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:27:47,921-Speed 126.60 samples/sec Loss 11.5778 Epoch: 4 Global Step: 49315 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:28:08,035-Speed 127.29 samples/sec Loss 11.4067 Epoch: 4 Global Step: 49320 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:28:28,828-Speed 123.12 samples/sec Loss 11.1011 Epoch: 4 Global Step: 49325 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:28:48,972-Speed 127.09 samples/sec Loss 11.3236 Epoch: 4 Global Step: 49330 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:29:09,244-Speed 126.29 samples/sec Loss 11.5261 Epoch: 4 Global Step: 49335 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:29:29,335-Speed 127.42 samples/sec Loss 11.3813 Epoch: 4 Global Step: 49340 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 05:29:49,523-Speed 126.82 samples/sec Loss 10.9672 Epoch: 4 Global Step: 49345 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:30:09,648-Speed 127.21 samples/sec Loss 11.0259 Epoch: 4 Global Step: 49350 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:30:29,754-Speed 127.33 samples/sec Loss 11.0028 Epoch: 4 Global Step: 49355 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:30:49,769-Speed 127.92 samples/sec Loss 11.0862 Epoch: 4 Global Step: 49360 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:31:09,882-Speed 127.29 samples/sec Loss 11.2267 Epoch: 4 Global Step: 49365 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:31:30,793-Speed 122.43 samples/sec Loss 11.4938 Epoch: 4 Global Step: 49370 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:31:50,902-Speed 127.31 samples/sec Loss 11.4034 Epoch: 4 Global Step: 49375 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:32:11,117-Speed 126.64 samples/sec Loss 11.2142 Epoch: 4 Global Step: 49380 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:32:31,295-Speed 126.88 samples/sec Loss 11.3918 Epoch: 4 Global Step: 49385 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:32:51,393-Speed 127.39 samples/sec Loss 11.4097 Epoch: 4 Global Step: 49390 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:33:11,592-Speed 126.75 samples/sec Loss 11.4674 Epoch: 4 Global Step: 49395 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:33:31,822-Speed 126.55 samples/sec Loss 11.4223 Epoch: 4 Global Step: 49400 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:33:51,919-Speed 127.39 samples/sec Loss 11.3116 Epoch: 4 Global Step: 49405 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:34:12,049-Speed 127.18 samples/sec Loss 11.0272 Epoch: 4 Global Step: 49410 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:34:32,893-Speed 122.83 samples/sec Loss 11.3069 Epoch: 4 Global Step: 49415 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:34:53,083-Speed 126.80 samples/sec Loss 11.5409 Epoch: 4 Global Step: 49420 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:35:13,384-Speed 126.11 samples/sec Loss 11.2768 Epoch: 4 Global Step: 49425 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:35:33,511-Speed 127.20 samples/sec Loss 11.1230 Epoch: 4 Global Step: 49430 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:35:53,627-Speed 127.27 samples/sec Loss 11.1569 Epoch: 4 Global Step: 49435 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:36:13,821-Speed 126.78 samples/sec Loss 11.4586 Epoch: 4 Global Step: 49440 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:36:34,022-Speed 126.74 samples/sec Loss 11.3746 Epoch: 4 Global Step: 49445 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:36:54,127-Speed 127.34 samples/sec Loss 11.8623 Epoch: 4 Global Step: 49450 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:37:14,157-Speed 127.82 samples/sec Loss 10.8988 Epoch: 4 Global Step: 49455 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:37:34,828-Speed 123.85 samples/sec Loss 11.3744 Epoch: 4 Global Step: 49460 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:37:54,974-Speed 127.08 samples/sec Loss 11.4607 Epoch: 4 Global Step: 49465 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:38:15,127-Speed 127.03 samples/sec Loss 11.6497 Epoch: 4 Global Step: 49470 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:38:35,590-Speed 125.11 samples/sec Loss 11.4584 Epoch: 4 Global Step: 49475 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:38:55,657-Speed 127.58 samples/sec Loss 11.1932 Epoch: 4 Global Step: 49480 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:39:15,837-Speed 126.86 samples/sec Loss 11.2968 Epoch: 4 Global Step: 49485 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:39:35,820-Speed 128.12 samples/sec Loss 11.4985 Epoch: 4 Global Step: 49490 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:39:56,008-Speed 126.82 samples/sec Loss 11.4484 Epoch: 4 Global Step: 49495 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:40:15,988-Speed 128.13 samples/sec Loss 11.3634 Epoch: 4 Global Step: 49500 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:40:36,684-Speed 123.70 samples/sec Loss 10.8436 Epoch: 4 Global Step: 49505 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:40:56,647-Speed 128.24 samples/sec Loss 11.5460 Epoch: 4 Global Step: 49510 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:41:16,661-Speed 127.92 samples/sec Loss 11.4108 Epoch: 4 Global Step: 49515 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:41:36,545-Speed 128.75 samples/sec Loss 11.2669 Epoch: 4 Global Step: 49520 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:41:56,708-Speed 126.97 samples/sec Loss 11.0142 Epoch: 4 Global Step: 49525 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:42:16,725-Speed 127.90 samples/sec Loss 11.3758 Epoch: 4 Global Step: 49530 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:42:36,937-Speed 126.66 samples/sec Loss 11.5251 Epoch: 4 Global Step: 49535 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:42:56,839-Speed 128.64 samples/sec Loss 11.4310 Epoch: 4 Global Step: 49540 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:43:16,961-Speed 127.23 samples/sec Loss 11.2492 Epoch: 4 Global Step: 49545 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:43:37,857-Speed 122.51 samples/sec Loss 11.3304 Epoch: 4 Global Step: 49550 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:43:57,956-Speed 127.38 samples/sec Loss 11.1584 Epoch: 4 Global Step: 49555 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:44:18,182-Speed 126.58 samples/sec Loss 11.0368 Epoch: 4 Global Step: 49560 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:44:38,328-Speed 127.08 samples/sec Loss 11.4277 Epoch: 4 Global Step: 49565 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:44:58,416-Speed 127.45 samples/sec Loss 11.5633 Epoch: 4 Global Step: 49570 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:45:18,624-Speed 126.69 samples/sec Loss 11.5651 Epoch: 4 Global Step: 49575 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:45:38,716-Speed 127.42 samples/sec Loss 11.6470 Epoch: 4 Global Step: 49580 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:45:58,806-Speed 127.44 samples/sec Loss 11.2408 Epoch: 4 Global Step: 49585 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:46:19,319-Speed 124.80 samples/sec Loss 11.3952 Epoch: 4 Global Step: 49590 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:46:40,161-Speed 122.83 samples/sec Loss 11.3151 Epoch: 4 Global Step: 49595 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:47:00,394-Speed 126.53 samples/sec Loss 11.2779 Epoch: 4 Global Step: 49600 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:47:20,568-Speed 126.90 samples/sec Loss 11.3444 Epoch: 4 Global Step: 49605 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:47:40,666-Speed 127.39 samples/sec Loss 11.4193 Epoch: 4 Global Step: 49610 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:48:01,615-Speed 122.21 samples/sec Loss 11.3658 Epoch: 4 Global Step: 49615 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:48:21,758-Speed 127.10 samples/sec Loss 11.1565 Epoch: 4 Global Step: 49620 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:48:41,932-Speed 126.90 samples/sec Loss 11.3376 Epoch: 4 Global Step: 49625 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:49:01,861-Speed 128.46 samples/sec Loss 10.8950 Epoch: 4 Global Step: 49630 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:49:21,933-Speed 127.55 samples/sec Loss 11.2880 Epoch: 4 Global Step: 49635 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:49:42,793-Speed 122.72 samples/sec Loss 11.2505 Epoch: 4 Global Step: 49640 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:50:02,863-Speed 127.56 samples/sec Loss 11.3575 Epoch: 4 Global Step: 49645 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:50:23,003-Speed 127.11 samples/sec Loss 11.7233 Epoch: 4 Global Step: 49650 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:50:43,211-Speed 126.69 samples/sec Loss 11.2942 Epoch: 4 Global Step: 49655 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:51:03,303-Speed 127.42 samples/sec Loss 11.2433 Epoch: 4 Global Step: 49660 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:51:23,493-Speed 126.81 samples/sec Loss 11.1429 Epoch: 4 Global Step: 49665 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:51:43,692-Speed 126.75 samples/sec Loss 11.4703 Epoch: 4 Global Step: 49670 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:52:04,586-Speed 122.53 samples/sec Loss 11.4353 Epoch: 4 Global Step: 49675 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:52:24,708-Speed 127.23 samples/sec Loss 11.0879 Epoch: 4 Global Step: 49680 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:52:45,672-Speed 122.12 samples/sec Loss 11.2437 Epoch: 4 Global Step: 49685 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:53:05,868-Speed 126.76 samples/sec Loss 11.1210 Epoch: 4 Global Step: 49690 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:53:26,273-Speed 125.47 samples/sec Loss 11.6514 Epoch: 4 Global Step: 49695 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:53:46,413-Speed 127.12 samples/sec Loss 11.1186 Epoch: 4 Global Step: 49700 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:54:06,795-Speed 125.61 samples/sec Loss 11.2531 Epoch: 4 Global Step: 49705 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:54:27,046-Speed 126.42 samples/sec Loss 11.4330 Epoch: 4 Global Step: 49710 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:54:47,160-Speed 127.28 samples/sec Loss 11.4962 Epoch: 4 Global Step: 49715 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:55:07,063-Speed 128.63 samples/sec Loss 11.4633 Epoch: 4 Global Step: 49720 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:55:27,503-Speed 125.25 samples/sec Loss 11.0124 Epoch: 4 Global Step: 49725 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:55:48,691-Speed 120.83 samples/sec Loss 11.2475 Epoch: 4 Global Step: 49730 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:56:08,787-Speed 127.40 samples/sec Loss 11.6182 Epoch: 4 Global Step: 49735 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:56:28,722-Speed 128.42 samples/sec Loss 11.2142 Epoch: 4 Global Step: 49740 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:56:48,920-Speed 126.75 samples/sec Loss 11.5343 Epoch: 4 Global Step: 49745 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:57:08,832-Speed 128.57 samples/sec Loss 10.8835 Epoch: 4 Global Step: 49750 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:57:29,170-Speed 125.88 samples/sec Loss 11.0360 Epoch: 4 Global Step: 49755 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:57:49,380-Speed 126.68 samples/sec Loss 11.3339 Epoch: 4 Global Step: 49760 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:58:09,537-Speed 127.01 samples/sec Loss 11.4081 Epoch: 4 Global Step: 49765 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:58:29,615-Speed 127.51 samples/sec Loss 11.3530 Epoch: 4 Global Step: 49770 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:58:50,389-Speed 123.24 samples/sec Loss 11.2330 Epoch: 4 Global Step: 49775 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:59:10,488-Speed 127.37 samples/sec Loss 11.2598 Epoch: 4 Global Step: 49780 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:59:30,632-Speed 127.09 samples/sec Loss 10.8648 Epoch: 4 Global Step: 49785 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 05:59:50,708-Speed 127.52 samples/sec Loss 11.3064 Epoch: 4 Global Step: 49790 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:00:10,843-Speed 127.15 samples/sec Loss 11.4464 Epoch: 4 Global Step: 49795 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:00:31,364-Speed 124.76 samples/sec Loss 11.4404 Epoch: 4 Global Step: 49800 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:00:51,554-Speed 126.81 samples/sec Loss 11.1735 Epoch: 4 Global Step: 49805 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:01:11,687-Speed 127.16 samples/sec Loss 11.4169 Epoch: 4 Global Step: 49810 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:01:31,682-Speed 128.04 samples/sec Loss 11.6064 Epoch: 4 Global Step: 49815 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:01:52,109-Speed 125.33 samples/sec Loss 11.5969 Epoch: 4 Global Step: 49820 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:02:28,070-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-09 06:02:55,011-Speed 40.70 samples/sec Loss 10.5054 Epoch: 5 Global Step: 49825 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:03:15,541-Speed 124.71 samples/sec Loss 10.2292 Epoch: 5 Global Step: 49830 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:03:35,677-Speed 127.14 samples/sec Loss 10.4856 Epoch: 5 Global Step: 49835 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:03:55,990-Speed 126.04 samples/sec Loss 10.8060 Epoch: 5 Global Step: 49840 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:04:16,140-Speed 127.06 samples/sec Loss 10.5064 Epoch: 5 Global Step: 49845 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:04:36,493-Speed 125.79 samples/sec Loss 10.5736 Epoch: 5 Global Step: 49850 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:04:56,577-Speed 127.47 samples/sec Loss 10.2428 Epoch: 5 Global Step: 49855 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:05:16,732-Speed 127.03 samples/sec Loss 10.4598 Epoch: 5 Global Step: 49860 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:05:37,614-Speed 122.60 samples/sec Loss 10.8791 Epoch: 5 Global Step: 49865 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:05:57,743-Speed 127.19 samples/sec Loss 10.6399 Epoch: 5 Global Step: 49870 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:06:18,237-Speed 124.93 samples/sec Loss 10.9334 Epoch: 5 Global Step: 49875 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:06:38,584-Speed 125.82 samples/sec Loss 10.4637 Epoch: 5 Global Step: 49880 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:06:58,800-Speed 126.64 samples/sec Loss 10.7142 Epoch: 5 Global Step: 49885 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:07:18,987-Speed 126.82 samples/sec Loss 10.6161 Epoch: 5 Global Step: 49890 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:07:39,256-Speed 126.31 samples/sec Loss 10.8392 Epoch: 5 Global Step: 49895 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:07:59,428-Speed 126.91 samples/sec Loss 10.3915 Epoch: 5 Global Step: 49900 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:08:19,748-Speed 125.99 samples/sec Loss 10.8196 Epoch: 5 Global Step: 49905 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:08:40,223-Speed 125.04 samples/sec Loss 10.6238 Epoch: 5 Global Step: 49910 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:09:00,466-Speed 126.47 samples/sec Loss 10.4225 Epoch: 5 Global Step: 49915 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:09:21,329-Speed 122.71 samples/sec Loss 10.4503 Epoch: 5 Global Step: 49920 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:09:41,401-Speed 127.55 samples/sec Loss 10.6729 Epoch: 5 Global Step: 49925 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:10:01,405-Speed 127.97 samples/sec Loss 10.6366 Epoch: 5 Global Step: 49930 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:10:21,557-Speed 127.04 samples/sec Loss 10.4545 Epoch: 5 Global Step: 49935 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:10:41,752-Speed 126.77 samples/sec Loss 10.6649 Epoch: 5 Global Step: 49940 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:11:01,997-Speed 126.46 samples/sec Loss 10.6990 Epoch: 5 Global Step: 49945 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:11:22,067-Speed 127.56 samples/sec Loss 10.6995 Epoch: 5 Global Step: 49950 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:11:42,526-Speed 125.13 samples/sec Loss 10.7341 Epoch: 5 Global Step: 49955 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:12:02,686-Speed 126.99 samples/sec Loss 10.7904 Epoch: 5 Global Step: 49960 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:12:23,066-Speed 125.62 samples/sec Loss 10.7832 Epoch: 5 Global Step: 49965 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:12:43,305-Speed 126.49 samples/sec Loss 10.7865 Epoch: 5 Global Step: 49970 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:13:03,269-Speed 128.24 samples/sec Loss 11.1602 Epoch: 5 Global Step: 49975 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:13:23,305-Speed 127.77 samples/sec Loss 10.3677 Epoch: 5 Global Step: 49980 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:13:43,250-Speed 128.36 samples/sec Loss 10.8029 Epoch: 5 Global Step: 49985 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:14:03,409-Speed 127.00 samples/sec Loss 10.6075 Epoch: 5 Global Step: 49990 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:14:23,521-Speed 127.29 samples/sec Loss 10.8086 Epoch: 5 Global Step: 49995 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:14:44,121-Speed 124.28 samples/sec Loss 10.9406 Epoch: 5 Global Step: 50000 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 06:29:35,536-[lfw][50000]XNorm: 23.342901 Training: 2021-06-09 06:29:35,537-[lfw][50000]Accuracy-Flip: 0.99683+-0.00217 Training: 2021-06-09 06:29:35,537-[lfw][50000]Accuracy-Highest: 0.99700 Training: 2021-06-09 06:46:50,458-[cfp_fp][50000]XNorm: 21.014315 Training: 2021-06-09 06:46:50,458-[cfp_fp][50000]Accuracy-Flip: 0.97514+-0.00658 Training: 2021-06-09 06:46:50,458-[cfp_fp][50000]Accuracy-Highest: 0.97514 Training: 2021-06-09 07:01:41,672-[agedb_30][50000]XNorm: 23.181468 Training: 2021-06-09 07:01:41,672-[agedb_30][50000]Accuracy-Flip: 0.97167+-0.00910 Training: 2021-06-09 07:01:41,672-[agedb_30][50000]Accuracy-Highest: 0.97167 Training: 2021-06-09 07:02:01,993-Speed 0.90 samples/sec Loss 10.8675 Epoch: 5 Global Step: 50005 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:02:22,002-Speed 127.95 samples/sec Loss 10.5531 Epoch: 5 Global Step: 50010 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:02:42,583-Speed 124.39 samples/sec Loss 10.7462 Epoch: 5 Global Step: 50015 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:03:02,337-Speed 129.60 samples/sec Loss 10.8078 Epoch: 5 Global Step: 50020 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:03:22,696-Speed 125.75 samples/sec Loss 10.7778 Epoch: 5 Global Step: 50025 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:03:42,889-Speed 126.78 samples/sec Loss 11.1111 Epoch: 5 Global Step: 50030 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:04:03,002-Speed 127.29 samples/sec Loss 10.8493 Epoch: 5 Global Step: 50035 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:04:23,116-Speed 127.28 samples/sec Loss 11.1279 Epoch: 5 Global Step: 50040 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:04:43,767-Speed 123.97 samples/sec Loss 10.9388 Epoch: 5 Global Step: 50045 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:05:03,758-Speed 128.07 samples/sec Loss 10.6487 Epoch: 5 Global Step: 50050 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:05:23,919-Speed 126.99 samples/sec Loss 10.6968 Epoch: 5 Global Step: 50055 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:05:44,075-Speed 127.01 samples/sec Loss 10.6834 Epoch: 5 Global Step: 50060 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:06:04,091-Speed 127.90 samples/sec Loss 10.7885 Epoch: 5 Global Step: 50065 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:06:24,593-Speed 124.88 samples/sec Loss 10.6472 Epoch: 5 Global Step: 50070 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:06:44,763-Speed 126.92 samples/sec Loss 10.7691 Epoch: 5 Global Step: 50075 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:07:04,956-Speed 126.78 samples/sec Loss 10.8602 Epoch: 5 Global Step: 50080 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:07:25,092-Speed 127.15 samples/sec Loss 11.1943 Epoch: 5 Global Step: 50085 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:07:45,546-Speed 125.16 samples/sec Loss 10.8116 Epoch: 5 Global Step: 50090 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:08:05,642-Speed 127.39 samples/sec Loss 10.7038 Epoch: 5 Global Step: 50095 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:08:25,899-Speed 126.39 samples/sec Loss 10.8368 Epoch: 5 Global Step: 50100 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:08:45,861-Speed 128.25 samples/sec Loss 10.9242 Epoch: 5 Global Step: 50105 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:09:06,167-Speed 126.08 samples/sec Loss 11.1012 Epoch: 5 Global Step: 50110 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:09:26,417-Speed 126.43 samples/sec Loss 10.4933 Epoch: 5 Global Step: 50115 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:09:46,305-Speed 128.72 samples/sec Loss 10.9604 Epoch: 5 Global Step: 50120 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:10:06,425-Speed 127.24 samples/sec Loss 10.8507 Epoch: 5 Global Step: 50125 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:10:26,538-Speed 127.29 samples/sec Loss 10.9489 Epoch: 5 Global Step: 50130 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:10:47,152-Speed 124.19 samples/sec Loss 10.6366 Epoch: 5 Global Step: 50135 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:11:07,417-Speed 126.33 samples/sec Loss 10.9613 Epoch: 5 Global Step: 50140 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:11:27,481-Speed 127.60 samples/sec Loss 10.4328 Epoch: 5 Global Step: 50145 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:11:47,729-Speed 126.44 samples/sec Loss 11.0854 Epoch: 5 Global Step: 50150 Fp16 Grad Scale: 8192 Required: 239 hours Training: 2021-06-09 07:12:07,923-Speed 126.78 samples/sec Loss 10.7173 Epoch: 5 Global Step: 50155 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:12:28,887-Speed 122.13 samples/sec Loss 11.1446 Epoch: 5 Global Step: 50160 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:12:49,204-Speed 126.00 samples/sec Loss 10.7633 Epoch: 5 Global Step: 50165 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:13:09,302-Speed 127.38 samples/sec Loss 11.0155 Epoch: 5 Global Step: 50170 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:13:29,366-Speed 127.59 samples/sec Loss 10.7877 Epoch: 5 Global Step: 50175 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:13:49,495-Speed 127.19 samples/sec Loss 10.6888 Epoch: 5 Global Step: 50180 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:14:09,638-Speed 127.10 samples/sec Loss 10.8628 Epoch: 5 Global Step: 50185 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:14:29,500-Speed 128.89 samples/sec Loss 10.7275 Epoch: 5 Global Step: 50190 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:14:49,519-Speed 127.88 samples/sec Loss 11.1395 Epoch: 5 Global Step: 50195 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:15:09,277-Speed 129.58 samples/sec Loss 10.9133 Epoch: 5 Global Step: 50200 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:15:29,557-Speed 126.24 samples/sec Loss 11.1070 Epoch: 5 Global Step: 50205 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:15:49,377-Speed 129.17 samples/sec Loss 10.9517 Epoch: 5 Global Step: 50210 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:16:09,431-Speed 127.66 samples/sec Loss 10.5850 Epoch: 5 Global Step: 50215 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:16:29,348-Speed 128.54 samples/sec Loss 11.1152 Epoch: 5 Global Step: 50220 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:16:49,508-Speed 126.99 samples/sec Loss 10.8517 Epoch: 5 Global Step: 50225 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:17:09,429-Speed 128.51 samples/sec Loss 11.1282 Epoch: 5 Global Step: 50230 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:17:29,285-Speed 128.94 samples/sec Loss 10.9250 Epoch: 5 Global Step: 50235 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:17:49,957-Speed 123.85 samples/sec Loss 10.5936 Epoch: 5 Global Step: 50240 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:18:09,893-Speed 128.41 samples/sec Loss 11.1264 Epoch: 5 Global Step: 50245 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:18:30,116-Speed 126.60 samples/sec Loss 10.7245 Epoch: 5 Global Step: 50250 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:18:50,012-Speed 128.67 samples/sec Loss 11.0838 Epoch: 5 Global Step: 50255 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:19:10,010-Speed 128.02 samples/sec Loss 10.9738 Epoch: 5 Global Step: 50260 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:19:29,882-Speed 128.83 samples/sec Loss 10.9927 Epoch: 5 Global Step: 50265 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:19:50,185-Speed 126.10 samples/sec Loss 11.4199 Epoch: 5 Global Step: 50270 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:20:10,036-Speed 128.96 samples/sec Loss 11.1094 Epoch: 5 Global Step: 50275 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:20:30,067-Speed 127.81 samples/sec Loss 11.3454 Epoch: 5 Global Step: 50280 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:20:49,956-Speed 128.72 samples/sec Loss 11.1634 Epoch: 5 Global Step: 50285 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:21:09,822-Speed 128.87 samples/sec Loss 11.3450 Epoch: 5 Global Step: 50290 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:21:30,005-Speed 126.85 samples/sec Loss 10.8031 Epoch: 5 Global Step: 50295 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:21:50,081-Speed 127.52 samples/sec Loss 10.8159 Epoch: 5 Global Step: 50300 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:22:09,930-Speed 128.98 samples/sec Loss 10.8834 Epoch: 5 Global Step: 50305 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:22:30,131-Speed 126.73 samples/sec Loss 11.1285 Epoch: 5 Global Step: 50310 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:22:50,292-Speed 126.98 samples/sec Loss 11.1573 Epoch: 5 Global Step: 50315 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:23:10,325-Speed 127.80 samples/sec Loss 11.0643 Epoch: 5 Global Step: 50320 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:23:30,120-Speed 129.33 samples/sec Loss 10.9874 Epoch: 5 Global Step: 50325 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:23:50,021-Speed 128.64 samples/sec Loss 10.7729 Epoch: 5 Global Step: 50330 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:24:09,963-Speed 128.39 samples/sec Loss 10.8813 Epoch: 5 Global Step: 50335 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:24:30,182-Speed 126.62 samples/sec Loss 11.0122 Epoch: 5 Global Step: 50340 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:24:50,177-Speed 128.04 samples/sec Loss 10.7193 Epoch: 5 Global Step: 50345 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:25:10,156-Speed 128.14 samples/sec Loss 10.9813 Epoch: 5 Global Step: 50350 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:25:30,168-Speed 127.93 samples/sec Loss 11.1360 Epoch: 5 Global Step: 50355 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:25:50,363-Speed 126.77 samples/sec Loss 11.3136 Epoch: 5 Global Step: 50360 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:26:10,359-Speed 128.03 samples/sec Loss 10.9850 Epoch: 5 Global Step: 50365 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:26:30,299-Speed 128.39 samples/sec Loss 11.2489 Epoch: 5 Global Step: 50370 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:26:50,335-Speed 127.78 samples/sec Loss 11.1279 Epoch: 5 Global Step: 50375 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:27:10,188-Speed 128.95 samples/sec Loss 11.2266 Epoch: 5 Global Step: 50380 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:27:30,404-Speed 126.64 samples/sec Loss 11.0309 Epoch: 5 Global Step: 50385 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:27:50,291-Speed 128.73 samples/sec Loss 11.5945 Epoch: 5 Global Step: 50390 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:28:10,247-Speed 128.29 samples/sec Loss 11.1805 Epoch: 5 Global Step: 50395 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:28:30,475-Speed 126.56 samples/sec Loss 11.1400 Epoch: 5 Global Step: 50400 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:28:50,775-Speed 126.11 samples/sec Loss 10.9736 Epoch: 5 Global Step: 50405 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:29:10,690-Speed 128.56 samples/sec Loss 10.9041 Epoch: 5 Global Step: 50410 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:29:30,573-Speed 128.76 samples/sec Loss 10.9858 Epoch: 5 Global Step: 50415 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:29:50,399-Speed 129.13 samples/sec Loss 10.8126 Epoch: 5 Global Step: 50420 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:30:10,255-Speed 128.94 samples/sec Loss 11.3011 Epoch: 5 Global Step: 50425 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:30:30,473-Speed 126.63 samples/sec Loss 11.0121 Epoch: 5 Global Step: 50430 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:30:50,477-Speed 127.98 samples/sec Loss 10.9804 Epoch: 5 Global Step: 50435 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:31:10,498-Speed 127.87 samples/sec Loss 10.9411 Epoch: 5 Global Step: 50440 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:31:30,358-Speed 128.91 samples/sec Loss 11.1978 Epoch: 5 Global Step: 50445 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:31:50,629-Speed 126.29 samples/sec Loss 10.9850 Epoch: 5 Global Step: 50450 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:32:10,552-Speed 128.50 samples/sec Loss 10.8203 Epoch: 5 Global Step: 50455 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:32:30,511-Speed 128.27 samples/sec Loss 11.2559 Epoch: 5 Global Step: 50460 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:32:50,444-Speed 128.44 samples/sec Loss 10.9587 Epoch: 5 Global Step: 50465 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:33:10,354-Speed 128.58 samples/sec Loss 11.0398 Epoch: 5 Global Step: 50470 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:33:30,597-Speed 126.47 samples/sec Loss 11.1707 Epoch: 5 Global Step: 50475 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:33:50,629-Speed 127.80 samples/sec Loss 11.1860 Epoch: 5 Global Step: 50480 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:34:10,388-Speed 129.58 samples/sec Loss 11.3803 Epoch: 5 Global Step: 50485 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:34:30,444-Speed 127.65 samples/sec Loss 11.0961 Epoch: 5 Global Step: 50490 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:34:50,679-Speed 126.52 samples/sec Loss 11.1191 Epoch: 5 Global Step: 50495 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:35:10,583-Speed 128.63 samples/sec Loss 11.2656 Epoch: 5 Global Step: 50500 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:35:30,547-Speed 128.24 samples/sec Loss 10.9792 Epoch: 5 Global Step: 50505 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:35:50,380-Speed 129.09 samples/sec Loss 10.7839 Epoch: 5 Global Step: 50510 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:36:10,496-Speed 127.27 samples/sec Loss 11.1390 Epoch: 5 Global Step: 50515 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:36:30,547-Speed 127.68 samples/sec Loss 11.3016 Epoch: 5 Global Step: 50520 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:36:50,587-Speed 127.75 samples/sec Loss 11.1440 Epoch: 5 Global Step: 50525 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:37:10,453-Speed 128.87 samples/sec Loss 11.3011 Epoch: 5 Global Step: 50530 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:37:30,515-Speed 127.61 samples/sec Loss 11.0894 Epoch: 5 Global Step: 50535 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:37:50,707-Speed 126.79 samples/sec Loss 11.2096 Epoch: 5 Global Step: 50540 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:38:10,651-Speed 128.36 samples/sec Loss 11.4433 Epoch: 5 Global Step: 50545 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:38:30,511-Speed 128.91 samples/sec Loss 11.3492 Epoch: 5 Global Step: 50550 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:38:50,514-Speed 127.99 samples/sec Loss 11.1346 Epoch: 5 Global Step: 50555 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:39:10,430-Speed 128.55 samples/sec Loss 11.2156 Epoch: 5 Global Step: 50560 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:39:30,779-Speed 125.81 samples/sec Loss 11.0986 Epoch: 5 Global Step: 50565 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:39:50,518-Speed 129.70 samples/sec Loss 11.1840 Epoch: 5 Global Step: 50570 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:40:10,617-Speed 127.37 samples/sec Loss 11.2041 Epoch: 5 Global Step: 50575 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:40:30,612-Speed 128.04 samples/sec Loss 11.1976 Epoch: 5 Global Step: 50580 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:40:50,689-Speed 127.51 samples/sec Loss 11.0419 Epoch: 5 Global Step: 50585 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:41:10,680-Speed 128.07 samples/sec Loss 11.0987 Epoch: 5 Global Step: 50590 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:41:30,533-Speed 128.95 samples/sec Loss 10.9921 Epoch: 5 Global Step: 50595 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:41:50,554-Speed 127.87 samples/sec Loss 10.9829 Epoch: 5 Global Step: 50600 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:42:10,397-Speed 129.02 samples/sec Loss 11.1046 Epoch: 5 Global Step: 50605 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:42:30,563-Speed 126.95 samples/sec Loss 10.9073 Epoch: 5 Global Step: 50610 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:42:50,460-Speed 128.66 samples/sec Loss 11.0009 Epoch: 5 Global Step: 50615 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:43:10,537-Speed 127.52 samples/sec Loss 11.0353 Epoch: 5 Global Step: 50620 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:43:30,300-Speed 129.54 samples/sec Loss 11.1951 Epoch: 5 Global Step: 50625 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:43:50,572-Speed 126.29 samples/sec Loss 11.2424 Epoch: 5 Global Step: 50630 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:44:10,422-Speed 128.97 samples/sec Loss 10.9428 Epoch: 5 Global Step: 50635 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:44:30,592-Speed 126.93 samples/sec Loss 11.0643 Epoch: 5 Global Step: 50640 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:44:50,461-Speed 128.85 samples/sec Loss 11.3422 Epoch: 5 Global Step: 50645 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:45:10,298-Speed 129.06 samples/sec Loss 11.3190 Epoch: 5 Global Step: 50650 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:45:30,442-Speed 127.10 samples/sec Loss 10.8759 Epoch: 5 Global Step: 50655 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:45:50,480-Speed 127.76 samples/sec Loss 11.2873 Epoch: 5 Global Step: 50660 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:46:10,362-Speed 128.77 samples/sec Loss 11.3348 Epoch: 5 Global Step: 50665 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:46:30,342-Speed 128.13 samples/sec Loss 11.5163 Epoch: 5 Global Step: 50670 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:46:50,402-Speed 127.62 samples/sec Loss 10.9471 Epoch: 5 Global Step: 50675 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:47:10,409-Speed 127.96 samples/sec Loss 10.9276 Epoch: 5 Global Step: 50680 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:47:30,275-Speed 128.87 samples/sec Loss 11.0683 Epoch: 5 Global Step: 50685 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:47:50,019-Speed 129.67 samples/sec Loss 10.8584 Epoch: 5 Global Step: 50690 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:48:09,965-Speed 128.35 samples/sec Loss 11.3260 Epoch: 5 Global Step: 50695 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:48:30,168-Speed 126.72 samples/sec Loss 11.3742 Epoch: 5 Global Step: 50700 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:48:50,101-Speed 128.44 samples/sec Loss 11.5552 Epoch: 5 Global Step: 50705 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:49:09,994-Speed 128.69 samples/sec Loss 11.0171 Epoch: 5 Global Step: 50710 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:49:29,936-Speed 128.37 samples/sec Loss 11.2483 Epoch: 5 Global Step: 50715 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:49:50,078-Speed 127.11 samples/sec Loss 11.5737 Epoch: 5 Global Step: 50720 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:50:10,164-Speed 127.46 samples/sec Loss 11.2236 Epoch: 5 Global Step: 50725 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:50:29,922-Speed 129.58 samples/sec Loss 11.1235 Epoch: 5 Global Step: 50730 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:50:49,839-Speed 128.54 samples/sec Loss 11.0386 Epoch: 5 Global Step: 50735 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:51:09,714-Speed 128.82 samples/sec Loss 11.0874 Epoch: 5 Global Step: 50740 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:51:29,902-Speed 126.81 samples/sec Loss 11.2679 Epoch: 5 Global Step: 50745 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:51:49,769-Speed 128.86 samples/sec Loss 11.1058 Epoch: 5 Global Step: 50750 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:52:09,539-Speed 129.50 samples/sec Loss 11.0627 Epoch: 5 Global Step: 50755 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:52:29,458-Speed 128.52 samples/sec Loss 11.1726 Epoch: 5 Global Step: 50760 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:52:49,735-Speed 126.26 samples/sec Loss 11.3802 Epoch: 5 Global Step: 50765 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:53:09,758-Speed 127.86 samples/sec Loss 11.3382 Epoch: 5 Global Step: 50770 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:53:29,481-Speed 129.81 samples/sec Loss 10.9548 Epoch: 5 Global Step: 50775 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:53:49,519-Speed 127.76 samples/sec Loss 11.4077 Epoch: 5 Global Step: 50780 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:54:09,374-Speed 128.95 samples/sec Loss 11.1904 Epoch: 5 Global Step: 50785 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:54:29,634-Speed 126.36 samples/sec Loss 11.3263 Epoch: 5 Global Step: 50790 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:54:49,363-Speed 129.76 samples/sec Loss 11.1426 Epoch: 5 Global Step: 50795 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:55:09,381-Speed 127.89 samples/sec Loss 10.7732 Epoch: 5 Global Step: 50800 Fp16 Grad Scale: 8192 Required: 238 hours Training: 2021-06-09 07:55:29,232-Speed 128.97 samples/sec Loss 10.9974 Epoch: 5 Global Step: 50805 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:55:49,458-Speed 126.57 samples/sec Loss 11.4674 Epoch: 5 Global Step: 50810 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:56:09,365-Speed 128.61 samples/sec Loss 11.1167 Epoch: 5 Global Step: 50815 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:56:29,227-Speed 128.90 samples/sec Loss 11.2707 Epoch: 5 Global Step: 50820 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:56:49,249-Speed 127.87 samples/sec Loss 11.4598 Epoch: 5 Global Step: 50825 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:57:09,164-Speed 128.55 samples/sec Loss 11.2590 Epoch: 5 Global Step: 50830 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:57:29,264-Speed 127.37 samples/sec Loss 11.1718 Epoch: 5 Global Step: 50835 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:57:49,160-Speed 128.67 samples/sec Loss 11.2252 Epoch: 5 Global Step: 50840 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:58:09,128-Speed 128.22 samples/sec Loss 11.4576 Epoch: 5 Global Step: 50845 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:58:29,111-Speed 128.12 samples/sec Loss 11.0368 Epoch: 5 Global Step: 50850 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:58:49,309-Speed 126.75 samples/sec Loss 10.9844 Epoch: 5 Global Step: 50855 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:59:09,144-Speed 129.07 samples/sec Loss 11.3038 Epoch: 5 Global Step: 50860 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:59:29,172-Speed 127.83 samples/sec Loss 11.2620 Epoch: 5 Global Step: 50865 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 07:59:49,085-Speed 128.57 samples/sec Loss 11.0081 Epoch: 5 Global Step: 50870 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:00:08,985-Speed 128.65 samples/sec Loss 11.1665 Epoch: 5 Global Step: 50875 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:00:29,025-Speed 127.75 samples/sec Loss 11.0773 Epoch: 5 Global Step: 50880 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:00:49,024-Speed 128.02 samples/sec Loss 11.2677 Epoch: 5 Global Step: 50885 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:01:08,934-Speed 128.58 samples/sec Loss 11.2186 Epoch: 5 Global Step: 50890 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:01:29,166-Speed 126.54 samples/sec Loss 11.3436 Epoch: 5 Global Step: 50895 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:01:49,094-Speed 128.47 samples/sec Loss 11.2294 Epoch: 5 Global Step: 50900 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:02:09,122-Speed 127.83 samples/sec Loss 10.9105 Epoch: 5 Global Step: 50905 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:02:29,049-Speed 128.48 samples/sec Loss 11.1211 Epoch: 5 Global Step: 50910 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:02:48,991-Speed 128.38 samples/sec Loss 11.1234 Epoch: 5 Global Step: 50915 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:03:08,863-Speed 128.83 samples/sec Loss 11.0945 Epoch: 5 Global Step: 50920 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:03:29,065-Speed 126.73 samples/sec Loss 11.2814 Epoch: 5 Global Step: 50925 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:03:48,932-Speed 128.86 samples/sec Loss 11.3673 Epoch: 5 Global Step: 50930 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:04:09,052-Speed 127.24 samples/sec Loss 11.0360 Epoch: 5 Global Step: 50935 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:04:28,799-Speed 129.65 samples/sec Loss 11.0967 Epoch: 5 Global Step: 50940 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:04:48,864-Speed 127.59 samples/sec Loss 11.1490 Epoch: 5 Global Step: 50945 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:05:08,928-Speed 127.60 samples/sec Loss 11.4780 Epoch: 5 Global Step: 50950 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:05:28,782-Speed 128.95 samples/sec Loss 11.3433 Epoch: 5 Global Step: 50955 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:05:48,685-Speed 128.63 samples/sec Loss 11.3191 Epoch: 5 Global Step: 50960 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:06:08,468-Speed 129.41 samples/sec Loss 11.5054 Epoch: 5 Global Step: 50965 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:06:28,692-Speed 126.59 samples/sec Loss 11.1892 Epoch: 5 Global Step: 50970 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:06:48,654-Speed 128.25 samples/sec Loss 11.2376 Epoch: 5 Global Step: 50975 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:07:08,657-Speed 127.99 samples/sec Loss 11.2208 Epoch: 5 Global Step: 50980 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:07:28,406-Speed 129.63 samples/sec Loss 11.4402 Epoch: 5 Global Step: 50985 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:07:48,730-Speed 125.97 samples/sec Loss 11.3374 Epoch: 5 Global Step: 50990 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:08:08,550-Speed 129.17 samples/sec Loss 11.3194 Epoch: 5 Global Step: 50995 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:08:28,421-Speed 128.84 samples/sec Loss 11.1639 Epoch: 5 Global Step: 51000 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:08:48,854-Speed 125.29 samples/sec Loss 11.2270 Epoch: 5 Global Step: 51005 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:09:08,713-Speed 128.91 samples/sec Loss 11.1492 Epoch: 5 Global Step: 51010 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:09:28,927-Speed 126.65 samples/sec Loss 11.1584 Epoch: 5 Global Step: 51015 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:09:48,953-Speed 127.84 samples/sec Loss 11.5091 Epoch: 5 Global Step: 51020 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:10:08,721-Speed 129.50 samples/sec Loss 11.8053 Epoch: 5 Global Step: 51025 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:10:28,512-Speed 129.36 samples/sec Loss 11.1929 Epoch: 5 Global Step: 51030 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:10:48,896-Speed 125.59 samples/sec Loss 11.3754 Epoch: 5 Global Step: 51035 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:11:08,749-Speed 128.95 samples/sec Loss 11.2992 Epoch: 5 Global Step: 51040 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:11:28,686-Speed 128.41 samples/sec Loss 11.3776 Epoch: 5 Global Step: 51045 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:11:48,549-Speed 128.90 samples/sec Loss 11.8214 Epoch: 5 Global Step: 51050 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:12:08,479-Speed 128.46 samples/sec Loss 11.3753 Epoch: 5 Global Step: 51055 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:12:28,802-Speed 125.97 samples/sec Loss 11.4487 Epoch: 5 Global Step: 51060 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:12:48,807-Speed 127.98 samples/sec Loss 10.9734 Epoch: 5 Global Step: 51065 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:13:08,545-Speed 129.70 samples/sec Loss 11.2395 Epoch: 5 Global Step: 51070 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:13:28,772-Speed 126.57 samples/sec Loss 11.0308 Epoch: 5 Global Step: 51075 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:13:48,664-Speed 128.70 samples/sec Loss 10.9644 Epoch: 5 Global Step: 51080 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:14:08,656-Speed 128.06 samples/sec Loss 11.0897 Epoch: 5 Global Step: 51085 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:14:28,503-Speed 128.99 samples/sec Loss 10.9466 Epoch: 5 Global Step: 51090 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:14:48,344-Speed 129.03 samples/sec Loss 11.0141 Epoch: 5 Global Step: 51095 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:15:08,400-Speed 127.65 samples/sec Loss 10.9435 Epoch: 5 Global Step: 51100 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:15:28,612-Speed 126.66 samples/sec Loss 11.2590 Epoch: 5 Global Step: 51105 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:15:48,377-Speed 129.53 samples/sec Loss 10.8386 Epoch: 5 Global Step: 51110 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:16:08,417-Speed 127.75 samples/sec Loss 11.4043 Epoch: 5 Global Step: 51115 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:16:28,225-Speed 129.25 samples/sec Loss 11.1990 Epoch: 5 Global Step: 51120 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:16:48,760-Speed 124.68 samples/sec Loss 11.1534 Epoch: 5 Global Step: 51125 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:17:08,747-Speed 128.09 samples/sec Loss 11.1826 Epoch: 5 Global Step: 51130 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:17:28,541-Speed 129.33 samples/sec Loss 11.1691 Epoch: 5 Global Step: 51135 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:17:48,567-Speed 127.84 samples/sec Loss 11.0762 Epoch: 5 Global Step: 51140 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:18:08,378-Speed 129.23 samples/sec Loss 11.2513 Epoch: 5 Global Step: 51145 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:18:28,633-Speed 126.39 samples/sec Loss 11.4774 Epoch: 5 Global Step: 51150 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:18:48,424-Speed 129.36 samples/sec Loss 11.0128 Epoch: 5 Global Step: 51155 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:19:08,417-Speed 128.05 samples/sec Loss 11.1500 Epoch: 5 Global Step: 51160 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:19:28,523-Speed 127.33 samples/sec Loss 11.1504 Epoch: 5 Global Step: 51165 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:19:48,773-Speed 126.42 samples/sec Loss 11.2638 Epoch: 5 Global Step: 51170 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:20:08,589-Speed 129.20 samples/sec Loss 11.0276 Epoch: 5 Global Step: 51175 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:20:28,585-Speed 128.03 samples/sec Loss 11.2996 Epoch: 5 Global Step: 51180 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:20:48,501-Speed 128.55 samples/sec Loss 10.8870 Epoch: 5 Global Step: 51185 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:21:08,365-Speed 128.89 samples/sec Loss 11.2324 Epoch: 5 Global Step: 51190 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:21:28,549-Speed 126.84 samples/sec Loss 11.0205 Epoch: 5 Global Step: 51195 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:21:48,476-Speed 128.48 samples/sec Loss 11.4176 Epoch: 5 Global Step: 51200 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:22:08,271-Speed 129.33 samples/sec Loss 11.1192 Epoch: 5 Global Step: 51205 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:22:28,230-Speed 128.26 samples/sec Loss 11.2439 Epoch: 5 Global Step: 51210 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:22:48,396-Speed 126.96 samples/sec Loss 11.5483 Epoch: 5 Global Step: 51215 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:23:08,391-Speed 128.03 samples/sec Loss 11.1674 Epoch: 5 Global Step: 51220 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:23:28,323-Speed 128.45 samples/sec Loss 11.3157 Epoch: 5 Global Step: 51225 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:23:48,195-Speed 128.83 samples/sec Loss 11.0050 Epoch: 5 Global Step: 51230 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:24:08,245-Speed 127.69 samples/sec Loss 10.9459 Epoch: 5 Global Step: 51235 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:24:28,414-Speed 126.93 samples/sec Loss 10.9348 Epoch: 5 Global Step: 51240 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:24:48,203-Speed 129.37 samples/sec Loss 11.3848 Epoch: 5 Global Step: 51245 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:25:08,152-Speed 128.33 samples/sec Loss 11.0100 Epoch: 5 Global Step: 51250 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:25:28,203-Speed 127.68 samples/sec Loss 10.8264 Epoch: 5 Global Step: 51255 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:25:48,336-Speed 127.16 samples/sec Loss 11.4153 Epoch: 5 Global Step: 51260 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:26:08,390-Speed 127.66 samples/sec Loss 11.1143 Epoch: 5 Global Step: 51265 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:26:28,322-Speed 128.44 samples/sec Loss 11.4102 Epoch: 5 Global Step: 51270 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:26:48,355-Speed 127.79 samples/sec Loss 11.4748 Epoch: 5 Global Step: 51275 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:27:08,320-Speed 128.23 samples/sec Loss 11.1057 Epoch: 5 Global Step: 51280 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:27:28,520-Speed 126.74 samples/sec Loss 11.2094 Epoch: 5 Global Step: 51285 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:27:48,335-Speed 129.20 samples/sec Loss 11.2777 Epoch: 5 Global Step: 51290 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:28:08,407-Speed 127.55 samples/sec Loss 11.3110 Epoch: 5 Global Step: 51295 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:28:28,261-Speed 128.95 samples/sec Loss 11.0951 Epoch: 5 Global Step: 51300 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:28:48,438-Speed 126.88 samples/sec Loss 11.3311 Epoch: 5 Global Step: 51305 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:29:08,477-Speed 127.76 samples/sec Loss 10.9030 Epoch: 5 Global Step: 51310 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:29:28,353-Speed 128.81 samples/sec Loss 11.3666 Epoch: 5 Global Step: 51315 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:29:48,332-Speed 128.14 samples/sec Loss 11.1179 Epoch: 5 Global Step: 51320 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:30:08,207-Speed 128.81 samples/sec Loss 11.2820 Epoch: 5 Global Step: 51325 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:30:28,432-Speed 126.58 samples/sec Loss 10.9900 Epoch: 5 Global Step: 51330 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:30:48,394-Speed 128.25 samples/sec Loss 11.3072 Epoch: 5 Global Step: 51335 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:31:08,413-Speed 127.89 samples/sec Loss 11.2658 Epoch: 5 Global Step: 51340 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:31:28,176-Speed 129.54 samples/sec Loss 11.0451 Epoch: 5 Global Step: 51345 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:31:48,444-Speed 126.31 samples/sec Loss 11.0872 Epoch: 5 Global Step: 51350 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:32:08,292-Speed 128.99 samples/sec Loss 11.2565 Epoch: 5 Global Step: 51355 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:32:28,235-Speed 128.37 samples/sec Loss 11.1879 Epoch: 5 Global Step: 51360 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:32:48,098-Speed 128.89 samples/sec Loss 11.4602 Epoch: 5 Global Step: 51365 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:33:07,961-Speed 128.89 samples/sec Loss 11.4097 Epoch: 5 Global Step: 51370 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:33:28,228-Speed 126.32 samples/sec Loss 11.1923 Epoch: 5 Global Step: 51375 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:33:48,274-Speed 127.71 samples/sec Loss 11.1908 Epoch: 5 Global Step: 51380 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:34:08,007-Speed 129.73 samples/sec Loss 11.0703 Epoch: 5 Global Step: 51385 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:34:27,988-Speed 128.13 samples/sec Loss 11.4456 Epoch: 5 Global Step: 51390 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:34:48,414-Speed 125.34 samples/sec Loss 11.6005 Epoch: 5 Global Step: 51395 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:35:08,258-Speed 129.01 samples/sec Loss 11.3734 Epoch: 5 Global Step: 51400 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:35:28,277-Speed 127.89 samples/sec Loss 11.3201 Epoch: 5 Global Step: 51405 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:35:48,206-Speed 128.46 samples/sec Loss 11.1459 Epoch: 5 Global Step: 51410 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:36:08,175-Speed 128.21 samples/sec Loss 11.4719 Epoch: 5 Global Step: 51415 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:36:28,315-Speed 127.11 samples/sec Loss 10.9004 Epoch: 5 Global Step: 51420 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:36:48,217-Speed 128.64 samples/sec Loss 11.3724 Epoch: 5 Global Step: 51425 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:37:08,124-Speed 128.60 samples/sec Loss 11.6453 Epoch: 5 Global Step: 51430 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:37:28,116-Speed 128.06 samples/sec Loss 11.3774 Epoch: 5 Global Step: 51435 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:37:48,265-Speed 127.06 samples/sec Loss 11.3440 Epoch: 5 Global Step: 51440 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:38:08,202-Speed 128.41 samples/sec Loss 10.9611 Epoch: 5 Global Step: 51445 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:38:28,229-Speed 127.83 samples/sec Loss 11.3195 Epoch: 5 Global Step: 51450 Fp16 Grad Scale: 8192 Required: 237 hours Training: 2021-06-09 08:38:48,026-Speed 129.32 samples/sec Loss 11.3696 Epoch: 5 Global Step: 51455 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:39:08,084-Speed 127.64 samples/sec Loss 10.9181 Epoch: 5 Global Step: 51460 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:39:28,295-Speed 126.67 samples/sec Loss 11.0547 Epoch: 5 Global Step: 51465 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:39:48,134-Speed 129.04 samples/sec Loss 11.3327 Epoch: 5 Global Step: 51470 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:40:07,988-Speed 128.95 samples/sec Loss 11.6501 Epoch: 5 Global Step: 51475 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:40:27,929-Speed 128.39 samples/sec Loss 11.2252 Epoch: 5 Global Step: 51480 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:40:48,084-Speed 127.03 samples/sec Loss 11.3508 Epoch: 5 Global Step: 51485 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:41:08,068-Speed 128.11 samples/sec Loss 10.9880 Epoch: 5 Global Step: 51490 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:41:27,908-Speed 129.04 samples/sec Loss 10.8682 Epoch: 5 Global Step: 51495 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:41:47,991-Speed 127.48 samples/sec Loss 11.4074 Epoch: 5 Global Step: 51500 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:42:07,845-Speed 128.95 samples/sec Loss 11.4371 Epoch: 5 Global Step: 51505 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:42:28,051-Speed 126.70 samples/sec Loss 11.4794 Epoch: 5 Global Step: 51510 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:42:47,789-Speed 129.70 samples/sec Loss 11.2639 Epoch: 5 Global Step: 51515 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:43:07,819-Speed 127.81 samples/sec Loss 11.0980 Epoch: 5 Global Step: 51520 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:43:27,699-Speed 128.78 samples/sec Loss 11.4217 Epoch: 5 Global Step: 51525 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:43:48,052-Speed 125.79 samples/sec Loss 11.2995 Epoch: 5 Global Step: 51530 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:44:07,892-Speed 129.04 samples/sec Loss 11.3926 Epoch: 5 Global Step: 51535 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:44:27,830-Speed 128.40 samples/sec Loss 11.1143 Epoch: 5 Global Step: 51540 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:44:47,806-Speed 128.16 samples/sec Loss 11.3981 Epoch: 5 Global Step: 51545 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:45:07,721-Speed 128.56 samples/sec Loss 11.2322 Epoch: 5 Global Step: 51550 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:45:28,007-Speed 126.20 samples/sec Loss 11.2574 Epoch: 5 Global Step: 51555 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:45:48,031-Speed 127.85 samples/sec Loss 11.1554 Epoch: 5 Global Step: 51560 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:46:07,829-Speed 129.31 samples/sec Loss 11.1731 Epoch: 5 Global Step: 51565 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:46:27,873-Speed 127.72 samples/sec Loss 11.2598 Epoch: 5 Global Step: 51570 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:46:48,000-Speed 127.20 samples/sec Loss 11.0996 Epoch: 5 Global Step: 51575 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:47:07,929-Speed 128.46 samples/sec Loss 11.4716 Epoch: 5 Global Step: 51580 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:47:27,842-Speed 128.57 samples/sec Loss 10.8013 Epoch: 5 Global Step: 51585 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:47:47,691-Speed 128.98 samples/sec Loss 11.4167 Epoch: 5 Global Step: 51590 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:48:07,716-Speed 127.85 samples/sec Loss 11.4243 Epoch: 5 Global Step: 51595 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:48:27,910-Speed 126.78 samples/sec Loss 11.4385 Epoch: 5 Global Step: 51600 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:48:48,947-Speed 121.69 samples/sec Loss 11.2504 Epoch: 5 Global Step: 51605 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:49:08,841-Speed 128.69 samples/sec Loss 11.3439 Epoch: 5 Global Step: 51610 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:49:28,856-Speed 127.91 samples/sec Loss 11.4475 Epoch: 5 Global Step: 51615 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:49:49,596-Speed 123.44 samples/sec Loss 10.9196 Epoch: 5 Global Step: 51620 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:50:09,654-Speed 127.64 samples/sec Loss 11.0549 Epoch: 5 Global Step: 51625 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:50:29,518-Speed 128.88 samples/sec Loss 11.1476 Epoch: 5 Global Step: 51630 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:50:49,459-Speed 128.39 samples/sec Loss 11.3838 Epoch: 5 Global Step: 51635 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:51:09,397-Speed 128.40 samples/sec Loss 11.1263 Epoch: 5 Global Step: 51640 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:51:29,656-Speed 126.37 samples/sec Loss 11.1832 Epoch: 5 Global Step: 51645 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:51:49,388-Speed 129.74 samples/sec Loss 11.0678 Epoch: 5 Global Step: 51650 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:52:09,405-Speed 127.90 samples/sec Loss 10.9281 Epoch: 5 Global Step: 51655 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:52:29,257-Speed 128.96 samples/sec Loss 11.2699 Epoch: 5 Global Step: 51660 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:52:49,374-Speed 127.26 samples/sec Loss 11.1248 Epoch: 5 Global Step: 51665 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:53:09,335-Speed 128.26 samples/sec Loss 11.3642 Epoch: 5 Global Step: 51670 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:53:29,194-Speed 128.91 samples/sec Loss 11.2819 Epoch: 5 Global Step: 51675 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:53:49,221-Speed 127.84 samples/sec Loss 11.2942 Epoch: 5 Global Step: 51680 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:54:09,126-Speed 128.62 samples/sec Loss 11.0868 Epoch: 5 Global Step: 51685 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:54:29,234-Speed 127.32 samples/sec Loss 11.1033 Epoch: 5 Global Step: 51690 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:54:49,125-Speed 128.71 samples/sec Loss 11.6748 Epoch: 5 Global Step: 51695 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:55:09,120-Speed 128.04 samples/sec Loss 11.1343 Epoch: 5 Global Step: 51700 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:55:28,988-Speed 128.86 samples/sec Loss 11.4980 Epoch: 5 Global Step: 51705 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:55:49,211-Speed 126.59 samples/sec Loss 11.0952 Epoch: 5 Global Step: 51710 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:56:09,059-Speed 128.99 samples/sec Loss 11.3707 Epoch: 5 Global Step: 51715 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:56:28,957-Speed 128.66 samples/sec Loss 11.2641 Epoch: 5 Global Step: 51720 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:56:49,109-Speed 127.04 samples/sec Loss 11.4644 Epoch: 5 Global Step: 51725 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:57:09,053-Speed 128.37 samples/sec Loss 11.0965 Epoch: 5 Global Step: 51730 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:57:29,262-Speed 126.68 samples/sec Loss 10.8751 Epoch: 5 Global Step: 51735 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:57:49,295-Speed 127.79 samples/sec Loss 10.9592 Epoch: 5 Global Step: 51740 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:58:09,128-Speed 129.08 samples/sec Loss 11.0929 Epoch: 5 Global Step: 51745 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:58:29,205-Speed 127.52 samples/sec Loss 11.2600 Epoch: 5 Global Step: 51750 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:58:49,336-Speed 127.18 samples/sec Loss 11.1810 Epoch: 5 Global Step: 51755 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:59:09,112-Speed 129.46 samples/sec Loss 11.2229 Epoch: 5 Global Step: 51760 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:59:29,115-Speed 127.99 samples/sec Loss 11.1900 Epoch: 5 Global Step: 51765 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 08:59:49,022-Speed 128.61 samples/sec Loss 11.3916 Epoch: 5 Global Step: 51770 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:00:09,048-Speed 127.84 samples/sec Loss 11.1134 Epoch: 5 Global Step: 51775 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:00:29,145-Speed 127.39 samples/sec Loss 11.3488 Epoch: 5 Global Step: 51780 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:00:49,103-Speed 128.27 samples/sec Loss 11.2537 Epoch: 5 Global Step: 51785 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:01:09,061-Speed 128.28 samples/sec Loss 10.8544 Epoch: 5 Global Step: 51790 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:01:29,127-Speed 127.58 samples/sec Loss 11.2301 Epoch: 5 Global Step: 51795 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:01:49,261-Speed 127.16 samples/sec Loss 11.2791 Epoch: 5 Global Step: 51800 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:02:09,302-Speed 127.74 samples/sec Loss 11.3489 Epoch: 5 Global Step: 51805 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:02:29,021-Speed 129.83 samples/sec Loss 11.0590 Epoch: 5 Global Step: 51810 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:02:48,969-Speed 128.34 samples/sec Loss 11.2561 Epoch: 5 Global Step: 51815 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:03:08,954-Speed 128.10 samples/sec Loss 11.4116 Epoch: 5 Global Step: 51820 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:03:29,180-Speed 126.58 samples/sec Loss 11.5249 Epoch: 5 Global Step: 51825 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:03:49,017-Speed 129.06 samples/sec Loss 11.2935 Epoch: 5 Global Step: 51830 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:04:08,908-Speed 128.71 samples/sec Loss 11.3810 Epoch: 5 Global Step: 51835 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:04:28,944-Speed 127.78 samples/sec Loss 11.1622 Epoch: 5 Global Step: 51840 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:04:49,174-Speed 126.55 samples/sec Loss 11.3126 Epoch: 5 Global Step: 51845 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:05:09,195-Speed 127.87 samples/sec Loss 11.1327 Epoch: 5 Global Step: 51850 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:05:29,090-Speed 128.68 samples/sec Loss 11.1203 Epoch: 5 Global Step: 51855 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:05:48,958-Speed 128.86 samples/sec Loss 11.0844 Epoch: 5 Global Step: 51860 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:06:08,845-Speed 128.73 samples/sec Loss 11.2012 Epoch: 5 Global Step: 51865 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:06:29,045-Speed 126.74 samples/sec Loss 11.0892 Epoch: 5 Global Step: 51870 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:06:48,788-Speed 129.67 samples/sec Loss 11.3049 Epoch: 5 Global Step: 51875 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:07:08,944-Speed 127.02 samples/sec Loss 11.6297 Epoch: 5 Global Step: 51880 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:07:28,909-Speed 128.23 samples/sec Loss 11.4902 Epoch: 5 Global Step: 51885 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:07:49,111-Speed 126.72 samples/sec Loss 11.2936 Epoch: 5 Global Step: 51890 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:08:09,141-Speed 127.82 samples/sec Loss 11.4762 Epoch: 5 Global Step: 51895 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:08:29,065-Speed 128.50 samples/sec Loss 11.4079 Epoch: 5 Global Step: 51900 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:08:48,980-Speed 128.55 samples/sec Loss 11.3185 Epoch: 5 Global Step: 51905 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:09:08,832-Speed 128.97 samples/sec Loss 11.1138 Epoch: 5 Global Step: 51910 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:09:28,917-Speed 127.46 samples/sec Loss 11.2860 Epoch: 5 Global Step: 51915 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:09:48,690-Speed 129.48 samples/sec Loss 11.2709 Epoch: 5 Global Step: 51920 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:10:08,740-Speed 127.68 samples/sec Loss 11.2417 Epoch: 5 Global Step: 51925 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:10:28,749-Speed 127.95 samples/sec Loss 11.1710 Epoch: 5 Global Step: 51930 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:10:48,854-Speed 127.33 samples/sec Loss 11.7445 Epoch: 5 Global Step: 51935 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:11:08,891-Speed 127.77 samples/sec Loss 10.9971 Epoch: 5 Global Step: 51940 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:11:28,727-Speed 129.06 samples/sec Loss 11.3718 Epoch: 5 Global Step: 51945 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:11:48,754-Speed 127.84 samples/sec Loss 11.3128 Epoch: 5 Global Step: 51950 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:12:08,588-Speed 129.08 samples/sec Loss 11.3191 Epoch: 5 Global Step: 51955 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:12:28,694-Speed 127.33 samples/sec Loss 11.5942 Epoch: 5 Global Step: 51960 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:12:48,725-Speed 127.81 samples/sec Loss 11.6206 Epoch: 5 Global Step: 51965 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:13:08,628-Speed 128.63 samples/sec Loss 11.3714 Epoch: 5 Global Step: 51970 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:13:28,503-Speed 128.81 samples/sec Loss 11.0761 Epoch: 5 Global Step: 51975 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:13:48,988-Speed 124.97 samples/sec Loss 10.9452 Epoch: 5 Global Step: 51980 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:14:08,897-Speed 128.59 samples/sec Loss 11.0862 Epoch: 5 Global Step: 51985 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:14:28,926-Speed 127.82 samples/sec Loss 11.0211 Epoch: 5 Global Step: 51990 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:14:48,760-Speed 129.08 samples/sec Loss 10.9748 Epoch: 5 Global Step: 51995 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:15:08,578-Speed 129.18 samples/sec Loss 11.2558 Epoch: 5 Global Step: 52000 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:15:28,774-Speed 126.77 samples/sec Loss 11.3954 Epoch: 5 Global Step: 52005 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:15:48,809-Speed 127.78 samples/sec Loss 11.0793 Epoch: 5 Global Step: 52010 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:16:08,667-Speed 128.92 samples/sec Loss 11.0726 Epoch: 5 Global Step: 52015 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:16:28,703-Speed 127.77 samples/sec Loss 11.0263 Epoch: 5 Global Step: 52020 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:16:50,034-Speed 120.02 samples/sec Loss 11.2804 Epoch: 5 Global Step: 52025 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:17:10,232-Speed 126.75 samples/sec Loss 11.1104 Epoch: 5 Global Step: 52030 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:17:30,083-Speed 128.97 samples/sec Loss 11.2230 Epoch: 5 Global Step: 52035 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:17:49,995-Speed 128.57 samples/sec Loss 11.2220 Epoch: 5 Global Step: 52040 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:18:10,048-Speed 127.67 samples/sec Loss 11.3572 Epoch: 5 Global Step: 52045 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:18:30,163-Speed 127.27 samples/sec Loss 11.3252 Epoch: 5 Global Step: 52050 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:18:50,076-Speed 128.57 samples/sec Loss 11.2567 Epoch: 5 Global Step: 52055 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:19:10,076-Speed 128.00 samples/sec Loss 11.0684 Epoch: 5 Global Step: 52060 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:19:29,923-Speed 129.00 samples/sec Loss 11.0836 Epoch: 5 Global Step: 52065 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:19:50,481-Speed 124.53 samples/sec Loss 11.2518 Epoch: 5 Global Step: 52070 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:20:10,396-Speed 128.55 samples/sec Loss 11.2981 Epoch: 5 Global Step: 52075 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:20:30,199-Speed 129.28 samples/sec Loss 11.1309 Epoch: 5 Global Step: 52080 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:20:50,131-Speed 128.44 samples/sec Loss 11.4883 Epoch: 5 Global Step: 52085 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:21:10,045-Speed 128.56 samples/sec Loss 11.3527 Epoch: 5 Global Step: 52090 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:21:30,250-Speed 126.71 samples/sec Loss 11.0605 Epoch: 5 Global Step: 52095 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:21:50,310-Speed 127.62 samples/sec Loss 11.2823 Epoch: 5 Global Step: 52100 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:22:10,144-Speed 129.08 samples/sec Loss 11.3684 Epoch: 5 Global Step: 52105 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:22:29,997-Speed 128.95 samples/sec Loss 11.3922 Epoch: 5 Global Step: 52110 Fp16 Grad Scale: 8192 Required: 236 hours Training: 2021-06-09 09:22:50,362-Speed 125.71 samples/sec Loss 11.3942 Epoch: 5 Global Step: 52115 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:23:10,169-Speed 129.26 samples/sec Loss 11.3295 Epoch: 5 Global Step: 52120 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:23:30,365-Speed 126.76 samples/sec Loss 11.1845 Epoch: 5 Global Step: 52125 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:23:50,245-Speed 128.78 samples/sec Loss 11.3013 Epoch: 5 Global Step: 52130 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:24:10,111-Speed 128.87 samples/sec Loss 11.4284 Epoch: 5 Global Step: 52135 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:24:30,293-Speed 126.85 samples/sec Loss 11.4618 Epoch: 5 Global Step: 52140 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:24:50,228-Speed 128.42 samples/sec Loss 11.0582 Epoch: 5 Global Step: 52145 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:25:10,070-Speed 129.03 samples/sec Loss 11.2677 Epoch: 5 Global Step: 52150 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:25:30,195-Speed 127.21 samples/sec Loss 11.2031 Epoch: 5 Global Step: 52155 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:25:50,761-Speed 124.48 samples/sec Loss 11.4310 Epoch: 5 Global Step: 52160 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:26:10,779-Speed 127.89 samples/sec Loss 11.5642 Epoch: 5 Global Step: 52165 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:26:30,867-Speed 127.44 samples/sec Loss 10.9280 Epoch: 5 Global Step: 52170 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:26:50,750-Speed 128.76 samples/sec Loss 10.9088 Epoch: 5 Global Step: 52175 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:27:10,641-Speed 128.71 samples/sec Loss 11.0148 Epoch: 5 Global Step: 52180 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:27:30,769-Speed 127.19 samples/sec Loss 11.2410 Epoch: 5 Global Step: 52185 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:27:50,612-Speed 129.02 samples/sec Loss 11.3526 Epoch: 5 Global Step: 52190 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:28:10,582-Speed 128.20 samples/sec Loss 11.1804 Epoch: 5 Global Step: 52195 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:28:30,556-Speed 128.17 samples/sec Loss 10.8863 Epoch: 5 Global Step: 52200 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:28:50,625-Speed 127.57 samples/sec Loss 11.2937 Epoch: 5 Global Step: 52205 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:29:10,623-Speed 128.02 samples/sec Loss 10.8274 Epoch: 5 Global Step: 52210 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:29:30,524-Speed 128.64 samples/sec Loss 11.5014 Epoch: 5 Global Step: 52215 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:29:50,471-Speed 128.35 samples/sec Loss 11.1371 Epoch: 5 Global Step: 52220 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:30:10,271-Speed 129.30 samples/sec Loss 11.4843 Epoch: 5 Global Step: 52225 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:30:30,373-Speed 127.35 samples/sec Loss 11.1952 Epoch: 5 Global Step: 52230 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:30:50,206-Speed 129.09 samples/sec Loss 11.2893 Epoch: 5 Global Step: 52235 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:31:10,238-Speed 127.80 samples/sec Loss 11.3841 Epoch: 5 Global Step: 52240 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:31:30,367-Speed 127.19 samples/sec Loss 11.1849 Epoch: 5 Global Step: 52245 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:31:50,561-Speed 126.78 samples/sec Loss 11.0349 Epoch: 5 Global Step: 52250 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:32:10,350-Speed 129.37 samples/sec Loss 10.9086 Epoch: 5 Global Step: 52255 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:32:30,244-Speed 128.69 samples/sec Loss 11.2357 Epoch: 5 Global Step: 52260 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:32:50,221-Speed 128.16 samples/sec Loss 11.4067 Epoch: 5 Global Step: 52265 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:33:09,976-Speed 129.59 samples/sec Loss 11.7101 Epoch: 5 Global Step: 52270 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:33:30,870-Speed 122.53 samples/sec Loss 11.1218 Epoch: 5 Global Step: 52275 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:33:50,897-Speed 127.84 samples/sec Loss 11.1307 Epoch: 5 Global Step: 52280 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:34:10,759-Speed 128.89 samples/sec Loss 10.9496 Epoch: 5 Global Step: 52285 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:34:30,893-Speed 127.15 samples/sec Loss 11.3790 Epoch: 5 Global Step: 52290 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:34:51,078-Speed 126.83 samples/sec Loss 10.9719 Epoch: 5 Global Step: 52295 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:35:10,827-Speed 129.63 samples/sec Loss 11.2549 Epoch: 5 Global Step: 52300 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:35:30,786-Speed 128.27 samples/sec Loss 11.1598 Epoch: 5 Global Step: 52305 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:35:50,612-Speed 129.13 samples/sec Loss 10.8473 Epoch: 5 Global Step: 52310 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:36:10,646-Speed 127.79 samples/sec Loss 11.1864 Epoch: 5 Global Step: 52315 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:36:30,772-Speed 127.21 samples/sec Loss 11.1080 Epoch: 5 Global Step: 52320 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:36:50,707-Speed 128.42 samples/sec Loss 11.2687 Epoch: 5 Global Step: 52325 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:37:10,421-Speed 129.86 samples/sec Loss 11.3672 Epoch: 5 Global Step: 52330 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:37:30,353-Speed 128.44 samples/sec Loss 11.1948 Epoch: 5 Global Step: 52335 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:37:50,452-Speed 127.38 samples/sec Loss 11.6279 Epoch: 5 Global Step: 52340 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:38:10,445-Speed 128.05 samples/sec Loss 11.4269 Epoch: 5 Global Step: 52345 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:38:30,251-Speed 129.27 samples/sec Loss 11.5999 Epoch: 5 Global Step: 52350 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:38:50,327-Speed 127.52 samples/sec Loss 11.1528 Epoch: 5 Global Step: 52355 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:39:10,197-Speed 128.84 samples/sec Loss 11.0599 Epoch: 5 Global Step: 52360 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:39:30,399-Speed 126.73 samples/sec Loss 11.2102 Epoch: 5 Global Step: 52365 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:39:50,222-Speed 129.15 samples/sec Loss 11.2378 Epoch: 5 Global Step: 52370 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:40:10,164-Speed 128.38 samples/sec Loss 11.0469 Epoch: 5 Global Step: 52375 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:40:30,093-Speed 128.46 samples/sec Loss 11.1621 Epoch: 5 Global Step: 52380 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:40:50,360-Speed 126.32 samples/sec Loss 11.4518 Epoch: 5 Global Step: 52385 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:41:10,277-Speed 128.54 samples/sec Loss 11.4100 Epoch: 5 Global Step: 52390 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:41:30,154-Speed 128.80 samples/sec Loss 11.3156 Epoch: 5 Global Step: 52395 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:41:50,163-Speed 127.95 samples/sec Loss 10.9416 Epoch: 5 Global Step: 52400 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:42:10,015-Speed 128.96 samples/sec Loss 11.3111 Epoch: 5 Global Step: 52405 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:42:30,230-Speed 126.65 samples/sec Loss 11.1597 Epoch: 5 Global Step: 52410 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:42:50,180-Speed 128.32 samples/sec Loss 11.0347 Epoch: 5 Global Step: 52415 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:43:10,106-Speed 128.48 samples/sec Loss 11.0230 Epoch: 5 Global Step: 52420 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:43:29,998-Speed 128.71 samples/sec Loss 11.2143 Epoch: 5 Global Step: 52425 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:43:50,199-Speed 126.73 samples/sec Loss 11.4032 Epoch: 5 Global Step: 52430 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:44:10,049-Speed 128.98 samples/sec Loss 10.9866 Epoch: 5 Global Step: 52435 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:44:30,134-Speed 127.47 samples/sec Loss 11.4902 Epoch: 5 Global Step: 52440 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:44:49,971-Speed 129.06 samples/sec Loss 11.2331 Epoch: 5 Global Step: 52445 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:45:09,877-Speed 128.62 samples/sec Loss 11.1860 Epoch: 5 Global Step: 52450 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:45:30,013-Speed 127.14 samples/sec Loss 11.1172 Epoch: 5 Global Step: 52455 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:45:50,050-Speed 127.77 samples/sec Loss 11.0793 Epoch: 5 Global Step: 52460 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:46:09,890-Speed 129.04 samples/sec Loss 11.2744 Epoch: 5 Global Step: 52465 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:46:29,818-Speed 128.47 samples/sec Loss 11.2815 Epoch: 5 Global Step: 52470 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:46:49,957-Speed 127.12 samples/sec Loss 11.6542 Epoch: 5 Global Step: 52475 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:47:10,024-Speed 127.58 samples/sec Loss 11.1348 Epoch: 5 Global Step: 52480 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:47:29,882-Speed 128.92 samples/sec Loss 11.0930 Epoch: 5 Global Step: 52485 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:47:49,655-Speed 129.47 samples/sec Loss 11.3863 Epoch: 5 Global Step: 52490 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:48:09,639-Speed 128.11 samples/sec Loss 11.1551 Epoch: 5 Global Step: 52495 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:48:29,722-Speed 127.48 samples/sec Loss 11.7223 Epoch: 5 Global Step: 52500 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:48:49,637-Speed 128.55 samples/sec Loss 11.1003 Epoch: 5 Global Step: 52505 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:49:09,510-Speed 128.83 samples/sec Loss 10.9547 Epoch: 5 Global Step: 52510 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:49:29,381-Speed 128.83 samples/sec Loss 11.4869 Epoch: 5 Global Step: 52515 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:49:49,433-Speed 127.68 samples/sec Loss 11.2437 Epoch: 5 Global Step: 52520 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:50:09,581-Speed 127.06 samples/sec Loss 10.8309 Epoch: 5 Global Step: 52525 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:50:29,331-Speed 129.62 samples/sec Loss 11.4522 Epoch: 5 Global Step: 52530 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:50:49,383-Speed 127.67 samples/sec Loss 11.0337 Epoch: 5 Global Step: 52535 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:51:09,189-Speed 129.26 samples/sec Loss 10.9096 Epoch: 5 Global Step: 52540 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:51:29,460-Speed 126.30 samples/sec Loss 11.1684 Epoch: 5 Global Step: 52545 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:51:49,297-Speed 129.06 samples/sec Loss 11.2120 Epoch: 5 Global Step: 52550 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:52:09,305-Speed 127.95 samples/sec Loss 11.3791 Epoch: 5 Global Step: 52555 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:52:29,161-Speed 128.94 samples/sec Loss 11.1887 Epoch: 5 Global Step: 52560 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:52:49,367-Speed 126.70 samples/sec Loss 11.1227 Epoch: 5 Global Step: 52565 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:53:09,271-Speed 128.62 samples/sec Loss 11.6755 Epoch: 5 Global Step: 52570 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:53:29,048-Speed 129.45 samples/sec Loss 11.3195 Epoch: 5 Global Step: 52575 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:53:48,981-Speed 128.43 samples/sec Loss 10.6944 Epoch: 5 Global Step: 52580 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:54:08,801-Speed 129.17 samples/sec Loss 11.3651 Epoch: 5 Global Step: 52585 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:54:29,009-Speed 126.69 samples/sec Loss 11.2629 Epoch: 5 Global Step: 52590 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:54:48,959-Speed 128.33 samples/sec Loss 11.2029 Epoch: 5 Global Step: 52595 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:55:08,877-Speed 128.53 samples/sec Loss 11.2169 Epoch: 5 Global Step: 52600 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:55:28,700-Speed 129.14 samples/sec Loss 11.4422 Epoch: 5 Global Step: 52605 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:55:49,051-Speed 125.80 samples/sec Loss 11.2836 Epoch: 5 Global Step: 52610 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:56:08,785-Speed 129.73 samples/sec Loss 11.2871 Epoch: 5 Global Step: 52615 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:56:28,795-Speed 127.94 samples/sec Loss 10.9779 Epoch: 5 Global Step: 52620 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:56:48,616-Speed 129.16 samples/sec Loss 11.2455 Epoch: 5 Global Step: 52625 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:57:08,604-Speed 128.09 samples/sec Loss 11.2650 Epoch: 5 Global Step: 52630 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:57:28,868-Speed 126.34 samples/sec Loss 11.4469 Epoch: 5 Global Step: 52635 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:57:49,408-Speed 124.64 samples/sec Loss 11.5898 Epoch: 5 Global Step: 52640 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:58:09,261-Speed 128.96 samples/sec Loss 11.3056 Epoch: 5 Global Step: 52645 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:58:29,353-Speed 127.41 samples/sec Loss 11.3379 Epoch: 5 Global Step: 52650 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:58:49,447-Speed 127.41 samples/sec Loss 11.3691 Epoch: 5 Global Step: 52655 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:59:09,482-Speed 127.78 samples/sec Loss 11.4618 Epoch: 5 Global Step: 52660 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:59:29,224-Speed 129.68 samples/sec Loss 11.1097 Epoch: 5 Global Step: 52665 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 09:59:49,220-Speed 128.04 samples/sec Loss 11.1437 Epoch: 5 Global Step: 52670 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:00:09,350-Speed 127.18 samples/sec Loss 11.2279 Epoch: 5 Global Step: 52675 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:00:29,601-Speed 126.42 samples/sec Loss 10.9547 Epoch: 5 Global Step: 52680 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:00:49,464-Speed 128.89 samples/sec Loss 11.1072 Epoch: 5 Global Step: 52685 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:01:09,247-Speed 129.41 samples/sec Loss 11.3618 Epoch: 5 Global Step: 52690 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:01:29,311-Speed 127.60 samples/sec Loss 11.1241 Epoch: 5 Global Step: 52695 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:01:49,464-Speed 127.03 samples/sec Loss 11.5049 Epoch: 5 Global Step: 52700 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:02:09,478-Speed 127.92 samples/sec Loss 11.1130 Epoch: 5 Global Step: 52705 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:02:29,382-Speed 128.62 samples/sec Loss 11.4449 Epoch: 5 Global Step: 52710 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:02:49,454-Speed 127.55 samples/sec Loss 11.2339 Epoch: 5 Global Step: 52715 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:03:09,295-Speed 129.03 samples/sec Loss 10.8520 Epoch: 5 Global Step: 52720 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:03:29,425-Speed 127.18 samples/sec Loss 11.2248 Epoch: 5 Global Step: 52725 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:03:49,236-Speed 129.23 samples/sec Loss 11.0029 Epoch: 5 Global Step: 52730 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:04:09,320-Speed 127.47 samples/sec Loss 11.3054 Epoch: 5 Global Step: 52735 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:04:29,162-Speed 129.02 samples/sec Loss 11.2229 Epoch: 5 Global Step: 52740 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:04:49,338-Speed 126.89 samples/sec Loss 11.2231 Epoch: 5 Global Step: 52745 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:05:09,316-Speed 128.15 samples/sec Loss 11.1711 Epoch: 5 Global Step: 52750 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:05:29,166-Speed 128.97 samples/sec Loss 11.0319 Epoch: 5 Global Step: 52755 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:05:49,275-Speed 127.32 samples/sec Loss 11.1093 Epoch: 5 Global Step: 52760 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:06:09,119-Speed 129.01 samples/sec Loss 11.4286 Epoch: 5 Global Step: 52765 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:06:29,327-Speed 126.69 samples/sec Loss 11.1595 Epoch: 5 Global Step: 52770 Fp16 Grad Scale: 8192 Required: 235 hours Training: 2021-06-09 10:06:49,332-Speed 127.97 samples/sec Loss 11.5195 Epoch: 5 Global Step: 52775 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:07:09,179-Speed 129.00 samples/sec Loss 11.5056 Epoch: 5 Global Step: 52780 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:07:29,039-Speed 128.91 samples/sec Loss 11.3574 Epoch: 5 Global Step: 52785 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:07:49,345-Speed 126.07 samples/sec Loss 11.2861 Epoch: 5 Global Step: 52790 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:08:09,939-Speed 124.32 samples/sec Loss 11.2150 Epoch: 5 Global Step: 52795 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:08:29,908-Speed 128.20 samples/sec Loss 11.3824 Epoch: 5 Global Step: 52800 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:08:49,835-Speed 128.48 samples/sec Loss 11.0918 Epoch: 5 Global Step: 52805 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:09:09,601-Speed 129.52 samples/sec Loss 11.0331 Epoch: 5 Global Step: 52810 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:09:29,867-Speed 126.33 samples/sec Loss 11.2286 Epoch: 5 Global Step: 52815 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:09:49,790-Speed 128.50 samples/sec Loss 10.8047 Epoch: 5 Global Step: 52820 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:10:09,669-Speed 128.78 samples/sec Loss 11.0108 Epoch: 5 Global Step: 52825 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:10:29,674-Speed 127.97 samples/sec Loss 11.2907 Epoch: 5 Global Step: 52830 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:10:49,881-Speed 126.70 samples/sec Loss 11.3347 Epoch: 5 Global Step: 52835 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:11:09,748-Speed 128.87 samples/sec Loss 11.2713 Epoch: 5 Global Step: 52840 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:11:29,645-Speed 128.67 samples/sec Loss 11.1852 Epoch: 5 Global Step: 52845 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:11:49,497-Speed 128.96 samples/sec Loss 11.4289 Epoch: 5 Global Step: 52850 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:12:09,855-Speed 125.75 samples/sec Loss 11.2424 Epoch: 5 Global Step: 52855 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:12:30,025-Speed 126.93 samples/sec Loss 11.3678 Epoch: 5 Global Step: 52860 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:12:50,024-Speed 128.01 samples/sec Loss 11.5204 Epoch: 5 Global Step: 52865 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:13:09,878-Speed 128.95 samples/sec Loss 11.0767 Epoch: 5 Global Step: 52870 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:13:29,981-Speed 127.35 samples/sec Loss 10.9329 Epoch: 5 Global Step: 52875 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:13:50,109-Speed 127.19 samples/sec Loss 11.2589 Epoch: 5 Global Step: 52880 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:14:10,134-Speed 127.85 samples/sec Loss 10.9440 Epoch: 5 Global Step: 52885 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:14:29,978-Speed 129.01 samples/sec Loss 11.1262 Epoch: 5 Global Step: 52890 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:14:49,872-Speed 128.69 samples/sec Loss 10.9585 Epoch: 5 Global Step: 52895 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:15:09,934-Speed 127.61 samples/sec Loss 11.0296 Epoch: 5 Global Step: 52900 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:15:30,141-Speed 126.70 samples/sec Loss 11.4862 Epoch: 5 Global Step: 52905 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:15:49,931-Speed 129.36 samples/sec Loss 11.4042 Epoch: 5 Global Step: 52910 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:16:09,988-Speed 127.64 samples/sec Loss 11.1124 Epoch: 5 Global Step: 52915 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:16:29,785-Speed 129.32 samples/sec Loss 11.2155 Epoch: 5 Global Step: 52920 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:16:49,964-Speed 126.87 samples/sec Loss 11.4426 Epoch: 5 Global Step: 52925 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:17:09,939-Speed 128.17 samples/sec Loss 11.1322 Epoch: 5 Global Step: 52930 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:17:29,814-Speed 128.82 samples/sec Loss 10.7819 Epoch: 5 Global Step: 52935 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:17:49,878-Speed 127.60 samples/sec Loss 11.6043 Epoch: 5 Global Step: 52940 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:18:09,733-Speed 128.94 samples/sec Loss 11.2399 Epoch: 5 Global Step: 52945 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:18:29,843-Speed 127.31 samples/sec Loss 11.2347 Epoch: 5 Global Step: 52950 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:18:49,934-Speed 127.43 samples/sec Loss 11.2736 Epoch: 5 Global Step: 52955 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:19:09,825-Speed 128.71 samples/sec Loss 11.2088 Epoch: 5 Global Step: 52960 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:19:29,741-Speed 128.54 samples/sec Loss 11.1982 Epoch: 5 Global Step: 52965 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:19:49,935-Speed 126.78 samples/sec Loss 11.4592 Epoch: 5 Global Step: 52970 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:20:09,802-Speed 128.87 samples/sec Loss 11.0272 Epoch: 5 Global Step: 52975 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:20:29,743-Speed 128.38 samples/sec Loss 11.2166 Epoch: 5 Global Step: 52980 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:20:49,667-Speed 128.50 samples/sec Loss 10.9380 Epoch: 5 Global Step: 52985 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:21:09,491-Speed 129.14 samples/sec Loss 11.1269 Epoch: 5 Global Step: 52990 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:21:29,629-Speed 127.13 samples/sec Loss 11.1148 Epoch: 5 Global Step: 52995 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:21:49,658-Speed 127.82 samples/sec Loss 11.0412 Epoch: 5 Global Step: 53000 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:22:09,493-Speed 129.07 samples/sec Loss 11.1952 Epoch: 5 Global Step: 53005 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:22:29,585-Speed 127.42 samples/sec Loss 10.8815 Epoch: 5 Global Step: 53010 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:22:49,609-Speed 127.85 samples/sec Loss 11.1174 Epoch: 5 Global Step: 53015 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:23:09,493-Speed 128.76 samples/sec Loss 11.5159 Epoch: 5 Global Step: 53020 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:23:29,542-Speed 127.69 samples/sec Loss 11.3194 Epoch: 5 Global Step: 53025 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:23:49,388-Speed 129.00 samples/sec Loss 11.2955 Epoch: 5 Global Step: 53030 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:24:09,339-Speed 128.32 samples/sec Loss 11.2074 Epoch: 5 Global Step: 53035 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:24:29,303-Speed 128.23 samples/sec Loss 11.4593 Epoch: 5 Global Step: 53040 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:24:49,259-Speed 128.28 samples/sec Loss 11.1961 Epoch: 5 Global Step: 53045 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:25:09,175-Speed 128.55 samples/sec Loss 11.4485 Epoch: 5 Global Step: 53050 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:25:29,336-Speed 126.99 samples/sec Loss 11.3985 Epoch: 5 Global Step: 53055 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:25:49,475-Speed 127.12 samples/sec Loss 11.2504 Epoch: 5 Global Step: 53060 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:26:09,564-Speed 127.44 samples/sec Loss 11.2283 Epoch: 5 Global Step: 53065 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:26:29,343-Speed 129.44 samples/sec Loss 10.9851 Epoch: 5 Global Step: 53070 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:26:49,221-Speed 128.80 samples/sec Loss 11.5056 Epoch: 5 Global Step: 53075 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:27:09,227-Speed 127.97 samples/sec Loss 11.0704 Epoch: 5 Global Step: 53080 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:27:29,439-Speed 126.67 samples/sec Loss 11.0857 Epoch: 5 Global Step: 53085 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:27:49,333-Speed 128.69 samples/sec Loss 11.2876 Epoch: 5 Global Step: 53090 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:28:09,102-Speed 129.50 samples/sec Loss 11.0574 Epoch: 5 Global Step: 53095 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:28:28,997-Speed 128.68 samples/sec Loss 11.1529 Epoch: 5 Global Step: 53100 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:28:49,066-Speed 127.56 samples/sec Loss 11.1601 Epoch: 5 Global Step: 53105 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:29:09,139-Speed 127.55 samples/sec Loss 11.2861 Epoch: 5 Global Step: 53110 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:29:28,898-Speed 129.57 samples/sec Loss 11.5357 Epoch: 5 Global Step: 53115 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:29:48,913-Speed 127.91 samples/sec Loss 11.1704 Epoch: 5 Global Step: 53120 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:30:08,744-Speed 129.09 samples/sec Loss 11.0779 Epoch: 5 Global Step: 53125 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:30:28,928-Speed 126.84 samples/sec Loss 11.1647 Epoch: 5 Global Step: 53130 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:30:48,795-Speed 128.86 samples/sec Loss 11.1929 Epoch: 5 Global Step: 53135 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:31:08,795-Speed 128.01 samples/sec Loss 11.2972 Epoch: 5 Global Step: 53140 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:31:28,649-Speed 128.95 samples/sec Loss 10.8310 Epoch: 5 Global Step: 53145 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:31:48,881-Speed 126.54 samples/sec Loss 11.3950 Epoch: 5 Global Step: 53150 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:32:08,745-Speed 128.88 samples/sec Loss 11.2843 Epoch: 5 Global Step: 53155 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:32:28,578-Speed 129.08 samples/sec Loss 11.0164 Epoch: 5 Global Step: 53160 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:32:48,589-Speed 127.94 samples/sec Loss 11.1337 Epoch: 5 Global Step: 53165 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:33:08,439-Speed 128.97 samples/sec Loss 11.1985 Epoch: 5 Global Step: 53170 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:33:28,680-Speed 126.49 samples/sec Loss 10.9955 Epoch: 5 Global Step: 53175 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:33:48,542-Speed 128.89 samples/sec Loss 11.0947 Epoch: 5 Global Step: 53180 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:34:08,497-Speed 128.29 samples/sec Loss 11.0047 Epoch: 5 Global Step: 53185 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:34:28,364-Speed 128.86 samples/sec Loss 11.1888 Epoch: 5 Global Step: 53190 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:34:48,687-Speed 125.97 samples/sec Loss 10.9270 Epoch: 5 Global Step: 53195 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:35:08,519-Speed 129.09 samples/sec Loss 11.4875 Epoch: 5 Global Step: 53200 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:35:28,589-Speed 127.56 samples/sec Loss 10.9250 Epoch: 5 Global Step: 53205 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:35:48,353-Speed 129.53 samples/sec Loss 11.4860 Epoch: 5 Global Step: 53210 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:36:08,173-Speed 129.17 samples/sec Loss 11.3654 Epoch: 5 Global Step: 53215 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:36:28,396-Speed 126.59 samples/sec Loss 11.1954 Epoch: 5 Global Step: 53220 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:36:48,304-Speed 128.60 samples/sec Loss 11.0609 Epoch: 5 Global Step: 53225 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:37:08,152-Speed 128.99 samples/sec Loss 11.0001 Epoch: 5 Global Step: 53230 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:37:29,485-Speed 120.00 samples/sec Loss 11.3716 Epoch: 5 Global Step: 53235 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:37:50,216-Speed 123.50 samples/sec Loss 11.4043 Epoch: 5 Global Step: 53240 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:38:10,092-Speed 128.81 samples/sec Loss 11.3980 Epoch: 5 Global Step: 53245 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:38:30,025-Speed 128.43 samples/sec Loss 11.0067 Epoch: 5 Global Step: 53250 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:38:49,826-Speed 129.29 samples/sec Loss 11.1413 Epoch: 5 Global Step: 53255 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:39:09,880-Speed 127.67 samples/sec Loss 11.2160 Epoch: 5 Global Step: 53260 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:39:30,085-Speed 126.71 samples/sec Loss 11.6385 Epoch: 5 Global Step: 53265 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:39:49,942-Speed 128.93 samples/sec Loss 11.3012 Epoch: 5 Global Step: 53270 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:40:09,798-Speed 128.94 samples/sec Loss 11.2754 Epoch: 5 Global Step: 53275 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:40:29,678-Speed 128.78 samples/sec Loss 11.1508 Epoch: 5 Global Step: 53280 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:40:49,781-Speed 127.35 samples/sec Loss 11.3186 Epoch: 5 Global Step: 53285 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:41:09,800-Speed 127.88 samples/sec Loss 11.1048 Epoch: 5 Global Step: 53290 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:41:29,649-Speed 128.98 samples/sec Loss 11.5072 Epoch: 5 Global Step: 53295 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:41:49,704-Speed 127.66 samples/sec Loss 11.3771 Epoch: 5 Global Step: 53300 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:42:09,534-Speed 129.10 samples/sec Loss 11.2857 Epoch: 5 Global Step: 53305 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:42:29,774-Speed 126.49 samples/sec Loss 11.3519 Epoch: 5 Global Step: 53310 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:42:49,613-Speed 129.04 samples/sec Loss 11.5804 Epoch: 5 Global Step: 53315 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:43:09,640-Speed 127.83 samples/sec Loss 11.2162 Epoch: 5 Global Step: 53320 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:43:29,554-Speed 128.56 samples/sec Loss 11.0683 Epoch: 5 Global Step: 53325 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:43:49,725-Speed 126.92 samples/sec Loss 11.3555 Epoch: 5 Global Step: 53330 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:44:09,660-Speed 128.42 samples/sec Loss 11.0633 Epoch: 5 Global Step: 53335 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:44:29,450-Speed 129.37 samples/sec Loss 11.0058 Epoch: 5 Global Step: 53340 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:44:49,551-Speed 127.37 samples/sec Loss 11.1970 Epoch: 5 Global Step: 53345 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:45:09,385-Speed 129.08 samples/sec Loss 11.3943 Epoch: 5 Global Step: 53350 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:45:29,598-Speed 126.66 samples/sec Loss 11.4365 Epoch: 5 Global Step: 53355 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:45:49,471-Speed 128.82 samples/sec Loss 11.2676 Epoch: 5 Global Step: 53360 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:46:09,450-Speed 128.14 samples/sec Loss 11.5467 Epoch: 5 Global Step: 53365 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:46:29,272-Speed 129.15 samples/sec Loss 11.0482 Epoch: 5 Global Step: 53370 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:46:49,496-Speed 126.59 samples/sec Loss 11.4392 Epoch: 5 Global Step: 53375 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:47:09,328-Speed 129.10 samples/sec Loss 11.1035 Epoch: 5 Global Step: 53380 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:47:29,265-Speed 128.41 samples/sec Loss 11.2491 Epoch: 5 Global Step: 53385 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:47:49,313-Speed 127.70 samples/sec Loss 11.1977 Epoch: 5 Global Step: 53390 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:48:09,043-Speed 129.76 samples/sec Loss 11.1224 Epoch: 5 Global Step: 53395 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:48:29,221-Speed 126.87 samples/sec Loss 11.2843 Epoch: 5 Global Step: 53400 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:48:49,095-Speed 128.82 samples/sec Loss 11.5852 Epoch: 5 Global Step: 53405 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:49:09,127-Speed 127.80 samples/sec Loss 11.0807 Epoch: 5 Global Step: 53410 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:49:28,936-Speed 129.24 samples/sec Loss 10.9719 Epoch: 5 Global Step: 53415 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:49:49,178-Speed 126.48 samples/sec Loss 11.0406 Epoch: 5 Global Step: 53420 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:50:09,087-Speed 128.59 samples/sec Loss 11.0374 Epoch: 5 Global Step: 53425 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:50:29,143-Speed 127.65 samples/sec Loss 11.1564 Epoch: 5 Global Step: 53430 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:50:48,998-Speed 128.94 samples/sec Loss 11.3862 Epoch: 5 Global Step: 53435 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:51:08,867-Speed 128.85 samples/sec Loss 11.0797 Epoch: 5 Global Step: 53440 Fp16 Grad Scale: 8192 Required: 234 hours Training: 2021-06-09 10:51:29,023-Speed 127.02 samples/sec Loss 11.3352 Epoch: 5 Global Step: 53445 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:51:49,307-Speed 126.21 samples/sec Loss 10.8655 Epoch: 5 Global Step: 53450 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:52:09,144-Speed 129.06 samples/sec Loss 11.3899 Epoch: 5 Global Step: 53455 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:52:29,189-Speed 127.72 samples/sec Loss 11.2169 Epoch: 5 Global Step: 53460 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:52:49,372-Speed 126.84 samples/sec Loss 11.0879 Epoch: 5 Global Step: 53465 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:53:09,220-Speed 128.99 samples/sec Loss 11.3771 Epoch: 5 Global Step: 53470 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:53:29,156-Speed 128.42 samples/sec Loss 11.4431 Epoch: 5 Global Step: 53475 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:53:49,032-Speed 128.81 samples/sec Loss 11.0008 Epoch: 5 Global Step: 53480 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:54:09,062-Speed 127.82 samples/sec Loss 11.1536 Epoch: 5 Global Step: 53485 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:54:29,307-Speed 126.46 samples/sec Loss 11.1026 Epoch: 5 Global Step: 53490 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:54:49,162-Speed 128.94 samples/sec Loss 10.9101 Epoch: 5 Global Step: 53495 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:55:08,905-Speed 129.67 samples/sec Loss 11.1202 Epoch: 5 Global Step: 53500 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:55:28,912-Speed 127.96 samples/sec Loss 11.3301 Epoch: 5 Global Step: 53505 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:55:49,019-Speed 127.32 samples/sec Loss 11.1910 Epoch: 5 Global Step: 53510 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:56:09,029-Speed 127.94 samples/sec Loss 11.3358 Epoch: 5 Global Step: 53515 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:56:28,766-Speed 129.71 samples/sec Loss 11.3605 Epoch: 5 Global Step: 53520 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:56:48,652-Speed 128.74 samples/sec Loss 10.7149 Epoch: 5 Global Step: 53525 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:57:08,675-Speed 127.86 samples/sec Loss 11.0954 Epoch: 5 Global Step: 53530 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:57:28,983-Speed 126.07 samples/sec Loss 11.3314 Epoch: 5 Global Step: 53535 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:57:48,696-Speed 129.86 samples/sec Loss 11.1851 Epoch: 5 Global Step: 53540 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:58:08,821-Speed 127.21 samples/sec Loss 11.1892 Epoch: 5 Global Step: 53545 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:58:28,675-Speed 128.94 samples/sec Loss 11.2038 Epoch: 5 Global Step: 53550 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:58:48,915-Speed 126.49 samples/sec Loss 10.9996 Epoch: 5 Global Step: 53555 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:59:08,768-Speed 128.96 samples/sec Loss 11.3559 Epoch: 5 Global Step: 53560 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:59:28,599-Speed 129.10 samples/sec Loss 10.7698 Epoch: 5 Global Step: 53565 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 10:59:48,717-Speed 127.26 samples/sec Loss 11.6789 Epoch: 5 Global Step: 53570 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:00:08,576-Speed 128.91 samples/sec Loss 11.2810 Epoch: 5 Global Step: 53575 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:00:28,648-Speed 127.55 samples/sec Loss 11.2583 Epoch: 5 Global Step: 53580 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:00:48,554-Speed 128.61 samples/sec Loss 11.3450 Epoch: 5 Global Step: 53585 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:01:08,494-Speed 128.38 samples/sec Loss 11.0531 Epoch: 5 Global Step: 53590 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:01:28,402-Speed 128.60 samples/sec Loss 11.3878 Epoch: 5 Global Step: 53595 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:01:48,609-Speed 126.70 samples/sec Loss 11.3081 Epoch: 5 Global Step: 53600 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:02:08,419-Speed 129.23 samples/sec Loss 11.1910 Epoch: 5 Global Step: 53605 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:02:28,302-Speed 128.76 samples/sec Loss 11.2389 Epoch: 5 Global Step: 53610 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:02:48,336-Speed 127.79 samples/sec Loss 11.0723 Epoch: 5 Global Step: 53615 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:03:08,129-Speed 129.35 samples/sec Loss 11.1715 Epoch: 5 Global Step: 53620 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:03:29,736-Speed 118.49 samples/sec Loss 11.3721 Epoch: 5 Global Step: 53625 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:03:49,774-Speed 127.76 samples/sec Loss 11.4773 Epoch: 5 Global Step: 53630 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:04:09,598-Speed 129.14 samples/sec Loss 10.9433 Epoch: 5 Global Step: 53635 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:04:29,627-Speed 127.82 samples/sec Loss 10.8359 Epoch: 5 Global Step: 53640 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:04:49,659-Speed 127.80 samples/sec Loss 11.0779 Epoch: 5 Global Step: 53645 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:05:09,486-Speed 129.12 samples/sec Loss 11.0022 Epoch: 5 Global Step: 53650 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:05:29,552-Speed 127.59 samples/sec Loss 11.6267 Epoch: 5 Global Step: 53655 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:05:49,666-Speed 127.28 samples/sec Loss 11.1094 Epoch: 5 Global Step: 53660 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:06:09,584-Speed 128.53 samples/sec Loss 11.1528 Epoch: 5 Global Step: 53665 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:06:29,631-Speed 127.71 samples/sec Loss 11.3464 Epoch: 5 Global Step: 53670 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:06:49,567-Speed 128.41 samples/sec Loss 11.1897 Epoch: 5 Global Step: 53675 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:07:09,460-Speed 128.70 samples/sec Loss 11.3045 Epoch: 5 Global Step: 53680 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:07:29,377-Speed 128.53 samples/sec Loss 11.1172 Epoch: 5 Global Step: 53685 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:07:50,089-Speed 123.61 samples/sec Loss 11.3342 Epoch: 5 Global Step: 53690 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:08:10,189-Speed 127.37 samples/sec Loss 10.6473 Epoch: 5 Global Step: 53695 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:08:30,032-Speed 129.02 samples/sec Loss 10.8335 Epoch: 5 Global Step: 53700 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:08:49,805-Speed 129.48 samples/sec Loss 11.0925 Epoch: 5 Global Step: 53705 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:09:09,818-Speed 127.92 samples/sec Loss 11.2433 Epoch: 5 Global Step: 53710 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:09:30,022-Speed 126.71 samples/sec Loss 11.4038 Epoch: 5 Global Step: 53715 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:09:49,919-Speed 128.67 samples/sec Loss 11.3357 Epoch: 5 Global Step: 53720 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:10:09,884-Speed 128.23 samples/sec Loss 11.1483 Epoch: 5 Global Step: 53725 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:10:29,805-Speed 128.51 samples/sec Loss 11.4511 Epoch: 5 Global Step: 53730 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:10:49,843-Speed 127.77 samples/sec Loss 11.2663 Epoch: 5 Global Step: 53735 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:11:09,872-Speed 127.82 samples/sec Loss 11.1383 Epoch: 5 Global Step: 53740 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:11:29,625-Speed 129.61 samples/sec Loss 11.1665 Epoch: 5 Global Step: 53745 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:11:49,632-Speed 127.96 samples/sec Loss 11.0616 Epoch: 5 Global Step: 53750 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:12:09,480-Speed 128.99 samples/sec Loss 11.0795 Epoch: 5 Global Step: 53755 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:12:29,776-Speed 126.14 samples/sec Loss 11.5920 Epoch: 5 Global Step: 53760 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:12:49,645-Speed 128.85 samples/sec Loss 10.9727 Epoch: 5 Global Step: 53765 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:13:09,621-Speed 128.16 samples/sec Loss 11.1143 Epoch: 5 Global Step: 53770 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:13:29,469-Speed 128.99 samples/sec Loss 11.1810 Epoch: 5 Global Step: 53775 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:13:49,856-Speed 125.58 samples/sec Loss 11.1246 Epoch: 5 Global Step: 53780 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:14:09,689-Speed 129.08 samples/sec Loss 10.9675 Epoch: 5 Global Step: 53785 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:14:29,670-Speed 128.13 samples/sec Loss 11.1697 Epoch: 5 Global Step: 53790 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:14:49,510-Speed 129.04 samples/sec Loss 11.0942 Epoch: 5 Global Step: 53795 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:15:09,385-Speed 128.81 samples/sec Loss 11.0141 Epoch: 5 Global Step: 53800 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:15:29,666-Speed 126.23 samples/sec Loss 11.0273 Epoch: 5 Global Step: 53805 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:15:49,660-Speed 128.04 samples/sec Loss 10.7148 Epoch: 5 Global Step: 53810 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:16:09,394-Speed 129.73 samples/sec Loss 10.8189 Epoch: 5 Global Step: 53815 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:16:29,427-Speed 127.80 samples/sec Loss 11.0115 Epoch: 5 Global Step: 53820 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:16:49,557-Speed 127.18 samples/sec Loss 11.0250 Epoch: 5 Global Step: 53825 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:17:09,552-Speed 128.04 samples/sec Loss 11.3989 Epoch: 5 Global Step: 53830 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:17:29,426-Speed 128.82 samples/sec Loss 10.9673 Epoch: 5 Global Step: 53835 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:17:49,276-Speed 128.97 samples/sec Loss 11.0028 Epoch: 5 Global Step: 53840 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:18:09,377-Speed 127.36 samples/sec Loss 11.3426 Epoch: 5 Global Step: 53845 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:18:29,585-Speed 126.69 samples/sec Loss 11.5080 Epoch: 5 Global Step: 53850 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:18:49,463-Speed 128.79 samples/sec Loss 11.3507 Epoch: 5 Global Step: 53855 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:19:09,328-Speed 128.87 samples/sec Loss 11.4467 Epoch: 5 Global Step: 53860 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:19:29,353-Speed 127.85 samples/sec Loss 11.3307 Epoch: 5 Global Step: 53865 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:19:49,361-Speed 127.95 samples/sec Loss 11.1859 Epoch: 5 Global Step: 53870 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:20:10,065-Speed 123.65 samples/sec Loss 11.1667 Epoch: 5 Global Step: 53875 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:20:29,890-Speed 129.13 samples/sec Loss 11.1202 Epoch: 5 Global Step: 53880 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:20:49,835-Speed 128.36 samples/sec Loss 11.3649 Epoch: 5 Global Step: 53885 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:21:09,792-Speed 128.28 samples/sec Loss 10.9170 Epoch: 5 Global Step: 53890 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:21:29,985-Speed 126.78 samples/sec Loss 11.0289 Epoch: 5 Global Step: 53895 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:21:49,826-Speed 129.04 samples/sec Loss 11.3303 Epoch: 5 Global Step: 53900 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:22:09,716-Speed 128.71 samples/sec Loss 11.2559 Epoch: 5 Global Step: 53905 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:22:29,712-Speed 128.03 samples/sec Loss 10.9387 Epoch: 5 Global Step: 53910 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:22:49,854-Speed 127.10 samples/sec Loss 10.9821 Epoch: 5 Global Step: 53915 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:23:09,837-Speed 128.12 samples/sec Loss 11.3787 Epoch: 5 Global Step: 53920 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:23:29,727-Speed 128.71 samples/sec Loss 11.2483 Epoch: 5 Global Step: 53925 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:23:49,661-Speed 128.43 samples/sec Loss 11.3640 Epoch: 5 Global Step: 53930 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:24:09,500-Speed 129.05 samples/sec Loss 11.2703 Epoch: 5 Global Step: 53935 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:24:29,710-Speed 126.68 samples/sec Loss 11.0970 Epoch: 5 Global Step: 53940 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:24:49,455-Speed 129.66 samples/sec Loss 11.5490 Epoch: 5 Global Step: 53945 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:25:09,681-Speed 126.58 samples/sec Loss 11.2345 Epoch: 5 Global Step: 53950 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:25:29,448-Speed 129.51 samples/sec Loss 10.9998 Epoch: 5 Global Step: 53955 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:25:49,739-Speed 126.17 samples/sec Loss 11.0243 Epoch: 5 Global Step: 53960 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:26:09,556-Speed 129.19 samples/sec Loss 11.0977 Epoch: 5 Global Step: 53965 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:26:29,435-Speed 128.79 samples/sec Loss 11.3840 Epoch: 5 Global Step: 53970 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:26:49,415-Speed 128.14 samples/sec Loss 11.3513 Epoch: 5 Global Step: 53975 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:27:09,176-Speed 129.55 samples/sec Loss 11.3099 Epoch: 5 Global Step: 53980 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:27:29,384-Speed 126.69 samples/sec Loss 11.3782 Epoch: 5 Global Step: 53985 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:27:49,454-Speed 127.56 samples/sec Loss 11.4508 Epoch: 5 Global Step: 53990 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:28:09,268-Speed 129.20 samples/sec Loss 11.0305 Epoch: 5 Global Step: 53995 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:28:29,068-Speed 129.31 samples/sec Loss 11.2131 Epoch: 5 Global Step: 54000 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:28:49,329-Speed 126.36 samples/sec Loss 11.0585 Epoch: 5 Global Step: 54005 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:29:09,231-Speed 128.63 samples/sec Loss 11.1866 Epoch: 5 Global Step: 54010 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:29:29,525-Speed 126.16 samples/sec Loss 11.2712 Epoch: 5 Global Step: 54015 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:29:49,454-Speed 128.46 samples/sec Loss 11.3955 Epoch: 5 Global Step: 54020 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:30:09,325-Speed 128.84 samples/sec Loss 11.2600 Epoch: 5 Global Step: 54025 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:30:29,544-Speed 126.61 samples/sec Loss 11.5782 Epoch: 5 Global Step: 54030 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:30:49,590-Speed 127.71 samples/sec Loss 11.0435 Epoch: 5 Global Step: 54035 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:31:09,432-Speed 129.02 samples/sec Loss 10.9989 Epoch: 5 Global Step: 54040 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:31:29,465-Speed 127.79 samples/sec Loss 11.0464 Epoch: 5 Global Step: 54045 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:31:49,650-Speed 126.83 samples/sec Loss 11.2550 Epoch: 5 Global Step: 54050 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:32:09,706-Speed 127.65 samples/sec Loss 11.1180 Epoch: 5 Global Step: 54055 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:32:29,670-Speed 128.24 samples/sec Loss 11.3426 Epoch: 5 Global Step: 54060 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:32:49,518-Speed 128.99 samples/sec Loss 11.2774 Epoch: 5 Global Step: 54065 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:33:09,536-Speed 127.89 samples/sec Loss 11.3253 Epoch: 5 Global Step: 54070 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:33:29,624-Speed 127.44 samples/sec Loss 11.0491 Epoch: 5 Global Step: 54075 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:33:49,441-Speed 129.19 samples/sec Loss 11.1935 Epoch: 5 Global Step: 54080 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:34:09,286-Speed 129.00 samples/sec Loss 11.3289 Epoch: 5 Global Step: 54085 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:34:29,203-Speed 128.54 samples/sec Loss 11.1924 Epoch: 5 Global Step: 54090 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:34:49,276-Speed 127.54 samples/sec Loss 11.1674 Epoch: 5 Global Step: 54095 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:35:09,283-Speed 127.96 samples/sec Loss 11.0969 Epoch: 5 Global Step: 54100 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:35:29,115-Speed 129.09 samples/sec Loss 11.3541 Epoch: 5 Global Step: 54105 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:35:48,976-Speed 128.91 samples/sec Loss 11.3194 Epoch: 5 Global Step: 54110 Fp16 Grad Scale: 8192 Required: 233 hours Training: 2021-06-09 11:36:08,964-Speed 128.08 samples/sec Loss 11.0385 Epoch: 5 Global Step: 54115 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:36:29,116-Speed 127.04 samples/sec Loss 11.3913 Epoch: 5 Global Step: 54120 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:36:48,939-Speed 129.15 samples/sec Loss 11.0702 Epoch: 5 Global Step: 54125 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:37:08,827-Speed 128.72 samples/sec Loss 11.0207 Epoch: 5 Global Step: 54130 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:37:28,847-Speed 127.88 samples/sec Loss 11.0850 Epoch: 5 Global Step: 54135 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:37:48,887-Speed 127.75 samples/sec Loss 11.3787 Epoch: 5 Global Step: 54140 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:38:08,904-Speed 127.90 samples/sec Loss 11.3757 Epoch: 5 Global Step: 54145 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:38:28,654-Speed 129.63 samples/sec Loss 11.0665 Epoch: 5 Global Step: 54150 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:38:48,635-Speed 128.13 samples/sec Loss 11.2213 Epoch: 5 Global Step: 54155 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:39:08,430-Speed 129.33 samples/sec Loss 10.7052 Epoch: 5 Global Step: 54160 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:39:28,614-Speed 126.84 samples/sec Loss 10.9195 Epoch: 5 Global Step: 54165 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:39:48,435-Speed 129.16 samples/sec Loss 11.1929 Epoch: 5 Global Step: 54170 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:40:08,320-Speed 128.75 samples/sec Loss 11.1738 Epoch: 5 Global Step: 54175 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:40:28,264-Speed 128.37 samples/sec Loss 11.1800 Epoch: 5 Global Step: 54180 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:40:48,254-Speed 128.07 samples/sec Loss 11.2844 Epoch: 5 Global Step: 54185 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:41:08,262-Speed 127.96 samples/sec Loss 11.0344 Epoch: 5 Global Step: 54190 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:41:28,184-Speed 128.51 samples/sec Loss 11.3308 Epoch: 5 Global Step: 54195 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:41:48,581-Speed 125.52 samples/sec Loss 11.0640 Epoch: 5 Global Step: 54200 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:42:08,318-Speed 129.71 samples/sec Loss 11.3410 Epoch: 5 Global Step: 54205 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:42:28,572-Speed 126.40 samples/sec Loss 11.2451 Epoch: 5 Global Step: 54210 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:42:48,473-Speed 128.64 samples/sec Loss 11.1845 Epoch: 5 Global Step: 54215 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:43:08,486-Speed 127.93 samples/sec Loss 11.2232 Epoch: 5 Global Step: 54220 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:43:28,195-Speed 129.90 samples/sec Loss 11.3034 Epoch: 5 Global Step: 54225 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:43:48,378-Speed 126.84 samples/sec Loss 11.0340 Epoch: 5 Global Step: 54230 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:44:08,341-Speed 128.25 samples/sec Loss 10.9742 Epoch: 5 Global Step: 54235 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:44:28,180-Speed 129.05 samples/sec Loss 11.3902 Epoch: 5 Global Step: 54240 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:44:48,237-Speed 127.64 samples/sec Loss 11.3717 Epoch: 5 Global Step: 54245 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:45:08,055-Speed 129.18 samples/sec Loss 11.3962 Epoch: 5 Global Step: 54250 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:45:28,427-Speed 125.67 samples/sec Loss 11.1358 Epoch: 5 Global Step: 54255 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:45:48,465-Speed 127.76 samples/sec Loss 11.4273 Epoch: 5 Global Step: 54260 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:46:08,226-Speed 129.56 samples/sec Loss 11.4024 Epoch: 5 Global Step: 54265 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:46:28,068-Speed 129.02 samples/sec Loss 11.4054 Epoch: 5 Global Step: 54270 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:46:48,299-Speed 126.54 samples/sec Loss 11.5813 Epoch: 5 Global Step: 54275 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:47:08,220-Speed 128.52 samples/sec Loss 11.1844 Epoch: 5 Global Step: 54280 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:47:28,100-Speed 128.78 samples/sec Loss 11.0738 Epoch: 5 Global Step: 54285 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:47:47,839-Speed 129.70 samples/sec Loss 11.0002 Epoch: 5 Global Step: 54290 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:48:07,860-Speed 127.87 samples/sec Loss 10.7806 Epoch: 5 Global Step: 54295 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:48:28,031-Speed 126.92 samples/sec Loss 11.2851 Epoch: 5 Global Step: 54300 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:48:47,968-Speed 128.41 samples/sec Loss 11.1483 Epoch: 5 Global Step: 54305 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:49:07,735-Speed 129.52 samples/sec Loss 11.3231 Epoch: 5 Global Step: 54310 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:49:27,894-Speed 127.00 samples/sec Loss 11.1432 Epoch: 5 Global Step: 54315 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:49:48,107-Speed 126.65 samples/sec Loss 11.3784 Epoch: 5 Global Step: 54320 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:50:08,124-Speed 127.90 samples/sec Loss 11.4027 Epoch: 5 Global Step: 54325 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:50:27,879-Speed 129.59 samples/sec Loss 11.4322 Epoch: 5 Global Step: 54330 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:50:48,008-Speed 127.19 samples/sec Loss 10.9784 Epoch: 5 Global Step: 54335 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:51:07,833-Speed 129.13 samples/sec Loss 11.0315 Epoch: 5 Global Step: 54340 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:51:28,017-Speed 126.84 samples/sec Loss 11.2653 Epoch: 5 Global Step: 54345 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:51:47,749-Speed 129.75 samples/sec Loss 11.1731 Epoch: 5 Global Step: 54350 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:52:07,647-Speed 128.66 samples/sec Loss 10.9589 Epoch: 5 Global Step: 54355 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:52:27,533-Speed 128.74 samples/sec Loss 11.0922 Epoch: 5 Global Step: 54360 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:52:47,985-Speed 125.18 samples/sec Loss 11.2026 Epoch: 5 Global Step: 54365 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:53:07,710-Speed 129.79 samples/sec Loss 11.2466 Epoch: 5 Global Step: 54370 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:53:27,628-Speed 128.53 samples/sec Loss 11.2915 Epoch: 5 Global Step: 54375 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:53:47,587-Speed 128.27 samples/sec Loss 11.3033 Epoch: 5 Global Step: 54380 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:54:07,527-Speed 128.39 samples/sec Loss 10.9453 Epoch: 5 Global Step: 54385 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:54:27,706-Speed 126.87 samples/sec Loss 11.4015 Epoch: 5 Global Step: 54390 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:54:47,677-Speed 128.19 samples/sec Loss 11.4460 Epoch: 5 Global Step: 54395 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:55:07,569-Speed 128.70 samples/sec Loss 10.9333 Epoch: 5 Global Step: 54400 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:55:27,569-Speed 128.01 samples/sec Loss 11.0802 Epoch: 5 Global Step: 54405 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:55:47,998-Speed 125.31 samples/sec Loss 11.3447 Epoch: 5 Global Step: 54410 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:56:07,823-Speed 129.13 samples/sec Loss 11.1264 Epoch: 5 Global Step: 54415 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:56:28,347-Speed 124.74 samples/sec Loss 11.1815 Epoch: 5 Global Step: 54420 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:56:48,245-Speed 128.66 samples/sec Loss 11.3666 Epoch: 5 Global Step: 54425 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:57:08,269-Speed 127.86 samples/sec Loss 11.2499 Epoch: 5 Global Step: 54430 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:57:28,240-Speed 128.19 samples/sec Loss 11.3210 Epoch: 5 Global Step: 54435 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:57:48,306-Speed 127.58 samples/sec Loss 10.8360 Epoch: 5 Global Step: 54440 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:58:08,208-Speed 128.63 samples/sec Loss 11.0369 Epoch: 5 Global Step: 54445 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:58:28,265-Speed 127.64 samples/sec Loss 11.4003 Epoch: 5 Global Step: 54450 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:58:48,194-Speed 128.46 samples/sec Loss 11.1176 Epoch: 5 Global Step: 54455 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:59:08,215-Speed 127.88 samples/sec Loss 11.1909 Epoch: 5 Global Step: 54460 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:59:28,110-Speed 128.68 samples/sec Loss 11.3118 Epoch: 5 Global Step: 54465 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 11:59:47,993-Speed 128.76 samples/sec Loss 11.1061 Epoch: 5 Global Step: 54470 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:00:07,895-Speed 128.64 samples/sec Loss 11.1726 Epoch: 5 Global Step: 54475 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:00:27,977-Speed 127.49 samples/sec Loss 11.1273 Epoch: 5 Global Step: 54480 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:00:47,813-Speed 129.07 samples/sec Loss 11.3316 Epoch: 5 Global Step: 54485 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:01:07,820-Speed 127.96 samples/sec Loss 11.1591 Epoch: 5 Global Step: 54490 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:01:27,719-Speed 128.66 samples/sec Loss 11.1096 Epoch: 5 Global Step: 54495 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:01:47,977-Speed 126.37 samples/sec Loss 11.5180 Epoch: 5 Global Step: 54500 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:02:08,045-Speed 127.57 samples/sec Loss 11.0370 Epoch: 5 Global Step: 54505 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:02:27,826-Speed 129.43 samples/sec Loss 11.0021 Epoch: 5 Global Step: 54510 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:02:47,802-Speed 128.16 samples/sec Loss 11.4349 Epoch: 5 Global Step: 54515 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:03:07,508-Speed 129.92 samples/sec Loss 10.9610 Epoch: 5 Global Step: 54520 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:03:27,709-Speed 126.73 samples/sec Loss 10.9730 Epoch: 5 Global Step: 54525 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:03:47,617-Speed 128.60 samples/sec Loss 11.6257 Epoch: 5 Global Step: 54530 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:04:07,630-Speed 127.93 samples/sec Loss 11.2678 Epoch: 5 Global Step: 54535 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:04:27,349-Speed 129.83 samples/sec Loss 11.1244 Epoch: 5 Global Step: 54540 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:04:47,549-Speed 126.74 samples/sec Loss 11.1018 Epoch: 5 Global Step: 54545 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:05:07,387-Speed 129.05 samples/sec Loss 11.0800 Epoch: 5 Global Step: 54550 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:05:27,244-Speed 128.93 samples/sec Loss 11.2866 Epoch: 5 Global Step: 54555 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:05:47,585-Speed 125.87 samples/sec Loss 11.1037 Epoch: 5 Global Step: 54560 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:06:07,406-Speed 129.16 samples/sec Loss 11.4649 Epoch: 5 Global Step: 54565 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:06:27,654-Speed 126.44 samples/sec Loss 11.5320 Epoch: 5 Global Step: 54570 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:06:47,469-Speed 129.20 samples/sec Loss 11.2587 Epoch: 5 Global Step: 54575 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:07:07,356-Speed 128.73 samples/sec Loss 10.9041 Epoch: 5 Global Step: 54580 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:07:27,118-Speed 129.55 samples/sec Loss 11.4408 Epoch: 5 Global Step: 54585 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:07:47,428-Speed 126.05 samples/sec Loss 11.2905 Epoch: 5 Global Step: 54590 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:08:07,250-Speed 129.15 samples/sec Loss 11.0413 Epoch: 5 Global Step: 54595 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:08:26,978-Speed 129.78 samples/sec Loss 10.8835 Epoch: 5 Global Step: 54600 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:08:47,114-Speed 127.14 samples/sec Loss 11.0840 Epoch: 5 Global Step: 54605 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:09:07,002-Speed 128.73 samples/sec Loss 10.9831 Epoch: 5 Global Step: 54610 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:09:27,189-Speed 126.82 samples/sec Loss 11.0907 Epoch: 5 Global Step: 54615 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:09:47,094-Speed 128.61 samples/sec Loss 11.2171 Epoch: 5 Global Step: 54620 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:10:06,890-Speed 129.33 samples/sec Loss 11.4843 Epoch: 5 Global Step: 54625 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:10:26,733-Speed 129.02 samples/sec Loss 10.9836 Epoch: 5 Global Step: 54630 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:10:47,136-Speed 125.47 samples/sec Loss 10.7287 Epoch: 5 Global Step: 54635 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:11:06,894-Speed 129.58 samples/sec Loss 11.1589 Epoch: 5 Global Step: 54640 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:11:26,843-Speed 128.33 samples/sec Loss 11.2699 Epoch: 5 Global Step: 54645 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:11:46,854-Speed 127.93 samples/sec Loss 11.0353 Epoch: 5 Global Step: 54650 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:12:06,795-Speed 128.39 samples/sec Loss 11.3052 Epoch: 5 Global Step: 54655 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:12:26,900-Speed 127.34 samples/sec Loss 11.1348 Epoch: 5 Global Step: 54660 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:12:47,188-Speed 126.19 samples/sec Loss 11.1206 Epoch: 5 Global Step: 54665 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:13:07,070-Speed 128.77 samples/sec Loss 11.0511 Epoch: 5 Global Step: 54670 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:13:27,177-Speed 127.32 samples/sec Loss 11.0252 Epoch: 5 Global Step: 54675 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:13:47,209-Speed 127.80 samples/sec Loss 11.4242 Epoch: 5 Global Step: 54680 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:14:07,235-Speed 127.84 samples/sec Loss 11.1097 Epoch: 5 Global Step: 54685 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:14:27,058-Speed 129.15 samples/sec Loss 11.3030 Epoch: 5 Global Step: 54690 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:14:46,980-Speed 128.51 samples/sec Loss 11.3121 Epoch: 5 Global Step: 54695 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:15:06,963-Speed 128.12 samples/sec Loss 11.6005 Epoch: 5 Global Step: 54700 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:15:27,055-Speed 127.42 samples/sec Loss 11.0222 Epoch: 5 Global Step: 54705 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:15:46,872-Speed 129.19 samples/sec Loss 10.9694 Epoch: 5 Global Step: 54710 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:16:06,941-Speed 127.57 samples/sec Loss 10.9603 Epoch: 5 Global Step: 54715 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:16:26,815-Speed 128.81 samples/sec Loss 11.1062 Epoch: 5 Global Step: 54720 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:16:46,830-Speed 127.91 samples/sec Loss 10.9293 Epoch: 5 Global Step: 54725 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:17:06,852-Speed 127.86 samples/sec Loss 11.0469 Epoch: 5 Global Step: 54730 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:17:26,743-Speed 128.71 samples/sec Loss 11.2011 Epoch: 5 Global Step: 54735 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:17:46,755-Speed 127.93 samples/sec Loss 10.8099 Epoch: 5 Global Step: 54740 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:18:06,476-Speed 129.82 samples/sec Loss 11.1360 Epoch: 5 Global Step: 54745 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:18:26,557-Speed 127.49 samples/sec Loss 11.2796 Epoch: 5 Global Step: 54750 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:18:46,449-Speed 128.70 samples/sec Loss 11.3224 Epoch: 5 Global Step: 54755 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:19:06,661-Speed 126.66 samples/sec Loss 11.3087 Epoch: 5 Global Step: 54760 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:19:26,484-Speed 129.15 samples/sec Loss 11.2517 Epoch: 5 Global Step: 54765 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:19:46,693-Speed 126.69 samples/sec Loss 11.2025 Epoch: 5 Global Step: 54770 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:20:06,539-Speed 129.00 samples/sec Loss 11.1219 Epoch: 5 Global Step: 54775 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:20:26,374-Speed 129.07 samples/sec Loss 11.0685 Epoch: 5 Global Step: 54780 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:20:46,343-Speed 128.20 samples/sec Loss 11.1365 Epoch: 5 Global Step: 54785 Fp16 Grad Scale: 8192 Required: 232 hours Training: 2021-06-09 12:21:06,055-Speed 129.88 samples/sec Loss 11.4438 Epoch: 5 Global Step: 54790 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:21:26,250-Speed 126.77 samples/sec Loss 11.1084 Epoch: 5 Global Step: 54795 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:21:46,344-Speed 127.41 samples/sec Loss 10.9820 Epoch: 5 Global Step: 54800 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:22:06,155-Speed 129.22 samples/sec Loss 11.3167 Epoch: 5 Global Step: 54805 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:22:26,069-Speed 128.56 samples/sec Loss 11.2011 Epoch: 5 Global Step: 54810 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:22:46,140-Speed 127.55 samples/sec Loss 11.2925 Epoch: 5 Global Step: 54815 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:23:05,982-Speed 129.03 samples/sec Loss 11.5963 Epoch: 5 Global Step: 54820 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:23:25,964-Speed 128.12 samples/sec Loss 11.4072 Epoch: 5 Global Step: 54825 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:23:45,711-Speed 129.65 samples/sec Loss 11.1275 Epoch: 5 Global Step: 54830 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:24:05,693-Speed 128.12 samples/sec Loss 10.9451 Epoch: 5 Global Step: 54835 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:24:26,232-Speed 124.65 samples/sec Loss 11.3140 Epoch: 5 Global Step: 54840 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:24:46,158-Speed 128.48 samples/sec Loss 11.3427 Epoch: 5 Global Step: 54845 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:25:06,041-Speed 128.75 samples/sec Loss 11.0590 Epoch: 5 Global Step: 54850 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:25:25,989-Speed 128.34 samples/sec Loss 11.1957 Epoch: 5 Global Step: 54855 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:25:46,182-Speed 126.79 samples/sec Loss 11.1814 Epoch: 5 Global Step: 54860 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:26:06,050-Speed 128.85 samples/sec Loss 11.2268 Epoch: 5 Global Step: 54865 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:26:26,102-Speed 127.68 samples/sec Loss 11.4246 Epoch: 5 Global Step: 54870 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:26:45,961-Speed 128.91 samples/sec Loss 11.4366 Epoch: 5 Global Step: 54875 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:27:06,021-Speed 127.63 samples/sec Loss 11.2245 Epoch: 5 Global Step: 54880 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:27:26,297-Speed 126.27 samples/sec Loss 11.0903 Epoch: 5 Global Step: 54885 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:27:46,121-Speed 129.14 samples/sec Loss 11.4886 Epoch: 5 Global Step: 54890 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:28:06,080-Speed 128.27 samples/sec Loss 11.2712 Epoch: 5 Global Step: 54895 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:28:25,903-Speed 129.15 samples/sec Loss 11.0760 Epoch: 5 Global Step: 54900 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:28:45,996-Speed 127.41 samples/sec Loss 11.2292 Epoch: 5 Global Step: 54905 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:29:05,966-Speed 128.20 samples/sec Loss 11.3514 Epoch: 5 Global Step: 54910 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:29:25,689-Speed 129.80 samples/sec Loss 11.0569 Epoch: 5 Global Step: 54915 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:29:45,750-Speed 127.62 samples/sec Loss 11.0013 Epoch: 5 Global Step: 54920 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:30:05,595-Speed 129.01 samples/sec Loss 11.2819 Epoch: 5 Global Step: 54925 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:30:25,785-Speed 126.80 samples/sec Loss 10.9011 Epoch: 5 Global Step: 54930 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:30:45,642-Speed 128.93 samples/sec Loss 11.1157 Epoch: 5 Global Step: 54935 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:31:05,515-Speed 128.82 samples/sec Loss 11.3486 Epoch: 5 Global Step: 54940 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:31:25,409-Speed 128.69 samples/sec Loss 10.9050 Epoch: 5 Global Step: 54945 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:31:45,644-Speed 126.52 samples/sec Loss 11.2194 Epoch: 5 Global Step: 54950 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:32:05,476-Speed 129.09 samples/sec Loss 10.9935 Epoch: 5 Global Step: 54955 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:32:25,459-Speed 128.12 samples/sec Loss 11.3498 Epoch: 5 Global Step: 54960 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:32:45,355-Speed 128.68 samples/sec Loss 11.0849 Epoch: 5 Global Step: 54965 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:33:05,389-Speed 127.79 samples/sec Loss 11.0083 Epoch: 5 Global Step: 54970 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:33:25,544-Speed 127.03 samples/sec Loss 11.2830 Epoch: 5 Global Step: 54975 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:33:45,468-Speed 128.49 samples/sec Loss 11.5454 Epoch: 5 Global Step: 54980 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:34:05,356-Speed 128.73 samples/sec Loss 11.1237 Epoch: 5 Global Step: 54985 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:34:25,357-Speed 128.00 samples/sec Loss 11.0074 Epoch: 5 Global Step: 54990 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:34:45,386-Speed 127.82 samples/sec Loss 11.4097 Epoch: 5 Global Step: 54995 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:35:05,322-Speed 128.42 samples/sec Loss 11.0583 Epoch: 5 Global Step: 55000 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:35:25,297-Speed 128.16 samples/sec Loss 11.5108 Epoch: 5 Global Step: 55005 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:35:45,130-Speed 129.09 samples/sec Loss 11.3910 Epoch: 5 Global Step: 55010 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:36:05,035-Speed 128.62 samples/sec Loss 11.1098 Epoch: 5 Global Step: 55015 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:36:25,241-Speed 126.70 samples/sec Loss 11.0309 Epoch: 5 Global Step: 55020 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:36:45,122-Speed 128.77 samples/sec Loss 11.4579 Epoch: 5 Global Step: 55025 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:37:05,004-Speed 128.76 samples/sec Loss 11.4727 Epoch: 5 Global Step: 55030 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:37:24,938-Speed 128.43 samples/sec Loss 11.2161 Epoch: 5 Global Step: 55035 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:37:44,990-Speed 127.67 samples/sec Loss 11.4919 Epoch: 5 Global Step: 55040 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:38:05,197-Speed 126.70 samples/sec Loss 11.0503 Epoch: 5 Global Step: 55045 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:38:24,978-Speed 129.43 samples/sec Loss 11.2278 Epoch: 5 Global Step: 55050 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:38:44,893-Speed 128.55 samples/sec Loss 11.2647 Epoch: 5 Global Step: 55055 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:39:04,689-Speed 129.32 samples/sec Loss 11.0610 Epoch: 5 Global Step: 55060 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:39:24,867-Speed 126.88 samples/sec Loss 11.0483 Epoch: 5 Global Step: 55065 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:39:44,689-Speed 129.15 samples/sec Loss 11.0007 Epoch: 5 Global Step: 55070 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:40:04,938-Speed 126.43 samples/sec Loss 10.9082 Epoch: 5 Global Step: 55075 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:40:24,753-Speed 129.20 samples/sec Loss 11.0646 Epoch: 5 Global Step: 55080 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:40:45,125-Speed 125.67 samples/sec Loss 10.9555 Epoch: 5 Global Step: 55085 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:41:04,969-Speed 129.02 samples/sec Loss 11.2143 Epoch: 5 Global Step: 55090 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:41:24,746-Speed 129.45 samples/sec Loss 11.3833 Epoch: 5 Global Step: 55095 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:41:44,753-Speed 127.97 samples/sec Loss 11.4393 Epoch: 5 Global Step: 55100 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:42:04,588-Speed 129.07 samples/sec Loss 11.1711 Epoch: 5 Global Step: 55105 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:42:24,972-Speed 125.60 samples/sec Loss 11.1738 Epoch: 5 Global Step: 55110 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:42:44,993-Speed 127.87 samples/sec Loss 11.0054 Epoch: 5 Global Step: 55115 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:43:04,768-Speed 129.47 samples/sec Loss 11.1969 Epoch: 5 Global Step: 55120 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:43:24,511-Speed 129.66 samples/sec Loss 11.7080 Epoch: 5 Global Step: 55125 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:43:44,717-Speed 126.70 samples/sec Loss 11.4845 Epoch: 5 Global Step: 55130 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:44:04,540-Speed 129.15 samples/sec Loss 11.2400 Epoch: 5 Global Step: 55135 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:44:24,536-Speed 128.03 samples/sec Loss 11.0754 Epoch: 5 Global Step: 55140 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:44:44,378-Speed 129.02 samples/sec Loss 11.0552 Epoch: 5 Global Step: 55145 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:45:04,692-Speed 126.03 samples/sec Loss 11.4052 Epoch: 5 Global Step: 55150 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:45:24,787-Speed 127.40 samples/sec Loss 11.2606 Epoch: 5 Global Step: 55155 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:45:44,764-Speed 128.15 samples/sec Loss 11.2473 Epoch: 5 Global Step: 55160 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:46:04,485-Speed 129.81 samples/sec Loss 11.3296 Epoch: 5 Global Step: 55165 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:46:24,557-Speed 127.55 samples/sec Loss 11.4464 Epoch: 5 Global Step: 55170 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:46:44,476-Speed 128.53 samples/sec Loss 10.8956 Epoch: 5 Global Step: 55175 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:47:04,273-Speed 129.33 samples/sec Loss 10.9864 Epoch: 5 Global Step: 55180 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:47:24,157-Speed 128.75 samples/sec Loss 11.2677 Epoch: 5 Global Step: 55185 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:47:43,970-Speed 129.21 samples/sec Loss 10.7675 Epoch: 5 Global Step: 55190 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:48:04,167-Speed 126.76 samples/sec Loss 11.1010 Epoch: 5 Global Step: 55195 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:48:24,353-Speed 126.83 samples/sec Loss 11.1070 Epoch: 5 Global Step: 55200 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:48:44,244-Speed 128.71 samples/sec Loss 11.1961 Epoch: 5 Global Step: 55205 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:49:04,174-Speed 128.45 samples/sec Loss 11.1016 Epoch: 5 Global Step: 55210 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:49:24,064-Speed 128.71 samples/sec Loss 10.9107 Epoch: 5 Global Step: 55215 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:49:44,178-Speed 127.28 samples/sec Loss 11.1921 Epoch: 5 Global Step: 55220 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:50:04,180-Speed 127.99 samples/sec Loss 11.1754 Epoch: 5 Global Step: 55225 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:50:24,019-Speed 129.04 samples/sec Loss 10.8437 Epoch: 5 Global Step: 55230 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:50:44,013-Speed 128.05 samples/sec Loss 11.3505 Epoch: 5 Global Step: 55235 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:51:04,296-Speed 126.22 samples/sec Loss 10.9892 Epoch: 5 Global Step: 55240 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:51:24,499-Speed 126.72 samples/sec Loss 10.9518 Epoch: 5 Global Step: 55245 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:51:44,389-Speed 128.71 samples/sec Loss 11.1746 Epoch: 5 Global Step: 55250 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:52:04,392-Speed 127.98 samples/sec Loss 10.7908 Epoch: 5 Global Step: 55255 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:52:24,145-Speed 129.61 samples/sec Loss 10.9754 Epoch: 5 Global Step: 55260 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:52:44,317-Speed 126.91 samples/sec Loss 11.3242 Epoch: 5 Global Step: 55265 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:53:04,205-Speed 128.73 samples/sec Loss 11.2962 Epoch: 5 Global Step: 55270 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:53:23,919-Speed 129.86 samples/sec Loss 11.3099 Epoch: 5 Global Step: 55275 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:53:43,908-Speed 128.08 samples/sec Loss 11.2651 Epoch: 5 Global Step: 55280 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:54:03,744-Speed 129.07 samples/sec Loss 11.1884 Epoch: 5 Global Step: 55285 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:54:23,988-Speed 126.46 samples/sec Loss 11.1351 Epoch: 5 Global Step: 55290 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:54:43,977-Speed 128.08 samples/sec Loss 10.9111 Epoch: 5 Global Step: 55295 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:55:03,695-Speed 129.84 samples/sec Loss 11.1265 Epoch: 5 Global Step: 55300 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:55:23,538-Speed 129.02 samples/sec Loss 11.2257 Epoch: 5 Global Step: 55305 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:55:43,868-Speed 125.92 samples/sec Loss 11.2403 Epoch: 5 Global Step: 55310 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:56:03,698-Speed 129.11 samples/sec Loss 11.3451 Epoch: 5 Global Step: 55315 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:56:23,603-Speed 128.62 samples/sec Loss 11.1928 Epoch: 5 Global Step: 55320 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:56:43,531-Speed 128.47 samples/sec Loss 11.0946 Epoch: 5 Global Step: 55325 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:57:03,385-Speed 128.95 samples/sec Loss 11.2258 Epoch: 5 Global Step: 55330 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:57:23,501-Speed 127.27 samples/sec Loss 11.3544 Epoch: 5 Global Step: 55335 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:57:43,522-Speed 127.87 samples/sec Loss 11.0572 Epoch: 5 Global Step: 55340 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:58:03,341-Speed 129.18 samples/sec Loss 10.8955 Epoch: 5 Global Step: 55345 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:58:23,368-Speed 127.84 samples/sec Loss 11.2006 Epoch: 5 Global Step: 55350 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:58:43,447-Speed 127.50 samples/sec Loss 11.5678 Epoch: 5 Global Step: 55355 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:59:03,351-Speed 128.63 samples/sec Loss 11.2670 Epoch: 5 Global Step: 55360 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:59:23,278-Speed 128.47 samples/sec Loss 11.2777 Epoch: 5 Global Step: 55365 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 12:59:43,100-Speed 129.16 samples/sec Loss 11.1534 Epoch: 5 Global Step: 55370 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:00:03,163-Speed 127.60 samples/sec Loss 10.9900 Epoch: 5 Global Step: 55375 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:00:23,275-Speed 127.30 samples/sec Loss 11.0390 Epoch: 5 Global Step: 55380 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:00:43,208-Speed 128.44 samples/sec Loss 11.2992 Epoch: 5 Global Step: 55385 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:01:03,111-Speed 128.63 samples/sec Loss 11.2125 Epoch: 5 Global Step: 55390 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:01:23,120-Speed 127.95 samples/sec Loss 11.2732 Epoch: 5 Global Step: 55395 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:01:43,156-Speed 127.77 samples/sec Loss 11.3060 Epoch: 5 Global Step: 55400 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:02:03,165-Speed 127.95 samples/sec Loss 11.3711 Epoch: 5 Global Step: 55405 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:02:23,005-Speed 129.04 samples/sec Loss 11.2836 Epoch: 5 Global Step: 55410 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:02:43,069-Speed 127.60 samples/sec Loss 10.9245 Epoch: 5 Global Step: 55415 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:03:02,784-Speed 129.86 samples/sec Loss 10.9964 Epoch: 5 Global Step: 55420 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:03:22,928-Speed 127.09 samples/sec Loss 11.1850 Epoch: 5 Global Step: 55425 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:03:42,752-Speed 129.14 samples/sec Loss 11.0015 Epoch: 5 Global Step: 55430 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:04:02,813-Speed 127.62 samples/sec Loss 11.0941 Epoch: 5 Global Step: 55435 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:04:22,548-Speed 129.72 samples/sec Loss 11.2012 Epoch: 5 Global Step: 55440 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:04:42,770-Speed 126.60 samples/sec Loss 11.2883 Epoch: 5 Global Step: 55445 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:05:02,614-Speed 129.02 samples/sec Loss 11.4781 Epoch: 5 Global Step: 55450 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:05:22,505-Speed 128.70 samples/sec Loss 11.1564 Epoch: 5 Global Step: 55455 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:05:42,361-Speed 128.94 samples/sec Loss 11.8389 Epoch: 5 Global Step: 55460 Fp16 Grad Scale: 8192 Required: 231 hours Training: 2021-06-09 13:06:02,201-Speed 129.04 samples/sec Loss 11.4849 Epoch: 5 Global Step: 55465 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:06:22,380-Speed 126.87 samples/sec Loss 11.2727 Epoch: 5 Global Step: 55470 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:06:42,456-Speed 127.52 samples/sec Loss 11.2643 Epoch: 5 Global Step: 55475 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:07:02,210-Speed 129.60 samples/sec Loss 11.8156 Epoch: 5 Global Step: 55480 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:07:21,972-Speed 129.55 samples/sec Loss 11.4187 Epoch: 5 Global Step: 55485 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:07:42,183-Speed 126.67 samples/sec Loss 11.2696 Epoch: 5 Global Step: 55490 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:08:02,072-Speed 128.72 samples/sec Loss 11.1416 Epoch: 5 Global Step: 55495 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:08:21,972-Speed 128.66 samples/sec Loss 11.1676 Epoch: 5 Global Step: 55500 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:08:41,693-Speed 129.82 samples/sec Loss 11.3315 Epoch: 5 Global Step: 55505 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:09:01,536-Speed 129.02 samples/sec Loss 11.4000 Epoch: 5 Global Step: 55510 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:09:21,694-Speed 127.00 samples/sec Loss 10.7218 Epoch: 5 Global Step: 55515 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:09:41,873-Speed 126.87 samples/sec Loss 11.2038 Epoch: 5 Global Step: 55520 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:10:01,571-Speed 129.97 samples/sec Loss 11.5260 Epoch: 5 Global Step: 55525 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:10:21,515-Speed 128.36 samples/sec Loss 10.9752 Epoch: 5 Global Step: 55530 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:10:41,680-Speed 126.96 samples/sec Loss 11.1039 Epoch: 5 Global Step: 55535 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:11:01,632-Speed 128.31 samples/sec Loss 10.9174 Epoch: 5 Global Step: 55540 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:11:21,335-Speed 129.94 samples/sec Loss 11.3473 Epoch: 5 Global Step: 55545 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:11:41,174-Speed 129.05 samples/sec Loss 11.1055 Epoch: 5 Global Step: 55550 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:12:01,197-Speed 127.86 samples/sec Loss 11.1104 Epoch: 5 Global Step: 55555 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:12:21,446-Speed 126.43 samples/sec Loss 11.0741 Epoch: 5 Global Step: 55560 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:12:41,180-Speed 129.73 samples/sec Loss 11.0416 Epoch: 5 Global Step: 55565 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:13:01,396-Speed 126.64 samples/sec Loss 11.2091 Epoch: 5 Global Step: 55570 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:13:21,229-Speed 129.08 samples/sec Loss 11.2516 Epoch: 5 Global Step: 55575 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:13:41,425-Speed 126.77 samples/sec Loss 10.9687 Epoch: 5 Global Step: 55580 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:14:01,284-Speed 128.92 samples/sec Loss 11.3451 Epoch: 5 Global Step: 55585 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:14:20,986-Speed 129.94 samples/sec Loss 11.1641 Epoch: 5 Global Step: 55590 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:14:40,972-Speed 128.10 samples/sec Loss 11.2177 Epoch: 5 Global Step: 55595 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:15:00,809-Speed 129.06 samples/sec Loss 11.2790 Epoch: 5 Global Step: 55600 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:15:20,880-Speed 127.56 samples/sec Loss 11.2977 Epoch: 5 Global Step: 55605 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:15:40,639-Speed 129.56 samples/sec Loss 11.1509 Epoch: 5 Global Step: 55610 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:16:00,526-Speed 128.73 samples/sec Loss 11.1130 Epoch: 5 Global Step: 55615 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:16:21,714-Speed 120.83 samples/sec Loss 11.3592 Epoch: 5 Global Step: 55620 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:16:41,924-Speed 126.68 samples/sec Loss 11.2654 Epoch: 5 Global Step: 55625 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:17:01,615-Speed 130.01 samples/sec Loss 11.4056 Epoch: 5 Global Step: 55630 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:17:21,472-Speed 128.93 samples/sec Loss 11.0589 Epoch: 5 Global Step: 55635 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:17:41,496-Speed 127.86 samples/sec Loss 11.0239 Epoch: 5 Global Step: 55640 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:18:01,327-Speed 129.10 samples/sec Loss 11.1783 Epoch: 5 Global Step: 55645 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:18:21,386-Speed 127.63 samples/sec Loss 10.8465 Epoch: 5 Global Step: 55650 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:18:41,287-Speed 128.65 samples/sec Loss 11.1922 Epoch: 5 Global Step: 55655 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:19:01,105-Speed 129.18 samples/sec Loss 11.5117 Epoch: 5 Global Step: 55660 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:19:20,998-Speed 128.70 samples/sec Loss 11.1147 Epoch: 5 Global Step: 55665 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:19:41,127-Speed 127.19 samples/sec Loss 11.0604 Epoch: 5 Global Step: 55670 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:20:00,863-Speed 129.72 samples/sec Loss 11.2032 Epoch: 5 Global Step: 55675 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:20:20,941-Speed 127.51 samples/sec Loss 11.0255 Epoch: 5 Global Step: 55680 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:20:40,797-Speed 128.94 samples/sec Loss 11.3876 Epoch: 5 Global Step: 55685 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:21:00,679-Speed 128.77 samples/sec Loss 11.2167 Epoch: 5 Global Step: 55690 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:21:20,751-Speed 127.55 samples/sec Loss 11.3092 Epoch: 5 Global Step: 55695 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:21:40,672-Speed 128.52 samples/sec Loss 11.2704 Epoch: 5 Global Step: 55700 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:22:00,588-Speed 128.54 samples/sec Loss 11.3323 Epoch: 5 Global Step: 55705 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:22:20,511-Speed 128.50 samples/sec Loss 10.9509 Epoch: 5 Global Step: 55710 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:22:40,958-Speed 125.20 samples/sec Loss 10.9982 Epoch: 5 Global Step: 55715 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:23:01,024-Speed 127.59 samples/sec Loss 10.7940 Epoch: 5 Global Step: 55720 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:23:20,851-Speed 129.13 samples/sec Loss 11.5088 Epoch: 5 Global Step: 55725 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:23:40,765-Speed 128.55 samples/sec Loss 11.1017 Epoch: 5 Global Step: 55730 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:24:00,474-Speed 129.90 samples/sec Loss 11.2853 Epoch: 5 Global Step: 55735 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:24:20,643-Speed 126.93 samples/sec Loss 11.5225 Epoch: 5 Global Step: 55740 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:24:40,587-Speed 128.37 samples/sec Loss 11.1308 Epoch: 5 Global Step: 55745 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:25:00,568-Speed 128.12 samples/sec Loss 10.8860 Epoch: 5 Global Step: 55750 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:25:20,289-Speed 129.82 samples/sec Loss 11.1640 Epoch: 5 Global Step: 55755 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:25:40,662-Speed 125.66 samples/sec Loss 10.9492 Epoch: 5 Global Step: 55760 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:26:00,460-Speed 129.31 samples/sec Loss 11.1074 Epoch: 5 Global Step: 55765 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:26:20,279-Speed 129.18 samples/sec Loss 11.1446 Epoch: 5 Global Step: 55770 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:26:40,228-Speed 128.34 samples/sec Loss 10.9360 Epoch: 5 Global Step: 55775 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:26:59,938-Speed 129.89 samples/sec Loss 10.8729 Epoch: 5 Global Step: 55780 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:27:20,192-Speed 126.40 samples/sec Loss 11.1019 Epoch: 5 Global Step: 55785 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:27:40,114-Speed 128.51 samples/sec Loss 11.1710 Epoch: 5 Global Step: 55790 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:27:59,953-Speed 129.04 samples/sec Loss 11.0147 Epoch: 5 Global Step: 55795 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:28:19,694-Speed 129.69 samples/sec Loss 11.0401 Epoch: 5 Global Step: 55800 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:28:40,028-Speed 125.90 samples/sec Loss 11.3426 Epoch: 5 Global Step: 55805 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:28:59,862-Speed 129.08 samples/sec Loss 11.0920 Epoch: 5 Global Step: 55810 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:29:19,861-Speed 128.02 samples/sec Loss 11.0637 Epoch: 5 Global Step: 55815 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:29:39,578-Speed 129.84 samples/sec Loss 11.2560 Epoch: 5 Global Step: 55820 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:29:59,473-Speed 128.68 samples/sec Loss 10.7150 Epoch: 5 Global Step: 55825 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:30:19,663-Speed 126.81 samples/sec Loss 11.1598 Epoch: 5 Global Step: 55830 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:30:39,536-Speed 128.82 samples/sec Loss 11.3032 Epoch: 5 Global Step: 55835 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:30:59,266-Speed 129.76 samples/sec Loss 10.9949 Epoch: 5 Global Step: 55840 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:31:19,137-Speed 128.84 samples/sec Loss 10.7997 Epoch: 5 Global Step: 55845 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:31:39,329-Speed 126.79 samples/sec Loss 11.4991 Epoch: 5 Global Step: 55850 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:31:59,327-Speed 128.01 samples/sec Loss 11.1403 Epoch: 5 Global Step: 55855 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:32:19,017-Speed 130.03 samples/sec Loss 11.2317 Epoch: 5 Global Step: 55860 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:32:38,868-Speed 128.97 samples/sec Loss 10.9460 Epoch: 5 Global Step: 55865 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:32:58,835-Speed 128.22 samples/sec Loss 11.4528 Epoch: 5 Global Step: 55870 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:33:18,956-Speed 127.23 samples/sec Loss 11.0171 Epoch: 5 Global Step: 55875 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:33:38,766-Speed 129.23 samples/sec Loss 10.9301 Epoch: 5 Global Step: 55880 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:33:58,516-Speed 129.63 samples/sec Loss 11.0338 Epoch: 5 Global Step: 55885 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:34:18,551-Speed 127.78 samples/sec Loss 11.1945 Epoch: 5 Global Step: 55890 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:34:38,570-Speed 127.88 samples/sec Loss 11.3744 Epoch: 5 Global Step: 55895 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:34:58,554-Speed 128.11 samples/sec Loss 11.2000 Epoch: 5 Global Step: 55900 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:35:18,325-Speed 129.49 samples/sec Loss 11.2574 Epoch: 5 Global Step: 55905 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:35:38,380-Speed 127.66 samples/sec Loss 10.9570 Epoch: 5 Global Step: 55910 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:35:58,240-Speed 128.91 samples/sec Loss 10.8507 Epoch: 5 Global Step: 55915 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:36:18,424-Speed 126.84 samples/sec Loss 11.1348 Epoch: 5 Global Step: 55920 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:36:38,119-Speed 129.99 samples/sec Loss 11.4158 Epoch: 5 Global Step: 55925 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:36:58,109-Speed 128.07 samples/sec Loss 10.9799 Epoch: 5 Global Step: 55930 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:37:17,957-Speed 128.98 samples/sec Loss 11.1433 Epoch: 5 Global Step: 55935 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:37:38,249-Speed 126.17 samples/sec Loss 11.2694 Epoch: 5 Global Step: 55940 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:37:57,973-Speed 129.80 samples/sec Loss 11.2625 Epoch: 5 Global Step: 55945 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:38:17,823-Speed 128.98 samples/sec Loss 11.1910 Epoch: 5 Global Step: 55950 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:38:37,828-Speed 127.98 samples/sec Loss 11.3329 Epoch: 5 Global Step: 55955 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:38:57,573-Speed 129.66 samples/sec Loss 11.0818 Epoch: 5 Global Step: 55960 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:39:17,734-Speed 126.98 samples/sec Loss 10.8590 Epoch: 5 Global Step: 55965 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:39:37,470-Speed 129.72 samples/sec Loss 10.8490 Epoch: 5 Global Step: 55970 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:39:57,526-Speed 127.65 samples/sec Loss 11.3700 Epoch: 5 Global Step: 55975 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:40:17,337-Speed 129.23 samples/sec Loss 11.3740 Epoch: 5 Global Step: 55980 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:40:38,328-Speed 121.96 samples/sec Loss 11.3347 Epoch: 5 Global Step: 55985 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:40:58,272-Speed 128.36 samples/sec Loss 11.3063 Epoch: 5 Global Step: 55990 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:41:18,189-Speed 128.54 samples/sec Loss 11.1100 Epoch: 5 Global Step: 55995 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:41:38,015-Speed 129.13 samples/sec Loss 11.0787 Epoch: 5 Global Step: 56000 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:41:57,870-Speed 128.94 samples/sec Loss 10.8568 Epoch: 5 Global Step: 56005 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:42:17,896-Speed 127.84 samples/sec Loss 11.2791 Epoch: 5 Global Step: 56010 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:42:37,979-Speed 127.48 samples/sec Loss 11.2804 Epoch: 5 Global Step: 56015 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:42:57,805-Speed 129.13 samples/sec Loss 11.2988 Epoch: 5 Global Step: 56020 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:43:17,815-Speed 127.94 samples/sec Loss 11.1810 Epoch: 5 Global Step: 56025 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:43:37,717-Speed 128.63 samples/sec Loss 11.1785 Epoch: 5 Global Step: 56030 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:43:57,626-Speed 128.60 samples/sec Loss 11.2163 Epoch: 5 Global Step: 56035 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:44:17,614-Speed 128.08 samples/sec Loss 11.3858 Epoch: 5 Global Step: 56040 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:44:37,379-Speed 129.53 samples/sec Loss 11.0822 Epoch: 5 Global Step: 56045 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:44:57,271-Speed 128.70 samples/sec Loss 10.9865 Epoch: 5 Global Step: 56050 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:45:17,766-Speed 124.92 samples/sec Loss 11.0747 Epoch: 5 Global Step: 56055 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:45:37,682-Speed 128.55 samples/sec Loss 11.0452 Epoch: 5 Global Step: 56060 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:45:57,524-Speed 129.02 samples/sec Loss 11.1140 Epoch: 5 Global Step: 56065 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:46:17,437-Speed 128.57 samples/sec Loss 11.0888 Epoch: 5 Global Step: 56070 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:46:37,529-Speed 127.42 samples/sec Loss 11.1192 Epoch: 5 Global Step: 56075 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:46:57,528-Speed 128.01 samples/sec Loss 11.3771 Epoch: 5 Global Step: 56080 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:47:17,343-Speed 129.21 samples/sec Loss 11.0076 Epoch: 5 Global Step: 56085 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:47:37,243-Speed 128.65 samples/sec Loss 11.0273 Epoch: 5 Global Step: 56090 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:47:57,256-Speed 127.93 samples/sec Loss 11.2874 Epoch: 5 Global Step: 56095 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:48:17,485-Speed 126.56 samples/sec Loss 10.8186 Epoch: 5 Global Step: 56100 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:48:37,329-Speed 129.01 samples/sec Loss 11.3473 Epoch: 5 Global Step: 56105 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:48:57,181-Speed 128.96 samples/sec Loss 11.2984 Epoch: 5 Global Step: 56110 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:49:17,220-Speed 127.76 samples/sec Loss 11.0747 Epoch: 5 Global Step: 56115 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:49:37,179-Speed 128.27 samples/sec Loss 10.9357 Epoch: 5 Global Step: 56120 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:49:57,160-Speed 128.13 samples/sec Loss 11.2591 Epoch: 5 Global Step: 56125 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:50:16,867-Speed 129.91 samples/sec Loss 11.3857 Epoch: 5 Global Step: 56130 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:50:36,848-Speed 128.12 samples/sec Loss 11.3299 Epoch: 5 Global Step: 56135 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:50:56,673-Speed 129.14 samples/sec Loss 10.9907 Epoch: 5 Global Step: 56140 Fp16 Grad Scale: 8192 Required: 230 hours Training: 2021-06-09 13:51:16,999-Speed 125.95 samples/sec Loss 11.4760 Epoch: 5 Global Step: 56145 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:51:36,831-Speed 129.09 samples/sec Loss 11.4703 Epoch: 5 Global Step: 56150 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:51:56,878-Speed 127.71 samples/sec Loss 11.1153 Epoch: 5 Global Step: 56155 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:52:16,756-Speed 128.79 samples/sec Loss 11.0850 Epoch: 5 Global Step: 56160 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:52:36,946-Speed 126.80 samples/sec Loss 11.0814 Epoch: 5 Global Step: 56165 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:52:56,758-Speed 129.22 samples/sec Loss 10.9066 Epoch: 5 Global Step: 56170 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:53:16,907-Speed 127.06 samples/sec Loss 11.2761 Epoch: 5 Global Step: 56175 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:53:36,899-Speed 128.06 samples/sec Loss 11.4563 Epoch: 5 Global Step: 56180 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:53:57,042-Speed 127.10 samples/sec Loss 10.9585 Epoch: 5 Global Step: 56185 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:54:17,137-Speed 127.40 samples/sec Loss 11.2330 Epoch: 5 Global Step: 56190 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:54:37,099-Speed 128.25 samples/sec Loss 11.2725 Epoch: 5 Global Step: 56195 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:54:56,987-Speed 128.73 samples/sec Loss 11.1708 Epoch: 5 Global Step: 56200 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:55:16,872-Speed 128.74 samples/sec Loss 11.2742 Epoch: 5 Global Step: 56205 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:55:36,976-Speed 127.34 samples/sec Loss 10.9516 Epoch: 5 Global Step: 56210 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:55:57,024-Speed 127.70 samples/sec Loss 11.1824 Epoch: 5 Global Step: 56215 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:56:16,736-Speed 129.88 samples/sec Loss 10.8616 Epoch: 5 Global Step: 56220 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:56:36,581-Speed 129.00 samples/sec Loss 11.2409 Epoch: 5 Global Step: 56225 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:56:56,546-Speed 128.23 samples/sec Loss 11.3502 Epoch: 5 Global Step: 56230 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:57:16,721-Speed 126.90 samples/sec Loss 11.2638 Epoch: 5 Global Step: 56235 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:57:36,599-Speed 128.79 samples/sec Loss 11.2116 Epoch: 5 Global Step: 56240 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:57:56,380-Speed 129.42 samples/sec Loss 11.1582 Epoch: 5 Global Step: 56245 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:58:16,332-Speed 128.32 samples/sec Loss 11.1330 Epoch: 5 Global Step: 56250 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:58:36,508-Speed 126.89 samples/sec Loss 11.4410 Epoch: 5 Global Step: 56255 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:58:56,361-Speed 128.95 samples/sec Loss 11.5697 Epoch: 5 Global Step: 56260 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:59:16,164-Speed 129.28 samples/sec Loss 10.8399 Epoch: 5 Global Step: 56265 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:59:36,182-Speed 127.89 samples/sec Loss 10.9471 Epoch: 5 Global Step: 56270 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 13:59:55,988-Speed 129.26 samples/sec Loss 11.3190 Epoch: 5 Global Step: 56275 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:00:16,234-Speed 126.45 samples/sec Loss 11.1855 Epoch: 5 Global Step: 56280 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:00:35,990-Speed 129.58 samples/sec Loss 10.9545 Epoch: 5 Global Step: 56285 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:00:55,957-Speed 128.21 samples/sec Loss 11.2446 Epoch: 5 Global Step: 56290 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:01:15,731-Speed 129.47 samples/sec Loss 11.1485 Epoch: 5 Global Step: 56295 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:01:36,029-Speed 126.13 samples/sec Loss 10.9376 Epoch: 5 Global Step: 56300 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:01:55,736-Speed 129.91 samples/sec Loss 11.1496 Epoch: 5 Global Step: 56305 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:02:15,864-Speed 127.20 samples/sec Loss 10.9570 Epoch: 5 Global Step: 56310 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:02:35,551-Speed 130.04 samples/sec Loss 11.1765 Epoch: 5 Global Step: 56315 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:02:55,411-Speed 128.91 samples/sec Loss 11.0464 Epoch: 5 Global Step: 56320 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:03:15,484-Speed 127.56 samples/sec Loss 11.3683 Epoch: 5 Global Step: 56325 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:03:35,391-Speed 128.60 samples/sec Loss 11.3402 Epoch: 5 Global Step: 56330 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:03:55,210-Speed 129.17 samples/sec Loss 11.3993 Epoch: 5 Global Step: 56335 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:04:15,036-Speed 129.13 samples/sec Loss 11.1318 Epoch: 5 Global Step: 56340 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:04:35,193-Speed 127.01 samples/sec Loss 11.1550 Epoch: 5 Global Step: 56345 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:04:54,986-Speed 129.35 samples/sec Loss 11.0539 Epoch: 5 Global Step: 56350 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:05:15,120-Speed 127.16 samples/sec Loss 11.1580 Epoch: 5 Global Step: 56355 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:05:34,968-Speed 128.99 samples/sec Loss 11.3632 Epoch: 5 Global Step: 56360 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:05:54,887-Speed 128.52 samples/sec Loss 11.0588 Epoch: 5 Global Step: 56365 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:06:14,894-Speed 127.97 samples/sec Loss 11.4537 Epoch: 5 Global Step: 56370 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:06:34,851-Speed 128.28 samples/sec Loss 11.4409 Epoch: 5 Global Step: 56375 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:06:54,604-Speed 129.60 samples/sec Loss 10.9820 Epoch: 5 Global Step: 56380 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:07:14,655-Speed 127.68 samples/sec Loss 11.3682 Epoch: 5 Global Step: 56385 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:07:34,566-Speed 128.58 samples/sec Loss 11.1848 Epoch: 5 Global Step: 56390 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:07:54,568-Speed 127.99 samples/sec Loss 11.2862 Epoch: 5 Global Step: 56395 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:08:14,435-Speed 128.86 samples/sec Loss 11.1548 Epoch: 5 Global Step: 56400 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:08:34,308-Speed 128.83 samples/sec Loss 11.2222 Epoch: 5 Global Step: 56405 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:08:54,985-Speed 123.82 samples/sec Loss 10.9738 Epoch: 5 Global Step: 56410 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:09:15,158-Speed 126.91 samples/sec Loss 10.9158 Epoch: 5 Global Step: 56415 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:09:34,981-Speed 129.15 samples/sec Loss 10.8673 Epoch: 5 Global Step: 56420 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:09:54,879-Speed 128.66 samples/sec Loss 11.2827 Epoch: 5 Global Step: 56425 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:10:14,800-Speed 128.51 samples/sec Loss 11.0582 Epoch: 5 Global Step: 56430 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:10:34,862-Speed 127.61 samples/sec Loss 10.9162 Epoch: 5 Global Step: 56435 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:10:54,822-Speed 128.26 samples/sec Loss 10.8720 Epoch: 5 Global Step: 56440 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:11:14,701-Speed 128.79 samples/sec Loss 11.3016 Epoch: 5 Global Step: 56445 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:11:34,601-Speed 128.66 samples/sec Loss 11.0945 Epoch: 5 Global Step: 56450 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:11:54,406-Speed 129.26 samples/sec Loss 11.3611 Epoch: 5 Global Step: 56455 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:12:14,583-Speed 126.89 samples/sec Loss 11.0137 Epoch: 5 Global Step: 56460 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:12:34,444-Speed 128.90 samples/sec Loss 11.1567 Epoch: 5 Global Step: 56465 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:12:54,481-Speed 127.77 samples/sec Loss 11.0564 Epoch: 5 Global Step: 56470 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:13:14,188-Speed 129.91 samples/sec Loss 11.4497 Epoch: 5 Global Step: 56475 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:13:34,385-Speed 126.75 samples/sec Loss 11.4926 Epoch: 5 Global Step: 56480 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:13:54,269-Speed 128.75 samples/sec Loss 11.4836 Epoch: 5 Global Step: 56485 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:14:14,773-Speed 124.86 samples/sec Loss 11.3365 Epoch: 5 Global Step: 56490 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:14:34,597-Speed 129.15 samples/sec Loss 10.9175 Epoch: 5 Global Step: 56495 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:14:54,456-Speed 128.91 samples/sec Loss 10.9895 Epoch: 5 Global Step: 56500 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:15:14,600-Speed 127.09 samples/sec Loss 11.6156 Epoch: 5 Global Step: 56505 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:15:34,689-Speed 127.44 samples/sec Loss 11.4537 Epoch: 5 Global Step: 56510 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:15:54,449-Speed 129.56 samples/sec Loss 10.8599 Epoch: 5 Global Step: 56515 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:16:14,462-Speed 127.92 samples/sec Loss 11.0827 Epoch: 5 Global Step: 56520 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:16:34,580-Speed 127.26 samples/sec Loss 11.1961 Epoch: 5 Global Step: 56525 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:16:54,385-Speed 129.27 samples/sec Loss 11.3954 Epoch: 5 Global Step: 56530 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:17:14,271-Speed 128.74 samples/sec Loss 11.2626 Epoch: 5 Global Step: 56535 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:17:34,092-Speed 129.17 samples/sec Loss 11.0255 Epoch: 5 Global Step: 56540 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:17:53,986-Speed 128.69 samples/sec Loss 11.2092 Epoch: 5 Global Step: 56545 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:18:14,226-Speed 126.49 samples/sec Loss 10.8822 Epoch: 5 Global Step: 56550 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:18:34,066-Speed 129.04 samples/sec Loss 11.1998 Epoch: 5 Global Step: 56555 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:18:54,048-Speed 128.12 samples/sec Loss 11.1548 Epoch: 5 Global Step: 56560 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:19:14,048-Speed 128.01 samples/sec Loss 10.8770 Epoch: 5 Global Step: 56565 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:19:34,107-Speed 127.63 samples/sec Loss 10.9961 Epoch: 5 Global Step: 56570 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:19:53,996-Speed 128.72 samples/sec Loss 11.1644 Epoch: 5 Global Step: 56575 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:20:13,834-Speed 129.05 samples/sec Loss 10.9488 Epoch: 5 Global Step: 56580 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:20:33,857-Speed 127.86 samples/sec Loss 11.0693 Epoch: 5 Global Step: 56585 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:20:53,734-Speed 128.80 samples/sec Loss 11.2342 Epoch: 5 Global Step: 56590 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:21:13,803-Speed 127.56 samples/sec Loss 11.2866 Epoch: 5 Global Step: 56595 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:21:33,536-Speed 129.74 samples/sec Loss 10.9090 Epoch: 5 Global Step: 56600 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:21:53,500-Speed 128.24 samples/sec Loss 11.1393 Epoch: 5 Global Step: 56605 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:22:13,513-Speed 127.92 samples/sec Loss 11.1402 Epoch: 5 Global Step: 56610 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:22:33,688-Speed 126.90 samples/sec Loss 11.1509 Epoch: 5 Global Step: 56615 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:22:53,505-Speed 129.19 samples/sec Loss 11.0037 Epoch: 5 Global Step: 56620 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:23:13,386-Speed 128.77 samples/sec Loss 11.0918 Epoch: 5 Global Step: 56625 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:23:33,428-Speed 127.74 samples/sec Loss 11.2213 Epoch: 5 Global Step: 56630 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:23:53,143-Speed 129.86 samples/sec Loss 11.3772 Epoch: 5 Global Step: 56635 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:24:13,303-Speed 126.99 samples/sec Loss 10.9181 Epoch: 5 Global Step: 56640 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:24:33,086-Speed 129.41 samples/sec Loss 11.2704 Epoch: 5 Global Step: 56645 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:24:53,050-Speed 128.23 samples/sec Loss 11.0077 Epoch: 5 Global Step: 56650 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:25:12,882-Speed 129.09 samples/sec Loss 10.9653 Epoch: 5 Global Step: 56655 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:25:33,090-Speed 126.69 samples/sec Loss 10.6281 Epoch: 5 Global Step: 56660 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:25:53,149-Speed 127.63 samples/sec Loss 11.2319 Epoch: 5 Global Step: 56665 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:26:13,269-Speed 127.25 samples/sec Loss 11.1839 Epoch: 5 Global Step: 56670 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:26:33,092-Speed 129.15 samples/sec Loss 11.3708 Epoch: 5 Global Step: 56675 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:26:52,785-Speed 130.00 samples/sec Loss 11.4336 Epoch: 5 Global Step: 56680 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:27:12,919-Speed 127.15 samples/sec Loss 11.1025 Epoch: 5 Global Step: 56685 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:27:33,007-Speed 127.44 samples/sec Loss 11.0685 Epoch: 5 Global Step: 56690 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:27:52,893-Speed 128.74 samples/sec Loss 11.5185 Epoch: 5 Global Step: 56695 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:28:12,754-Speed 128.91 samples/sec Loss 10.7622 Epoch: 5 Global Step: 56700 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:28:32,989-Speed 126.52 samples/sec Loss 11.0633 Epoch: 5 Global Step: 56705 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:28:52,811-Speed 129.16 samples/sec Loss 11.2124 Epoch: 5 Global Step: 56710 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:29:12,758-Speed 128.35 samples/sec Loss 10.9995 Epoch: 5 Global Step: 56715 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:29:32,483-Speed 129.79 samples/sec Loss 11.0460 Epoch: 5 Global Step: 56720 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:29:52,452-Speed 128.21 samples/sec Loss 11.3323 Epoch: 5 Global Step: 56725 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:30:12,740-Speed 126.19 samples/sec Loss 11.3348 Epoch: 5 Global Step: 56730 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:30:32,748-Speed 127.95 samples/sec Loss 10.8624 Epoch: 5 Global Step: 56735 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:30:52,573-Speed 129.14 samples/sec Loss 11.2636 Epoch: 5 Global Step: 56740 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:31:12,462-Speed 128.72 samples/sec Loss 11.5119 Epoch: 5 Global Step: 56745 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:31:32,540-Speed 127.51 samples/sec Loss 11.0055 Epoch: 5 Global Step: 56750 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:31:53,228-Speed 123.75 samples/sec Loss 11.1718 Epoch: 5 Global Step: 56755 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:32:13,038-Speed 129.23 samples/sec Loss 11.0362 Epoch: 5 Global Step: 56760 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:32:32,881-Speed 129.02 samples/sec Loss 11.1406 Epoch: 5 Global Step: 56765 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:32:52,853-Speed 128.20 samples/sec Loss 11.3238 Epoch: 5 Global Step: 56770 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:33:13,005-Speed 127.04 samples/sec Loss 10.9321 Epoch: 5 Global Step: 56775 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:33:32,986-Speed 128.13 samples/sec Loss 11.2759 Epoch: 5 Global Step: 56780 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:33:52,989-Speed 127.99 samples/sec Loss 11.2862 Epoch: 5 Global Step: 56785 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:34:12,730-Speed 129.68 samples/sec Loss 11.2357 Epoch: 5 Global Step: 56790 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:34:32,923-Speed 126.79 samples/sec Loss 10.6830 Epoch: 5 Global Step: 56795 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:34:52,865-Speed 128.38 samples/sec Loss 10.9260 Epoch: 5 Global Step: 56800 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:35:12,662-Speed 129.32 samples/sec Loss 11.0399 Epoch: 5 Global Step: 56805 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:35:32,687-Speed 127.85 samples/sec Loss 11.3120 Epoch: 5 Global Step: 56810 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:35:52,411-Speed 129.80 samples/sec Loss 11.2319 Epoch: 5 Global Step: 56815 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:36:12,782-Speed 125.67 samples/sec Loss 10.8851 Epoch: 5 Global Step: 56820 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:36:32,792-Speed 127.95 samples/sec Loss 11.1227 Epoch: 5 Global Step: 56825 Fp16 Grad Scale: 8192 Required: 229 hours Training: 2021-06-09 14:36:52,701-Speed 128.59 samples/sec Loss 11.3493 Epoch: 5 Global Step: 56830 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:37:12,535-Speed 129.07 samples/sec Loss 10.8248 Epoch: 5 Global Step: 56835 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:37:32,826-Speed 126.17 samples/sec Loss 11.2321 Epoch: 5 Global Step: 56840 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:37:52,643-Speed 129.19 samples/sec Loss 11.2693 Epoch: 5 Global Step: 56845 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:38:12,599-Speed 128.29 samples/sec Loss 11.1748 Epoch: 5 Global Step: 56850 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:38:32,541-Speed 128.38 samples/sec Loss 11.1388 Epoch: 5 Global Step: 56855 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:38:52,354-Speed 129.21 samples/sec Loss 11.2797 Epoch: 5 Global Step: 56860 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:39:12,533-Speed 126.87 samples/sec Loss 10.9720 Epoch: 5 Global Step: 56865 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:39:32,543-Speed 127.94 samples/sec Loss 11.3505 Epoch: 5 Global Step: 56870 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:39:52,378-Speed 129.07 samples/sec Loss 11.2784 Epoch: 5 Global Step: 56875 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:40:12,244-Speed 128.87 samples/sec Loss 11.2880 Epoch: 5 Global Step: 56880 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:40:32,554-Speed 126.05 samples/sec Loss 11.4983 Epoch: 5 Global Step: 56885 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:40:52,354-Speed 129.30 samples/sec Loss 11.3115 Epoch: 5 Global Step: 56890 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:41:12,355-Speed 128.00 samples/sec Loss 11.0271 Epoch: 5 Global Step: 56895 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:41:32,193-Speed 129.06 samples/sec Loss 10.9786 Epoch: 5 Global Step: 56900 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:41:52,209-Speed 127.90 samples/sec Loss 11.0096 Epoch: 5 Global Step: 56905 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:42:12,240-Speed 127.81 samples/sec Loss 11.0953 Epoch: 5 Global Step: 56910 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:42:32,245-Speed 127.98 samples/sec Loss 11.4011 Epoch: 5 Global Step: 56915 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:42:52,025-Speed 129.43 samples/sec Loss 10.9371 Epoch: 5 Global Step: 56920 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:43:12,085-Speed 127.62 samples/sec Loss 10.7993 Epoch: 5 Global Step: 56925 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:43:32,116-Speed 127.81 samples/sec Loss 11.2741 Epoch: 5 Global Step: 56930 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:43:52,225-Speed 127.31 samples/sec Loss 11.4433 Epoch: 5 Global Step: 56935 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:44:12,055-Speed 129.10 samples/sec Loss 11.0724 Epoch: 5 Global Step: 56940 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:44:31,966-Speed 128.58 samples/sec Loss 11.1361 Epoch: 5 Global Step: 56945 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:44:51,906-Speed 128.39 samples/sec Loss 11.4598 Epoch: 5 Global Step: 56950 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:45:11,973-Speed 127.58 samples/sec Loss 10.9250 Epoch: 5 Global Step: 56955 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:45:31,911-Speed 128.41 samples/sec Loss 10.8674 Epoch: 5 Global Step: 56960 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:45:51,904-Speed 128.05 samples/sec Loss 11.1944 Epoch: 5 Global Step: 56965 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:46:11,898-Speed 128.04 samples/sec Loss 10.9101 Epoch: 5 Global Step: 56970 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:46:31,897-Speed 128.01 samples/sec Loss 11.0018 Epoch: 5 Global Step: 56975 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:46:51,912-Speed 127.91 samples/sec Loss 10.9605 Epoch: 5 Global Step: 56980 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:47:11,906-Speed 128.04 samples/sec Loss 11.1285 Epoch: 5 Global Step: 56985 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:47:31,737-Speed 129.10 samples/sec Loss 11.1392 Epoch: 5 Global Step: 56990 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:47:51,609-Speed 128.84 samples/sec Loss 11.3021 Epoch: 5 Global Step: 56995 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:48:11,886-Speed 126.26 samples/sec Loss 10.9086 Epoch: 5 Global Step: 57000 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:48:31,672-Speed 129.38 samples/sec Loss 11.1948 Epoch: 5 Global Step: 57005 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:48:51,740-Speed 127.57 samples/sec Loss 11.4477 Epoch: 5 Global Step: 57010 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:49:11,588-Speed 128.99 samples/sec Loss 11.4227 Epoch: 5 Global Step: 57015 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:49:31,697-Speed 127.31 samples/sec Loss 11.1647 Epoch: 5 Global Step: 57020 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:49:51,710-Speed 127.93 samples/sec Loss 11.3013 Epoch: 5 Global Step: 57025 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:50:11,453-Speed 129.68 samples/sec Loss 11.4144 Epoch: 5 Global Step: 57030 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:50:31,439-Speed 128.10 samples/sec Loss 11.1056 Epoch: 5 Global Step: 57035 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:50:51,249-Speed 129.23 samples/sec Loss 11.2176 Epoch: 5 Global Step: 57040 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:51:11,467-Speed 126.63 samples/sec Loss 11.3635 Epoch: 5 Global Step: 57045 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:51:31,395-Speed 128.46 samples/sec Loss 11.1697 Epoch: 5 Global Step: 57050 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:51:51,359-Speed 128.24 samples/sec Loss 10.9961 Epoch: 5 Global Step: 57055 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:52:11,172-Speed 129.21 samples/sec Loss 11.0913 Epoch: 5 Global Step: 57060 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:52:31,525-Speed 125.78 samples/sec Loss 11.2114 Epoch: 5 Global Step: 57065 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:52:51,351-Speed 129.13 samples/sec Loss 11.3745 Epoch: 5 Global Step: 57070 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:53:11,134-Speed 129.42 samples/sec Loss 11.4793 Epoch: 5 Global Step: 57075 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:53:31,184-Speed 127.68 samples/sec Loss 10.9691 Epoch: 5 Global Step: 57080 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:53:51,022-Speed 129.06 samples/sec Loss 11.2978 Epoch: 5 Global Step: 57085 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:54:11,257-Speed 126.52 samples/sec Loss 10.9132 Epoch: 5 Global Step: 57090 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:54:31,211-Speed 128.30 samples/sec Loss 11.2682 Epoch: 5 Global Step: 57095 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:54:51,036-Speed 129.14 samples/sec Loss 11.1095 Epoch: 5 Global Step: 57100 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:55:10,819-Speed 129.41 samples/sec Loss 10.8945 Epoch: 5 Global Step: 57105 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:55:31,086-Speed 126.31 samples/sec Loss 10.8160 Epoch: 5 Global Step: 57110 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:55:50,798-Speed 129.88 samples/sec Loss 10.9030 Epoch: 5 Global Step: 57115 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:56:10,542-Speed 129.67 samples/sec Loss 11.2397 Epoch: 5 Global Step: 57120 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:56:30,496-Speed 128.30 samples/sec Loss 11.2360 Epoch: 5 Global Step: 57125 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:56:50,405-Speed 128.59 samples/sec Loss 11.2239 Epoch: 5 Global Step: 57130 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:57:10,535-Speed 127.18 samples/sec Loss 10.9666 Epoch: 5 Global Step: 57135 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:57:30,536-Speed 128.00 samples/sec Loss 11.0196 Epoch: 5 Global Step: 57140 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:57:50,327-Speed 129.35 samples/sec Loss 11.1152 Epoch: 5 Global Step: 57145 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:58:10,350-Speed 127.86 samples/sec Loss 10.8519 Epoch: 5 Global Step: 57150 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:58:30,460-Speed 127.30 samples/sec Loss 10.7783 Epoch: 5 Global Step: 57155 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:58:50,179-Speed 129.83 samples/sec Loss 10.8501 Epoch: 5 Global Step: 57160 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:59:10,134-Speed 128.29 samples/sec Loss 11.1484 Epoch: 5 Global Step: 57165 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:59:29,952-Speed 129.19 samples/sec Loss 11.2811 Epoch: 5 Global Step: 57170 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 14:59:49,986-Speed 127.79 samples/sec Loss 11.3000 Epoch: 5 Global Step: 57175 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:00:09,911-Speed 128.49 samples/sec Loss 11.3963 Epoch: 5 Global Step: 57180 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:00:29,836-Speed 128.49 samples/sec Loss 10.9087 Epoch: 5 Global Step: 57185 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:00:49,644-Speed 129.25 samples/sec Loss 11.0076 Epoch: 5 Global Step: 57190 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:01:09,659-Speed 127.91 samples/sec Loss 11.0045 Epoch: 5 Global Step: 57195 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:01:29,707-Speed 127.70 samples/sec Loss 11.0191 Epoch: 5 Global Step: 57200 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:01:49,701-Speed 128.04 samples/sec Loss 10.6599 Epoch: 5 Global Step: 57205 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:02:09,647-Speed 128.35 samples/sec Loss 11.4047 Epoch: 5 Global Step: 57210 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:02:29,507-Speed 128.91 samples/sec Loss 11.1111 Epoch: 5 Global Step: 57215 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:02:49,452-Speed 128.36 samples/sec Loss 11.0952 Epoch: 5 Global Step: 57220 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:03:09,628-Speed 126.88 samples/sec Loss 11.0545 Epoch: 5 Global Step: 57225 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:03:29,445-Speed 129.19 samples/sec Loss 10.8859 Epoch: 5 Global Step: 57230 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:03:49,410-Speed 128.23 samples/sec Loss 11.1327 Epoch: 5 Global Step: 57235 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:04:09,301-Speed 128.71 samples/sec Loss 10.8112 Epoch: 5 Global Step: 57240 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:04:29,418-Speed 127.26 samples/sec Loss 11.3648 Epoch: 5 Global Step: 57245 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:04:49,412-Speed 128.05 samples/sec Loss 11.0388 Epoch: 5 Global Step: 57250 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:05:09,244-Speed 129.09 samples/sec Loss 11.0421 Epoch: 5 Global Step: 57255 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:05:29,306-Speed 127.61 samples/sec Loss 11.2846 Epoch: 5 Global Step: 57260 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:05:49,033-Speed 129.78 samples/sec Loss 11.4796 Epoch: 5 Global Step: 57265 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:06:09,263-Speed 126.55 samples/sec Loss 11.2730 Epoch: 5 Global Step: 57270 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:06:29,122-Speed 128.91 samples/sec Loss 10.9329 Epoch: 5 Global Step: 57275 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:06:49,079-Speed 128.28 samples/sec Loss 10.8090 Epoch: 5 Global Step: 57280 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:07:08,926-Speed 128.99 samples/sec Loss 10.7955 Epoch: 5 Global Step: 57285 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:07:29,198-Speed 126.29 samples/sec Loss 11.2656 Epoch: 5 Global Step: 57290 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:07:48,772-Speed 130.79 samples/sec Loss 10.5647 Epoch: 5 Global Step: 57295 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:08:09,244-Speed 125.06 samples/sec Loss 10.9914 Epoch: 5 Global Step: 57300 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:08:29,238-Speed 128.06 samples/sec Loss 10.8913 Epoch: 5 Global Step: 57305 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:08:49,043-Speed 129.26 samples/sec Loss 11.2212 Epoch: 5 Global Step: 57310 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:09:09,226-Speed 126.85 samples/sec Loss 11.2193 Epoch: 5 Global Step: 57315 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:09:29,281-Speed 127.66 samples/sec Loss 11.3445 Epoch: 5 Global Step: 57320 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:09:49,155-Speed 128.81 samples/sec Loss 11.3557 Epoch: 5 Global Step: 57325 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:10:08,859-Speed 129.93 samples/sec Loss 10.8722 Epoch: 5 Global Step: 57330 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:10:29,164-Speed 126.08 samples/sec Loss 10.8549 Epoch: 5 Global Step: 57335 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:10:48,981-Speed 129.19 samples/sec Loss 11.0223 Epoch: 5 Global Step: 57340 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:11:08,970-Speed 128.08 samples/sec Loss 11.0695 Epoch: 5 Global Step: 57345 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:11:28,764-Speed 129.33 samples/sec Loss 10.9573 Epoch: 5 Global Step: 57350 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:11:48,675-Speed 128.58 samples/sec Loss 11.3866 Epoch: 5 Global Step: 57355 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:12:08,788-Speed 127.29 samples/sec Loss 11.2333 Epoch: 5 Global Step: 57360 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:12:28,861-Speed 127.54 samples/sec Loss 11.2060 Epoch: 5 Global Step: 57365 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:12:48,560-Speed 129.96 samples/sec Loss 11.2899 Epoch: 5 Global Step: 57370 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:13:08,560-Speed 128.01 samples/sec Loss 11.0644 Epoch: 5 Global Step: 57375 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:13:28,609-Speed 127.69 samples/sec Loss 11.3614 Epoch: 5 Global Step: 57380 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:13:48,542-Speed 128.44 samples/sec Loss 11.3016 Epoch: 5 Global Step: 57385 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:14:08,479-Speed 128.41 samples/sec Loss 11.2920 Epoch: 5 Global Step: 57390 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:14:28,168-Speed 130.03 samples/sec Loss 11.0671 Epoch: 5 Global Step: 57395 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:14:48,095-Speed 128.48 samples/sec Loss 10.9478 Epoch: 5 Global Step: 57400 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:15:08,182-Speed 127.45 samples/sec Loss 11.3128 Epoch: 5 Global Step: 57405 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:15:28,039-Speed 128.93 samples/sec Loss 11.3439 Epoch: 5 Global Step: 57410 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:15:47,856-Speed 129.19 samples/sec Loss 11.3598 Epoch: 5 Global Step: 57415 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:16:07,780-Speed 128.49 samples/sec Loss 11.2163 Epoch: 5 Global Step: 57420 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:16:27,882-Speed 127.36 samples/sec Loss 11.2991 Epoch: 5 Global Step: 57425 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:16:47,983-Speed 127.36 samples/sec Loss 11.0865 Epoch: 5 Global Step: 57430 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:17:07,803-Speed 129.17 samples/sec Loss 11.3512 Epoch: 5 Global Step: 57435 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:17:27,774-Speed 128.19 samples/sec Loss 11.2622 Epoch: 5 Global Step: 57440 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:17:47,646-Speed 128.83 samples/sec Loss 11.5494 Epoch: 5 Global Step: 57445 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:18:07,778-Speed 127.17 samples/sec Loss 11.2728 Epoch: 5 Global Step: 57450 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:18:27,581-Speed 129.28 samples/sec Loss 11.1949 Epoch: 5 Global Step: 57455 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:18:47,598-Speed 127.89 samples/sec Loss 11.1067 Epoch: 5 Global Step: 57460 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:19:07,522-Speed 128.50 samples/sec Loss 11.0643 Epoch: 5 Global Step: 57465 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:19:27,715-Speed 126.78 samples/sec Loss 10.8782 Epoch: 5 Global Step: 57470 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:19:47,834-Speed 127.25 samples/sec Loss 11.0790 Epoch: 5 Global Step: 57475 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:20:07,546-Speed 129.88 samples/sec Loss 11.1576 Epoch: 5 Global Step: 57480 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:20:27,598-Speed 127.68 samples/sec Loss 10.9742 Epoch: 5 Global Step: 57485 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:20:47,406-Speed 129.25 samples/sec Loss 11.4743 Epoch: 5 Global Step: 57490 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:21:07,609-Speed 126.72 samples/sec Loss 11.2506 Epoch: 5 Global Step: 57495 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:21:27,601-Speed 128.06 samples/sec Loss 11.1412 Epoch: 5 Global Step: 57500 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:21:47,427-Speed 129.13 samples/sec Loss 11.1895 Epoch: 5 Global Step: 57505 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:22:07,307-Speed 128.78 samples/sec Loss 11.1686 Epoch: 5 Global Step: 57510 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:22:27,523-Speed 126.64 samples/sec Loss 10.9896 Epoch: 5 Global Step: 57515 Fp16 Grad Scale: 8192 Required: 228 hours Training: 2021-06-09 15:22:47,347-Speed 129.15 samples/sec Loss 11.0404 Epoch: 5 Global Step: 57520 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:23:07,345-Speed 128.02 samples/sec Loss 11.0253 Epoch: 5 Global Step: 57525 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:23:27,337-Speed 128.06 samples/sec Loss 11.3854 Epoch: 5 Global Step: 57530 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:23:47,239-Speed 128.63 samples/sec Loss 11.0596 Epoch: 5 Global Step: 57535 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:24:07,351-Speed 127.30 samples/sec Loss 11.1641 Epoch: 5 Global Step: 57540 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:24:27,269-Speed 128.53 samples/sec Loss 11.1737 Epoch: 5 Global Step: 57545 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:24:47,137-Speed 128.86 samples/sec Loss 11.2436 Epoch: 5 Global Step: 57550 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:25:07,149-Speed 127.93 samples/sec Loss 10.8764 Epoch: 5 Global Step: 57555 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:25:27,191-Speed 127.74 samples/sec Loss 11.1253 Epoch: 5 Global Step: 57560 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:25:47,083-Speed 128.70 samples/sec Loss 11.0601 Epoch: 5 Global Step: 57565 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:26:06,965-Speed 128.77 samples/sec Loss 11.1752 Epoch: 5 Global Step: 57570 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:26:26,813-Speed 128.99 samples/sec Loss 11.3145 Epoch: 5 Global Step: 57575 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:26:46,662-Speed 128.98 samples/sec Loss 10.8558 Epoch: 5 Global Step: 57580 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:27:06,852-Speed 126.80 samples/sec Loss 11.3748 Epoch: 5 Global Step: 57585 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:27:26,566-Speed 129.86 samples/sec Loss 11.5078 Epoch: 5 Global Step: 57590 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:27:46,548-Speed 128.12 samples/sec Loss 11.4073 Epoch: 5 Global Step: 57595 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:28:06,375-Speed 129.12 samples/sec Loss 11.1401 Epoch: 5 Global Step: 57600 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:28:26,443-Speed 127.57 samples/sec Loss 11.0245 Epoch: 5 Global Step: 57605 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:28:46,428-Speed 128.11 samples/sec Loss 10.7883 Epoch: 5 Global Step: 57610 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:29:06,310-Speed 128.77 samples/sec Loss 11.5329 Epoch: 5 Global Step: 57615 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:29:26,241-Speed 128.45 samples/sec Loss 11.6423 Epoch: 5 Global Step: 57620 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:29:46,112-Speed 128.84 samples/sec Loss 10.8913 Epoch: 5 Global Step: 57625 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:30:06,237-Speed 127.21 samples/sec Loss 11.2673 Epoch: 5 Global Step: 57630 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:30:26,150-Speed 128.56 samples/sec Loss 10.8397 Epoch: 5 Global Step: 57635 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:30:46,109-Speed 128.27 samples/sec Loss 10.9544 Epoch: 5 Global Step: 57640 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:31:05,821-Speed 129.87 samples/sec Loss 11.1140 Epoch: 5 Global Step: 57645 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:31:26,110-Speed 126.18 samples/sec Loss 11.3514 Epoch: 5 Global Step: 57650 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:31:45,972-Speed 128.90 samples/sec Loss 11.0419 Epoch: 5 Global Step: 57655 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:32:05,830-Speed 128.92 samples/sec Loss 11.2881 Epoch: 5 Global Step: 57660 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:32:25,789-Speed 128.28 samples/sec Loss 11.6071 Epoch: 5 Global Step: 57665 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:32:45,595-Speed 129.26 samples/sec Loss 11.1313 Epoch: 5 Global Step: 57670 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:33:05,746-Speed 127.05 samples/sec Loss 11.0803 Epoch: 5 Global Step: 57675 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:33:25,810-Speed 127.60 samples/sec Loss 11.4985 Epoch: 5 Global Step: 57680 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:33:45,549-Speed 129.70 samples/sec Loss 10.8668 Epoch: 5 Global Step: 57685 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:34:05,416-Speed 128.86 samples/sec Loss 11.3012 Epoch: 5 Global Step: 57690 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:34:25,717-Speed 126.11 samples/sec Loss 11.0400 Epoch: 5 Global Step: 57695 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:34:45,522-Speed 129.27 samples/sec Loss 10.9737 Epoch: 5 Global Step: 57700 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:35:05,412-Speed 128.71 samples/sec Loss 10.9788 Epoch: 5 Global Step: 57705 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:35:25,226-Speed 129.21 samples/sec Loss 11.2038 Epoch: 5 Global Step: 57710 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:35:45,167-Speed 128.38 samples/sec Loss 11.1700 Epoch: 5 Global Step: 57715 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:36:05,189-Speed 127.87 samples/sec Loss 11.1274 Epoch: 5 Global Step: 57720 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:36:25,188-Speed 128.01 samples/sec Loss 10.9581 Epoch: 5 Global Step: 57725 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:36:44,882-Speed 129.99 samples/sec Loss 11.2238 Epoch: 5 Global Step: 57730 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:37:04,885-Speed 127.99 samples/sec Loss 11.3158 Epoch: 5 Global Step: 57735 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:37:25,007-Speed 127.23 samples/sec Loss 10.6559 Epoch: 5 Global Step: 57740 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:37:44,971-Speed 128.24 samples/sec Loss 11.3186 Epoch: 5 Global Step: 57745 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:38:04,802-Speed 129.10 samples/sec Loss 11.4025 Epoch: 5 Global Step: 57750 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:38:24,653-Speed 128.97 samples/sec Loss 11.2637 Epoch: 5 Global Step: 57755 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:38:44,662-Speed 127.95 samples/sec Loss 11.2368 Epoch: 5 Global Step: 57760 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:39:04,841-Speed 126.87 samples/sec Loss 11.4289 Epoch: 5 Global Step: 57765 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:39:24,634-Speed 129.34 samples/sec Loss 11.3971 Epoch: 5 Global Step: 57770 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:39:44,615-Speed 128.13 samples/sec Loss 10.6417 Epoch: 5 Global Step: 57775 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:40:04,442-Speed 129.13 samples/sec Loss 10.9830 Epoch: 5 Global Step: 57780 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:40:24,599-Speed 127.01 samples/sec Loss 11.0501 Epoch: 5 Global Step: 57785 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:40:44,546-Speed 128.35 samples/sec Loss 11.1905 Epoch: 5 Global Step: 57790 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:41:04,320-Speed 129.46 samples/sec Loss 10.8402 Epoch: 5 Global Step: 57795 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:41:24,368-Speed 127.71 samples/sec Loss 11.0773 Epoch: 5 Global Step: 57800 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:41:44,172-Speed 129.27 samples/sec Loss 10.9266 Epoch: 5 Global Step: 57805 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:42:04,297-Speed 127.21 samples/sec Loss 11.2471 Epoch: 5 Global Step: 57810 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:42:24,164-Speed 128.86 samples/sec Loss 11.1151 Epoch: 5 Global Step: 57815 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:42:44,128-Speed 128.23 samples/sec Loss 11.3146 Epoch: 5 Global Step: 57820 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:43:03,985-Speed 128.93 samples/sec Loss 10.9207 Epoch: 5 Global Step: 57825 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:43:24,234-Speed 126.43 samples/sec Loss 10.9555 Epoch: 5 Global Step: 57830 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:43:44,052-Speed 129.19 samples/sec Loss 10.8755 Epoch: 5 Global Step: 57835 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:44:04,054-Speed 127.99 samples/sec Loss 11.2497 Epoch: 5 Global Step: 57840 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:44:23,915-Speed 128.90 samples/sec Loss 11.2327 Epoch: 5 Global Step: 57845 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:44:43,714-Speed 129.31 samples/sec Loss 11.2800 Epoch: 5 Global Step: 57850 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:45:03,836-Speed 127.23 samples/sec Loss 11.0489 Epoch: 5 Global Step: 57855 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:45:23,831-Speed 128.04 samples/sec Loss 10.8153 Epoch: 5 Global Step: 57860 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:45:43,723-Speed 128.70 samples/sec Loss 10.8975 Epoch: 5 Global Step: 57865 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:46:03,729-Speed 127.97 samples/sec Loss 11.2201 Epoch: 5 Global Step: 57870 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:46:23,701-Speed 128.18 samples/sec Loss 11.0710 Epoch: 5 Global Step: 57875 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:46:43,664-Speed 128.24 samples/sec Loss 11.3415 Epoch: 5 Global Step: 57880 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:47:03,710-Speed 127.71 samples/sec Loss 11.1048 Epoch: 5 Global Step: 57885 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:47:23,547-Speed 129.06 samples/sec Loss 11.0410 Epoch: 5 Global Step: 57890 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:47:43,501-Speed 128.30 samples/sec Loss 11.2779 Epoch: 5 Global Step: 57895 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:48:03,645-Speed 127.09 samples/sec Loss 11.1511 Epoch: 5 Global Step: 57900 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:48:23,532-Speed 128.73 samples/sec Loss 10.9804 Epoch: 5 Global Step: 57905 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:48:43,436-Speed 128.62 samples/sec Loss 11.5187 Epoch: 5 Global Step: 57910 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:49:03,258-Speed 129.16 samples/sec Loss 11.3995 Epoch: 5 Global Step: 57915 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:49:23,405-Speed 127.07 samples/sec Loss 11.3384 Epoch: 5 Global Step: 57920 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:49:43,417-Speed 127.93 samples/sec Loss 11.0142 Epoch: 5 Global Step: 57925 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:50:03,243-Speed 129.13 samples/sec Loss 11.2153 Epoch: 5 Global Step: 57930 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:50:23,143-Speed 128.65 samples/sec Loss 11.2024 Epoch: 5 Global Step: 57935 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:50:42,879-Speed 129.72 samples/sec Loss 11.1997 Epoch: 5 Global Step: 57940 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:51:03,124-Speed 126.45 samples/sec Loss 11.0495 Epoch: 5 Global Step: 57945 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:51:22,957-Speed 129.09 samples/sec Loss 10.9003 Epoch: 5 Global Step: 57950 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:51:42,934-Speed 128.15 samples/sec Loss 11.3216 Epoch: 5 Global Step: 57955 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:52:02,677-Speed 129.67 samples/sec Loss 11.6221 Epoch: 5 Global Step: 57960 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:52:22,913-Speed 126.51 samples/sec Loss 11.1836 Epoch: 5 Global Step: 57965 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:52:42,701-Speed 129.38 samples/sec Loss 10.9356 Epoch: 5 Global Step: 57970 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:53:02,555-Speed 128.95 samples/sec Loss 11.5680 Epoch: 5 Global Step: 57975 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:53:22,433-Speed 128.79 samples/sec Loss 11.2271 Epoch: 5 Global Step: 57980 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:53:42,347-Speed 128.57 samples/sec Loss 11.2045 Epoch: 5 Global Step: 57985 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:54:02,695-Speed 125.81 samples/sec Loss 11.1275 Epoch: 5 Global Step: 57990 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:54:22,706-Speed 127.94 samples/sec Loss 11.0232 Epoch: 5 Global Step: 57995 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:54:42,409-Speed 129.93 samples/sec Loss 11.2962 Epoch: 5 Global Step: 58000 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:55:02,414-Speed 127.97 samples/sec Loss 11.0988 Epoch: 5 Global Step: 58005 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:55:22,588-Speed 126.90 samples/sec Loss 11.1011 Epoch: 5 Global Step: 58010 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:55:42,438-Speed 128.97 samples/sec Loss 10.9913 Epoch: 5 Global Step: 58015 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:56:02,406-Speed 128.21 samples/sec Loss 10.8826 Epoch: 5 Global Step: 58020 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:56:22,255-Speed 128.98 samples/sec Loss 11.0863 Epoch: 5 Global Step: 58025 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:56:42,227-Speed 128.19 samples/sec Loss 10.9317 Epoch: 5 Global Step: 58030 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:57:02,299-Speed 127.55 samples/sec Loss 10.9868 Epoch: 5 Global Step: 58035 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:57:22,053-Speed 129.60 samples/sec Loss 11.1478 Epoch: 5 Global Step: 58040 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:57:41,907-Speed 128.95 samples/sec Loss 10.8640 Epoch: 5 Global Step: 58045 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:58:01,929-Speed 127.86 samples/sec Loss 11.1435 Epoch: 5 Global Step: 58050 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:58:22,006-Speed 127.52 samples/sec Loss 10.8992 Epoch: 5 Global Step: 58055 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:58:41,882-Speed 128.80 samples/sec Loss 11.0641 Epoch: 5 Global Step: 58060 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:59:01,690-Speed 129.25 samples/sec Loss 11.6721 Epoch: 5 Global Step: 58065 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:59:21,736-Speed 127.71 samples/sec Loss 10.8020 Epoch: 5 Global Step: 58070 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 15:59:41,441-Speed 129.92 samples/sec Loss 10.7596 Epoch: 5 Global Step: 58075 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:00:01,601-Speed 126.99 samples/sec Loss 11.3314 Epoch: 5 Global Step: 58080 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:00:21,317-Speed 129.85 samples/sec Loss 11.3456 Epoch: 5 Global Step: 58085 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:00:41,385-Speed 127.57 samples/sec Loss 10.9648 Epoch: 5 Global Step: 58090 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:01:01,203-Speed 129.18 samples/sec Loss 11.1849 Epoch: 5 Global Step: 58095 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:01:21,248-Speed 127.72 samples/sec Loss 11.2745 Epoch: 5 Global Step: 58100 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:01:41,184-Speed 128.42 samples/sec Loss 10.8681 Epoch: 5 Global Step: 58105 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:02:01,078-Speed 128.69 samples/sec Loss 11.3294 Epoch: 5 Global Step: 58110 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:02:21,058-Speed 128.14 samples/sec Loss 11.2558 Epoch: 5 Global Step: 58115 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:02:40,791-Speed 129.74 samples/sec Loss 10.6825 Epoch: 5 Global Step: 58120 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:03:00,951-Speed 126.99 samples/sec Loss 11.1357 Epoch: 5 Global Step: 58125 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:03:20,801-Speed 128.97 samples/sec Loss 10.9486 Epoch: 5 Global Step: 58130 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:03:40,759-Speed 128.28 samples/sec Loss 11.0880 Epoch: 5 Global Step: 58135 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:04:00,596-Speed 129.06 samples/sec Loss 11.3737 Epoch: 5 Global Step: 58140 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:04:20,764-Speed 126.94 samples/sec Loss 11.2022 Epoch: 5 Global Step: 58145 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:04:40,686-Speed 128.51 samples/sec Loss 11.1263 Epoch: 5 Global Step: 58150 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:05:00,543-Speed 128.93 samples/sec Loss 11.0630 Epoch: 5 Global Step: 58155 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:05:20,488-Speed 128.36 samples/sec Loss 10.9247 Epoch: 5 Global Step: 58160 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:05:40,312-Speed 129.15 samples/sec Loss 10.8128 Epoch: 5 Global Step: 58165 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:06:00,473-Speed 126.98 samples/sec Loss 11.1723 Epoch: 5 Global Step: 58170 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:06:20,530-Speed 127.64 samples/sec Loss 11.0189 Epoch: 5 Global Step: 58175 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:06:40,361-Speed 129.09 samples/sec Loss 11.2991 Epoch: 5 Global Step: 58180 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:07:00,110-Speed 129.63 samples/sec Loss 11.0429 Epoch: 5 Global Step: 58185 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:07:20,329-Speed 126.62 samples/sec Loss 11.2123 Epoch: 5 Global Step: 58190 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:07:40,188-Speed 128.91 samples/sec Loss 10.9741 Epoch: 5 Global Step: 58195 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:08:00,171-Speed 128.12 samples/sec Loss 11.3690 Epoch: 5 Global Step: 58200 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:08:19,995-Speed 129.14 samples/sec Loss 10.9116 Epoch: 5 Global Step: 58205 Fp16 Grad Scale: 8192 Required: 227 hours Training: 2021-06-09 16:08:39,800-Speed 129.27 samples/sec Loss 11.0098 Epoch: 5 Global Step: 58210 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:09:00,202-Speed 125.48 samples/sec Loss 10.8427 Epoch: 5 Global Step: 58215 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:09:20,138-Speed 128.42 samples/sec Loss 11.3198 Epoch: 5 Global Step: 58220 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:09:39,926-Speed 129.38 samples/sec Loss 11.2503 Epoch: 5 Global Step: 58225 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:09:59,958-Speed 127.80 samples/sec Loss 10.6388 Epoch: 5 Global Step: 58230 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:10:20,140-Speed 126.85 samples/sec Loss 10.9342 Epoch: 5 Global Step: 58235 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:10:40,049-Speed 128.59 samples/sec Loss 10.9709 Epoch: 5 Global Step: 58240 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:10:59,873-Speed 129.14 samples/sec Loss 11.1624 Epoch: 5 Global Step: 58245 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:11:19,667-Speed 129.34 samples/sec Loss 11.4069 Epoch: 5 Global Step: 58250 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:11:39,650-Speed 128.12 samples/sec Loss 11.2221 Epoch: 5 Global Step: 58255 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:11:59,883-Speed 126.53 samples/sec Loss 10.9158 Epoch: 5 Global Step: 58260 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:12:19,575-Speed 130.01 samples/sec Loss 11.4508 Epoch: 5 Global Step: 58265 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:12:39,397-Speed 129.15 samples/sec Loss 11.1443 Epoch: 5 Global Step: 58270 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:12:59,274-Speed 128.79 samples/sec Loss 11.4617 Epoch: 5 Global Step: 58275 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:13:19,431-Speed 127.01 samples/sec Loss 10.6844 Epoch: 5 Global Step: 58280 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:13:39,410-Speed 128.14 samples/sec Loss 11.2030 Epoch: 5 Global Step: 58285 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:13:59,222-Speed 129.22 samples/sec Loss 11.1184 Epoch: 5 Global Step: 58290 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:14:19,085-Speed 128.89 samples/sec Loss 11.2377 Epoch: 5 Global Step: 58295 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:14:39,051-Speed 128.23 samples/sec Loss 11.1737 Epoch: 5 Global Step: 58300 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:14:59,157-Speed 127.33 samples/sec Loss 11.1782 Epoch: 5 Global Step: 58305 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:15:18,989-Speed 129.09 samples/sec Loss 10.9426 Epoch: 5 Global Step: 58310 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:15:38,865-Speed 128.80 samples/sec Loss 10.9813 Epoch: 5 Global Step: 58315 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:15:58,665-Speed 129.30 samples/sec Loss 11.2386 Epoch: 5 Global Step: 58320 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:16:18,851-Speed 126.83 samples/sec Loss 11.0841 Epoch: 5 Global Step: 58325 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:16:38,668-Speed 129.19 samples/sec Loss 11.1339 Epoch: 5 Global Step: 58330 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:16:58,427-Speed 129.57 samples/sec Loss 11.0327 Epoch: 5 Global Step: 58335 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:17:18,443-Speed 127.90 samples/sec Loss 11.2785 Epoch: 5 Global Step: 58340 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:17:38,247-Speed 129.27 samples/sec Loss 11.0207 Epoch: 5 Global Step: 58345 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:17:58,416-Speed 126.93 samples/sec Loss 11.0554 Epoch: 5 Global Step: 58350 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:18:18,302-Speed 128.74 samples/sec Loss 11.0716 Epoch: 5 Global Step: 58355 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:18:38,108-Speed 129.26 samples/sec Loss 11.1269 Epoch: 5 Global Step: 58360 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:18:57,937-Speed 129.11 samples/sec Loss 10.7917 Epoch: 5 Global Step: 58365 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:19:18,075-Speed 127.13 samples/sec Loss 11.1804 Epoch: 5 Global Step: 58370 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:19:37,776-Speed 129.95 samples/sec Loss 11.0582 Epoch: 5 Global Step: 58375 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:19:57,803-Speed 127.84 samples/sec Loss 10.9307 Epoch: 5 Global Step: 58380 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:20:17,588-Speed 129.39 samples/sec Loss 11.0338 Epoch: 5 Global Step: 58385 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:20:37,354-Speed 129.53 samples/sec Loss 10.8683 Epoch: 5 Global Step: 58390 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:20:57,442-Speed 127.45 samples/sec Loss 11.1172 Epoch: 5 Global Step: 58395 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:21:17,413-Speed 128.19 samples/sec Loss 11.0935 Epoch: 5 Global Step: 58400 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:21:37,247-Speed 129.07 samples/sec Loss 10.8660 Epoch: 5 Global Step: 58405 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:21:57,250-Speed 127.99 samples/sec Loss 10.8563 Epoch: 5 Global Step: 58410 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:22:17,301-Speed 127.68 samples/sec Loss 11.3371 Epoch: 5 Global Step: 58415 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:22:37,152-Speed 128.97 samples/sec Loss 11.2421 Epoch: 5 Global Step: 58420 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:22:57,116-Speed 128.23 samples/sec Loss 11.1600 Epoch: 5 Global Step: 58425 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:23:16,856-Speed 129.70 samples/sec Loss 11.0939 Epoch: 5 Global Step: 58430 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:23:36,879-Speed 127.86 samples/sec Loss 11.4170 Epoch: 5 Global Step: 58435 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:23:57,025-Speed 127.08 samples/sec Loss 11.4858 Epoch: 5 Global Step: 58440 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:24:16,892-Speed 128.87 samples/sec Loss 11.1212 Epoch: 5 Global Step: 58445 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:24:36,793-Speed 128.64 samples/sec Loss 10.9036 Epoch: 5 Global Step: 58450 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:24:56,609-Speed 129.19 samples/sec Loss 11.1809 Epoch: 5 Global Step: 58455 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:25:16,707-Speed 127.38 samples/sec Loss 10.9699 Epoch: 5 Global Step: 58460 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:25:36,696-Speed 128.08 samples/sec Loss 11.0325 Epoch: 5 Global Step: 58465 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:25:56,500-Speed 129.28 samples/sec Loss 11.5047 Epoch: 5 Global Step: 58470 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:26:16,376-Speed 128.81 samples/sec Loss 10.8185 Epoch: 5 Global Step: 58475 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:26:36,196-Speed 129.17 samples/sec Loss 10.9322 Epoch: 5 Global Step: 58480 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:26:56,418-Speed 126.60 samples/sec Loss 10.9448 Epoch: 5 Global Step: 58485 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:27:16,234-Speed 129.19 samples/sec Loss 10.9243 Epoch: 5 Global Step: 58490 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:27:36,198-Speed 128.24 samples/sec Loss 11.2560 Epoch: 5 Global Step: 58495 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:27:55,999-Speed 129.29 samples/sec Loss 11.2906 Epoch: 5 Global Step: 58500 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:28:16,341-Speed 125.85 samples/sec Loss 11.3116 Epoch: 5 Global Step: 58505 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:28:36,720-Speed 125.63 samples/sec Loss 10.9699 Epoch: 5 Global Step: 58510 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:28:56,685-Speed 128.23 samples/sec Loss 10.9290 Epoch: 5 Global Step: 58515 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:29:16,517-Speed 129.09 samples/sec Loss 11.2980 Epoch: 5 Global Step: 58520 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:29:36,439-Speed 128.51 samples/sec Loss 10.9462 Epoch: 5 Global Step: 58525 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:29:56,471-Speed 127.80 samples/sec Loss 11.1716 Epoch: 5 Global Step: 58530 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:30:16,462-Speed 128.06 samples/sec Loss 10.8857 Epoch: 5 Global Step: 58535 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:30:36,266-Speed 129.28 samples/sec Loss 10.9484 Epoch: 5 Global Step: 58540 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:30:56,270-Speed 127.98 samples/sec Loss 11.7324 Epoch: 5 Global Step: 58545 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:31:16,433-Speed 126.97 samples/sec Loss 11.1041 Epoch: 5 Global Step: 58550 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:31:36,353-Speed 128.52 samples/sec Loss 10.8872 Epoch: 5 Global Step: 58555 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:31:56,230-Speed 128.81 samples/sec Loss 11.1374 Epoch: 5 Global Step: 58560 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:32:16,027-Speed 129.32 samples/sec Loss 10.9390 Epoch: 5 Global Step: 58565 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:32:35,994-Speed 128.21 samples/sec Loss 11.1502 Epoch: 5 Global Step: 58570 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:32:56,109-Speed 127.28 samples/sec Loss 10.8598 Epoch: 5 Global Step: 58575 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:33:15,834-Speed 129.79 samples/sec Loss 10.9473 Epoch: 5 Global Step: 58580 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:33:35,752-Speed 128.53 samples/sec Loss 11.2894 Epoch: 5 Global Step: 58585 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:33:55,767-Speed 127.91 samples/sec Loss 11.4140 Epoch: 5 Global Step: 58590 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:34:15,881-Speed 127.28 samples/sec Loss 10.7059 Epoch: 5 Global Step: 58595 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:34:35,891-Speed 127.95 samples/sec Loss 11.3042 Epoch: 5 Global Step: 58600 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:34:55,711-Speed 129.17 samples/sec Loss 11.2012 Epoch: 5 Global Step: 58605 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:35:15,759-Speed 127.70 samples/sec Loss 11.0466 Epoch: 5 Global Step: 58610 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:35:35,575-Speed 129.20 samples/sec Loss 10.9967 Epoch: 5 Global Step: 58615 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:35:55,729-Speed 127.02 samples/sec Loss 11.1582 Epoch: 5 Global Step: 58620 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:36:15,939-Speed 126.68 samples/sec Loss 11.0360 Epoch: 5 Global Step: 58625 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:36:35,928-Speed 128.07 samples/sec Loss 11.0369 Epoch: 5 Global Step: 58630 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:36:55,786-Speed 128.92 samples/sec Loss 11.0626 Epoch: 5 Global Step: 58635 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:37:16,024-Speed 126.50 samples/sec Loss 11.2027 Epoch: 5 Global Step: 58640 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:37:35,835-Speed 129.23 samples/sec Loss 10.8820 Epoch: 5 Global Step: 58645 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:37:55,564-Speed 129.76 samples/sec Loss 11.1350 Epoch: 5 Global Step: 58650 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:38:15,581-Speed 127.90 samples/sec Loss 11.2175 Epoch: 5 Global Step: 58655 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:38:35,382-Speed 129.29 samples/sec Loss 11.2480 Epoch: 5 Global Step: 58660 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:38:55,536-Speed 127.03 samples/sec Loss 11.2705 Epoch: 5 Global Step: 58665 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:39:15,449-Speed 128.56 samples/sec Loss 10.9863 Epoch: 5 Global Step: 58670 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:39:35,411-Speed 128.25 samples/sec Loss 11.4662 Epoch: 5 Global Step: 58675 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:39:55,373-Speed 128.25 samples/sec Loss 11.4147 Epoch: 5 Global Step: 58680 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:40:15,641-Speed 126.31 samples/sec Loss 11.1983 Epoch: 5 Global Step: 58685 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:40:35,432-Speed 129.36 samples/sec Loss 10.9856 Epoch: 5 Global Step: 58690 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:40:55,489-Speed 127.64 samples/sec Loss 11.3270 Epoch: 5 Global Step: 58695 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:41:15,313-Speed 129.14 samples/sec Loss 11.3624 Epoch: 5 Global Step: 58700 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:41:35,043-Speed 129.76 samples/sec Loss 11.2162 Epoch: 5 Global Step: 58705 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:41:55,228-Speed 126.84 samples/sec Loss 11.0030 Epoch: 5 Global Step: 58710 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:42:15,235-Speed 127.96 samples/sec Loss 10.9865 Epoch: 5 Global Step: 58715 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:42:35,058-Speed 129.15 samples/sec Loss 11.3329 Epoch: 5 Global Step: 58720 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:42:54,969-Speed 128.58 samples/sec Loss 10.8271 Epoch: 5 Global Step: 58725 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:43:14,963-Speed 128.05 samples/sec Loss 10.6530 Epoch: 5 Global Step: 58730 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:43:34,838-Speed 128.81 samples/sec Loss 11.3859 Epoch: 5 Global Step: 58735 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:43:54,847-Speed 127.95 samples/sec Loss 10.9435 Epoch: 5 Global Step: 58740 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:44:14,555-Speed 129.90 samples/sec Loss 11.2114 Epoch: 5 Global Step: 58745 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:44:34,541-Speed 128.10 samples/sec Loss 11.1223 Epoch: 5 Global Step: 58750 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:44:54,547-Speed 127.97 samples/sec Loss 11.0385 Epoch: 5 Global Step: 58755 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:45:14,532-Speed 128.10 samples/sec Loss 11.1241 Epoch: 5 Global Step: 58760 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:45:34,308-Speed 129.45 samples/sec Loss 11.3118 Epoch: 5 Global Step: 58765 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:45:54,247-Speed 128.40 samples/sec Loss 11.2503 Epoch: 5 Global Step: 58770 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:46:14,418-Speed 126.92 samples/sec Loss 11.3161 Epoch: 5 Global Step: 58775 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:46:34,439-Speed 127.87 samples/sec Loss 10.8163 Epoch: 5 Global Step: 58780 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:46:54,255-Speed 129.20 samples/sec Loss 11.0723 Epoch: 5 Global Step: 58785 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:47:14,241-Speed 128.09 samples/sec Loss 11.1542 Epoch: 5 Global Step: 58790 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:47:34,092-Speed 128.97 samples/sec Loss 11.2552 Epoch: 5 Global Step: 58795 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:47:54,328-Speed 126.51 samples/sec Loss 11.1482 Epoch: 5 Global Step: 58800 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:48:14,175-Speed 128.99 samples/sec Loss 11.1160 Epoch: 5 Global Step: 58805 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:48:34,147-Speed 128.18 samples/sec Loss 10.8957 Epoch: 5 Global Step: 58810 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:48:53,967-Speed 129.17 samples/sec Loss 11.0542 Epoch: 5 Global Step: 58815 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:49:14,101-Speed 127.16 samples/sec Loss 10.8372 Epoch: 5 Global Step: 58820 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:49:33,941-Speed 129.04 samples/sec Loss 11.0999 Epoch: 5 Global Step: 58825 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:49:53,751-Speed 129.23 samples/sec Loss 11.2429 Epoch: 5 Global Step: 58830 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:50:13,741-Speed 128.07 samples/sec Loss 11.1438 Epoch: 5 Global Step: 58835 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:50:33,619-Speed 128.79 samples/sec Loss 11.4676 Epoch: 5 Global Step: 58840 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:50:53,779-Speed 126.99 samples/sec Loss 11.3160 Epoch: 5 Global Step: 58845 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:51:13,528-Speed 129.63 samples/sec Loss 11.3152 Epoch: 5 Global Step: 58850 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:51:33,461-Speed 128.43 samples/sec Loss 11.2270 Epoch: 5 Global Step: 58855 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:51:53,278-Speed 129.19 samples/sec Loss 11.1844 Epoch: 5 Global Step: 58860 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:52:13,613-Speed 125.89 samples/sec Loss 11.1332 Epoch: 5 Global Step: 58865 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:52:33,396-Speed 129.41 samples/sec Loss 11.4312 Epoch: 5 Global Step: 58870 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:52:53,368-Speed 128.19 samples/sec Loss 11.2924 Epoch: 5 Global Step: 58875 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:53:13,219-Speed 128.97 samples/sec Loss 11.2458 Epoch: 5 Global Step: 58880 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:53:33,039-Speed 129.17 samples/sec Loss 10.9754 Epoch: 5 Global Step: 58885 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:53:53,170-Speed 127.17 samples/sec Loss 11.2453 Epoch: 5 Global Step: 58890 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:54:13,197-Speed 127.83 samples/sec Loss 11.2508 Epoch: 5 Global Step: 58895 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:54:33,001-Speed 129.27 samples/sec Loss 11.0946 Epoch: 5 Global Step: 58900 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 16:54:52,869-Speed 128.86 samples/sec Loss 11.0456 Epoch: 5 Global Step: 58905 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:55:12,976-Speed 127.32 samples/sec Loss 11.0566 Epoch: 5 Global Step: 58910 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:55:32,780-Speed 129.27 samples/sec Loss 10.6784 Epoch: 5 Global Step: 58915 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:55:52,734-Speed 128.30 samples/sec Loss 11.3108 Epoch: 5 Global Step: 58920 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:56:12,537-Speed 129.28 samples/sec Loss 11.1923 Epoch: 5 Global Step: 58925 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:56:32,539-Speed 128.00 samples/sec Loss 11.3460 Epoch: 5 Global Step: 58930 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:56:52,681-Speed 127.11 samples/sec Loss 11.4976 Epoch: 5 Global Step: 58935 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:57:12,500-Speed 129.17 samples/sec Loss 11.1999 Epoch: 5 Global Step: 58940 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:57:32,424-Speed 128.49 samples/sec Loss 11.0992 Epoch: 5 Global Step: 58945 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:57:52,304-Speed 128.78 samples/sec Loss 11.0165 Epoch: 5 Global Step: 58950 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:58:12,292-Speed 128.08 samples/sec Loss 10.9998 Epoch: 5 Global Step: 58955 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:58:32,277-Speed 128.10 samples/sec Loss 11.3540 Epoch: 5 Global Step: 58960 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:58:52,133-Speed 128.94 samples/sec Loss 10.8443 Epoch: 5 Global Step: 58965 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:59:11,996-Speed 128.89 samples/sec Loss 10.7053 Epoch: 5 Global Step: 58970 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:59:31,790-Speed 129.34 samples/sec Loss 11.0161 Epoch: 5 Global Step: 58975 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 16:59:51,890-Speed 127.37 samples/sec Loss 11.0798 Epoch: 5 Global Step: 58980 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:00:11,666-Speed 129.45 samples/sec Loss 11.1531 Epoch: 5 Global Step: 58985 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:00:31,718-Speed 127.67 samples/sec Loss 11.2016 Epoch: 5 Global Step: 58990 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:00:51,525-Speed 129.26 samples/sec Loss 11.2174 Epoch: 5 Global Step: 58995 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:01:11,648-Speed 127.22 samples/sec Loss 11.3732 Epoch: 5 Global Step: 59000 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:01:31,724-Speed 127.53 samples/sec Loss 11.1295 Epoch: 5 Global Step: 59005 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:01:51,566-Speed 129.03 samples/sec Loss 11.0742 Epoch: 5 Global Step: 59010 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:02:11,499-Speed 128.43 samples/sec Loss 11.1426 Epoch: 5 Global Step: 59015 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:02:31,200-Speed 129.96 samples/sec Loss 10.8683 Epoch: 5 Global Step: 59020 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:02:51,331-Speed 127.17 samples/sec Loss 11.1556 Epoch: 5 Global Step: 59025 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:03:11,292-Speed 128.26 samples/sec Loss 11.0290 Epoch: 5 Global Step: 59030 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:03:31,194-Speed 128.64 samples/sec Loss 11.0991 Epoch: 5 Global Step: 59035 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:03:51,075-Speed 128.77 samples/sec Loss 10.9077 Epoch: 5 Global Step: 59040 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:04:11,302-Speed 126.57 samples/sec Loss 11.1631 Epoch: 5 Global Step: 59045 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:04:31,119-Speed 129.19 samples/sec Loss 11.1137 Epoch: 5 Global Step: 59050 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:04:51,090-Speed 128.20 samples/sec Loss 11.1871 Epoch: 5 Global Step: 59055 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:05:10,827-Speed 129.71 samples/sec Loss 11.0819 Epoch: 5 Global Step: 59060 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:05:30,545-Speed 129.84 samples/sec Loss 11.2989 Epoch: 5 Global Step: 59065 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:05:50,725-Speed 126.86 samples/sec Loss 11.2320 Epoch: 5 Global Step: 59070 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:06:10,695-Speed 128.20 samples/sec Loss 11.0802 Epoch: 5 Global Step: 59075 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:06:30,497-Speed 129.29 samples/sec Loss 10.9281 Epoch: 5 Global Step: 59080 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:06:50,490-Speed 128.05 samples/sec Loss 10.7388 Epoch: 5 Global Step: 59085 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:07:10,628-Speed 127.13 samples/sec Loss 11.3234 Epoch: 5 Global Step: 59090 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:07:30,598-Speed 128.20 samples/sec Loss 11.0167 Epoch: 5 Global Step: 59095 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:07:50,453-Speed 128.94 samples/sec Loss 11.4401 Epoch: 5 Global Step: 59100 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:08:10,275-Speed 129.16 samples/sec Loss 11.0602 Epoch: 5 Global Step: 59105 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:08:30,296-Speed 127.87 samples/sec Loss 11.0891 Epoch: 5 Global Step: 59110 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:08:50,469-Speed 126.91 samples/sec Loss 11.0326 Epoch: 5 Global Step: 59115 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:09:10,174-Speed 129.92 samples/sec Loss 11.0145 Epoch: 5 Global Step: 59120 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:09:30,080-Speed 128.61 samples/sec Loss 11.0635 Epoch: 5 Global Step: 59125 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:09:49,990-Speed 128.59 samples/sec Loss 11.0384 Epoch: 5 Global Step: 59130 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:10:10,113-Speed 127.22 samples/sec Loss 11.0236 Epoch: 5 Global Step: 59135 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:10:30,086-Speed 128.18 samples/sec Loss 10.9858 Epoch: 5 Global Step: 59140 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:10:49,917-Speed 129.10 samples/sec Loss 10.9242 Epoch: 5 Global Step: 59145 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:11:09,951-Speed 127.79 samples/sec Loss 10.9639 Epoch: 5 Global Step: 59150 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:11:29,895-Speed 128.36 samples/sec Loss 11.2050 Epoch: 5 Global Step: 59155 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:11:50,054-Speed 127.00 samples/sec Loss 11.2927 Epoch: 5 Global Step: 59160 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:12:09,893-Speed 129.05 samples/sec Loss 11.0026 Epoch: 5 Global Step: 59165 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:12:29,895-Speed 127.99 samples/sec Loss 11.5526 Epoch: 5 Global Step: 59170 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:12:49,591-Speed 129.98 samples/sec Loss 10.9962 Epoch: 5 Global Step: 59175 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:13:09,806-Speed 126.65 samples/sec Loss 10.8306 Epoch: 5 Global Step: 59180 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:13:29,604-Speed 129.31 samples/sec Loss 10.7006 Epoch: 5 Global Step: 59185 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:13:49,578-Speed 128.17 samples/sec Loss 10.6946 Epoch: 5 Global Step: 59190 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:14:09,460-Speed 128.77 samples/sec Loss 10.9542 Epoch: 5 Global Step: 59195 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:14:29,237-Speed 129.45 samples/sec Loss 11.1844 Epoch: 5 Global Step: 59200 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:14:49,315-Speed 127.51 samples/sec Loss 11.3474 Epoch: 5 Global Step: 59205 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:15:09,312-Speed 128.02 samples/sec Loss 11.3964 Epoch: 5 Global Step: 59210 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:15:29,166-Speed 128.95 samples/sec Loss 11.1972 Epoch: 5 Global Step: 59215 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:15:49,310-Speed 127.09 samples/sec Loss 11.3583 Epoch: 5 Global Step: 59220 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:16:09,425-Speed 127.28 samples/sec Loss 11.0904 Epoch: 5 Global Step: 59225 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:16:29,155-Speed 129.76 samples/sec Loss 11.0742 Epoch: 5 Global Step: 59230 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:16:49,198-Speed 127.74 samples/sec Loss 10.9070 Epoch: 5 Global Step: 59235 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:17:09,030-Speed 129.09 samples/sec Loss 11.2725 Epoch: 5 Global Step: 59240 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:17:29,011-Speed 128.13 samples/sec Loss 11.2349 Epoch: 5 Global Step: 59245 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:17:49,038-Speed 127.84 samples/sec Loss 11.0710 Epoch: 5 Global Step: 59250 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:18:08,969-Speed 128.44 samples/sec Loss 11.2795 Epoch: 5 Global Step: 59255 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:18:28,787-Speed 129.18 samples/sec Loss 11.1001 Epoch: 5 Global Step: 59260 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:18:48,802-Speed 127.91 samples/sec Loss 10.7391 Epoch: 5 Global Step: 59265 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:19:08,801-Speed 128.02 samples/sec Loss 11.0150 Epoch: 5 Global Step: 59270 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:19:28,679-Speed 128.79 samples/sec Loss 11.1106 Epoch: 5 Global Step: 59275 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:19:48,617-Speed 128.41 samples/sec Loss 10.9678 Epoch: 5 Global Step: 59280 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:20:08,440-Speed 129.14 samples/sec Loss 10.9252 Epoch: 5 Global Step: 59285 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:20:28,526-Speed 127.46 samples/sec Loss 10.6128 Epoch: 5 Global Step: 59290 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:20:48,590-Speed 127.59 samples/sec Loss 10.8972 Epoch: 5 Global Step: 59295 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:21:08,468-Speed 128.79 samples/sec Loss 11.0612 Epoch: 5 Global Step: 59300 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:21:28,334-Speed 128.87 samples/sec Loss 11.1444 Epoch: 5 Global Step: 59305 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:21:48,290-Speed 128.29 samples/sec Loss 10.8767 Epoch: 5 Global Step: 59310 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:22:08,408-Speed 127.25 samples/sec Loss 10.9857 Epoch: 5 Global Step: 59315 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:22:28,374-Speed 128.23 samples/sec Loss 10.9168 Epoch: 5 Global Step: 59320 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:22:48,212-Speed 129.05 samples/sec Loss 11.3836 Epoch: 5 Global Step: 59325 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:23:08,181-Speed 128.20 samples/sec Loss 10.9526 Epoch: 5 Global Step: 59330 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:23:27,876-Speed 129.99 samples/sec Loss 10.7400 Epoch: 5 Global Step: 59335 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:23:48,089-Speed 126.66 samples/sec Loss 11.1762 Epoch: 5 Global Step: 59340 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:24:07,899-Speed 129.23 samples/sec Loss 11.2119 Epoch: 5 Global Step: 59345 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:24:27,902-Speed 127.98 samples/sec Loss 11.1264 Epoch: 5 Global Step: 59350 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:24:47,699-Speed 129.32 samples/sec Loss 10.9062 Epoch: 5 Global Step: 59355 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:25:07,844-Speed 127.09 samples/sec Loss 11.0746 Epoch: 5 Global Step: 59360 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:25:28,145-Speed 126.11 samples/sec Loss 11.0789 Epoch: 5 Global Step: 59365 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:25:47,879-Speed 129.74 samples/sec Loss 11.2776 Epoch: 5 Global Step: 59370 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:26:07,858-Speed 128.14 samples/sec Loss 11.0345 Epoch: 5 Global Step: 59375 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:26:27,572-Speed 129.86 samples/sec Loss 11.1265 Epoch: 5 Global Step: 59380 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:26:47,811-Speed 126.50 samples/sec Loss 10.9144 Epoch: 5 Global Step: 59385 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:27:07,718-Speed 128.61 samples/sec Loss 10.8224 Epoch: 5 Global Step: 59390 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:27:27,600-Speed 128.77 samples/sec Loss 10.8975 Epoch: 5 Global Step: 59395 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:27:47,289-Speed 130.03 samples/sec Loss 10.8270 Epoch: 5 Global Step: 59400 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:28:07,622-Speed 125.91 samples/sec Loss 11.0391 Epoch: 5 Global Step: 59405 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:28:27,423-Speed 129.29 samples/sec Loss 11.1333 Epoch: 5 Global Step: 59410 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:28:47,386-Speed 128.24 samples/sec Loss 11.1770 Epoch: 5 Global Step: 59415 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:29:07,081-Speed 129.99 samples/sec Loss 11.1770 Epoch: 5 Global Step: 59420 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:29:26,946-Speed 128.87 samples/sec Loss 11.1170 Epoch: 5 Global Step: 59425 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:29:47,140-Speed 126.78 samples/sec Loss 11.3809 Epoch: 5 Global Step: 59430 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:30:07,074-Speed 128.43 samples/sec Loss 10.9175 Epoch: 5 Global Step: 59435 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:30:26,774-Speed 129.95 samples/sec Loss 11.3827 Epoch: 5 Global Step: 59440 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:30:46,510-Speed 129.72 samples/sec Loss 11.1665 Epoch: 5 Global Step: 59445 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:31:06,905-Speed 125.53 samples/sec Loss 11.1554 Epoch: 5 Global Step: 59450 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:31:26,687-Speed 129.42 samples/sec Loss 11.3906 Epoch: 5 Global Step: 59455 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:31:46,558-Speed 128.83 samples/sec Loss 11.0359 Epoch: 5 Global Step: 59460 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:32:06,258-Speed 129.96 samples/sec Loss 11.4212 Epoch: 5 Global Step: 59465 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:32:26,167-Speed 128.60 samples/sec Loss 10.7856 Epoch: 5 Global Step: 59470 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:32:46,300-Speed 127.16 samples/sec Loss 11.2353 Epoch: 5 Global Step: 59475 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:33:06,297-Speed 128.03 samples/sec Loss 11.0454 Epoch: 5 Global Step: 59480 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:33:25,998-Speed 129.95 samples/sec Loss 11.1190 Epoch: 5 Global Step: 59485 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:33:46,044-Speed 127.71 samples/sec Loss 10.9927 Epoch: 5 Global Step: 59490 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:34:06,057-Speed 127.92 samples/sec Loss 11.0820 Epoch: 5 Global Step: 59495 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:34:26,001-Speed 128.37 samples/sec Loss 11.0852 Epoch: 5 Global Step: 59500 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:34:45,882-Speed 128.77 samples/sec Loss 11.0223 Epoch: 5 Global Step: 59505 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:35:05,955-Speed 127.54 samples/sec Loss 11.1796 Epoch: 5 Global Step: 59510 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:35:25,942-Speed 128.10 samples/sec Loss 11.3319 Epoch: 5 Global Step: 59515 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:35:46,034-Speed 127.42 samples/sec Loss 11.2481 Epoch: 5 Global Step: 59520 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:36:05,796-Speed 129.54 samples/sec Loss 11.3787 Epoch: 5 Global Step: 59525 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:36:25,550-Speed 129.60 samples/sec Loss 11.4087 Epoch: 5 Global Step: 59530 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:36:45,554-Speed 127.98 samples/sec Loss 10.9333 Epoch: 5 Global Step: 59535 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:37:05,573-Speed 127.88 samples/sec Loss 11.1564 Epoch: 5 Global Step: 59540 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:37:25,447-Speed 128.82 samples/sec Loss 11.1348 Epoch: 5 Global Step: 59545 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:37:45,301-Speed 128.95 samples/sec Loss 10.9385 Epoch: 5 Global Step: 59550 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:38:05,302-Speed 128.00 samples/sec Loss 11.0160 Epoch: 5 Global Step: 59555 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:38:25,192-Speed 128.71 samples/sec Loss 11.1108 Epoch: 5 Global Step: 59560 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:38:45,243-Speed 127.69 samples/sec Loss 11.2714 Epoch: 5 Global Step: 59565 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:39:04,968-Speed 129.79 samples/sec Loss 11.1214 Epoch: 5 Global Step: 59570 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:39:24,984-Speed 127.90 samples/sec Loss 11.2586 Epoch: 5 Global Step: 59575 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:39:44,811-Speed 129.13 samples/sec Loss 10.8374 Epoch: 5 Global Step: 59580 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:40:05,021-Speed 126.67 samples/sec Loss 11.1269 Epoch: 5 Global Step: 59585 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:40:24,802-Speed 129.43 samples/sec Loss 11.0314 Epoch: 5 Global Step: 59590 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:40:44,637-Speed 129.07 samples/sec Loss 11.1873 Epoch: 5 Global Step: 59595 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 17:41:04,580-Speed 128.37 samples/sec Loss 11.3576 Epoch: 5 Global Step: 59600 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:41:24,273-Speed 130.00 samples/sec Loss 11.2669 Epoch: 5 Global Step: 59605 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:41:44,326-Speed 127.67 samples/sec Loss 11.0598 Epoch: 5 Global Step: 59610 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:42:04,316-Speed 128.07 samples/sec Loss 11.4983 Epoch: 5 Global Step: 59615 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:42:24,129-Speed 129.21 samples/sec Loss 10.9312 Epoch: 5 Global Step: 59620 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:42:44,114-Speed 128.10 samples/sec Loss 11.1697 Epoch: 5 Global Step: 59625 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:43:04,133-Speed 127.89 samples/sec Loss 11.0256 Epoch: 5 Global Step: 59630 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:43:24,004-Speed 128.83 samples/sec Loss 11.1303 Epoch: 5 Global Step: 59635 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:43:43,993-Speed 128.08 samples/sec Loss 11.0668 Epoch: 5 Global Step: 59640 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:44:03,915-Speed 128.50 samples/sec Loss 11.1451 Epoch: 5 Global Step: 59645 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:44:23,650-Speed 129.73 samples/sec Loss 10.6493 Epoch: 5 Global Step: 59650 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:44:43,769-Speed 127.25 samples/sec Loss 11.0474 Epoch: 5 Global Step: 59655 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:45:03,751-Speed 128.12 samples/sec Loss 11.0784 Epoch: 5 Global Step: 59660 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:45:23,583-Speed 129.09 samples/sec Loss 11.0451 Epoch: 5 Global Step: 59665 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:45:43,583-Speed 128.01 samples/sec Loss 11.0639 Epoch: 5 Global Step: 59670 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:46:03,640-Speed 127.64 samples/sec Loss 11.1181 Epoch: 5 Global Step: 59675 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:46:23,470-Speed 129.11 samples/sec Loss 11.0331 Epoch: 5 Global Step: 59680 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:46:43,414-Speed 128.37 samples/sec Loss 11.0260 Epoch: 5 Global Step: 59685 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:47:03,099-Speed 130.06 samples/sec Loss 10.9028 Epoch: 5 Global Step: 59690 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:47:23,133-Speed 127.79 samples/sec Loss 10.8518 Epoch: 5 Global Step: 59695 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:47:43,148-Speed 127.91 samples/sec Loss 11.2698 Epoch: 5 Global Step: 59700 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:48:03,057-Speed 128.58 samples/sec Loss 11.3077 Epoch: 5 Global Step: 59705 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:48:22,905-Speed 128.99 samples/sec Loss 11.0336 Epoch: 5 Global Step: 59710 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:48:42,729-Speed 129.14 samples/sec Loss 11.3021 Epoch: 5 Global Step: 59715 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:49:02,972-Speed 126.47 samples/sec Loss 11.0854 Epoch: 5 Global Step: 59720 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:49:22,975-Speed 127.99 samples/sec Loss 11.4522 Epoch: 5 Global Step: 59725 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:49:42,543-Speed 130.83 samples/sec Loss 10.4738 Epoch: 5 Global Step: 59730 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:50:02,401-Speed 128.92 samples/sec Loss 11.3312 Epoch: 5 Global Step: 59735 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:50:22,255-Speed 128.95 samples/sec Loss 11.4954 Epoch: 5 Global Step: 59740 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:50:42,397-Speed 127.10 samples/sec Loss 10.9102 Epoch: 5 Global Step: 59745 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:51:02,206-Speed 129.24 samples/sec Loss 10.9830 Epoch: 5 Global Step: 59750 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:51:22,071-Speed 128.88 samples/sec Loss 11.2528 Epoch: 5 Global Step: 59755 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:51:41,892-Speed 129.16 samples/sec Loss 11.0427 Epoch: 5 Global Step: 59760 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:52:02,302-Speed 125.43 samples/sec Loss 11.1984 Epoch: 5 Global Step: 59765 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:52:21,993-Speed 130.02 samples/sec Loss 11.1032 Epoch: 5 Global Step: 59770 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:52:41,722-Speed 129.77 samples/sec Loss 11.2017 Epoch: 5 Global Step: 59775 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:53:01,651-Speed 128.46 samples/sec Loss 11.2528 Epoch: 5 Global Step: 59780 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:53:53,110-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-09 17:54:04,132-Speed 40.97 samples/sec Loss 10.9694 Epoch: 6 Global Step: 59785 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:54:24,190-Speed 127.64 samples/sec Loss 10.6693 Epoch: 6 Global Step: 59790 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:54:43,878-Speed 130.03 samples/sec Loss 10.3240 Epoch: 6 Global Step: 59795 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:55:03,746-Speed 128.86 samples/sec Loss 10.4246 Epoch: 6 Global Step: 59800 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:55:23,819-Speed 127.54 samples/sec Loss 9.8659 Epoch: 6 Global Step: 59805 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:55:43,455-Speed 130.38 samples/sec Loss 10.4008 Epoch: 6 Global Step: 59810 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:56:03,003-Speed 130.97 samples/sec Loss 10.3125 Epoch: 6 Global Step: 59815 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:56:22,691-Speed 130.04 samples/sec Loss 10.4371 Epoch: 6 Global Step: 59820 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:56:42,329-Speed 130.37 samples/sec Loss 10.4839 Epoch: 6 Global Step: 59825 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:57:01,898-Speed 130.83 samples/sec Loss 10.3836 Epoch: 6 Global Step: 59830 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:57:21,445-Speed 130.97 samples/sec Loss 10.4864 Epoch: 6 Global Step: 59835 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:57:40,980-Speed 131.06 samples/sec Loss 9.8926 Epoch: 6 Global Step: 59840 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:58:00,860-Speed 128.78 samples/sec Loss 10.3505 Epoch: 6 Global Step: 59845 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:58:20,556-Speed 129.98 samples/sec Loss 10.5793 Epoch: 6 Global Step: 59850 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:58:40,072-Speed 131.19 samples/sec Loss 10.4045 Epoch: 6 Global Step: 59855 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:58:59,638-Speed 130.85 samples/sec Loss 10.7024 Epoch: 6 Global Step: 59860 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:59:19,461-Speed 129.15 samples/sec Loss 10.6257 Epoch: 6 Global Step: 59865 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:59:39,044-Speed 130.73 samples/sec Loss 10.3359 Epoch: 6 Global Step: 59870 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 17:59:58,763-Speed 129.83 samples/sec Loss 10.6361 Epoch: 6 Global Step: 59875 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:00:18,370-Speed 130.58 samples/sec Loss 10.3094 Epoch: 6 Global Step: 59880 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:00:37,925-Speed 130.92 samples/sec Loss 10.5276 Epoch: 6 Global Step: 59885 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:00:57,816-Speed 128.71 samples/sec Loss 10.6844 Epoch: 6 Global Step: 59890 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:01:17,506-Speed 130.02 samples/sec Loss 10.3927 Epoch: 6 Global Step: 59895 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:01:37,125-Speed 130.50 samples/sec Loss 10.5102 Epoch: 6 Global Step: 59900 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:01:56,868-Speed 129.67 samples/sec Loss 10.7329 Epoch: 6 Global Step: 59905 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:02:16,467-Speed 130.63 samples/sec Loss 10.5434 Epoch: 6 Global Step: 59910 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:02:36,079-Speed 130.54 samples/sec Loss 10.5356 Epoch: 6 Global Step: 59915 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:02:55,758-Speed 130.10 samples/sec Loss 10.6467 Epoch: 6 Global Step: 59920 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:03:15,388-Speed 130.42 samples/sec Loss 10.5896 Epoch: 6 Global Step: 59925 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:03:35,081-Speed 130.00 samples/sec Loss 10.6107 Epoch: 6 Global Step: 59930 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:03:55,043-Speed 128.25 samples/sec Loss 10.6107 Epoch: 6 Global Step: 59935 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:04:14,790-Speed 129.65 samples/sec Loss 10.4818 Epoch: 6 Global Step: 59940 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:04:34,405-Speed 130.52 samples/sec Loss 10.4794 Epoch: 6 Global Step: 59945 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:04:54,089-Speed 130.06 samples/sec Loss 10.6773 Epoch: 6 Global Step: 59950 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:05:13,600-Speed 131.22 samples/sec Loss 10.6978 Epoch: 6 Global Step: 59955 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:05:33,275-Speed 130.12 samples/sec Loss 10.6969 Epoch: 6 Global Step: 59960 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:05:52,893-Speed 130.50 samples/sec Loss 10.5958 Epoch: 6 Global Step: 59965 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:06:12,501-Speed 130.57 samples/sec Loss 10.5553 Epoch: 6 Global Step: 59970 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:06:32,201-Speed 129.96 samples/sec Loss 10.7236 Epoch: 6 Global Step: 59975 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:06:52,049-Speed 128.99 samples/sec Loss 10.3551 Epoch: 6 Global Step: 59980 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:07:11,799-Speed 129.63 samples/sec Loss 10.7109 Epoch: 6 Global Step: 59985 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:07:31,441-Speed 130.34 samples/sec Loss 10.1630 Epoch: 6 Global Step: 59990 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:07:51,130-Speed 130.03 samples/sec Loss 10.7913 Epoch: 6 Global Step: 59995 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:08:10,753-Speed 130.46 samples/sec Loss 10.5075 Epoch: 6 Global Step: 60000 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 18:22:58,802-[lfw][60000]XNorm: 22.582449 Training: 2021-06-09 18:22:58,802-[lfw][60000]Accuracy-Flip: 0.99700+-0.00306 Training: 2021-06-09 18:22:58,802-[lfw][60000]Accuracy-Highest: 0.99700 Training: 2021-06-09 18:40:15,560-[cfp_fp][60000]XNorm: 20.772449 Training: 2021-06-09 18:40:15,561-[cfp_fp][60000]Accuracy-Flip: 0.97086+-0.00950 Training: 2021-06-09 18:40:15,561-[cfp_fp][60000]Accuracy-Highest: 0.97514 Training: 2021-06-09 18:55:04,698-[agedb_30][60000]XNorm: 22.272163 Training: 2021-06-09 18:55:04,698-[agedb_30][60000]Accuracy-Flip: 0.97183+-0.00751 Training: 2021-06-09 18:55:04,698-[agedb_30][60000]Accuracy-Highest: 0.97183 Training: 2021-06-09 18:55:24,445-Speed 0.90 samples/sec Loss 10.5044 Epoch: 6 Global Step: 60005 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:55:44,304-Speed 128.92 samples/sec Loss 10.5616 Epoch: 6 Global Step: 60010 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:56:03,936-Speed 130.41 samples/sec Loss 10.5937 Epoch: 6 Global Step: 60015 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:56:23,879-Speed 128.37 samples/sec Loss 10.3967 Epoch: 6 Global Step: 60020 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:56:43,524-Speed 130.32 samples/sec Loss 10.7083 Epoch: 6 Global Step: 60025 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:57:03,716-Speed 126.79 samples/sec Loss 10.3891 Epoch: 6 Global Step: 60030 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:57:23,415-Speed 129.96 samples/sec Loss 10.8987 Epoch: 6 Global Step: 60035 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:57:43,107-Speed 130.01 samples/sec Loss 10.8161 Epoch: 6 Global Step: 60040 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:58:02,865-Speed 129.57 samples/sec Loss 10.6677 Epoch: 6 Global Step: 60045 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:58:22,571-Speed 129.92 samples/sec Loss 10.2256 Epoch: 6 Global Step: 60050 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:58:42,464-Speed 128.69 samples/sec Loss 10.7405 Epoch: 6 Global Step: 60055 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:59:02,132-Speed 130.17 samples/sec Loss 10.8877 Epoch: 6 Global Step: 60060 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:59:22,233-Speed 127.36 samples/sec Loss 10.9178 Epoch: 6 Global Step: 60065 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 18:59:41,859-Speed 130.45 samples/sec Loss 10.7299 Epoch: 6 Global Step: 60070 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:00:02,021-Speed 126.97 samples/sec Loss 10.7125 Epoch: 6 Global Step: 60075 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:00:22,075-Speed 127.66 samples/sec Loss 10.6317 Epoch: 6 Global Step: 60080 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:00:41,616-Speed 131.01 samples/sec Loss 10.8472 Epoch: 6 Global Step: 60085 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:01:01,462-Speed 129.01 samples/sec Loss 10.8748 Epoch: 6 Global Step: 60090 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:01:21,110-Speed 130.30 samples/sec Loss 10.8634 Epoch: 6 Global Step: 60095 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:01:40,915-Speed 129.27 samples/sec Loss 10.6030 Epoch: 6 Global Step: 60100 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:02:00,618-Speed 129.94 samples/sec Loss 10.5631 Epoch: 6 Global Step: 60105 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:02:20,403-Speed 129.40 samples/sec Loss 10.6359 Epoch: 6 Global Step: 60110 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:02:40,117-Speed 129.86 samples/sec Loss 10.9831 Epoch: 6 Global Step: 60115 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:03:00,173-Speed 127.65 samples/sec Loss 10.6436 Epoch: 6 Global Step: 60120 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:03:20,157-Speed 128.11 samples/sec Loss 10.8430 Epoch: 6 Global Step: 60125 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:03:39,799-Speed 130.33 samples/sec Loss 10.6965 Epoch: 6 Global Step: 60130 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:03:59,753-Speed 128.30 samples/sec Loss 10.7745 Epoch: 6 Global Step: 60135 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:04:19,338-Speed 130.72 samples/sec Loss 10.6967 Epoch: 6 Global Step: 60140 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:04:39,192-Speed 128.95 samples/sec Loss 10.8137 Epoch: 6 Global Step: 60145 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:04:58,925-Speed 129.74 samples/sec Loss 10.9201 Epoch: 6 Global Step: 60150 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:05:18,904-Speed 128.14 samples/sec Loss 10.9887 Epoch: 6 Global Step: 60155 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:05:38,600-Speed 129.98 samples/sec Loss 10.6312 Epoch: 6 Global Step: 60160 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:05:58,691-Speed 127.43 samples/sec Loss 10.8905 Epoch: 6 Global Step: 60165 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:06:18,645-Speed 128.31 samples/sec Loss 11.0143 Epoch: 6 Global Step: 60170 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:06:38,203-Speed 130.89 samples/sec Loss 10.5446 Epoch: 6 Global Step: 60175 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:06:58,181-Speed 128.15 samples/sec Loss 10.8248 Epoch: 6 Global Step: 60180 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:07:17,817-Speed 130.38 samples/sec Loss 10.8042 Epoch: 6 Global Step: 60185 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:07:37,715-Speed 128.66 samples/sec Loss 10.7036 Epoch: 6 Global Step: 60190 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:07:57,359-Speed 130.33 samples/sec Loss 10.5642 Epoch: 6 Global Step: 60195 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:08:17,090-Speed 129.75 samples/sec Loss 10.8810 Epoch: 6 Global Step: 60200 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:08:36,737-Speed 130.30 samples/sec Loss 11.1253 Epoch: 6 Global Step: 60205 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:08:56,823-Speed 127.46 samples/sec Loss 11.1373 Epoch: 6 Global Step: 60210 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:09:16,779-Speed 128.29 samples/sec Loss 10.7756 Epoch: 6 Global Step: 60215 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:09:36,397-Speed 130.49 samples/sec Loss 10.8976 Epoch: 6 Global Step: 60220 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:09:56,398-Speed 128.00 samples/sec Loss 10.5735 Epoch: 6 Global Step: 60225 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:10:16,169-Speed 129.49 samples/sec Loss 11.0646 Epoch: 6 Global Step: 60230 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:10:36,161-Speed 128.06 samples/sec Loss 10.6219 Epoch: 6 Global Step: 60235 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:10:55,783-Speed 130.48 samples/sec Loss 10.6798 Epoch: 6 Global Step: 60240 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:11:15,736-Speed 128.31 samples/sec Loss 10.8676 Epoch: 6 Global Step: 60245 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:11:35,427-Speed 130.02 samples/sec Loss 10.9788 Epoch: 6 Global Step: 60250 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:11:55,475-Speed 127.70 samples/sec Loss 10.7345 Epoch: 6 Global Step: 60255 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:12:15,202-Speed 129.78 samples/sec Loss 10.7001 Epoch: 6 Global Step: 60260 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:12:34,983-Speed 129.42 samples/sec Loss 10.6492 Epoch: 6 Global Step: 60265 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:12:54,801-Speed 129.18 samples/sec Loss 10.7314 Epoch: 6 Global Step: 60270 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:13:14,482-Speed 130.08 samples/sec Loss 10.8665 Epoch: 6 Global Step: 60275 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:13:34,278-Speed 129.33 samples/sec Loss 10.8764 Epoch: 6 Global Step: 60280 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:13:53,962-Speed 130.07 samples/sec Loss 10.6315 Epoch: 6 Global Step: 60285 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:14:13,766-Speed 129.27 samples/sec Loss 10.6648 Epoch: 6 Global Step: 60290 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:14:33,395-Speed 130.43 samples/sec Loss 11.1081 Epoch: 6 Global Step: 60295 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:14:53,560-Speed 126.96 samples/sec Loss 10.8127 Epoch: 6 Global Step: 60300 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:15:13,667-Speed 127.33 samples/sec Loss 10.9924 Epoch: 6 Global Step: 60305 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:15:33,347-Speed 130.08 samples/sec Loss 10.8777 Epoch: 6 Global Step: 60310 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:15:53,247-Speed 128.65 samples/sec Loss 11.1981 Epoch: 6 Global Step: 60315 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:16:12,874-Speed 130.44 samples/sec Loss 10.8142 Epoch: 6 Global Step: 60320 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:16:32,603-Speed 129.77 samples/sec Loss 10.5259 Epoch: 6 Global Step: 60325 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:16:52,256-Speed 130.27 samples/sec Loss 10.6588 Epoch: 6 Global Step: 60330 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:17:12,930-Speed 123.83 samples/sec Loss 10.9315 Epoch: 6 Global Step: 60335 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:17:32,619-Speed 130.03 samples/sec Loss 11.0682 Epoch: 6 Global Step: 60340 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:17:52,666-Speed 127.71 samples/sec Loss 10.8615 Epoch: 6 Global Step: 60345 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:18:12,633-Speed 128.21 samples/sec Loss 10.5023 Epoch: 6 Global Step: 60350 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:18:32,241-Speed 130.56 samples/sec Loss 10.9826 Epoch: 6 Global Step: 60355 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:18:52,216-Speed 128.17 samples/sec Loss 10.7683 Epoch: 6 Global Step: 60360 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:19:11,840-Speed 130.46 samples/sec Loss 11.1698 Epoch: 6 Global Step: 60365 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:19:31,831-Speed 128.07 samples/sec Loss 10.7213 Epoch: 6 Global Step: 60370 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:19:51,508-Speed 130.11 samples/sec Loss 11.0918 Epoch: 6 Global Step: 60375 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:20:11,424-Speed 128.55 samples/sec Loss 11.0342 Epoch: 6 Global Step: 60380 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:20:31,178-Speed 129.60 samples/sec Loss 10.5760 Epoch: 6 Global Step: 60385 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:20:51,301-Speed 127.23 samples/sec Loss 10.7789 Epoch: 6 Global Step: 60390 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:21:11,133-Speed 129.09 samples/sec Loss 10.8863 Epoch: 6 Global Step: 60395 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:21:30,806-Speed 130.13 samples/sec Loss 11.0224 Epoch: 6 Global Step: 60400 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:21:50,741-Speed 128.42 samples/sec Loss 10.9285 Epoch: 6 Global Step: 60405 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:22:10,354-Speed 130.53 samples/sec Loss 10.6595 Epoch: 6 Global Step: 60410 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:22:30,551-Speed 126.76 samples/sec Loss 11.0699 Epoch: 6 Global Step: 60415 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:22:50,230-Speed 130.09 samples/sec Loss 10.9836 Epoch: 6 Global Step: 60420 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:23:10,310-Speed 127.50 samples/sec Loss 10.6064 Epoch: 6 Global Step: 60425 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:23:29,978-Speed 130.17 samples/sec Loss 10.8550 Epoch: 6 Global Step: 60430 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:23:50,120-Speed 127.11 samples/sec Loss 11.3907 Epoch: 6 Global Step: 60435 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:24:10,174-Speed 127.66 samples/sec Loss 10.7392 Epoch: 6 Global Step: 60440 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:24:29,777-Speed 130.60 samples/sec Loss 11.2596 Epoch: 6 Global Step: 60445 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:24:49,520-Speed 129.67 samples/sec Loss 10.8011 Epoch: 6 Global Step: 60450 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:25:09,207-Speed 130.04 samples/sec Loss 10.7712 Epoch: 6 Global Step: 60455 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:25:28,977-Speed 129.50 samples/sec Loss 11.1019 Epoch: 6 Global Step: 60460 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:25:48,655-Speed 130.10 samples/sec Loss 10.8318 Epoch: 6 Global Step: 60465 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:26:08,457-Speed 129.29 samples/sec Loss 10.8677 Epoch: 6 Global Step: 60470 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:26:28,067-Speed 130.56 samples/sec Loss 10.9952 Epoch: 6 Global Step: 60475 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:26:48,234-Speed 126.94 samples/sec Loss 11.0557 Epoch: 6 Global Step: 60480 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:27:08,302-Speed 127.57 samples/sec Loss 11.1421 Epoch: 6 Global Step: 60485 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:27:27,939-Speed 130.37 samples/sec Loss 10.9301 Epoch: 6 Global Step: 60490 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:27:47,832-Speed 128.70 samples/sec Loss 11.0752 Epoch: 6 Global Step: 60495 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:28:07,653-Speed 129.17 samples/sec Loss 10.9909 Epoch: 6 Global Step: 60500 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:28:27,610-Speed 128.28 samples/sec Loss 10.5301 Epoch: 6 Global Step: 60505 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:28:47,255-Speed 130.32 samples/sec Loss 11.1478 Epoch: 6 Global Step: 60510 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:29:07,061-Speed 129.26 samples/sec Loss 10.8278 Epoch: 6 Global Step: 60515 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:29:26,686-Speed 130.45 samples/sec Loss 10.9150 Epoch: 6 Global Step: 60520 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:29:46,787-Speed 127.36 samples/sec Loss 10.6647 Epoch: 6 Global Step: 60525 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:30:06,812-Speed 127.85 samples/sec Loss 10.9751 Epoch: 6 Global Step: 60530 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:30:26,421-Speed 130.56 samples/sec Loss 10.9999 Epoch: 6 Global Step: 60535 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:30:46,364-Speed 128.38 samples/sec Loss 11.1203 Epoch: 6 Global Step: 60540 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:31:06,268-Speed 128.63 samples/sec Loss 10.8166 Epoch: 6 Global Step: 60545 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:31:26,020-Speed 129.61 samples/sec Loss 11.0073 Epoch: 6 Global Step: 60550 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:31:45,711-Speed 130.01 samples/sec Loss 10.7945 Epoch: 6 Global Step: 60555 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:32:05,515-Speed 129.27 samples/sec Loss 10.9830 Epoch: 6 Global Step: 60560 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:32:25,272-Speed 129.58 samples/sec Loss 10.9656 Epoch: 6 Global Step: 60565 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:32:45,281-Speed 127.95 samples/sec Loss 10.5314 Epoch: 6 Global Step: 60570 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:33:05,225-Speed 128.36 samples/sec Loss 11.1707 Epoch: 6 Global Step: 60575 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:33:24,883-Speed 130.24 samples/sec Loss 11.1257 Epoch: 6 Global Step: 60580 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:33:44,757-Speed 128.82 samples/sec Loss 11.0557 Epoch: 6 Global Step: 60585 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:34:04,494-Speed 129.71 samples/sec Loss 11.0144 Epoch: 6 Global Step: 60590 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:34:24,373-Speed 128.78 samples/sec Loss 10.6315 Epoch: 6 Global Step: 60595 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:34:44,005-Speed 130.41 samples/sec Loss 11.0121 Epoch: 6 Global Step: 60600 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:35:03,971-Speed 128.23 samples/sec Loss 11.0925 Epoch: 6 Global Step: 60605 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:35:23,581-Speed 130.55 samples/sec Loss 10.8290 Epoch: 6 Global Step: 60610 Fp16 Grad Scale: 8192 Required: 226 hours Training: 2021-06-09 19:35:43,760-Speed 126.87 samples/sec Loss 10.8728 Epoch: 6 Global Step: 60615 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:36:03,534-Speed 129.47 samples/sec Loss 10.7709 Epoch: 6 Global Step: 60620 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:36:23,229-Speed 129.99 samples/sec Loss 10.6473 Epoch: 6 Global Step: 60625 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:36:43,017-Speed 129.38 samples/sec Loss 11.2327 Epoch: 6 Global Step: 60630 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:37:02,647-Speed 130.42 samples/sec Loss 10.5740 Epoch: 6 Global Step: 60635 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:37:22,630-Speed 128.12 samples/sec Loss 10.8777 Epoch: 6 Global Step: 60640 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:37:42,292-Speed 130.21 samples/sec Loss 10.9610 Epoch: 6 Global Step: 60645 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:38:02,181-Speed 128.73 samples/sec Loss 11.3484 Epoch: 6 Global Step: 60650 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:38:21,807-Speed 130.44 samples/sec Loss 11.0408 Epoch: 6 Global Step: 60655 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:38:41,954-Speed 127.07 samples/sec Loss 10.9317 Epoch: 6 Global Step: 60660 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:39:01,833-Speed 128.79 samples/sec Loss 10.8770 Epoch: 6 Global Step: 60665 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:39:21,554-Speed 129.81 samples/sec Loss 10.9009 Epoch: 6 Global Step: 60670 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:39:41,342-Speed 129.38 samples/sec Loss 10.9467 Epoch: 6 Global Step: 60675 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:40:00,960-Speed 130.50 samples/sec Loss 10.9511 Epoch: 6 Global Step: 60680 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:40:20,856-Speed 128.67 samples/sec Loss 11.1878 Epoch: 6 Global Step: 60685 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:40:40,549-Speed 130.00 samples/sec Loss 10.9908 Epoch: 6 Global Step: 60690 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:41:00,521-Speed 128.19 samples/sec Loss 10.9673 Epoch: 6 Global Step: 60695 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:41:20,045-Speed 131.13 samples/sec Loss 11.1472 Epoch: 6 Global Step: 60700 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:41:40,205-Speed 126.99 samples/sec Loss 10.9359 Epoch: 6 Global Step: 60705 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:42:00,118-Speed 128.57 samples/sec Loss 11.0452 Epoch: 6 Global Step: 60710 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:42:19,770-Speed 130.27 samples/sec Loss 11.0495 Epoch: 6 Global Step: 60715 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:42:39,510-Speed 129.69 samples/sec Loss 10.8096 Epoch: 6 Global Step: 60720 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:42:59,180-Speed 130.16 samples/sec Loss 11.0882 Epoch: 6 Global Step: 60725 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:43:18,994-Speed 129.21 samples/sec Loss 11.1803 Epoch: 6 Global Step: 60730 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:43:38,647-Speed 130.27 samples/sec Loss 10.8280 Epoch: 6 Global Step: 60735 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:43:58,543-Speed 128.68 samples/sec Loss 10.8869 Epoch: 6 Global Step: 60740 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:44:18,167-Speed 130.46 samples/sec Loss 10.9618 Epoch: 6 Global Step: 60745 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:44:38,326-Speed 127.00 samples/sec Loss 11.1706 Epoch: 6 Global Step: 60750 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:44:58,454-Speed 127.19 samples/sec Loss 11.1728 Epoch: 6 Global Step: 60755 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:45:18,056-Speed 130.60 samples/sec Loss 10.9533 Epoch: 6 Global Step: 60760 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:45:37,962-Speed 128.61 samples/sec Loss 11.1977 Epoch: 6 Global Step: 60765 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:45:57,645-Speed 130.07 samples/sec Loss 11.2112 Epoch: 6 Global Step: 60770 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:46:17,348-Speed 129.94 samples/sec Loss 11.1505 Epoch: 6 Global Step: 60775 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:46:37,052-Speed 129.93 samples/sec Loss 11.1580 Epoch: 6 Global Step: 60780 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:46:56,832-Speed 129.43 samples/sec Loss 11.0469 Epoch: 6 Global Step: 60785 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:47:16,463-Speed 130.41 samples/sec Loss 10.9911 Epoch: 6 Global Step: 60790 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:47:36,652-Speed 126.81 samples/sec Loss 11.2047 Epoch: 6 Global Step: 60795 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:47:56,630-Speed 128.15 samples/sec Loss 11.1187 Epoch: 6 Global Step: 60800 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:48:16,447-Speed 129.19 samples/sec Loss 11.1595 Epoch: 6 Global Step: 60805 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:48:36,412-Speed 128.23 samples/sec Loss 10.8131 Epoch: 6 Global Step: 60810 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:48:56,058-Speed 130.31 samples/sec Loss 11.0420 Epoch: 6 Global Step: 60815 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:49:15,917-Speed 128.91 samples/sec Loss 10.6648 Epoch: 6 Global Step: 60820 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:49:35,594-Speed 130.11 samples/sec Loss 11.0105 Epoch: 6 Global Step: 60825 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:49:55,322-Speed 129.77 samples/sec Loss 10.8600 Epoch: 6 Global Step: 60830 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:50:15,124-Speed 129.28 samples/sec Loss 10.6379 Epoch: 6 Global Step: 60835 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:50:35,110-Speed 128.09 samples/sec Loss 11.1422 Epoch: 6 Global Step: 60840 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:50:55,103-Speed 128.05 samples/sec Loss 10.8903 Epoch: 6 Global Step: 60845 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:51:14,935-Speed 129.09 samples/sec Loss 10.8730 Epoch: 6 Global Step: 60850 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:51:34,925-Speed 128.07 samples/sec Loss 11.4162 Epoch: 6 Global Step: 60855 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:51:54,542-Speed 130.51 samples/sec Loss 11.0558 Epoch: 6 Global Step: 60860 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:52:14,433-Speed 128.71 samples/sec Loss 11.0428 Epoch: 6 Global Step: 60865 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:52:33,987-Speed 130.92 samples/sec Loss 11.2355 Epoch: 6 Global Step: 60870 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:52:53,998-Speed 127.94 samples/sec Loss 11.0699 Epoch: 6 Global Step: 60875 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:53:13,648-Speed 130.29 samples/sec Loss 10.8318 Epoch: 6 Global Step: 60880 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:53:33,744-Speed 127.40 samples/sec Loss 10.7988 Epoch: 6 Global Step: 60885 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:53:53,506-Speed 129.55 samples/sec Loss 10.8494 Epoch: 6 Global Step: 60890 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:54:13,197-Speed 130.01 samples/sec Loss 11.0535 Epoch: 6 Global Step: 60895 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:54:33,262-Speed 127.59 samples/sec Loss 11.1620 Epoch: 6 Global Step: 60900 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:54:52,871-Speed 130.56 samples/sec Loss 10.8743 Epoch: 6 Global Step: 60905 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:55:12,862-Speed 128.06 samples/sec Loss 11.4827 Epoch: 6 Global Step: 60910 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:55:32,486-Speed 130.46 samples/sec Loss 10.8326 Epoch: 6 Global Step: 60915 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:55:52,344-Speed 128.92 samples/sec Loss 11.2257 Epoch: 6 Global Step: 60920 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:56:12,011-Speed 130.18 samples/sec Loss 11.1689 Epoch: 6 Global Step: 60925 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:56:32,089-Speed 127.51 samples/sec Loss 10.8793 Epoch: 6 Global Step: 60930 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:56:51,896-Speed 129.25 samples/sec Loss 11.2865 Epoch: 6 Global Step: 60935 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:57:11,570-Speed 130.13 samples/sec Loss 11.3452 Epoch: 6 Global Step: 60940 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:57:31,391-Speed 129.16 samples/sec Loss 10.8880 Epoch: 6 Global Step: 60945 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:57:50,995-Speed 130.60 samples/sec Loss 11.1024 Epoch: 6 Global Step: 60950 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:58:10,853-Speed 128.92 samples/sec Loss 11.1694 Epoch: 6 Global Step: 60955 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:58:30,571-Speed 129.83 samples/sec Loss 11.0651 Epoch: 6 Global Step: 60960 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:58:50,433-Speed 128.90 samples/sec Loss 11.0649 Epoch: 6 Global Step: 60965 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:59:10,066-Speed 130.40 samples/sec Loss 10.9597 Epoch: 6 Global Step: 60970 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:59:30,230-Speed 126.96 samples/sec Loss 11.0961 Epoch: 6 Global Step: 60975 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 19:59:50,185-Speed 128.30 samples/sec Loss 11.1217 Epoch: 6 Global Step: 60980 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:00:09,844-Speed 130.23 samples/sec Loss 10.8195 Epoch: 6 Global Step: 60985 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:00:29,659-Speed 129.20 samples/sec Loss 10.8372 Epoch: 6 Global Step: 60990 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:00:49,337-Speed 130.11 samples/sec Loss 10.7356 Epoch: 6 Global Step: 60995 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:01:09,123-Speed 129.39 samples/sec Loss 11.1959 Epoch: 6 Global Step: 61000 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:01:28,757-Speed 130.39 samples/sec Loss 10.9421 Epoch: 6 Global Step: 61005 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:01:48,749-Speed 128.06 samples/sec Loss 10.8905 Epoch: 6 Global Step: 61010 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:02:08,390-Speed 130.35 samples/sec Loss 11.2829 Epoch: 6 Global Step: 61015 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:02:28,451-Speed 127.62 samples/sec Loss 11.1417 Epoch: 6 Global Step: 61020 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:02:48,360-Speed 128.59 samples/sec Loss 11.1737 Epoch: 6 Global Step: 61025 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:03:08,012-Speed 130.27 samples/sec Loss 10.9486 Epoch: 6 Global Step: 61030 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:03:27,688-Speed 130.12 samples/sec Loss 10.8511 Epoch: 6 Global Step: 61035 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:03:47,364-Speed 130.12 samples/sec Loss 11.2654 Epoch: 6 Global Step: 61040 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:04:07,168-Speed 129.27 samples/sec Loss 11.0162 Epoch: 6 Global Step: 61045 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:04:26,787-Speed 130.50 samples/sec Loss 10.7333 Epoch: 6 Global Step: 61050 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:04:46,728-Speed 128.38 samples/sec Loss 11.1229 Epoch: 6 Global Step: 61055 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:05:06,263-Speed 131.06 samples/sec Loss 11.0908 Epoch: 6 Global Step: 61060 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:05:26,488-Speed 126.58 samples/sec Loss 11.1548 Epoch: 6 Global Step: 61065 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:05:46,580-Speed 127.42 samples/sec Loss 10.7705 Epoch: 6 Global Step: 61070 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:06:06,188-Speed 130.56 samples/sec Loss 10.8139 Epoch: 6 Global Step: 61075 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:06:26,021-Speed 129.09 samples/sec Loss 11.0988 Epoch: 6 Global Step: 61080 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:06:45,670-Speed 130.29 samples/sec Loss 10.7513 Epoch: 6 Global Step: 61085 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:07:05,390-Speed 129.83 samples/sec Loss 11.1292 Epoch: 6 Global Step: 61090 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:07:25,081-Speed 130.02 samples/sec Loss 10.9927 Epoch: 6 Global Step: 61095 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:07:44,829-Speed 129.64 samples/sec Loss 11.1880 Epoch: 6 Global Step: 61100 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:08:04,568-Speed 129.70 samples/sec Loss 10.7923 Epoch: 6 Global Step: 61105 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:08:24,619-Speed 127.68 samples/sec Loss 10.6974 Epoch: 6 Global Step: 61110 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:08:44,630-Speed 127.94 samples/sec Loss 10.8105 Epoch: 6 Global Step: 61115 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:09:04,158-Speed 131.10 samples/sec Loss 10.8079 Epoch: 6 Global Step: 61120 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:09:24,210-Speed 127.67 samples/sec Loss 11.2266 Epoch: 6 Global Step: 61125 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:09:43,829-Speed 130.50 samples/sec Loss 11.1554 Epoch: 6 Global Step: 61130 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:10:03,797-Speed 128.21 samples/sec Loss 10.8875 Epoch: 6 Global Step: 61135 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:10:23,411-Speed 130.53 samples/sec Loss 10.7381 Epoch: 6 Global Step: 61140 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:10:43,215-Speed 129.27 samples/sec Loss 11.1904 Epoch: 6 Global Step: 61145 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:11:03,087-Speed 128.84 samples/sec Loss 11.2690 Epoch: 6 Global Step: 61150 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:11:23,307-Speed 126.61 samples/sec Loss 11.0970 Epoch: 6 Global Step: 61155 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:11:43,000-Speed 130.00 samples/sec Loss 10.7740 Epoch: 6 Global Step: 61160 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:12:02,670-Speed 130.15 samples/sec Loss 10.9151 Epoch: 6 Global Step: 61165 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:12:22,587-Speed 128.54 samples/sec Loss 11.1626 Epoch: 6 Global Step: 61170 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:12:42,222-Speed 130.39 samples/sec Loss 11.1033 Epoch: 6 Global Step: 61175 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:13:02,183-Speed 128.26 samples/sec Loss 10.9627 Epoch: 6 Global Step: 61180 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:13:21,708-Speed 131.12 samples/sec Loss 11.0388 Epoch: 6 Global Step: 61185 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:13:41,754-Speed 127.71 samples/sec Loss 10.9182 Epoch: 6 Global Step: 61190 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:14:01,380-Speed 130.45 samples/sec Loss 11.2808 Epoch: 6 Global Step: 61195 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:14:21,558-Speed 126.87 samples/sec Loss 11.3325 Epoch: 6 Global Step: 61200 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:14:41,349-Speed 129.36 samples/sec Loss 11.2924 Epoch: 6 Global Step: 61205 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:15:01,073-Speed 129.79 samples/sec Loss 10.9996 Epoch: 6 Global Step: 61210 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:15:20,870-Speed 129.32 samples/sec Loss 11.1787 Epoch: 6 Global Step: 61215 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:15:40,418-Speed 130.97 samples/sec Loss 10.9280 Epoch: 6 Global Step: 61220 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:16:00,236-Speed 129.19 samples/sec Loss 10.8935 Epoch: 6 Global Step: 61225 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:16:19,848-Speed 130.54 samples/sec Loss 10.8806 Epoch: 6 Global Step: 61230 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:16:39,820-Speed 128.19 samples/sec Loss 11.2632 Epoch: 6 Global Step: 61235 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:16:59,449-Speed 130.43 samples/sec Loss 11.1956 Epoch: 6 Global Step: 61240 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:17:19,610-Speed 126.98 samples/sec Loss 10.9317 Epoch: 6 Global Step: 61245 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:17:39,873-Speed 126.35 samples/sec Loss 11.2293 Epoch: 6 Global Step: 61250 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:17:59,683-Speed 129.23 samples/sec Loss 11.0044 Epoch: 6 Global Step: 61255 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:18:19,549-Speed 128.87 samples/sec Loss 11.1293 Epoch: 6 Global Step: 61260 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:18:39,203-Speed 130.26 samples/sec Loss 11.0722 Epoch: 6 Global Step: 61265 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:18:58,984-Speed 129.42 samples/sec Loss 10.8026 Epoch: 6 Global Step: 61270 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:19:18,682-Speed 129.97 samples/sec Loss 10.9540 Epoch: 6 Global Step: 61275 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:19:38,466-Speed 129.41 samples/sec Loss 10.7644 Epoch: 6 Global Step: 61280 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:19:58,039-Speed 130.80 samples/sec Loss 10.9299 Epoch: 6 Global Step: 61285 Fp16 Grad Scale: 8192 Required: 225 hours Training: 2021-06-09 20:20:18,053-Speed 127.92 samples/sec Loss 11.4504 Epoch: 6 Global Step: 61290 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:20:38,171-Speed 127.26 samples/sec Loss 10.7333 Epoch: 6 Global Step: 61295 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:20:57,783-Speed 130.53 samples/sec Loss 11.2043 Epoch: 6 Global Step: 61300 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:21:17,642-Speed 128.92 samples/sec Loss 10.8975 Epoch: 6 Global Step: 61305 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:21:37,273-Speed 130.41 samples/sec Loss 11.0854 Epoch: 6 Global Step: 61310 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:21:57,028-Speed 129.60 samples/sec Loss 10.9594 Epoch: 6 Global Step: 61315 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:22:16,711-Speed 130.07 samples/sec Loss 11.2437 Epoch: 6 Global Step: 61320 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:22:36,517-Speed 129.26 samples/sec Loss 10.7797 Epoch: 6 Global Step: 61325 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:22:56,139-Speed 130.48 samples/sec Loss 11.0975 Epoch: 6 Global Step: 61330 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:23:16,295-Speed 127.01 samples/sec Loss 11.0165 Epoch: 6 Global Step: 61335 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:23:36,422-Speed 127.20 samples/sec Loss 10.9687 Epoch: 6 Global Step: 61340 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:23:56,084-Speed 130.21 samples/sec Loss 11.2266 Epoch: 6 Global Step: 61345 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:24:15,806-Speed 129.81 samples/sec Loss 10.9861 Epoch: 6 Global Step: 61350 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:24:35,467-Speed 130.22 samples/sec Loss 10.8339 Epoch: 6 Global Step: 61355 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:24:55,314-Speed 128.99 samples/sec Loss 11.5500 Epoch: 6 Global Step: 61360 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:25:14,991-Speed 130.11 samples/sec Loss 10.9949 Epoch: 6 Global Step: 61365 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:25:34,768-Speed 129.45 samples/sec Loss 11.2592 Epoch: 6 Global Step: 61370 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:25:54,398-Speed 130.42 samples/sec Loss 10.8866 Epoch: 6 Global Step: 61375 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:26:14,416-Speed 127.89 samples/sec Loss 11.2911 Epoch: 6 Global Step: 61380 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:26:34,585-Speed 126.93 samples/sec Loss 10.9564 Epoch: 6 Global Step: 61385 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:26:54,190-Speed 130.59 samples/sec Loss 10.5419 Epoch: 6 Global Step: 61390 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:27:13,999-Speed 129.24 samples/sec Loss 11.2368 Epoch: 6 Global Step: 61395 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:27:33,687-Speed 130.04 samples/sec Loss 10.7294 Epoch: 6 Global Step: 61400 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:27:53,503-Speed 129.19 samples/sec Loss 11.0897 Epoch: 6 Global Step: 61405 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:28:13,175-Speed 130.14 samples/sec Loss 11.2305 Epoch: 6 Global Step: 61410 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:28:32,948-Speed 129.48 samples/sec Loss 11.0360 Epoch: 6 Global Step: 61415 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:28:52,674-Speed 129.78 samples/sec Loss 10.9796 Epoch: 6 Global Step: 61420 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:29:12,867-Speed 126.79 samples/sec Loss 11.4564 Epoch: 6 Global Step: 61425 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:29:32,900-Speed 127.79 samples/sec Loss 11.0460 Epoch: 6 Global Step: 61430 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:29:52,500-Speed 130.62 samples/sec Loss 11.0709 Epoch: 6 Global Step: 61435 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:30:12,449-Speed 128.33 samples/sec Loss 10.8727 Epoch: 6 Global Step: 61440 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:30:32,080-Speed 130.42 samples/sec Loss 11.0107 Epoch: 6 Global Step: 61445 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:30:52,071-Speed 128.06 samples/sec Loss 11.0154 Epoch: 6 Global Step: 61450 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:31:11,574-Speed 131.27 samples/sec Loss 11.2066 Epoch: 6 Global Step: 61455 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:31:31,360-Speed 129.40 samples/sec Loss 11.0566 Epoch: 6 Global Step: 61460 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:31:51,030-Speed 130.15 samples/sec Loss 10.8491 Epoch: 6 Global Step: 61465 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:32:11,284-Speed 126.40 samples/sec Loss 11.0141 Epoch: 6 Global Step: 61470 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:32:31,072-Speed 129.38 samples/sec Loss 10.9407 Epoch: 6 Global Step: 61475 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:32:50,594-Speed 131.14 samples/sec Loss 10.8665 Epoch: 6 Global Step: 61480 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:33:10,613-Speed 127.88 samples/sec Loss 11.0469 Epoch: 6 Global Step: 61485 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:33:30,249-Speed 130.38 samples/sec Loss 11.0598 Epoch: 6 Global Step: 61490 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:33:50,099-Speed 128.98 samples/sec Loss 11.0611 Epoch: 6 Global Step: 61495 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:34:09,715-Speed 130.51 samples/sec Loss 11.2246 Epoch: 6 Global Step: 61500 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:34:29,688-Speed 128.18 samples/sec Loss 10.9511 Epoch: 6 Global Step: 61505 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:34:49,330-Speed 130.34 samples/sec Loss 11.1688 Epoch: 6 Global Step: 61510 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:35:09,459-Speed 127.19 samples/sec Loss 11.6593 Epoch: 6 Global Step: 61515 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:35:29,139-Speed 130.08 samples/sec Loss 11.1427 Epoch: 6 Global Step: 61520 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:35:48,816-Speed 130.11 samples/sec Loss 10.8846 Epoch: 6 Global Step: 61525 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:36:08,642-Speed 129.13 samples/sec Loss 11.1477 Epoch: 6 Global Step: 61530 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:36:28,248-Speed 130.58 samples/sec Loss 10.9860 Epoch: 6 Global Step: 61535 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:36:48,076-Speed 129.12 samples/sec Loss 10.9697 Epoch: 6 Global Step: 61540 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:37:07,735-Speed 130.23 samples/sec Loss 10.9639 Epoch: 6 Global Step: 61545 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:37:27,764-Speed 127.82 samples/sec Loss 11.0878 Epoch: 6 Global Step: 61550 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:37:47,293-Speed 131.09 samples/sec Loss 10.9411 Epoch: 6 Global Step: 61555 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:38:07,429-Speed 127.15 samples/sec Loss 11.1924 Epoch: 6 Global Step: 61560 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:38:27,289-Speed 128.90 samples/sec Loss 10.8638 Epoch: 6 Global Step: 61565 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:38:46,973-Speed 130.06 samples/sec Loss 10.9696 Epoch: 6 Global Step: 61570 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:39:06,790-Speed 129.19 samples/sec Loss 10.9677 Epoch: 6 Global Step: 61575 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:39:26,458-Speed 130.17 samples/sec Loss 11.3193 Epoch: 6 Global Step: 61580 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:39:46,253-Speed 129.34 samples/sec Loss 10.7042 Epoch: 6 Global Step: 61585 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:40:05,935-Speed 130.07 samples/sec Loss 10.9659 Epoch: 6 Global Step: 61590 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:40:25,746-Speed 129.23 samples/sec Loss 11.1868 Epoch: 6 Global Step: 61595 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:40:45,377-Speed 130.41 samples/sec Loss 10.9918 Epoch: 6 Global Step: 61600 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:41:05,428-Speed 127.68 samples/sec Loss 10.9933 Epoch: 6 Global Step: 61605 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:41:25,506-Speed 127.50 samples/sec Loss 11.3242 Epoch: 6 Global Step: 61610 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:41:45,125-Speed 130.49 samples/sec Loss 11.0012 Epoch: 6 Global Step: 61615 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:42:05,099-Speed 128.17 samples/sec Loss 10.7186 Epoch: 6 Global Step: 61620 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:42:24,696-Speed 130.64 samples/sec Loss 10.9257 Epoch: 6 Global Step: 61625 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:42:44,503-Speed 129.25 samples/sec Loss 11.1812 Epoch: 6 Global Step: 61630 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:43:04,169-Speed 130.18 samples/sec Loss 10.8804 Epoch: 6 Global Step: 61635 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:43:23,979-Speed 129.23 samples/sec Loss 10.9963 Epoch: 6 Global Step: 61640 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:43:43,653-Speed 130.13 samples/sec Loss 11.2112 Epoch: 6 Global Step: 61645 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:44:03,701-Speed 127.70 samples/sec Loss 10.9054 Epoch: 6 Global Step: 61650 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:44:23,643-Speed 128.37 samples/sec Loss 11.1509 Epoch: 6 Global Step: 61655 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:44:43,276-Speed 130.41 samples/sec Loss 10.9826 Epoch: 6 Global Step: 61660 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:45:03,252-Speed 128.16 samples/sec Loss 11.0377 Epoch: 6 Global Step: 61665 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:45:22,757-Speed 131.26 samples/sec Loss 11.2214 Epoch: 6 Global Step: 61670 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:45:42,804-Speed 127.71 samples/sec Loss 10.8871 Epoch: 6 Global Step: 61675 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:46:02,432-Speed 130.43 samples/sec Loss 11.1263 Epoch: 6 Global Step: 61680 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:46:22,269-Speed 129.06 samples/sec Loss 11.4374 Epoch: 6 Global Step: 61685 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:46:41,918-Speed 130.29 samples/sec Loss 10.8335 Epoch: 6 Global Step: 61690 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:47:02,044-Speed 127.20 samples/sec Loss 11.2232 Epoch: 6 Global Step: 61695 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:47:21,852-Speed 129.25 samples/sec Loss 11.3599 Epoch: 6 Global Step: 61700 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:47:41,471-Speed 130.49 samples/sec Loss 11.0565 Epoch: 6 Global Step: 61705 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:48:01,299-Speed 129.12 samples/sec Loss 11.0203 Epoch: 6 Global Step: 61710 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:48:20,975-Speed 130.12 samples/sec Loss 11.6272 Epoch: 6 Global Step: 61715 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:48:40,932-Speed 128.28 samples/sec Loss 10.8986 Epoch: 6 Global Step: 61720 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:49:00,560-Speed 130.44 samples/sec Loss 11.2828 Epoch: 6 Global Step: 61725 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:49:20,524-Speed 128.24 samples/sec Loss 10.9625 Epoch: 6 Global Step: 61730 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:49:40,033-Speed 131.23 samples/sec Loss 11.1523 Epoch: 6 Global Step: 61735 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:50:00,876-Speed 122.83 samples/sec Loss 10.9968 Epoch: 6 Global Step: 61740 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:50:20,612-Speed 129.72 samples/sec Loss 11.1295 Epoch: 6 Global Step: 61745 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:50:40,308-Speed 129.98 samples/sec Loss 11.0459 Epoch: 6 Global Step: 61750 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:51:00,089-Speed 129.42 samples/sec Loss 11.0824 Epoch: 6 Global Step: 61755 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:51:19,848-Speed 129.57 samples/sec Loss 11.1505 Epoch: 6 Global Step: 61760 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:51:39,742-Speed 128.69 samples/sec Loss 11.3240 Epoch: 6 Global Step: 61765 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:51:59,449-Speed 129.91 samples/sec Loss 11.0392 Epoch: 6 Global Step: 61770 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:52:19,270-Speed 129.16 samples/sec Loss 10.9833 Epoch: 6 Global Step: 61775 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:52:38,960-Speed 130.02 samples/sec Loss 10.9025 Epoch: 6 Global Step: 61780 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:52:59,101-Speed 127.11 samples/sec Loss 10.6426 Epoch: 6 Global Step: 61785 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:53:18,930-Speed 129.11 samples/sec Loss 10.9685 Epoch: 6 Global Step: 61790 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:53:38,575-Speed 130.32 samples/sec Loss 10.7908 Epoch: 6 Global Step: 61795 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:53:58,422-Speed 128.99 samples/sec Loss 10.5980 Epoch: 6 Global Step: 61800 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:54:18,138-Speed 129.86 samples/sec Loss 10.8320 Epoch: 6 Global Step: 61805 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:54:37,950-Speed 129.22 samples/sec Loss 10.9209 Epoch: 6 Global Step: 61810 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:54:57,634-Speed 130.06 samples/sec Loss 11.3430 Epoch: 6 Global Step: 61815 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:55:17,402-Speed 129.51 samples/sec Loss 11.1001 Epoch: 6 Global Step: 61820 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:55:37,037-Speed 130.38 samples/sec Loss 11.3054 Epoch: 6 Global Step: 61825 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:55:57,167-Speed 127.18 samples/sec Loss 10.8846 Epoch: 6 Global Step: 61830 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:56:17,127-Speed 128.26 samples/sec Loss 11.0050 Epoch: 6 Global Step: 61835 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:56:36,801-Speed 130.15 samples/sec Loss 10.9738 Epoch: 6 Global Step: 61840 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:56:56,780-Speed 128.14 samples/sec Loss 11.0502 Epoch: 6 Global Step: 61845 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:57:16,383-Speed 130.60 samples/sec Loss 11.0094 Epoch: 6 Global Step: 61850 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:57:36,140-Speed 129.58 samples/sec Loss 11.2700 Epoch: 6 Global Step: 61855 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:57:55,750-Speed 130.55 samples/sec Loss 10.7400 Epoch: 6 Global Step: 61860 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:58:15,526-Speed 129.46 samples/sec Loss 11.0448 Epoch: 6 Global Step: 61865 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:58:35,205-Speed 130.10 samples/sec Loss 11.1531 Epoch: 6 Global Step: 61870 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:58:55,359-Speed 127.03 samples/sec Loss 10.9066 Epoch: 6 Global Step: 61875 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:59:15,165-Speed 129.26 samples/sec Loss 11.2531 Epoch: 6 Global Step: 61880 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:59:34,775-Speed 130.55 samples/sec Loss 11.2009 Epoch: 6 Global Step: 61885 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 20:59:54,716-Speed 128.38 samples/sec Loss 11.3472 Epoch: 6 Global Step: 61890 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:00:14,206-Speed 131.36 samples/sec Loss 11.1097 Epoch: 6 Global Step: 61895 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:00:34,292-Speed 127.46 samples/sec Loss 11.4162 Epoch: 6 Global Step: 61900 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:00:53,961-Speed 130.16 samples/sec Loss 10.7811 Epoch: 6 Global Step: 61905 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:01:13,877-Speed 128.55 samples/sec Loss 11.0739 Epoch: 6 Global Step: 61910 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:01:33,488-Speed 130.55 samples/sec Loss 10.9350 Epoch: 6 Global Step: 61915 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:01:53,514-Speed 127.84 samples/sec Loss 11.0908 Epoch: 6 Global Step: 61920 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:02:13,301-Speed 129.38 samples/sec Loss 10.8834 Epoch: 6 Global Step: 61925 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:02:32,931-Speed 130.42 samples/sec Loss 10.7936 Epoch: 6 Global Step: 61930 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:02:52,800-Speed 128.85 samples/sec Loss 10.9473 Epoch: 6 Global Step: 61935 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:03:12,537-Speed 129.71 samples/sec Loss 11.1184 Epoch: 6 Global Step: 61940 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:03:32,463-Speed 128.48 samples/sec Loss 10.7831 Epoch: 6 Global Step: 61945 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:03:52,096-Speed 130.40 samples/sec Loss 11.0795 Epoch: 6 Global Step: 61950 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:04:12,115-Speed 127.89 samples/sec Loss 10.9191 Epoch: 6 Global Step: 61955 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:04:31,661-Speed 130.98 samples/sec Loss 11.2660 Epoch: 6 Global Step: 61960 Fp16 Grad Scale: 8192 Required: 224 hours Training: 2021-06-09 21:04:51,690-Speed 127.82 samples/sec Loss 11.3697 Epoch: 6 Global Step: 61965 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:05:11,790-Speed 127.37 samples/sec Loss 11.2239 Epoch: 6 Global Step: 61970 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:05:31,435-Speed 130.32 samples/sec Loss 11.1177 Epoch: 6 Global Step: 61975 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:05:51,132-Speed 129.97 samples/sec Loss 10.9005 Epoch: 6 Global Step: 61980 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:06:10,839-Speed 129.91 samples/sec Loss 11.4247 Epoch: 6 Global Step: 61985 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:06:30,801-Speed 128.25 samples/sec Loss 11.3369 Epoch: 6 Global Step: 61990 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:06:50,418-Speed 130.51 samples/sec Loss 11.0268 Epoch: 6 Global Step: 61995 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:07:10,341-Speed 128.50 samples/sec Loss 10.8920 Epoch: 6 Global Step: 62000 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:07:29,859-Speed 131.17 samples/sec Loss 10.7191 Epoch: 6 Global Step: 62005 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:07:50,006-Speed 127.07 samples/sec Loss 10.9208 Epoch: 6 Global Step: 62010 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:08:09,938-Speed 128.45 samples/sec Loss 11.1182 Epoch: 6 Global Step: 62015 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:08:29,576-Speed 130.37 samples/sec Loss 11.0473 Epoch: 6 Global Step: 62020 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:08:49,255-Speed 130.10 samples/sec Loss 10.7668 Epoch: 6 Global Step: 62025 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:09:08,956-Speed 129.95 samples/sec Loss 10.8093 Epoch: 6 Global Step: 62030 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:09:28,745-Speed 129.37 samples/sec Loss 11.2517 Epoch: 6 Global Step: 62035 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:09:48,360-Speed 130.52 samples/sec Loss 11.1940 Epoch: 6 Global Step: 62040 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:10:08,186-Speed 129.13 samples/sec Loss 10.9126 Epoch: 6 Global Step: 62045 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:10:27,811-Speed 130.45 samples/sec Loss 10.9696 Epoch: 6 Global Step: 62050 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:10:48,047-Speed 126.52 samples/sec Loss 11.5191 Epoch: 6 Global Step: 62055 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:11:08,119-Speed 127.54 samples/sec Loss 10.8918 Epoch: 6 Global Step: 62060 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:11:27,611-Speed 131.34 samples/sec Loss 11.0707 Epoch: 6 Global Step: 62065 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:11:47,364-Speed 129.61 samples/sec Loss 10.9836 Epoch: 6 Global Step: 62070 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:12:07,012-Speed 130.30 samples/sec Loss 11.0172 Epoch: 6 Global Step: 62075 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:12:26,872-Speed 128.91 samples/sec Loss 10.9696 Epoch: 6 Global Step: 62080 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:12:46,691-Speed 129.18 samples/sec Loss 11.0399 Epoch: 6 Global Step: 62085 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:13:06,360-Speed 130.16 samples/sec Loss 10.7827 Epoch: 6 Global Step: 62090 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:13:26,031-Speed 130.15 samples/sec Loss 11.1946 Epoch: 6 Global Step: 62095 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:13:46,141-Speed 127.30 samples/sec Loss 10.8136 Epoch: 6 Global Step: 62100 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:14:06,083-Speed 128.38 samples/sec Loss 11.1155 Epoch: 6 Global Step: 62105 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:14:25,591-Speed 131.24 samples/sec Loss 11.0025 Epoch: 6 Global Step: 62110 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:14:45,624-Speed 127.80 samples/sec Loss 11.1753 Epoch: 6 Global Step: 62115 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:15:05,247-Speed 130.46 samples/sec Loss 11.1361 Epoch: 6 Global Step: 62120 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:15:25,201-Speed 128.30 samples/sec Loss 11.3821 Epoch: 6 Global Step: 62125 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:15:44,711-Speed 131.23 samples/sec Loss 11.0338 Epoch: 6 Global Step: 62130 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:16:04,517-Speed 129.26 samples/sec Loss 10.9003 Epoch: 6 Global Step: 62135 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:16:24,171-Speed 130.26 samples/sec Loss 11.5397 Epoch: 6 Global Step: 62140 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:16:44,071-Speed 128.65 samples/sec Loss 10.9896 Epoch: 6 Global Step: 62145 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:17:04,716-Speed 124.01 samples/sec Loss 11.0289 Epoch: 6 Global Step: 62150 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:17:24,382-Speed 130.18 samples/sec Loss 10.7098 Epoch: 6 Global Step: 62155 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:17:44,468-Speed 127.46 samples/sec Loss 10.7573 Epoch: 6 Global Step: 62160 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:18:04,116-Speed 130.30 samples/sec Loss 10.9923 Epoch: 6 Global Step: 62165 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:18:24,002-Speed 128.74 samples/sec Loss 11.0943 Epoch: 6 Global Step: 62170 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:18:43,594-Speed 130.67 samples/sec Loss 11.1677 Epoch: 6 Global Step: 62175 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:19:03,496-Speed 128.64 samples/sec Loss 10.6459 Epoch: 6 Global Step: 62180 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:19:23,169-Speed 130.14 samples/sec Loss 11.1718 Epoch: 6 Global Step: 62185 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:19:43,433-Speed 126.33 samples/sec Loss 10.9455 Epoch: 6 Global Step: 62190 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:20:03,205-Speed 129.48 samples/sec Loss 11.0763 Epoch: 6 Global Step: 62195 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:20:22,836-Speed 130.41 samples/sec Loss 10.9134 Epoch: 6 Global Step: 62200 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:20:42,737-Speed 128.65 samples/sec Loss 11.0529 Epoch: 6 Global Step: 62205 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:21:02,361-Speed 130.46 samples/sec Loss 10.8577 Epoch: 6 Global Step: 62210 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:21:22,219-Speed 128.92 samples/sec Loss 11.1807 Epoch: 6 Global Step: 62215 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:21:41,894-Speed 130.12 samples/sec Loss 10.9316 Epoch: 6 Global Step: 62220 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:22:01,772-Speed 128.79 samples/sec Loss 10.9797 Epoch: 6 Global Step: 62225 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:22:21,384-Speed 130.54 samples/sec Loss 11.1228 Epoch: 6 Global Step: 62230 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:22:41,597-Speed 126.65 samples/sec Loss 11.0146 Epoch: 6 Global Step: 62235 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:23:01,441-Speed 129.01 samples/sec Loss 11.0693 Epoch: 6 Global Step: 62240 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:23:21,222-Speed 129.42 samples/sec Loss 11.3089 Epoch: 6 Global Step: 62245 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:23:40,912-Speed 130.03 samples/sec Loss 10.7123 Epoch: 6 Global Step: 62250 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:24:00,576-Speed 130.19 samples/sec Loss 11.0932 Epoch: 6 Global Step: 62255 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:24:20,496-Speed 128.52 samples/sec Loss 10.8000 Epoch: 6 Global Step: 62260 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:24:40,983-Speed 124.97 samples/sec Loss 11.1476 Epoch: 6 Global Step: 62265 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:25:00,969-Speed 128.10 samples/sec Loss 10.8115 Epoch: 6 Global Step: 62270 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:25:20,588-Speed 130.49 samples/sec Loss 10.9423 Epoch: 6 Global Step: 62275 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:25:40,742-Speed 127.03 samples/sec Loss 11.0854 Epoch: 6 Global Step: 62280 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:26:00,578-Speed 129.06 samples/sec Loss 11.1084 Epoch: 6 Global Step: 62285 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:26:20,263-Speed 130.06 samples/sec Loss 10.8393 Epoch: 6 Global Step: 62290 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:26:40,054-Speed 129.36 samples/sec Loss 11.1281 Epoch: 6 Global Step: 62295 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:26:59,780-Speed 129.78 samples/sec Loss 10.8564 Epoch: 6 Global Step: 62300 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:27:19,614-Speed 129.08 samples/sec Loss 11.0478 Epoch: 6 Global Step: 62305 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:27:39,251-Speed 130.37 samples/sec Loss 10.9140 Epoch: 6 Global Step: 62310 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:27:59,117-Speed 128.88 samples/sec Loss 11.0022 Epoch: 6 Global Step: 62315 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:28:18,737-Speed 130.49 samples/sec Loss 10.8639 Epoch: 6 Global Step: 62320 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:28:38,890-Speed 127.04 samples/sec Loss 11.0175 Epoch: 6 Global Step: 62325 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:28:58,894-Speed 127.98 samples/sec Loss 11.0585 Epoch: 6 Global Step: 62330 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:29:18,525-Speed 130.41 samples/sec Loss 11.0052 Epoch: 6 Global Step: 62335 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:29:38,261-Speed 129.72 samples/sec Loss 10.9543 Epoch: 6 Global Step: 62340 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:29:57,940-Speed 130.10 samples/sec Loss 10.7743 Epoch: 6 Global Step: 62345 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:30:17,740-Speed 129.30 samples/sec Loss 11.0120 Epoch: 6 Global Step: 62350 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:30:37,284-Speed 130.99 samples/sec Loss 10.8304 Epoch: 6 Global Step: 62355 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:30:57,205-Speed 128.51 samples/sec Loss 10.7682 Epoch: 6 Global Step: 62360 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:31:16,849-Speed 130.33 samples/sec Loss 10.9517 Epoch: 6 Global Step: 62365 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:31:37,080-Speed 126.54 samples/sec Loss 11.1660 Epoch: 6 Global Step: 62370 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:31:57,084-Speed 127.98 samples/sec Loss 10.6998 Epoch: 6 Global Step: 62375 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:32:16,677-Speed 130.67 samples/sec Loss 10.6597 Epoch: 6 Global Step: 62380 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:32:36,635-Speed 128.27 samples/sec Loss 11.1319 Epoch: 6 Global Step: 62385 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:32:56,307-Speed 130.14 samples/sec Loss 11.1634 Epoch: 6 Global Step: 62390 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:33:16,109-Speed 129.29 samples/sec Loss 11.0844 Epoch: 6 Global Step: 62395 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:33:35,741-Speed 130.41 samples/sec Loss 11.4369 Epoch: 6 Global Step: 62400 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:33:55,399-Speed 130.23 samples/sec Loss 10.7001 Epoch: 6 Global Step: 62405 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:34:15,103-Speed 129.93 samples/sec Loss 11.2466 Epoch: 6 Global Step: 62410 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:34:35,149-Speed 127.72 samples/sec Loss 11.0968 Epoch: 6 Global Step: 62415 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:34:55,150-Speed 128.00 samples/sec Loss 11.4430 Epoch: 6 Global Step: 62420 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:35:14,981-Speed 129.10 samples/sec Loss 11.2111 Epoch: 6 Global Step: 62425 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:35:35,013-Speed 127.80 samples/sec Loss 11.1858 Epoch: 6 Global Step: 62430 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:35:54,636-Speed 130.47 samples/sec Loss 10.9667 Epoch: 6 Global Step: 62435 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:36:14,374-Speed 129.71 samples/sec Loss 11.2888 Epoch: 6 Global Step: 62440 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:36:34,012-Speed 130.36 samples/sec Loss 11.0570 Epoch: 6 Global Step: 62445 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:36:53,803-Speed 129.36 samples/sec Loss 11.1904 Epoch: 6 Global Step: 62450 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:37:13,493-Speed 130.03 samples/sec Loss 11.1466 Epoch: 6 Global Step: 62455 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:37:33,797-Speed 126.09 samples/sec Loss 11.2270 Epoch: 6 Global Step: 62460 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:37:53,614-Speed 129.19 samples/sec Loss 11.0606 Epoch: 6 Global Step: 62465 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:38:13,220-Speed 130.58 samples/sec Loss 11.3334 Epoch: 6 Global Step: 62470 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:38:33,243-Speed 127.86 samples/sec Loss 11.0915 Epoch: 6 Global Step: 62475 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:38:52,831-Speed 130.70 samples/sec Loss 11.0229 Epoch: 6 Global Step: 62480 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:39:12,806-Speed 128.17 samples/sec Loss 10.9721 Epoch: 6 Global Step: 62485 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:39:32,339-Speed 131.06 samples/sec Loss 10.9754 Epoch: 6 Global Step: 62490 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:39:53,070-Speed 123.50 samples/sec Loss 10.8715 Epoch: 6 Global Step: 62495 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:40:12,702-Speed 130.41 samples/sec Loss 10.5865 Epoch: 6 Global Step: 62500 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:40:32,850-Speed 127.06 samples/sec Loss 10.9467 Epoch: 6 Global Step: 62505 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:40:52,627-Speed 129.45 samples/sec Loss 10.9900 Epoch: 6 Global Step: 62510 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:41:12,308-Speed 130.08 samples/sec Loss 10.9419 Epoch: 6 Global Step: 62515 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:41:32,267-Speed 128.27 samples/sec Loss 11.0434 Epoch: 6 Global Step: 62520 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:41:51,874-Speed 130.57 samples/sec Loss 10.9012 Epoch: 6 Global Step: 62525 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:42:11,713-Speed 129.05 samples/sec Loss 10.9994 Epoch: 6 Global Step: 62530 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:42:31,406-Speed 130.00 samples/sec Loss 11.1587 Epoch: 6 Global Step: 62535 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:42:51,362-Speed 128.29 samples/sec Loss 11.1305 Epoch: 6 Global Step: 62540 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:43:10,983-Speed 130.48 samples/sec Loss 11.1270 Epoch: 6 Global Step: 62545 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:43:31,298-Speed 126.02 samples/sec Loss 10.8162 Epoch: 6 Global Step: 62550 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:43:51,149-Speed 128.96 samples/sec Loss 10.7288 Epoch: 6 Global Step: 62555 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:44:10,829-Speed 130.09 samples/sec Loss 11.0536 Epoch: 6 Global Step: 62560 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:44:30,655-Speed 129.13 samples/sec Loss 10.9436 Epoch: 6 Global Step: 62565 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:44:50,306-Speed 130.28 samples/sec Loss 10.9376 Epoch: 6 Global Step: 62570 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:45:10,159-Speed 128.96 samples/sec Loss 11.0727 Epoch: 6 Global Step: 62575 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:45:29,787-Speed 130.43 samples/sec Loss 11.0900 Epoch: 6 Global Step: 62580 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:45:49,747-Speed 128.26 samples/sec Loss 11.1122 Epoch: 6 Global Step: 62585 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:46:09,261-Speed 131.20 samples/sec Loss 11.0856 Epoch: 6 Global Step: 62590 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:46:29,329-Speed 127.57 samples/sec Loss 11.2649 Epoch: 6 Global Step: 62595 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:46:49,423-Speed 127.40 samples/sec Loss 11.3112 Epoch: 6 Global Step: 62600 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:47:09,030-Speed 130.57 samples/sec Loss 11.0580 Epoch: 6 Global Step: 62605 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:47:28,816-Speed 129.39 samples/sec Loss 11.3145 Epoch: 6 Global Step: 62610 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:47:49,142-Speed 125.96 samples/sec Loss 10.7576 Epoch: 6 Global Step: 62615 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:48:09,446-Speed 126.09 samples/sec Loss 11.2758 Epoch: 6 Global Step: 62620 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:48:29,647-Speed 126.73 samples/sec Loss 11.2586 Epoch: 6 Global Step: 62625 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:48:49,506-Speed 128.92 samples/sec Loss 11.2540 Epoch: 6 Global Step: 62630 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:49:09,133-Speed 130.44 samples/sec Loss 10.9280 Epoch: 6 Global Step: 62635 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:49:29,362-Speed 126.56 samples/sec Loss 10.9105 Epoch: 6 Global Step: 62640 Fp16 Grad Scale: 8192 Required: 223 hours Training: 2021-06-09 21:49:49,450-Speed 127.45 samples/sec Loss 11.1636 Epoch: 6 Global Step: 62645 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:50:08,953-Speed 131.27 samples/sec Loss 11.1452 Epoch: 6 Global Step: 62650 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:50:28,729-Speed 129.45 samples/sec Loss 10.9781 Epoch: 6 Global Step: 62655 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:50:48,407-Speed 130.10 samples/sec Loss 11.1896 Epoch: 6 Global Step: 62660 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:51:08,225-Speed 129.18 samples/sec Loss 10.9738 Epoch: 6 Global Step: 62665 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:51:27,881-Speed 130.25 samples/sec Loss 11.0780 Epoch: 6 Global Step: 62670 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:51:47,679-Speed 129.32 samples/sec Loss 10.9628 Epoch: 6 Global Step: 62675 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:52:07,371-Speed 130.00 samples/sec Loss 10.8544 Epoch: 6 Global Step: 62680 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:52:27,373-Speed 127.99 samples/sec Loss 10.7831 Epoch: 6 Global Step: 62685 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:52:47,318-Speed 128.36 samples/sec Loss 11.0237 Epoch: 6 Global Step: 62690 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:53:06,938-Speed 130.49 samples/sec Loss 11.3991 Epoch: 6 Global Step: 62695 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:53:26,996-Speed 127.63 samples/sec Loss 11.1968 Epoch: 6 Global Step: 62700 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:53:46,613-Speed 130.51 samples/sec Loss 11.1905 Epoch: 6 Global Step: 62705 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:54:06,514-Speed 128.64 samples/sec Loss 10.9284 Epoch: 6 Global Step: 62710 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:54:26,106-Speed 130.67 samples/sec Loss 10.9520 Epoch: 6 Global Step: 62715 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:54:45,936-Speed 129.11 samples/sec Loss 10.8262 Epoch: 6 Global Step: 62720 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:55:05,631-Speed 129.99 samples/sec Loss 10.9902 Epoch: 6 Global Step: 62725 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:55:25,809-Speed 126.88 samples/sec Loss 11.0056 Epoch: 6 Global Step: 62730 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:55:46,174-Speed 125.71 samples/sec Loss 11.0452 Epoch: 6 Global Step: 62735 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:56:05,845-Speed 130.15 samples/sec Loss 11.0500 Epoch: 6 Global Step: 62740 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:56:25,755-Speed 128.59 samples/sec Loss 10.9979 Epoch: 6 Global Step: 62745 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:56:45,667-Speed 128.58 samples/sec Loss 11.2090 Epoch: 6 Global Step: 62750 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:57:05,579-Speed 128.57 samples/sec Loss 10.6798 Epoch: 6 Global Step: 62755 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:57:25,203-Speed 130.46 samples/sec Loss 11.0303 Epoch: 6 Global Step: 62760 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:57:45,230-Speed 127.84 samples/sec Loss 10.8842 Epoch: 6 Global Step: 62765 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:58:04,845-Speed 130.52 samples/sec Loss 11.2106 Epoch: 6 Global Step: 62770 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:58:24,979-Speed 127.15 samples/sec Loss 11.2903 Epoch: 6 Global Step: 62775 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:58:44,765-Speed 129.39 samples/sec Loss 10.9430 Epoch: 6 Global Step: 62780 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:59:04,457-Speed 130.01 samples/sec Loss 10.8856 Epoch: 6 Global Step: 62785 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:59:24,350-Speed 128.70 samples/sec Loss 11.3479 Epoch: 6 Global Step: 62790 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 21:59:44,400-Speed 127.69 samples/sec Loss 10.9734 Epoch: 6 Global Step: 62795 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:00:04,328-Speed 128.48 samples/sec Loss 11.1966 Epoch: 6 Global Step: 62800 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:00:24,063-Speed 129.72 samples/sec Loss 10.7512 Epoch: 6 Global Step: 62805 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:00:44,025-Speed 128.25 samples/sec Loss 10.9668 Epoch: 6 Global Step: 62810 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:01:03,635-Speed 130.55 samples/sec Loss 10.8795 Epoch: 6 Global Step: 62815 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:01:23,687-Speed 127.67 samples/sec Loss 11.2751 Epoch: 6 Global Step: 62820 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:01:43,565-Speed 128.79 samples/sec Loss 11.1011 Epoch: 6 Global Step: 62825 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:02:03,240-Speed 130.12 samples/sec Loss 10.8199 Epoch: 6 Global Step: 62830 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:02:23,024-Speed 129.41 samples/sec Loss 11.0129 Epoch: 6 Global Step: 62835 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:02:42,605-Speed 130.75 samples/sec Loss 11.0310 Epoch: 6 Global Step: 62840 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:03:02,551-Speed 128.35 samples/sec Loss 10.6907 Epoch: 6 Global Step: 62845 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:03:22,244-Speed 130.00 samples/sec Loss 11.1489 Epoch: 6 Global Step: 62850 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:03:42,836-Speed 124.33 samples/sec Loss 10.8224 Epoch: 6 Global Step: 62855 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:04:02,456-Speed 130.49 samples/sec Loss 10.7962 Epoch: 6 Global Step: 62860 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:04:22,617-Speed 126.98 samples/sec Loss 11.2160 Epoch: 6 Global Step: 62865 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:04:42,561-Speed 128.37 samples/sec Loss 11.0787 Epoch: 6 Global Step: 62870 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:05:02,215-Speed 130.26 samples/sec Loss 11.2677 Epoch: 6 Global Step: 62875 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:05:21,889-Speed 130.13 samples/sec Loss 11.0370 Epoch: 6 Global Step: 62880 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:05:41,571-Speed 130.07 samples/sec Loss 11.2286 Epoch: 6 Global Step: 62885 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:06:01,423-Speed 128.97 samples/sec Loss 11.1130 Epoch: 6 Global Step: 62890 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:06:20,966-Speed 131.00 samples/sec Loss 10.9892 Epoch: 6 Global Step: 62895 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:06:40,901-Speed 128.43 samples/sec Loss 10.9491 Epoch: 6 Global Step: 62900 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:07:00,512-Speed 130.55 samples/sec Loss 10.9237 Epoch: 6 Global Step: 62905 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:07:20,675-Speed 126.97 samples/sec Loss 10.9664 Epoch: 6 Global Step: 62910 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:07:40,688-Speed 127.92 samples/sec Loss 10.8390 Epoch: 6 Global Step: 62915 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:08:00,293-Speed 130.59 samples/sec Loss 11.0209 Epoch: 6 Global Step: 62920 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:08:20,115-Speed 129.16 samples/sec Loss 11.0824 Epoch: 6 Global Step: 62925 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:08:39,787-Speed 130.14 samples/sec Loss 11.1701 Epoch: 6 Global Step: 62930 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:08:59,454-Speed 130.17 samples/sec Loss 11.0439 Epoch: 6 Global Step: 62935 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:09:19,143-Speed 130.02 samples/sec Loss 10.7604 Epoch: 6 Global Step: 62940 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:09:38,839-Speed 129.99 samples/sec Loss 10.8651 Epoch: 6 Global Step: 62945 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:09:58,434-Speed 130.65 samples/sec Loss 11.0182 Epoch: 6 Global Step: 62950 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:10:18,614-Speed 126.86 samples/sec Loss 11.0836 Epoch: 6 Global Step: 62955 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:10:38,590-Speed 128.16 samples/sec Loss 10.7082 Epoch: 6 Global Step: 62960 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:10:58,177-Speed 130.70 samples/sec Loss 10.9994 Epoch: 6 Global Step: 62965 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:11:18,061-Speed 128.75 samples/sec Loss 11.2797 Epoch: 6 Global Step: 62970 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:11:37,757-Speed 129.99 samples/sec Loss 11.0847 Epoch: 6 Global Step: 62975 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:11:57,565-Speed 129.25 samples/sec Loss 11.0601 Epoch: 6 Global Step: 62980 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:12:17,184-Speed 130.50 samples/sec Loss 11.0367 Epoch: 6 Global Step: 62985 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:12:36,980-Speed 129.32 samples/sec Loss 11.0686 Epoch: 6 Global Step: 62990 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:12:56,674-Speed 130.00 samples/sec Loss 10.7840 Epoch: 6 Global Step: 62995 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:13:16,824-Speed 127.05 samples/sec Loss 11.1298 Epoch: 6 Global Step: 63000 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:13:36,510-Speed 130.05 samples/sec Loss 11.1016 Epoch: 6 Global Step: 63005 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:13:56,114-Speed 130.59 samples/sec Loss 11.1703 Epoch: 6 Global Step: 63010 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:14:16,109-Speed 128.04 samples/sec Loss 11.1542 Epoch: 6 Global Step: 63015 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:14:35,626-Speed 131.17 samples/sec Loss 11.2412 Epoch: 6 Global Step: 63020 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:14:55,590-Speed 128.24 samples/sec Loss 11.0906 Epoch: 6 Global Step: 63025 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:15:15,172-Speed 130.74 samples/sec Loss 11.0595 Epoch: 6 Global Step: 63030 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:15:35,135-Speed 128.24 samples/sec Loss 10.9570 Epoch: 6 Global Step: 63035 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:15:54,810-Speed 130.13 samples/sec Loss 11.5464 Epoch: 6 Global Step: 63040 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:16:14,954-Speed 127.09 samples/sec Loss 11.3546 Epoch: 6 Global Step: 63045 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:16:34,736-Speed 129.42 samples/sec Loss 11.3021 Epoch: 6 Global Step: 63050 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:16:54,425-Speed 130.02 samples/sec Loss 10.9326 Epoch: 6 Global Step: 63055 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:17:14,214-Speed 129.38 samples/sec Loss 11.0801 Epoch: 6 Global Step: 63060 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:17:33,892-Speed 130.10 samples/sec Loss 11.1450 Epoch: 6 Global Step: 63065 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:17:53,577-Speed 130.06 samples/sec Loss 10.9786 Epoch: 6 Global Step: 63070 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:18:13,435-Speed 128.92 samples/sec Loss 10.8522 Epoch: 6 Global Step: 63075 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:18:33,419-Speed 128.11 samples/sec Loss 11.0929 Epoch: 6 Global Step: 63080 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:18:52,942-Speed 131.14 samples/sec Loss 11.1253 Epoch: 6 Global Step: 63085 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:19:13,121-Speed 126.87 samples/sec Loss 11.0215 Epoch: 6 Global Step: 63090 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:19:33,164-Speed 127.74 samples/sec Loss 10.8367 Epoch: 6 Global Step: 63095 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:19:52,826-Speed 130.20 samples/sec Loss 11.2222 Epoch: 6 Global Step: 63100 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:20:12,583-Speed 129.58 samples/sec Loss 11.1474 Epoch: 6 Global Step: 63105 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:20:32,551-Speed 128.22 samples/sec Loss 10.7361 Epoch: 6 Global Step: 63110 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:20:52,440-Speed 128.72 samples/sec Loss 10.9427 Epoch: 6 Global Step: 63115 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:21:12,103-Speed 130.20 samples/sec Loss 10.7989 Epoch: 6 Global Step: 63120 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:21:31,874-Speed 129.49 samples/sec Loss 10.9800 Epoch: 6 Global Step: 63125 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:21:51,551-Speed 130.11 samples/sec Loss 10.8553 Epoch: 6 Global Step: 63130 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:22:11,555-Speed 127.98 samples/sec Loss 11.2981 Epoch: 6 Global Step: 63135 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:22:31,604-Speed 127.70 samples/sec Loss 11.3100 Epoch: 6 Global Step: 63140 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:22:51,113-Speed 131.23 samples/sec Loss 10.9046 Epoch: 6 Global Step: 63145 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:23:11,099-Speed 128.09 samples/sec Loss 10.9200 Epoch: 6 Global Step: 63150 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:23:30,705-Speed 130.58 samples/sec Loss 11.2819 Epoch: 6 Global Step: 63155 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:23:50,580-Speed 128.81 samples/sec Loss 11.0716 Epoch: 6 Global Step: 63160 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:24:10,271-Speed 130.02 samples/sec Loss 11.1307 Epoch: 6 Global Step: 63165 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:24:29,968-Speed 129.98 samples/sec Loss 11.1012 Epoch: 6 Global Step: 63170 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:24:49,648-Speed 130.09 samples/sec Loss 10.8279 Epoch: 6 Global Step: 63175 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:25:09,754-Speed 127.33 samples/sec Loss 10.7880 Epoch: 6 Global Step: 63180 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:25:29,597-Speed 129.01 samples/sec Loss 10.6497 Epoch: 6 Global Step: 63185 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:25:49,220-Speed 130.47 samples/sec Loss 11.3469 Epoch: 6 Global Step: 63190 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:26:09,211-Speed 128.06 samples/sec Loss 10.8067 Epoch: 6 Global Step: 63195 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:26:28,808-Speed 130.64 samples/sec Loss 10.8133 Epoch: 6 Global Step: 63200 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:26:48,836-Speed 127.83 samples/sec Loss 11.0786 Epoch: 6 Global Step: 63205 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:27:08,358-Speed 131.14 samples/sec Loss 11.3747 Epoch: 6 Global Step: 63210 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:27:28,447-Speed 127.44 samples/sec Loss 11.1737 Epoch: 6 Global Step: 63215 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:27:48,138-Speed 130.01 samples/sec Loss 10.9390 Epoch: 6 Global Step: 63220 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:28:08,331-Speed 126.79 samples/sec Loss 11.1032 Epoch: 6 Global Step: 63225 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:28:28,418-Speed 127.45 samples/sec Loss 10.8419 Epoch: 6 Global Step: 63230 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:28:48,042-Speed 130.45 samples/sec Loss 11.3577 Epoch: 6 Global Step: 63235 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:29:07,896-Speed 128.95 samples/sec Loss 10.8442 Epoch: 6 Global Step: 63240 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:29:27,507-Speed 130.55 samples/sec Loss 11.0595 Epoch: 6 Global Step: 63245 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:29:47,473-Speed 128.23 samples/sec Loss 10.7932 Epoch: 6 Global Step: 63250 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:30:07,102-Speed 130.42 samples/sec Loss 11.0084 Epoch: 6 Global Step: 63255 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:30:27,065-Speed 128.25 samples/sec Loss 10.8041 Epoch: 6 Global Step: 63260 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:30:46,679-Speed 130.53 samples/sec Loss 10.9196 Epoch: 6 Global Step: 63265 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:31:06,892-Speed 126.65 samples/sec Loss 11.0512 Epoch: 6 Global Step: 63270 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:31:26,721-Speed 129.11 samples/sec Loss 10.8081 Epoch: 6 Global Step: 63275 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:31:46,369-Speed 130.30 samples/sec Loss 10.8307 Epoch: 6 Global Step: 63280 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:32:06,167-Speed 129.32 samples/sec Loss 11.1697 Epoch: 6 Global Step: 63285 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:32:25,862-Speed 129.99 samples/sec Loss 10.9625 Epoch: 6 Global Step: 63290 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:32:45,623-Speed 129.55 samples/sec Loss 10.4233 Epoch: 6 Global Step: 63295 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:33:05,153-Speed 131.09 samples/sec Loss 11.0819 Epoch: 6 Global Step: 63300 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:33:25,075-Speed 128.50 samples/sec Loss 11.0212 Epoch: 6 Global Step: 63305 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:33:44,806-Speed 129.75 samples/sec Loss 10.8077 Epoch: 6 Global Step: 63310 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:34:04,944-Speed 127.13 samples/sec Loss 10.9879 Epoch: 6 Global Step: 63315 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:34:25,008-Speed 127.60 samples/sec Loss 11.2087 Epoch: 6 Global Step: 63320 Fp16 Grad Scale: 8192 Required: 222 hours Training: 2021-06-09 22:34:44,621-Speed 130.53 samples/sec Loss 11.0659 Epoch: 6 Global Step: 63325 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:35:04,635-Speed 127.91 samples/sec Loss 10.7375 Epoch: 6 Global Step: 63330 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:35:24,242-Speed 130.58 samples/sec Loss 10.7332 Epoch: 6 Global Step: 63335 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:35:44,091-Speed 128.98 samples/sec Loss 10.9144 Epoch: 6 Global Step: 63340 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:36:03,850-Speed 129.57 samples/sec Loss 10.8598 Epoch: 6 Global Step: 63345 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:36:23,629-Speed 129.44 samples/sec Loss 10.8839 Epoch: 6 Global Step: 63350 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:36:43,293-Speed 130.20 samples/sec Loss 10.7836 Epoch: 6 Global Step: 63355 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:37:03,303-Speed 127.94 samples/sec Loss 11.2425 Epoch: 6 Global Step: 63360 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:37:23,223-Speed 128.52 samples/sec Loss 11.1594 Epoch: 6 Global Step: 63365 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:37:42,820-Speed 130.64 samples/sec Loss 11.2684 Epoch: 6 Global Step: 63370 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:38:02,773-Speed 128.31 samples/sec Loss 10.8794 Epoch: 6 Global Step: 63375 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:38:22,390-Speed 130.50 samples/sec Loss 10.9231 Epoch: 6 Global Step: 63380 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:38:42,309-Speed 128.53 samples/sec Loss 11.2374 Epoch: 6 Global Step: 63385 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:39:01,906-Speed 130.64 samples/sec Loss 10.8568 Epoch: 6 Global Step: 63390 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:39:21,767-Speed 128.91 samples/sec Loss 11.0045 Epoch: 6 Global Step: 63395 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:39:41,449-Speed 130.07 samples/sec Loss 10.7862 Epoch: 6 Global Step: 63400 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:40:01,611-Speed 126.98 samples/sec Loss 10.8416 Epoch: 6 Global Step: 63405 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:40:21,375-Speed 129.54 samples/sec Loss 10.6707 Epoch: 6 Global Step: 63410 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:40:41,041-Speed 130.18 samples/sec Loss 11.1209 Epoch: 6 Global Step: 63415 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:41:00,955-Speed 128.56 samples/sec Loss 10.9492 Epoch: 6 Global Step: 63420 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:41:20,565-Speed 130.55 samples/sec Loss 11.0974 Epoch: 6 Global Step: 63425 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:41:40,462-Speed 128.67 samples/sec Loss 10.4675 Epoch: 6 Global Step: 63430 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:42:00,110-Speed 130.30 samples/sec Loss 11.0778 Epoch: 6 Global Step: 63435 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:42:19,917-Speed 129.25 samples/sec Loss 10.9712 Epoch: 6 Global Step: 63440 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:42:39,572-Speed 130.25 samples/sec Loss 10.8803 Epoch: 6 Global Step: 63445 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:42:59,963-Speed 125.55 samples/sec Loss 11.1159 Epoch: 6 Global Step: 63450 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:43:19,822-Speed 128.92 samples/sec Loss 11.2964 Epoch: 6 Global Step: 63455 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:43:39,452-Speed 130.42 samples/sec Loss 10.9161 Epoch: 6 Global Step: 63460 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:43:59,369-Speed 128.55 samples/sec Loss 10.9708 Epoch: 6 Global Step: 63465 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:44:18,985-Speed 130.51 samples/sec Loss 10.6882 Epoch: 6 Global Step: 63470 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:44:38,923-Speed 128.40 samples/sec Loss 11.1201 Epoch: 6 Global Step: 63475 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:44:58,660-Speed 129.71 samples/sec Loss 11.0898 Epoch: 6 Global Step: 63480 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:45:18,621-Speed 128.26 samples/sec Loss 10.8547 Epoch: 6 Global Step: 63485 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:45:38,230-Speed 130.56 samples/sec Loss 11.0618 Epoch: 6 Global Step: 63490 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:45:58,399-Speed 126.93 samples/sec Loss 11.0378 Epoch: 6 Global Step: 63495 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:46:18,204-Speed 129.27 samples/sec Loss 10.8770 Epoch: 6 Global Step: 63500 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:46:37,873-Speed 130.16 samples/sec Loss 11.4282 Epoch: 6 Global Step: 63505 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:46:57,670-Speed 129.32 samples/sec Loss 11.0626 Epoch: 6 Global Step: 63510 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:47:17,346-Speed 130.12 samples/sec Loss 10.9176 Epoch: 6 Global Step: 63515 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:47:37,210-Speed 128.88 samples/sec Loss 11.3087 Epoch: 6 Global Step: 63520 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:47:56,735-Speed 131.13 samples/sec Loss 11.0709 Epoch: 6 Global Step: 63525 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:48:16,693-Speed 128.27 samples/sec Loss 11.2093 Epoch: 6 Global Step: 63530 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:48:36,286-Speed 130.67 samples/sec Loss 10.9086 Epoch: 6 Global Step: 63535 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:48:56,633-Speed 125.83 samples/sec Loss 10.8552 Epoch: 6 Global Step: 63540 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:49:16,544-Speed 128.58 samples/sec Loss 10.9321 Epoch: 6 Global Step: 63545 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:49:36,148-Speed 130.59 samples/sec Loss 11.2293 Epoch: 6 Global Step: 63550 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:49:55,967-Speed 129.18 samples/sec Loss 10.9875 Epoch: 6 Global Step: 63555 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:50:15,651-Speed 130.07 samples/sec Loss 10.8058 Epoch: 6 Global Step: 63560 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:50:35,522-Speed 128.83 samples/sec Loss 10.9336 Epoch: 6 Global Step: 63565 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:50:55,198-Speed 130.11 samples/sec Loss 11.1893 Epoch: 6 Global Step: 63570 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:51:15,034-Speed 129.07 samples/sec Loss 11.0442 Epoch: 6 Global Step: 63575 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:51:34,659-Speed 130.45 samples/sec Loss 11.1468 Epoch: 6 Global Step: 63580 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:51:54,786-Speed 127.20 samples/sec Loss 10.9454 Epoch: 6 Global Step: 63585 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:52:14,905-Speed 127.25 samples/sec Loss 11.2601 Epoch: 6 Global Step: 63590 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:52:34,503-Speed 130.63 samples/sec Loss 11.4712 Epoch: 6 Global Step: 63595 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:52:54,459-Speed 128.29 samples/sec Loss 11.3807 Epoch: 6 Global Step: 63600 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:53:14,097-Speed 130.37 samples/sec Loss 11.1533 Epoch: 6 Global Step: 63605 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:53:33,828-Speed 129.76 samples/sec Loss 10.8769 Epoch: 6 Global Step: 63610 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:53:53,453-Speed 130.45 samples/sec Loss 11.0097 Epoch: 6 Global Step: 63615 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:54:13,260-Speed 129.26 samples/sec Loss 10.9212 Epoch: 6 Global Step: 63620 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:54:32,930-Speed 130.16 samples/sec Loss 10.7695 Epoch: 6 Global Step: 63625 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:54:52,900-Speed 128.20 samples/sec Loss 10.9602 Epoch: 6 Global Step: 63630 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:55:12,914-Speed 127.92 samples/sec Loss 11.1012 Epoch: 6 Global Step: 63635 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:55:32,519-Speed 130.59 samples/sec Loss 11.0601 Epoch: 6 Global Step: 63640 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:55:52,561-Speed 127.73 samples/sec Loss 11.1459 Epoch: 6 Global Step: 63645 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:56:12,094-Speed 131.07 samples/sec Loss 11.2272 Epoch: 6 Global Step: 63650 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:56:32,069-Speed 128.17 samples/sec Loss 10.7411 Epoch: 6 Global Step: 63655 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:56:51,678-Speed 130.56 samples/sec Loss 10.8098 Epoch: 6 Global Step: 63660 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:57:11,513-Speed 129.07 samples/sec Loss 11.1858 Epoch: 6 Global Step: 63665 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:57:31,171-Speed 130.24 samples/sec Loss 11.0093 Epoch: 6 Global Step: 63670 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:57:51,201-Speed 127.82 samples/sec Loss 10.7357 Epoch: 6 Global Step: 63675 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:58:10,980-Speed 129.43 samples/sec Loss 11.3512 Epoch: 6 Global Step: 63680 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:58:30,649-Speed 130.16 samples/sec Loss 10.7852 Epoch: 6 Global Step: 63685 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:58:50,482-Speed 129.09 samples/sec Loss 10.7195 Epoch: 6 Global Step: 63690 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:59:10,001-Speed 131.16 samples/sec Loss 10.9032 Epoch: 6 Global Step: 63695 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:59:29,971-Speed 128.20 samples/sec Loss 11.0971 Epoch: 6 Global Step: 63700 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 22:59:49,652-Speed 130.08 samples/sec Loss 11.0644 Epoch: 6 Global Step: 63705 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:00:09,675-Speed 127.86 samples/sec Loss 10.9668 Epoch: 6 Global Step: 63710 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:00:29,188-Speed 131.21 samples/sec Loss 10.7295 Epoch: 6 Global Step: 63715 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:00:49,424-Speed 126.51 samples/sec Loss 11.1622 Epoch: 6 Global Step: 63720 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:01:09,155-Speed 129.75 samples/sec Loss 10.6649 Epoch: 6 Global Step: 63725 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:01:28,834-Speed 130.09 samples/sec Loss 11.1491 Epoch: 6 Global Step: 63730 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:01:48,537-Speed 129.94 samples/sec Loss 11.3097 Epoch: 6 Global Step: 63735 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:02:08,157-Speed 130.49 samples/sec Loss 10.9670 Epoch: 6 Global Step: 63740 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:02:28,057-Speed 128.65 samples/sec Loss 11.1697 Epoch: 6 Global Step: 63745 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:02:47,709-Speed 130.27 samples/sec Loss 11.0221 Epoch: 6 Global Step: 63750 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:03:07,592-Speed 128.77 samples/sec Loss 11.2148 Epoch: 6 Global Step: 63755 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:03:27,131-Speed 131.02 samples/sec Loss 10.8006 Epoch: 6 Global Step: 63760 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:03:47,278-Speed 127.07 samples/sec Loss 10.6816 Epoch: 6 Global Step: 63765 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:04:07,264-Speed 128.10 samples/sec Loss 10.9118 Epoch: 6 Global Step: 63770 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:04:26,917-Speed 130.26 samples/sec Loss 10.7970 Epoch: 6 Global Step: 63775 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:04:46,716-Speed 129.31 samples/sec Loss 11.2142 Epoch: 6 Global Step: 63780 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:05:06,386-Speed 130.16 samples/sec Loss 11.0370 Epoch: 6 Global Step: 63785 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:05:26,165-Speed 129.43 samples/sec Loss 10.8603 Epoch: 6 Global Step: 63790 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:05:45,854-Speed 130.03 samples/sec Loss 10.6909 Epoch: 6 Global Step: 63795 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:06:05,675-Speed 129.16 samples/sec Loss 11.0345 Epoch: 6 Global Step: 63800 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:06:25,350-Speed 130.12 samples/sec Loss 10.5843 Epoch: 6 Global Step: 63805 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:06:45,523-Speed 126.91 samples/sec Loss 11.1210 Epoch: 6 Global Step: 63810 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:07:05,648-Speed 127.21 samples/sec Loss 10.8973 Epoch: 6 Global Step: 63815 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:07:25,257-Speed 130.56 samples/sec Loss 10.9385 Epoch: 6 Global Step: 63820 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:07:45,090-Speed 129.08 samples/sec Loss 11.0740 Epoch: 6 Global Step: 63825 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:08:04,722-Speed 130.41 samples/sec Loss 11.1056 Epoch: 6 Global Step: 63830 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:08:24,574-Speed 128.96 samples/sec Loss 11.0504 Epoch: 6 Global Step: 63835 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:08:44,236-Speed 130.21 samples/sec Loss 10.8729 Epoch: 6 Global Step: 63840 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:09:04,033-Speed 129.32 samples/sec Loss 11.1304 Epoch: 6 Global Step: 63845 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:09:23,690-Speed 130.24 samples/sec Loss 10.8439 Epoch: 6 Global Step: 63850 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:09:43,710-Speed 127.88 samples/sec Loss 11.0980 Epoch: 6 Global Step: 63855 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:10:03,696-Speed 128.09 samples/sec Loss 11.0660 Epoch: 6 Global Step: 63860 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:10:23,284-Speed 130.70 samples/sec Loss 11.0256 Epoch: 6 Global Step: 63865 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:10:43,249-Speed 128.23 samples/sec Loss 11.0850 Epoch: 6 Global Step: 63870 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:11:02,896-Speed 130.31 samples/sec Loss 11.2905 Epoch: 6 Global Step: 63875 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:11:22,692-Speed 129.33 samples/sec Loss 11.3009 Epoch: 6 Global Step: 63880 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:11:42,282-Speed 130.69 samples/sec Loss 10.6852 Epoch: 6 Global Step: 63885 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:12:02,087-Speed 129.27 samples/sec Loss 10.9078 Epoch: 6 Global Step: 63890 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:12:21,718-Speed 130.41 samples/sec Loss 11.1734 Epoch: 6 Global Step: 63895 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:12:41,946-Speed 126.57 samples/sec Loss 11.0958 Epoch: 6 Global Step: 63900 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:13:01,718-Speed 129.48 samples/sec Loss 11.1853 Epoch: 6 Global Step: 63905 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:13:21,399-Speed 130.08 samples/sec Loss 11.4472 Epoch: 6 Global Step: 63910 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:13:41,232-Speed 129.09 samples/sec Loss 11.1002 Epoch: 6 Global Step: 63915 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:14:00,885-Speed 130.27 samples/sec Loss 11.0676 Epoch: 6 Global Step: 63920 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:14:20,850-Speed 128.23 samples/sec Loss 10.9779 Epoch: 6 Global Step: 63925 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:14:40,416-Speed 130.85 samples/sec Loss 11.0806 Epoch: 6 Global Step: 63930 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:15:00,213-Speed 129.32 samples/sec Loss 10.9358 Epoch: 6 Global Step: 63935 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:15:19,955-Speed 129.68 samples/sec Loss 11.1407 Epoch: 6 Global Step: 63940 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:15:40,042-Speed 127.45 samples/sec Loss 10.8652 Epoch: 6 Global Step: 63945 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:15:59,891-Speed 128.98 samples/sec Loss 11.1008 Epoch: 6 Global Step: 63950 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:16:19,429-Speed 131.03 samples/sec Loss 10.8695 Epoch: 6 Global Step: 63955 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:16:39,426-Speed 128.03 samples/sec Loss 11.1741 Epoch: 6 Global Step: 63960 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:16:59,030-Speed 130.59 samples/sec Loss 11.1371 Epoch: 6 Global Step: 63965 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:17:19,025-Speed 128.04 samples/sec Loss 11.0190 Epoch: 6 Global Step: 63970 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:17:38,667-Speed 130.34 samples/sec Loss 11.1641 Epoch: 6 Global Step: 63975 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:17:58,563-Speed 128.68 samples/sec Loss 11.0311 Epoch: 6 Global Step: 63980 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:18:18,306-Speed 129.67 samples/sec Loss 10.8000 Epoch: 6 Global Step: 63985 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:18:38,457-Speed 127.05 samples/sec Loss 11.0178 Epoch: 6 Global Step: 63990 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:18:58,235-Speed 129.44 samples/sec Loss 11.2702 Epoch: 6 Global Step: 63995 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:19:17,983-Speed 129.64 samples/sec Loss 11.2715 Epoch: 6 Global Step: 64000 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:19:37,998-Speed 127.91 samples/sec Loss 11.3700 Epoch: 6 Global Step: 64005 Fp16 Grad Scale: 8192 Required: 221 hours Training: 2021-06-09 23:19:57,616-Speed 130.50 samples/sec Loss 11.2657 Epoch: 6 Global Step: 64010 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:20:17,508-Speed 128.70 samples/sec Loss 11.0290 Epoch: 6 Global Step: 64015 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:20:37,044-Speed 131.05 samples/sec Loss 11.0165 Epoch: 6 Global Step: 64020 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:20:57,171-Speed 127.20 samples/sec Loss 11.1875 Epoch: 6 Global Step: 64025 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:21:16,846-Speed 130.12 samples/sec Loss 10.8128 Epoch: 6 Global Step: 64030 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:21:36,989-Speed 127.10 samples/sec Loss 11.3982 Epoch: 6 Global Step: 64035 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:21:56,789-Speed 129.30 samples/sec Loss 10.9250 Epoch: 6 Global Step: 64040 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:22:16,469-Speed 130.08 samples/sec Loss 10.8410 Epoch: 6 Global Step: 64045 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:22:36,219-Speed 129.63 samples/sec Loss 11.0131 Epoch: 6 Global Step: 64050 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:22:55,873-Speed 130.26 samples/sec Loss 11.4140 Epoch: 6 Global Step: 64055 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:23:15,689-Speed 129.20 samples/sec Loss 11.2357 Epoch: 6 Global Step: 64060 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:23:35,230-Speed 131.01 samples/sec Loss 10.9333 Epoch: 6 Global Step: 64065 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:23:55,206-Speed 128.17 samples/sec Loss 11.3481 Epoch: 6 Global Step: 64070 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:24:14,820-Speed 130.52 samples/sec Loss 10.8734 Epoch: 6 Global Step: 64075 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:24:34,885-Speed 127.59 samples/sec Loss 11.3272 Epoch: 6 Global Step: 64080 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:24:54,920-Speed 127.79 samples/sec Loss 10.9620 Epoch: 6 Global Step: 64085 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:25:14,510-Speed 130.68 samples/sec Loss 11.0167 Epoch: 6 Global Step: 64090 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:25:34,351-Speed 129.03 samples/sec Loss 11.3039 Epoch: 6 Global Step: 64095 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:25:53,998-Speed 130.31 samples/sec Loss 10.9227 Epoch: 6 Global Step: 64100 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:26:13,680-Speed 130.07 samples/sec Loss 11.0492 Epoch: 6 Global Step: 64105 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:26:33,357-Speed 130.11 samples/sec Loss 11.1084 Epoch: 6 Global Step: 64110 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:26:53,150-Speed 129.34 samples/sec Loss 11.0396 Epoch: 6 Global Step: 64115 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:27:12,702-Speed 130.94 samples/sec Loss 10.9309 Epoch: 6 Global Step: 64120 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:27:32,824-Speed 127.23 samples/sec Loss 11.0222 Epoch: 6 Global Step: 64125 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:27:52,879-Speed 127.66 samples/sec Loss 11.1903 Epoch: 6 Global Step: 64130 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:28:12,570-Speed 130.02 samples/sec Loss 10.8469 Epoch: 6 Global Step: 64135 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:28:32,418-Speed 128.98 samples/sec Loss 11.0705 Epoch: 6 Global Step: 64140 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:28:52,029-Speed 130.55 samples/sec Loss 10.8730 Epoch: 6 Global Step: 64145 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:29:11,770-Speed 129.69 samples/sec Loss 11.0724 Epoch: 6 Global Step: 64150 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:29:31,426-Speed 130.25 samples/sec Loss 10.9498 Epoch: 6 Global Step: 64155 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:29:51,130-Speed 129.93 samples/sec Loss 11.1101 Epoch: 6 Global Step: 64160 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:30:10,756-Speed 130.45 samples/sec Loss 10.7591 Epoch: 6 Global Step: 64165 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:30:30,808-Speed 127.67 samples/sec Loss 10.9666 Epoch: 6 Global Step: 64170 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:30:50,853-Speed 127.72 samples/sec Loss 10.7251 Epoch: 6 Global Step: 64175 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:31:10,357-Speed 131.26 samples/sec Loss 10.7944 Epoch: 6 Global Step: 64180 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:31:30,330-Speed 128.18 samples/sec Loss 10.8950 Epoch: 6 Global Step: 64185 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:31:49,903-Speed 130.81 samples/sec Loss 10.9472 Epoch: 6 Global Step: 64190 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:32:09,721-Speed 129.18 samples/sec Loss 10.9740 Epoch: 6 Global Step: 64195 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:32:29,367-Speed 130.31 samples/sec Loss 11.0007 Epoch: 6 Global Step: 64200 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:32:49,052-Speed 130.06 samples/sec Loss 11.3299 Epoch: 6 Global Step: 64205 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:33:08,741-Speed 130.03 samples/sec Loss 11.0272 Epoch: 6 Global Step: 64210 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:33:28,844-Speed 127.35 samples/sec Loss 11.0488 Epoch: 6 Global Step: 64215 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:33:48,717-Speed 128.82 samples/sec Loss 11.1125 Epoch: 6 Global Step: 64220 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:34:08,482-Speed 129.53 samples/sec Loss 10.8903 Epoch: 6 Global Step: 64225 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:34:28,392-Speed 128.59 samples/sec Loss 11.2673 Epoch: 6 Global Step: 64230 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:34:48,064-Speed 130.14 samples/sec Loss 11.3280 Epoch: 6 Global Step: 64235 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:35:08,055-Speed 128.07 samples/sec Loss 10.9276 Epoch: 6 Global Step: 64240 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:35:27,580-Speed 131.12 samples/sec Loss 11.0299 Epoch: 6 Global Step: 64245 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:35:47,541-Speed 128.26 samples/sec Loss 10.7526 Epoch: 6 Global Step: 64250 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:36:07,277-Speed 129.72 samples/sec Loss 11.0892 Epoch: 6 Global Step: 64255 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:36:27,470-Speed 126.78 samples/sec Loss 10.9761 Epoch: 6 Global Step: 64260 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:36:47,158-Speed 130.03 samples/sec Loss 11.3303 Epoch: 6 Global Step: 64265 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:37:06,772-Speed 130.53 samples/sec Loss 11.2000 Epoch: 6 Global Step: 64270 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:37:26,705-Speed 128.43 samples/sec Loss 10.8688 Epoch: 6 Global Step: 64275 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:37:46,320-Speed 130.52 samples/sec Loss 11.2089 Epoch: 6 Global Step: 64280 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:38:06,270-Speed 128.33 samples/sec Loss 11.1368 Epoch: 6 Global Step: 64285 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:38:25,861-Speed 130.68 samples/sec Loss 10.9091 Epoch: 6 Global Step: 64290 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:38:45,832-Speed 128.19 samples/sec Loss 10.8798 Epoch: 6 Global Step: 64295 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:39:05,534-Speed 129.94 samples/sec Loss 11.1784 Epoch: 6 Global Step: 64300 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:39:25,700-Speed 126.95 samples/sec Loss 10.8354 Epoch: 6 Global Step: 64305 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:39:45,621-Speed 128.51 samples/sec Loss 10.9761 Epoch: 6 Global Step: 64310 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:40:05,250-Speed 130.43 samples/sec Loss 10.9333 Epoch: 6 Global Step: 64315 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:40:25,030-Speed 129.43 samples/sec Loss 10.9484 Epoch: 6 Global Step: 64320 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:40:44,719-Speed 130.03 samples/sec Loss 11.1321 Epoch: 6 Global Step: 64325 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:41:04,377-Speed 130.24 samples/sec Loss 11.0562 Epoch: 6 Global Step: 64330 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:41:24,001-Speed 130.46 samples/sec Loss 10.7979 Epoch: 6 Global Step: 64335 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:41:43,953-Speed 128.32 samples/sec Loss 11.0869 Epoch: 6 Global Step: 64340 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:42:03,575-Speed 130.48 samples/sec Loss 10.9980 Epoch: 6 Global Step: 64345 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:42:23,751-Speed 126.88 samples/sec Loss 11.2037 Epoch: 6 Global Step: 64350 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:42:43,812-Speed 127.62 samples/sec Loss 10.9102 Epoch: 6 Global Step: 64355 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:43:03,482-Speed 130.16 samples/sec Loss 11.1857 Epoch: 6 Global Step: 64360 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:43:23,406-Speed 128.49 samples/sec Loss 11.2436 Epoch: 6 Global Step: 64365 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:43:43,071-Speed 130.19 samples/sec Loss 11.0238 Epoch: 6 Global Step: 64370 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:44:02,841-Speed 129.49 samples/sec Loss 10.9077 Epoch: 6 Global Step: 64375 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:44:22,518-Speed 130.11 samples/sec Loss 10.8095 Epoch: 6 Global Step: 64380 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:44:42,321-Speed 129.28 samples/sec Loss 11.3247 Epoch: 6 Global Step: 64385 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:45:01,931-Speed 130.55 samples/sec Loss 11.1549 Epoch: 6 Global Step: 64390 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:45:22,023-Speed 127.42 samples/sec Loss 10.8242 Epoch: 6 Global Step: 64395 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:45:42,067-Speed 127.73 samples/sec Loss 10.8159 Epoch: 6 Global Step: 64400 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:46:01,721-Speed 130.26 samples/sec Loss 11.0979 Epoch: 6 Global Step: 64405 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:46:21,697-Speed 128.16 samples/sec Loss 10.8126 Epoch: 6 Global Step: 64410 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:46:41,305-Speed 130.57 samples/sec Loss 10.9852 Epoch: 6 Global Step: 64415 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:47:01,170-Speed 128.88 samples/sec Loss 11.2403 Epoch: 6 Global Step: 64420 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:47:20,837-Speed 130.17 samples/sec Loss 10.6915 Epoch: 6 Global Step: 64425 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:47:40,519-Speed 130.08 samples/sec Loss 11.0205 Epoch: 6 Global Step: 64430 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:48:00,189-Speed 130.16 samples/sec Loss 11.2178 Epoch: 6 Global Step: 64435 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:48:20,163-Speed 128.17 samples/sec Loss 10.9302 Epoch: 6 Global Step: 64440 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:48:40,188-Speed 127.85 samples/sec Loss 10.9033 Epoch: 6 Global Step: 64445 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:48:59,791-Speed 130.59 samples/sec Loss 11.1590 Epoch: 6 Global Step: 64450 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:49:19,660-Speed 128.85 samples/sec Loss 11.1258 Epoch: 6 Global Step: 64455 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:49:39,269-Speed 130.56 samples/sec Loss 10.9814 Epoch: 6 Global Step: 64460 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:49:59,077-Speed 129.25 samples/sec Loss 11.0454 Epoch: 6 Global Step: 64465 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:50:18,713-Speed 130.38 samples/sec Loss 10.6283 Epoch: 6 Global Step: 64470 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:50:38,489-Speed 129.46 samples/sec Loss 10.9615 Epoch: 6 Global Step: 64475 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:50:58,243-Speed 129.60 samples/sec Loss 11.2575 Epoch: 6 Global Step: 64480 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:51:18,164-Speed 128.52 samples/sec Loss 11.2823 Epoch: 6 Global Step: 64485 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:51:38,144-Speed 128.14 samples/sec Loss 11.0386 Epoch: 6 Global Step: 64490 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:51:57,705-Speed 130.88 samples/sec Loss 10.8344 Epoch: 6 Global Step: 64495 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:52:17,670-Speed 128.23 samples/sec Loss 11.1710 Epoch: 6 Global Step: 64500 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:52:37,262-Speed 130.67 samples/sec Loss 10.7449 Epoch: 6 Global Step: 64505 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:52:57,406-Speed 127.09 samples/sec Loss 10.8088 Epoch: 6 Global Step: 64510 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:53:16,998-Speed 130.67 samples/sec Loss 11.1526 Epoch: 6 Global Step: 64515 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:53:36,804-Speed 129.26 samples/sec Loss 10.9522 Epoch: 6 Global Step: 64520 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:53:56,437-Speed 130.40 samples/sec Loss 11.1636 Epoch: 6 Global Step: 64525 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:54:16,602-Speed 126.96 samples/sec Loss 10.9061 Epoch: 6 Global Step: 64530 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:54:36,388-Speed 129.39 samples/sec Loss 11.1545 Epoch: 6 Global Step: 64535 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:54:56,058-Speed 130.16 samples/sec Loss 10.8224 Epoch: 6 Global Step: 64540 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:55:15,926-Speed 128.86 samples/sec Loss 11.0456 Epoch: 6 Global Step: 64545 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:55:35,610-Speed 130.06 samples/sec Loss 11.0933 Epoch: 6 Global Step: 64550 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:55:55,604-Speed 128.05 samples/sec Loss 11.4441 Epoch: 6 Global Step: 64555 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:56:15,162-Speed 130.90 samples/sec Loss 11.0136 Epoch: 6 Global Step: 64560 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:56:35,131-Speed 128.20 samples/sec Loss 10.8763 Epoch: 6 Global Step: 64565 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:56:54,803-Speed 130.14 samples/sec Loss 11.1376 Epoch: 6 Global Step: 64570 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:57:15,010-Speed 126.70 samples/sec Loss 10.9788 Epoch: 6 Global Step: 64575 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:57:34,969-Speed 128.27 samples/sec Loss 11.2462 Epoch: 6 Global Step: 64580 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:57:54,622-Speed 130.26 samples/sec Loss 10.8672 Epoch: 6 Global Step: 64585 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:58:14,401-Speed 129.44 samples/sec Loss 11.1037 Epoch: 6 Global Step: 64590 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:58:34,133-Speed 129.75 samples/sec Loss 11.1043 Epoch: 6 Global Step: 64595 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:58:54,053-Speed 128.52 samples/sec Loss 10.9869 Epoch: 6 Global Step: 64600 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:59:13,658-Speed 130.59 samples/sec Loss 10.8362 Epoch: 6 Global Step: 64605 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:59:33,534-Speed 128.80 samples/sec Loss 11.5027 Epoch: 6 Global Step: 64610 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-09 23:59:53,228-Speed 130.00 samples/sec Loss 11.1244 Epoch: 6 Global Step: 64615 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:00:13,359-Speed 127.17 samples/sec Loss 11.0461 Epoch: 6 Global Step: 64620 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:00:33,278-Speed 128.53 samples/sec Loss 10.9961 Epoch: 6 Global Step: 64625 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:00:52,930-Speed 130.27 samples/sec Loss 11.0559 Epoch: 6 Global Step: 64630 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:01:12,744-Speed 129.21 samples/sec Loss 11.0413 Epoch: 6 Global Step: 64635 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:01:32,541-Speed 129.32 samples/sec Loss 11.2827 Epoch: 6 Global Step: 64640 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:01:52,839-Speed 126.13 samples/sec Loss 11.1004 Epoch: 6 Global Step: 64645 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:02:12,382-Speed 131.00 samples/sec Loss 11.0402 Epoch: 6 Global Step: 64650 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:02:32,652-Speed 126.30 samples/sec Loss 11.0564 Epoch: 6 Global Step: 64655 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:02:52,239-Speed 130.71 samples/sec Loss 10.8441 Epoch: 6 Global Step: 64660 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:03:12,399-Speed 126.99 samples/sec Loss 11.1346 Epoch: 6 Global Step: 64665 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:03:32,443-Speed 127.72 samples/sec Loss 11.2422 Epoch: 6 Global Step: 64670 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:03:52,038-Speed 130.65 samples/sec Loss 11.0386 Epoch: 6 Global Step: 64675 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:04:11,750-Speed 129.88 samples/sec Loss 11.0920 Epoch: 6 Global Step: 64680 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:04:31,394-Speed 130.33 samples/sec Loss 11.2889 Epoch: 6 Global Step: 64685 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:04:51,179-Speed 129.40 samples/sec Loss 10.9252 Epoch: 6 Global Step: 64690 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:05:10,853-Speed 130.13 samples/sec Loss 10.8739 Epoch: 6 Global Step: 64695 Fp16 Grad Scale: 8192 Required: 220 hours Training: 2021-06-10 00:05:30,643-Speed 129.37 samples/sec Loss 11.2237 Epoch: 6 Global Step: 64700 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:05:50,336-Speed 130.00 samples/sec Loss 10.7218 Epoch: 6 Global Step: 64705 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:06:10,321-Speed 128.11 samples/sec Loss 11.1186 Epoch: 6 Global Step: 64710 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:06:30,418-Speed 127.38 samples/sec Loss 11.3231 Epoch: 6 Global Step: 64715 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:06:50,025-Speed 130.57 samples/sec Loss 11.4296 Epoch: 6 Global Step: 64720 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:07:10,013-Speed 128.08 samples/sec Loss 10.6452 Epoch: 6 Global Step: 64725 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:07:29,676-Speed 130.20 samples/sec Loss 11.2664 Epoch: 6 Global Step: 64730 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:07:49,544-Speed 128.86 samples/sec Loss 11.0288 Epoch: 6 Global Step: 64735 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:08:09,195-Speed 130.28 samples/sec Loss 11.0258 Epoch: 6 Global Step: 64740 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:08:28,971-Speed 129.45 samples/sec Loss 10.7729 Epoch: 6 Global Step: 64745 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:08:48,619-Speed 130.31 samples/sec Loss 11.0402 Epoch: 6 Global Step: 64750 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:09:08,767-Speed 127.06 samples/sec Loss 10.7605 Epoch: 6 Global Step: 64755 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:09:28,575-Speed 129.25 samples/sec Loss 10.8045 Epoch: 6 Global Step: 64760 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:09:48,177-Speed 130.60 samples/sec Loss 11.1415 Epoch: 6 Global Step: 64765 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:10:08,114-Speed 128.41 samples/sec Loss 10.9599 Epoch: 6 Global Step: 64770 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:10:27,730-Speed 130.51 samples/sec Loss 11.0336 Epoch: 6 Global Step: 64775 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:10:47,668-Speed 128.41 samples/sec Loss 11.2825 Epoch: 6 Global Step: 64780 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:11:07,282-Speed 130.53 samples/sec Loss 10.8089 Epoch: 6 Global Step: 64785 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:11:27,234-Speed 128.31 samples/sec Loss 10.9985 Epoch: 6 Global Step: 64790 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:11:46,842-Speed 130.57 samples/sec Loss 11.0652 Epoch: 6 Global Step: 64795 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:12:07,085-Speed 126.47 samples/sec Loss 11.2468 Epoch: 6 Global Step: 64800 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:12:26,756-Speed 130.15 samples/sec Loss 10.8482 Epoch: 6 Global Step: 64805 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:12:46,438-Speed 130.07 samples/sec Loss 11.0480 Epoch: 6 Global Step: 64810 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:13:06,225-Speed 129.38 samples/sec Loss 10.9942 Epoch: 6 Global Step: 64815 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:13:25,805-Speed 130.76 samples/sec Loss 11.0908 Epoch: 6 Global Step: 64820 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:13:45,761-Speed 128.29 samples/sec Loss 11.1364 Epoch: 6 Global Step: 64825 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:14:05,489-Speed 129.77 samples/sec Loss 11.1650 Epoch: 6 Global Step: 64830 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:14:25,410-Speed 128.51 samples/sec Loss 11.2093 Epoch: 6 Global Step: 64835 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:14:45,089-Speed 130.09 samples/sec Loss 10.8881 Epoch: 6 Global Step: 64840 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:15:05,156-Speed 127.58 samples/sec Loss 10.9608 Epoch: 6 Global Step: 64845 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:15:24,955-Speed 129.31 samples/sec Loss 10.5695 Epoch: 6 Global Step: 64850 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:15:44,623-Speed 130.17 samples/sec Loss 11.0932 Epoch: 6 Global Step: 64855 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:16:04,388-Speed 129.53 samples/sec Loss 11.1791 Epoch: 6 Global Step: 64860 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:16:24,060-Speed 130.14 samples/sec Loss 11.2124 Epoch: 6 Global Step: 64865 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:16:43,855-Speed 129.33 samples/sec Loss 10.9564 Epoch: 6 Global Step: 64870 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:17:03,558-Speed 129.94 samples/sec Loss 10.8484 Epoch: 6 Global Step: 64875 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:17:23,519-Speed 128.26 samples/sec Loss 11.1898 Epoch: 6 Global Step: 64880 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:17:43,044-Speed 131.12 samples/sec Loss 11.0009 Epoch: 6 Global Step: 64885 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:18:03,247-Speed 126.72 samples/sec Loss 11.1513 Epoch: 6 Global Step: 64890 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:18:23,113-Speed 128.86 samples/sec Loss 10.9093 Epoch: 6 Global Step: 64895 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:18:42,782-Speed 130.16 samples/sec Loss 11.0207 Epoch: 6 Global Step: 64900 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:19:02,463-Speed 130.08 samples/sec Loss 10.9258 Epoch: 6 Global Step: 64905 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:19:22,133-Speed 130.16 samples/sec Loss 11.0507 Epoch: 6 Global Step: 64910 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:19:41,890-Speed 129.58 samples/sec Loss 11.0320 Epoch: 6 Global Step: 64915 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:20:01,486-Speed 130.65 samples/sec Loss 10.9336 Epoch: 6 Global Step: 64920 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:20:21,466-Speed 128.13 samples/sec Loss 11.3905 Epoch: 6 Global Step: 64925 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:20:41,009-Speed 131.00 samples/sec Loss 11.4534 Epoch: 6 Global Step: 64930 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:21:01,157-Speed 127.07 samples/sec Loss 10.9476 Epoch: 6 Global Step: 64935 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:21:21,220-Speed 127.61 samples/sec Loss 11.1200 Epoch: 6 Global Step: 64940 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:21:40,796-Speed 130.77 samples/sec Loss 11.0140 Epoch: 6 Global Step: 64945 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:22:00,604-Speed 129.25 samples/sec Loss 11.1023 Epoch: 6 Global Step: 64950 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:22:20,232-Speed 130.43 samples/sec Loss 10.8948 Epoch: 6 Global Step: 64955 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:22:39,897-Speed 130.19 samples/sec Loss 11.1161 Epoch: 6 Global Step: 64960 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:22:59,559-Speed 130.21 samples/sec Loss 11.0452 Epoch: 6 Global Step: 64965 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:23:19,313-Speed 129.60 samples/sec Loss 10.5924 Epoch: 6 Global Step: 64970 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:23:38,940-Speed 130.44 samples/sec Loss 11.0555 Epoch: 6 Global Step: 64975 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:23:59,027-Speed 127.46 samples/sec Loss 11.0881 Epoch: 6 Global Step: 64980 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:24:19,061-Speed 127.79 samples/sec Loss 11.1625 Epoch: 6 Global Step: 64985 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:24:38,545-Speed 131.39 samples/sec Loss 11.0118 Epoch: 6 Global Step: 64990 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:24:58,531-Speed 128.10 samples/sec Loss 10.9839 Epoch: 6 Global Step: 64995 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:25:18,128-Speed 130.64 samples/sec Loss 10.7967 Epoch: 6 Global Step: 65000 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:25:38,074-Speed 128.35 samples/sec Loss 11.0997 Epoch: 6 Global Step: 65005 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:25:57,665-Speed 130.68 samples/sec Loss 10.7954 Epoch: 6 Global Step: 65010 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:26:17,425-Speed 129.57 samples/sec Loss 11.5802 Epoch: 6 Global Step: 65015 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:26:37,104-Speed 130.09 samples/sec Loss 11.2409 Epoch: 6 Global Step: 65020 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:26:57,004-Speed 128.65 samples/sec Loss 11.5952 Epoch: 6 Global Step: 65025 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:27:16,958-Speed 128.30 samples/sec Loss 11.1438 Epoch: 6 Global Step: 65030 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:27:36,560-Speed 130.61 samples/sec Loss 11.2491 Epoch: 6 Global Step: 65035 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:27:56,489-Speed 128.46 samples/sec Loss 11.0726 Epoch: 6 Global Step: 65040 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:28:16,072-Speed 130.73 samples/sec Loss 10.7968 Epoch: 6 Global Step: 65045 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:28:36,109-Speed 127.77 samples/sec Loss 11.0789 Epoch: 6 Global Step: 65050 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:28:55,649-Speed 131.02 samples/sec Loss 10.8410 Epoch: 6 Global Step: 65055 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:29:15,535-Speed 128.75 samples/sec Loss 10.7192 Epoch: 6 Global Step: 65060 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:29:35,124-Speed 130.69 samples/sec Loss 10.9705 Epoch: 6 Global Step: 65065 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:29:55,387-Speed 126.35 samples/sec Loss 10.9804 Epoch: 6 Global Step: 65070 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:30:15,251-Speed 128.88 samples/sec Loss 11.3085 Epoch: 6 Global Step: 65075 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:30:34,866-Speed 130.52 samples/sec Loss 11.2028 Epoch: 6 Global Step: 65080 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:30:54,766-Speed 128.65 samples/sec Loss 10.8191 Epoch: 6 Global Step: 65085 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:31:14,547-Speed 129.43 samples/sec Loss 11.1759 Epoch: 6 Global Step: 65090 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:31:34,405-Speed 128.92 samples/sec Loss 11.0042 Epoch: 6 Global Step: 65095 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:31:54,008-Speed 130.60 samples/sec Loss 10.8798 Epoch: 6 Global Step: 65100 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:32:14,102-Speed 127.41 samples/sec Loss 10.9780 Epoch: 6 Global Step: 65105 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:32:33,719-Speed 130.50 samples/sec Loss 11.0783 Epoch: 6 Global Step: 65110 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:32:53,906-Speed 126.82 samples/sec Loss 10.6423 Epoch: 6 Global Step: 65115 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:33:13,758-Speed 128.96 samples/sec Loss 10.8585 Epoch: 6 Global Step: 65120 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:33:33,427-Speed 130.16 samples/sec Loss 10.8582 Epoch: 6 Global Step: 65125 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:33:53,138-Speed 129.88 samples/sec Loss 11.1436 Epoch: 6 Global Step: 65130 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:34:12,810-Speed 130.14 samples/sec Loss 11.0763 Epoch: 6 Global Step: 65135 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:34:32,790-Speed 128.14 samples/sec Loss 10.8799 Epoch: 6 Global Step: 65140 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:34:52,405-Speed 130.51 samples/sec Loss 10.8202 Epoch: 6 Global Step: 65145 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:35:12,353-Speed 128.34 samples/sec Loss 10.7769 Epoch: 6 Global Step: 65150 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:35:32,020-Speed 130.18 samples/sec Loss 11.1351 Epoch: 6 Global Step: 65155 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:35:52,130-Speed 127.30 samples/sec Loss 10.9120 Epoch: 6 Global Step: 65160 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:36:12,039-Speed 128.59 samples/sec Loss 10.8245 Epoch: 6 Global Step: 65165 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:36:31,671-Speed 130.40 samples/sec Loss 11.1044 Epoch: 6 Global Step: 65170 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:36:51,531-Speed 128.92 samples/sec Loss 11.1114 Epoch: 6 Global Step: 65175 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:37:11,210-Speed 130.09 samples/sec Loss 10.8203 Epoch: 6 Global Step: 65180 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:37:31,129-Speed 128.52 samples/sec Loss 11.0857 Epoch: 6 Global Step: 65185 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:37:50,778-Speed 130.29 samples/sec Loss 11.1272 Epoch: 6 Global Step: 65190 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:38:10,500-Speed 129.81 samples/sec Loss 10.8779 Epoch: 6 Global Step: 65195 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:38:30,182-Speed 130.08 samples/sec Loss 10.8718 Epoch: 6 Global Step: 65200 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:38:50,188-Speed 127.97 samples/sec Loss 11.0244 Epoch: 6 Global Step: 65205 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:39:10,272-Speed 127.47 samples/sec Loss 11.0758 Epoch: 6 Global Step: 65210 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:39:29,927-Speed 130.25 samples/sec Loss 10.9338 Epoch: 6 Global Step: 65215 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:39:50,024-Speed 127.39 samples/sec Loss 10.9532 Epoch: 6 Global Step: 65220 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:40:09,596-Speed 130.81 samples/sec Loss 10.8964 Epoch: 6 Global Step: 65225 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:40:29,483-Speed 128.74 samples/sec Loss 11.2235 Epoch: 6 Global Step: 65230 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:40:49,125-Speed 130.34 samples/sec Loss 10.8176 Epoch: 6 Global Step: 65235 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:41:08,970-Speed 129.01 samples/sec Loss 11.1494 Epoch: 6 Global Step: 65240 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:41:28,494-Speed 131.13 samples/sec Loss 11.0025 Epoch: 6 Global Step: 65245 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:41:48,540-Speed 127.71 samples/sec Loss 10.7658 Epoch: 6 Global Step: 65250 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:42:08,611-Speed 127.55 samples/sec Loss 10.7654 Epoch: 6 Global Step: 65255 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:42:28,273-Speed 130.21 samples/sec Loss 11.1385 Epoch: 6 Global Step: 65260 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:42:48,192-Speed 128.53 samples/sec Loss 11.3309 Epoch: 6 Global Step: 65265 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:43:07,813-Speed 130.48 samples/sec Loss 11.0514 Epoch: 6 Global Step: 65270 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:43:27,548-Speed 129.72 samples/sec Loss 10.7510 Epoch: 6 Global Step: 65275 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:43:47,183-Speed 130.39 samples/sec Loss 11.1283 Epoch: 6 Global Step: 65280 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:44:06,928-Speed 129.66 samples/sec Loss 10.4494 Epoch: 6 Global Step: 65285 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:44:26,582-Speed 130.26 samples/sec Loss 10.8066 Epoch: 6 Global Step: 65290 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:44:46,603-Speed 127.87 samples/sec Loss 11.1037 Epoch: 6 Global Step: 65295 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:45:06,572-Speed 128.21 samples/sec Loss 10.8818 Epoch: 6 Global Step: 65300 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:45:26,138-Speed 130.85 samples/sec Loss 11.1161 Epoch: 6 Global Step: 65305 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:45:46,104-Speed 128.23 samples/sec Loss 10.9890 Epoch: 6 Global Step: 65310 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:46:05,712-Speed 130.56 samples/sec Loss 10.9565 Epoch: 6 Global Step: 65315 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:46:25,604-Speed 128.70 samples/sec Loss 10.9269 Epoch: 6 Global Step: 65320 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:46:45,274-Speed 130.16 samples/sec Loss 10.9178 Epoch: 6 Global Step: 65325 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:47:04,982-Speed 129.90 samples/sec Loss 10.9525 Epoch: 6 Global Step: 65330 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:47:24,727-Speed 129.66 samples/sec Loss 11.1885 Epoch: 6 Global Step: 65335 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:47:44,866-Speed 127.12 samples/sec Loss 11.0484 Epoch: 6 Global Step: 65340 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:48:04,617-Speed 129.63 samples/sec Loss 11.2632 Epoch: 6 Global Step: 65345 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:48:24,263-Speed 130.31 samples/sec Loss 11.0243 Epoch: 6 Global Step: 65350 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:48:44,064-Speed 129.30 samples/sec Loss 10.8352 Epoch: 6 Global Step: 65355 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:49:03,661-Speed 130.64 samples/sec Loss 10.7294 Epoch: 6 Global Step: 65360 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:49:23,616-Speed 128.30 samples/sec Loss 10.7735 Epoch: 6 Global Step: 65365 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:49:43,192-Speed 130.78 samples/sec Loss 11.2122 Epoch: 6 Global Step: 65370 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:50:03,146-Speed 128.30 samples/sec Loss 10.9941 Epoch: 6 Global Step: 65375 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:50:22,747-Speed 130.61 samples/sec Loss 11.0949 Epoch: 6 Global Step: 65380 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:50:42,895-Speed 127.06 samples/sec Loss 11.0536 Epoch: 6 Global Step: 65385 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:51:02,663-Speed 129.52 samples/sec Loss 11.0354 Epoch: 6 Global Step: 65390 Fp16 Grad Scale: 8192 Required: 219 hours Training: 2021-06-10 00:51:22,334-Speed 130.15 samples/sec Loss 10.9244 Epoch: 6 Global Step: 65395 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:51:42,146-Speed 129.22 samples/sec Loss 11.0491 Epoch: 6 Global Step: 65400 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:52:01,821-Speed 130.12 samples/sec Loss 11.1724 Epoch: 6 Global Step: 65405 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:52:21,547-Speed 129.78 samples/sec Loss 11.0801 Epoch: 6 Global Step: 65410 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:52:41,055-Speed 131.24 samples/sec Loss 10.7605 Epoch: 6 Global Step: 65415 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:53:01,455-Speed 125.50 samples/sec Loss 11.0983 Epoch: 6 Global Step: 65420 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:53:21,059-Speed 130.59 samples/sec Loss 10.8461 Epoch: 6 Global Step: 65425 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:53:41,414-Speed 125.78 samples/sec Loss 11.1285 Epoch: 6 Global Step: 65430 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:54:01,286-Speed 128.83 samples/sec Loss 10.8462 Epoch: 6 Global Step: 65435 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:54:20,899-Speed 130.53 samples/sec Loss 10.9013 Epoch: 6 Global Step: 65440 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:54:40,673-Speed 129.47 samples/sec Loss 10.9152 Epoch: 6 Global Step: 65445 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:55:00,341-Speed 130.17 samples/sec Loss 11.1379 Epoch: 6 Global Step: 65450 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:55:20,014-Speed 130.14 samples/sec Loss 11.0254 Epoch: 6 Global Step: 65455 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:55:39,645-Speed 130.41 samples/sec Loss 10.8037 Epoch: 6 Global Step: 65460 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:55:59,485-Speed 129.04 samples/sec Loss 10.6701 Epoch: 6 Global Step: 65465 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:56:19,208-Speed 129.81 samples/sec Loss 10.8467 Epoch: 6 Global Step: 65470 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:56:39,324-Speed 127.27 samples/sec Loss 11.0608 Epoch: 6 Global Step: 65475 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:56:59,305-Speed 128.13 samples/sec Loss 11.0835 Epoch: 6 Global Step: 65480 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:57:19,135-Speed 129.10 samples/sec Loss 10.7395 Epoch: 6 Global Step: 65485 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:57:39,002-Speed 128.86 samples/sec Loss 11.1272 Epoch: 6 Global Step: 65490 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:57:58,626-Speed 130.46 samples/sec Loss 10.6791 Epoch: 6 Global Step: 65495 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:58:18,587-Speed 128.25 samples/sec Loss 10.7984 Epoch: 6 Global Step: 65500 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:58:38,272-Speed 130.06 samples/sec Loss 11.1157 Epoch: 6 Global Step: 65505 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:58:58,054-Speed 129.42 samples/sec Loss 10.8970 Epoch: 6 Global Step: 65510 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:59:17,826-Speed 129.49 samples/sec Loss 11.3090 Epoch: 6 Global Step: 65515 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:59:37,753-Speed 128.47 samples/sec Loss 11.0035 Epoch: 6 Global Step: 65520 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 00:59:57,724-Speed 128.19 samples/sec Loss 11.0318 Epoch: 6 Global Step: 65525 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:00:17,319-Speed 130.65 samples/sec Loss 10.9550 Epoch: 6 Global Step: 65530 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:00:37,260-Speed 128.39 samples/sec Loss 11.0272 Epoch: 6 Global Step: 65535 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:00:56,748-Speed 131.37 samples/sec Loss 10.6956 Epoch: 6 Global Step: 65540 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:01:16,602-Speed 128.95 samples/sec Loss 10.6535 Epoch: 6 Global Step: 65545 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:01:36,242-Speed 130.35 samples/sec Loss 10.9961 Epoch: 6 Global Step: 65550 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:01:56,010-Speed 129.51 samples/sec Loss 11.1054 Epoch: 6 Global Step: 65555 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:02:15,813-Speed 129.29 samples/sec Loss 10.7558 Epoch: 6 Global Step: 65560 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:02:35,855-Speed 127.74 samples/sec Loss 10.9940 Epoch: 6 Global Step: 65565 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:02:56,399-Speed 124.62 samples/sec Loss 11.1121 Epoch: 6 Global Step: 65570 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:03:16,114-Speed 129.86 samples/sec Loss 11.0649 Epoch: 6 Global Step: 65575 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:03:35,971-Speed 128.93 samples/sec Loss 11.1387 Epoch: 6 Global Step: 65580 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:03:55,567-Speed 130.65 samples/sec Loss 11.1270 Epoch: 6 Global Step: 65585 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:04:15,533-Speed 128.23 samples/sec Loss 11.0174 Epoch: 6 Global Step: 65590 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:04:35,186-Speed 130.26 samples/sec Loss 11.0453 Epoch: 6 Global Step: 65595 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:04:54,972-Speed 129.39 samples/sec Loss 11.0539 Epoch: 6 Global Step: 65600 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:05:14,621-Speed 130.29 samples/sec Loss 10.6475 Epoch: 6 Global Step: 65605 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:05:34,775-Speed 127.02 samples/sec Loss 11.0458 Epoch: 6 Global Step: 65610 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:05:54,612-Speed 129.06 samples/sec Loss 11.0056 Epoch: 6 Global Step: 65615 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:06:14,130-Speed 131.16 samples/sec Loss 11.2710 Epoch: 6 Global Step: 65620 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:06:34,078-Speed 128.35 samples/sec Loss 11.1725 Epoch: 6 Global Step: 65625 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:06:53,748-Speed 130.15 samples/sec Loss 10.7796 Epoch: 6 Global Step: 65630 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:07:13,621-Speed 128.82 samples/sec Loss 11.0756 Epoch: 6 Global Step: 65635 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:07:33,290-Speed 130.16 samples/sec Loss 11.1161 Epoch: 6 Global Step: 65640 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:07:53,136-Speed 129.00 samples/sec Loss 10.9615 Epoch: 6 Global Step: 65645 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:08:12,737-Speed 130.61 samples/sec Loss 11.0370 Epoch: 6 Global Step: 65650 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:08:32,892-Speed 127.02 samples/sec Loss 11.0959 Epoch: 6 Global Step: 65655 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:08:52,721-Speed 129.11 samples/sec Loss 10.8026 Epoch: 6 Global Step: 65660 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:09:12,392-Speed 130.15 samples/sec Loss 11.0940 Epoch: 6 Global Step: 65665 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:09:32,171-Speed 129.43 samples/sec Loss 10.8835 Epoch: 6 Global Step: 65670 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:09:51,873-Speed 129.94 samples/sec Loss 10.5051 Epoch: 6 Global Step: 65675 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:10:11,810-Speed 128.41 samples/sec Loss 11.1890 Epoch: 6 Global Step: 65680 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:10:31,384-Speed 130.79 samples/sec Loss 10.9602 Epoch: 6 Global Step: 65685 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:10:51,346-Speed 128.25 samples/sec Loss 10.8868 Epoch: 6 Global Step: 65690 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:11:11,146-Speed 129.30 samples/sec Loss 11.0898 Epoch: 6 Global Step: 65695 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:11:31,310-Speed 126.97 samples/sec Loss 11.0151 Epoch: 6 Global Step: 65700 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:11:51,281-Speed 128.20 samples/sec Loss 11.0381 Epoch: 6 Global Step: 65705 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:12:10,918-Speed 130.37 samples/sec Loss 10.7518 Epoch: 6 Global Step: 65710 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:12:30,689-Speed 129.49 samples/sec Loss 11.1314 Epoch: 6 Global Step: 65715 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:12:50,361-Speed 130.14 samples/sec Loss 11.4079 Epoch: 6 Global Step: 65720 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:13:10,237-Speed 128.81 samples/sec Loss 11.0502 Epoch: 6 Global Step: 65725 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:13:29,829-Speed 130.67 samples/sec Loss 11.1262 Epoch: 6 Global Step: 65730 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:13:49,743-Speed 128.56 samples/sec Loss 11.0383 Epoch: 6 Global Step: 65735 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:14:09,382-Speed 130.36 samples/sec Loss 10.9333 Epoch: 6 Global Step: 65740 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:14:29,483-Speed 127.37 samples/sec Loss 11.2357 Epoch: 6 Global Step: 65745 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:14:49,488-Speed 127.97 samples/sec Loss 11.2364 Epoch: 6 Global Step: 65750 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:15:09,052-Speed 130.86 samples/sec Loss 11.1663 Epoch: 6 Global Step: 65755 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:15:28,880-Speed 129.11 samples/sec Loss 11.1949 Epoch: 6 Global Step: 65760 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:15:48,544-Speed 130.20 samples/sec Loss 11.5275 Epoch: 6 Global Step: 65765 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:16:08,330-Speed 129.39 samples/sec Loss 10.8913 Epoch: 6 Global Step: 65770 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:16:27,984-Speed 130.26 samples/sec Loss 11.1950 Epoch: 6 Global Step: 65775 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:16:47,780-Speed 129.32 samples/sec Loss 11.0033 Epoch: 6 Global Step: 65780 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:17:07,330-Speed 130.95 samples/sec Loss 10.9601 Epoch: 6 Global Step: 65785 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:17:27,517-Speed 126.82 samples/sec Loss 11.4212 Epoch: 6 Global Step: 65790 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:17:47,600-Speed 127.47 samples/sec Loss 10.9709 Epoch: 6 Global Step: 65795 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:18:07,222-Speed 130.48 samples/sec Loss 11.0276 Epoch: 6 Global Step: 65800 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:18:27,172-Speed 128.33 samples/sec Loss 10.6738 Epoch: 6 Global Step: 65805 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:18:46,831-Speed 130.23 samples/sec Loss 11.0117 Epoch: 6 Global Step: 65810 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:19:06,734-Speed 128.63 samples/sec Loss 10.6970 Epoch: 6 Global Step: 65815 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:19:26,329-Speed 130.66 samples/sec Loss 11.1511 Epoch: 6 Global Step: 65820 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:19:46,100-Speed 129.49 samples/sec Loss 11.1557 Epoch: 6 Global Step: 65825 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:20:05,805-Speed 129.92 samples/sec Loss 11.2469 Epoch: 6 Global Step: 65830 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:20:25,827-Speed 127.87 samples/sec Loss 10.9599 Epoch: 6 Global Step: 65835 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:20:45,715-Speed 128.73 samples/sec Loss 10.8332 Epoch: 6 Global Step: 65840 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:21:05,295-Speed 130.75 samples/sec Loss 11.0020 Epoch: 6 Global Step: 65845 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:21:25,301-Speed 127.97 samples/sec Loss 11.2734 Epoch: 6 Global Step: 65850 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:21:44,886-Speed 130.72 samples/sec Loss 10.9089 Epoch: 6 Global Step: 65855 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:22:04,857-Speed 128.19 samples/sec Loss 10.6998 Epoch: 6 Global Step: 65860 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:22:24,458-Speed 130.61 samples/sec Loss 11.2109 Epoch: 6 Global Step: 65865 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:22:44,260-Speed 129.29 samples/sec Loss 11.2514 Epoch: 6 Global Step: 65870 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:23:03,929-Speed 130.16 samples/sec Loss 10.8766 Epoch: 6 Global Step: 65875 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:23:24,078-Speed 127.05 samples/sec Loss 10.9939 Epoch: 6 Global Step: 65880 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:23:43,840-Speed 129.55 samples/sec Loss 11.1230 Epoch: 6 Global Step: 65885 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:24:03,447-Speed 130.57 samples/sec Loss 10.9125 Epoch: 6 Global Step: 65890 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:24:23,284-Speed 129.06 samples/sec Loss 10.7010 Epoch: 6 Global Step: 65895 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:24:42,894-Speed 130.55 samples/sec Loss 10.9859 Epoch: 6 Global Step: 65900 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:25:02,869-Speed 128.17 samples/sec Loss 11.1181 Epoch: 6 Global Step: 65905 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:25:22,463-Speed 130.66 samples/sec Loss 11.0381 Epoch: 6 Global Step: 65910 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:25:42,446-Speed 128.12 samples/sec Loss 10.9061 Epoch: 6 Global Step: 65915 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:26:02,097-Speed 130.28 samples/sec Loss 10.9991 Epoch: 6 Global Step: 65920 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:26:22,130-Speed 127.80 samples/sec Loss 10.8491 Epoch: 6 Global Step: 65925 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:26:41,908-Speed 129.44 samples/sec Loss 11.0634 Epoch: 6 Global Step: 65930 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:27:01,574-Speed 130.17 samples/sec Loss 10.8984 Epoch: 6 Global Step: 65935 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:27:21,269-Speed 129.99 samples/sec Loss 10.9536 Epoch: 6 Global Step: 65940 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:27:40,904-Speed 130.39 samples/sec Loss 10.6954 Epoch: 6 Global Step: 65945 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:28:00,578-Speed 130.13 samples/sec Loss 10.8876 Epoch: 6 Global Step: 65950 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:28:20,157-Speed 130.76 samples/sec Loss 10.7920 Epoch: 6 Global Step: 65955 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:28:40,153-Speed 128.04 samples/sec Loss 10.8114 Epoch: 6 Global Step: 65960 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:28:59,662-Speed 131.23 samples/sec Loss 10.7646 Epoch: 6 Global Step: 65965 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:29:19,814-Speed 127.04 samples/sec Loss 10.9876 Epoch: 6 Global Step: 65970 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:29:39,656-Speed 129.03 samples/sec Loss 10.8975 Epoch: 6 Global Step: 65975 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:29:59,269-Speed 130.53 samples/sec Loss 11.2756 Epoch: 6 Global Step: 65980 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:30:18,971-Speed 129.95 samples/sec Loss 10.5935 Epoch: 6 Global Step: 65985 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:30:38,651-Speed 130.09 samples/sec Loss 10.8030 Epoch: 6 Global Step: 65990 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:30:58,401-Speed 129.63 samples/sec Loss 11.2580 Epoch: 6 Global Step: 65995 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:31:18,095-Speed 130.00 samples/sec Loss 10.9324 Epoch: 6 Global Step: 66000 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:31:37,978-Speed 128.76 samples/sec Loss 10.8816 Epoch: 6 Global Step: 66005 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:31:57,595-Speed 130.51 samples/sec Loss 10.9476 Epoch: 6 Global Step: 66010 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:32:17,781-Speed 126.83 samples/sec Loss 10.7088 Epoch: 6 Global Step: 66015 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:32:37,812-Speed 127.81 samples/sec Loss 10.7394 Epoch: 6 Global Step: 66020 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:32:57,371-Speed 130.89 samples/sec Loss 11.1503 Epoch: 6 Global Step: 66025 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:33:17,168-Speed 129.32 samples/sec Loss 10.9888 Epoch: 6 Global Step: 66030 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:33:36,829-Speed 130.22 samples/sec Loss 10.8855 Epoch: 6 Global Step: 66035 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:33:56,656-Speed 129.12 samples/sec Loss 11.0192 Epoch: 6 Global Step: 66040 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:34:16,324-Speed 130.17 samples/sec Loss 10.9270 Epoch: 6 Global Step: 66045 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:34:36,107-Speed 129.41 samples/sec Loss 11.5903 Epoch: 6 Global Step: 66050 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:34:55,727-Speed 130.49 samples/sec Loss 10.7420 Epoch: 6 Global Step: 66055 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:35:15,846-Speed 127.25 samples/sec Loss 11.0975 Epoch: 6 Global Step: 66060 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:35:35,902-Speed 127.65 samples/sec Loss 10.8776 Epoch: 6 Global Step: 66065 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:35:55,525-Speed 130.46 samples/sec Loss 10.9695 Epoch: 6 Global Step: 66070 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:36:15,218-Speed 130.01 samples/sec Loss 10.9046 Epoch: 6 Global Step: 66075 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:36:34,849-Speed 130.41 samples/sec Loss 10.9913 Epoch: 6 Global Step: 66080 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:36:54,956-Speed 127.33 samples/sec Loss 11.3735 Epoch: 6 Global Step: 66085 Fp16 Grad Scale: 8192 Required: 218 hours Training: 2021-06-10 01:37:14,590-Speed 130.39 samples/sec Loss 11.0120 Epoch: 6 Global Step: 66090 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:37:34,382-Speed 129.36 samples/sec Loss 10.7887 Epoch: 6 Global Step: 66095 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:37:54,108-Speed 129.79 samples/sec Loss 11.2472 Epoch: 6 Global Step: 66100 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:38:14,228-Speed 127.24 samples/sec Loss 10.9536 Epoch: 6 Global Step: 66105 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:38:34,153-Speed 128.49 samples/sec Loss 11.2580 Epoch: 6 Global Step: 66110 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:38:53,700-Speed 130.97 samples/sec Loss 11.1761 Epoch: 6 Global Step: 66115 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:39:13,667-Speed 128.22 samples/sec Loss 10.9755 Epoch: 6 Global Step: 66120 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:39:33,345-Speed 130.11 samples/sec Loss 11.0281 Epoch: 6 Global Step: 66125 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:39:53,245-Speed 128.65 samples/sec Loss 10.8057 Epoch: 6 Global Step: 66130 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:40:12,859-Speed 130.52 samples/sec Loss 10.8853 Epoch: 6 Global Step: 66135 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:40:32,640-Speed 129.42 samples/sec Loss 11.1415 Epoch: 6 Global Step: 66140 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:40:52,265-Speed 130.46 samples/sec Loss 10.8235 Epoch: 6 Global Step: 66145 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:41:12,485-Speed 126.61 samples/sec Loss 10.8128 Epoch: 6 Global Step: 66150 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:41:32,288-Speed 129.28 samples/sec Loss 10.8891 Epoch: 6 Global Step: 66155 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:41:51,900-Speed 130.54 samples/sec Loss 10.6821 Epoch: 6 Global Step: 66160 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:42:11,827-Speed 128.47 samples/sec Loss 11.1630 Epoch: 6 Global Step: 66165 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:42:31,449-Speed 130.47 samples/sec Loss 11.0372 Epoch: 6 Global Step: 66170 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:42:51,289-Speed 129.04 samples/sec Loss 10.8854 Epoch: 6 Global Step: 66175 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:43:10,851-Speed 130.87 samples/sec Loss 11.1729 Epoch: 6 Global Step: 66180 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:43:30,716-Speed 128.87 samples/sec Loss 10.7515 Epoch: 6 Global Step: 66185 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:43:50,370-Speed 130.26 samples/sec Loss 10.8710 Epoch: 6 Global Step: 66190 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:44:10,473-Speed 127.35 samples/sec Loss 10.8774 Epoch: 6 Global Step: 66195 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:44:30,215-Speed 129.68 samples/sec Loss 11.3248 Epoch: 6 Global Step: 66200 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:44:49,869-Speed 130.26 samples/sec Loss 10.9582 Epoch: 6 Global Step: 66205 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:45:09,752-Speed 128.76 samples/sec Loss 10.8840 Epoch: 6 Global Step: 66210 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:45:29,293-Speed 131.01 samples/sec Loss 10.9462 Epoch: 6 Global Step: 66215 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:45:49,250-Speed 128.28 samples/sec Loss 10.7094 Epoch: 6 Global Step: 66220 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:46:08,930-Speed 130.09 samples/sec Loss 10.9185 Epoch: 6 Global Step: 66225 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:46:28,878-Speed 128.35 samples/sec Loss 10.7963 Epoch: 6 Global Step: 66230 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:46:48,489-Speed 130.54 samples/sec Loss 11.3597 Epoch: 6 Global Step: 66235 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:47:08,537-Speed 127.70 samples/sec Loss 10.8695 Epoch: 6 Global Step: 66240 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:47:28,269-Speed 129.75 samples/sec Loss 11.0103 Epoch: 6 Global Step: 66245 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:47:48,026-Speed 129.58 samples/sec Loss 10.8720 Epoch: 6 Global Step: 66250 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:48:07,729-Speed 129.94 samples/sec Loss 11.2720 Epoch: 6 Global Step: 66255 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:48:27,292-Speed 130.87 samples/sec Loss 11.0787 Epoch: 6 Global Step: 66260 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:48:47,219-Speed 128.47 samples/sec Loss 11.2455 Epoch: 6 Global Step: 66265 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:49:06,717-Speed 131.31 samples/sec Loss 11.1032 Epoch: 6 Global Step: 66270 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:49:26,717-Speed 128.01 samples/sec Loss 10.6763 Epoch: 6 Global Step: 66275 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:49:46,254-Speed 131.04 samples/sec Loss 11.1239 Epoch: 6 Global Step: 66280 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:50:06,402-Speed 127.06 samples/sec Loss 10.8354 Epoch: 6 Global Step: 66285 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:50:26,322-Speed 128.53 samples/sec Loss 10.7050 Epoch: 6 Global Step: 66290 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:50:45,980-Speed 130.23 samples/sec Loss 10.8936 Epoch: 6 Global Step: 66295 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:51:05,661-Speed 130.08 samples/sec Loss 10.8372 Epoch: 6 Global Step: 66300 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:51:25,337-Speed 130.12 samples/sec Loss 10.9756 Epoch: 6 Global Step: 66305 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:51:44,983-Speed 130.31 samples/sec Loss 11.2638 Epoch: 6 Global Step: 66310 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:52:04,693-Speed 129.90 samples/sec Loss 11.1654 Epoch: 6 Global Step: 66315 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:52:24,633-Speed 128.39 samples/sec Loss 11.2891 Epoch: 6 Global Step: 66320 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:52:44,258-Speed 130.45 samples/sec Loss 10.6464 Epoch: 6 Global Step: 66325 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:53:04,440-Speed 126.85 samples/sec Loss 11.1526 Epoch: 6 Global Step: 66330 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:53:24,383-Speed 128.38 samples/sec Loss 10.7846 Epoch: 6 Global Step: 66335 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:53:43,972-Speed 130.69 samples/sec Loss 11.0198 Epoch: 6 Global Step: 66340 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:54:03,742-Speed 129.50 samples/sec Loss 11.1332 Epoch: 6 Global Step: 66345 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:54:23,390-Speed 130.30 samples/sec Loss 11.0340 Epoch: 6 Global Step: 66350 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:54:43,098-Speed 129.90 samples/sec Loss 10.7780 Epoch: 6 Global Step: 66355 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:55:02,834-Speed 129.72 samples/sec Loss 11.1644 Epoch: 6 Global Step: 66360 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:55:22,501-Speed 130.18 samples/sec Loss 10.7829 Epoch: 6 Global Step: 66365 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:55:42,087-Speed 130.71 samples/sec Loss 10.9812 Epoch: 6 Global Step: 66370 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:56:02,472-Speed 125.59 samples/sec Loss 10.9733 Epoch: 6 Global Step: 66375 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:56:22,574-Speed 127.36 samples/sec Loss 10.9909 Epoch: 6 Global Step: 66380 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:56:42,144-Speed 130.82 samples/sec Loss 10.8629 Epoch: 6 Global Step: 66385 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:57:01,965-Speed 129.16 samples/sec Loss 11.2400 Epoch: 6 Global Step: 66390 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:57:21,578-Speed 130.53 samples/sec Loss 10.6656 Epoch: 6 Global Step: 66395 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:57:41,363-Speed 129.40 samples/sec Loss 10.7490 Epoch: 6 Global Step: 66400 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:58:01,027-Speed 130.20 samples/sec Loss 10.7193 Epoch: 6 Global Step: 66405 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:58:20,791-Speed 129.53 samples/sec Loss 11.5378 Epoch: 6 Global Step: 66410 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:58:40,400-Speed 130.56 samples/sec Loss 10.9602 Epoch: 6 Global Step: 66415 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:59:00,436-Speed 127.78 samples/sec Loss 10.9352 Epoch: 6 Global Step: 66420 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:59:20,426-Speed 128.07 samples/sec Loss 11.1779 Epoch: 6 Global Step: 66425 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:59:40,087-Speed 130.21 samples/sec Loss 11.0215 Epoch: 6 Global Step: 66430 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 01:59:59,927-Speed 129.04 samples/sec Loss 10.9827 Epoch: 6 Global Step: 66435 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:00:19,529-Speed 130.60 samples/sec Loss 11.2784 Epoch: 6 Global Step: 66440 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:00:39,333-Speed 129.28 samples/sec Loss 10.6991 Epoch: 6 Global Step: 66445 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:00:58,964-Speed 130.42 samples/sec Loss 11.2056 Epoch: 6 Global Step: 66450 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:01:18,736-Speed 129.48 samples/sec Loss 10.9852 Epoch: 6 Global Step: 66455 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:01:38,389-Speed 130.26 samples/sec Loss 10.8375 Epoch: 6 Global Step: 66460 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:01:58,545-Speed 127.02 samples/sec Loss 10.7732 Epoch: 6 Global Step: 66465 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:02:18,515-Speed 128.20 samples/sec Loss 10.7792 Epoch: 6 Global Step: 66470 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:02:38,126-Speed 130.54 samples/sec Loss 11.1604 Epoch: 6 Global Step: 66475 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:02:58,079-Speed 128.32 samples/sec Loss 11.2522 Epoch: 6 Global Step: 66480 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:03:17,740-Speed 130.21 samples/sec Loss 11.1044 Epoch: 6 Global Step: 66485 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:03:37,565-Speed 129.13 samples/sec Loss 10.9091 Epoch: 6 Global Step: 66490 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:03:57,152-Speed 130.71 samples/sec Loss 10.7961 Epoch: 6 Global Step: 66495 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:04:16,866-Speed 129.86 samples/sec Loss 10.9830 Epoch: 6 Global Step: 66500 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:04:36,514-Speed 130.30 samples/sec Loss 11.1151 Epoch: 6 Global Step: 66505 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:04:56,690-Speed 126.89 samples/sec Loss 10.8218 Epoch: 6 Global Step: 66510 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:05:16,558-Speed 128.85 samples/sec Loss 10.8661 Epoch: 6 Global Step: 66515 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:05:36,079-Speed 131.15 samples/sec Loss 10.9700 Epoch: 6 Global Step: 66520 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:05:55,994-Speed 128.55 samples/sec Loss 11.2020 Epoch: 6 Global Step: 66525 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:06:15,554-Speed 130.89 samples/sec Loss 10.8296 Epoch: 6 Global Step: 66530 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:06:35,513-Speed 128.27 samples/sec Loss 10.8722 Epoch: 6 Global Step: 66535 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:06:55,110-Speed 130.64 samples/sec Loss 11.0333 Epoch: 6 Global Step: 66540 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:07:14,965-Speed 128.94 samples/sec Loss 10.5411 Epoch: 6 Global Step: 66545 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:07:34,643-Speed 130.10 samples/sec Loss 10.8120 Epoch: 6 Global Step: 66550 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:07:54,707-Speed 127.60 samples/sec Loss 11.2468 Epoch: 6 Global Step: 66555 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:08:14,359-Speed 130.27 samples/sec Loss 10.6944 Epoch: 6 Global Step: 66560 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:08:34,023-Speed 130.19 samples/sec Loss 10.8687 Epoch: 6 Global Step: 66565 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:08:53,781-Speed 129.58 samples/sec Loss 11.2354 Epoch: 6 Global Step: 66570 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:09:13,430-Speed 130.29 samples/sec Loss 10.5975 Epoch: 6 Global Step: 66575 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:09:33,345-Speed 128.56 samples/sec Loss 11.1279 Epoch: 6 Global Step: 66580 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:09:52,911-Speed 130.84 samples/sec Loss 10.8282 Epoch: 6 Global Step: 66585 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:10:12,936-Speed 127.85 samples/sec Loss 10.8658 Epoch: 6 Global Step: 66590 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:10:32,435-Speed 131.29 samples/sec Loss 11.0908 Epoch: 6 Global Step: 66595 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:10:52,595-Speed 126.99 samples/sec Loss 11.0159 Epoch: 6 Global Step: 66600 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:11:12,353-Speed 129.58 samples/sec Loss 10.8823 Epoch: 6 Global Step: 66605 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:11:32,027-Speed 130.12 samples/sec Loss 10.7721 Epoch: 6 Global Step: 66610 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:11:51,781-Speed 129.60 samples/sec Loss 10.7346 Epoch: 6 Global Step: 66615 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:12:11,378-Speed 130.64 samples/sec Loss 10.7570 Epoch: 6 Global Step: 66620 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:12:31,211-Speed 129.08 samples/sec Loss 10.8483 Epoch: 6 Global Step: 66625 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:12:50,806-Speed 130.66 samples/sec Loss 10.6691 Epoch: 6 Global Step: 66630 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:13:10,742-Speed 128.42 samples/sec Loss 11.1894 Epoch: 6 Global Step: 66635 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:13:30,318-Speed 130.78 samples/sec Loss 11.1249 Epoch: 6 Global Step: 66640 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:13:50,494-Speed 126.89 samples/sec Loss 11.1447 Epoch: 6 Global Step: 66645 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:14:10,288-Speed 129.34 samples/sec Loss 11.0665 Epoch: 6 Global Step: 66650 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:14:29,938-Speed 130.28 samples/sec Loss 11.1036 Epoch: 6 Global Step: 66655 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:14:49,618-Speed 130.09 samples/sec Loss 11.2017 Epoch: 6 Global Step: 66660 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:15:09,244-Speed 130.45 samples/sec Loss 10.9534 Epoch: 6 Global Step: 66665 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:15:29,177-Speed 128.43 samples/sec Loss 10.7577 Epoch: 6 Global Step: 66670 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:15:48,856-Speed 130.10 samples/sec Loss 11.0647 Epoch: 6 Global Step: 66675 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:16:08,779-Speed 128.50 samples/sec Loss 11.2114 Epoch: 6 Global Step: 66680 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:16:28,394-Speed 130.52 samples/sec Loss 11.0536 Epoch: 6 Global Step: 66685 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:16:48,622-Speed 126.56 samples/sec Loss 10.9214 Epoch: 6 Global Step: 66690 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:17:08,636-Speed 127.92 samples/sec Loss 11.0692 Epoch: 6 Global Step: 66695 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:17:28,258-Speed 130.47 samples/sec Loss 11.1436 Epoch: 6 Global Step: 66700 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:17:48,037-Speed 129.44 samples/sec Loss 10.7125 Epoch: 6 Global Step: 66705 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:18:07,702-Speed 130.19 samples/sec Loss 11.1180 Epoch: 6 Global Step: 66710 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:18:27,461-Speed 129.56 samples/sec Loss 10.7674 Epoch: 6 Global Step: 66715 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:18:47,118-Speed 130.25 samples/sec Loss 11.0681 Epoch: 6 Global Step: 66720 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:19:06,785-Speed 130.17 samples/sec Loss 11.1944 Epoch: 6 Global Step: 66725 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:19:26,372-Speed 130.70 samples/sec Loss 10.9155 Epoch: 6 Global Step: 66730 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:19:46,474-Speed 127.36 samples/sec Loss 10.5404 Epoch: 6 Global Step: 66735 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:20:06,560-Speed 127.46 samples/sec Loss 11.1146 Epoch: 6 Global Step: 66740 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:20:26,154-Speed 130.66 samples/sec Loss 10.5817 Epoch: 6 Global Step: 66745 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:20:45,994-Speed 129.04 samples/sec Loss 11.0028 Epoch: 6 Global Step: 66750 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:21:05,583-Speed 130.69 samples/sec Loss 10.8008 Epoch: 6 Global Step: 66755 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:21:25,572-Speed 128.08 samples/sec Loss 11.0300 Epoch: 6 Global Step: 66760 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:21:45,279-Speed 129.91 samples/sec Loss 10.6773 Epoch: 6 Global Step: 66765 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:22:05,117-Speed 129.05 samples/sec Loss 10.8163 Epoch: 6 Global Step: 66770 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:22:24,822-Speed 129.93 samples/sec Loss 10.9986 Epoch: 6 Global Step: 66775 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:22:44,781-Speed 128.26 samples/sec Loss 10.8310 Epoch: 6 Global Step: 66780 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:23:04,777-Speed 128.03 samples/sec Loss 10.8170 Epoch: 6 Global Step: 66785 Fp16 Grad Scale: 8192 Required: 217 hours Training: 2021-06-10 02:23:24,296-Speed 131.16 samples/sec Loss 10.8745 Epoch: 6 Global Step: 66790 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:23:44,235-Speed 128.40 samples/sec Loss 10.8428 Epoch: 6 Global Step: 66795 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:24:03,843-Speed 130.57 samples/sec Loss 11.2403 Epoch: 6 Global Step: 66800 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:24:23,754-Speed 128.58 samples/sec Loss 11.1093 Epoch: 6 Global Step: 66805 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:24:43,353-Speed 130.62 samples/sec Loss 11.0375 Epoch: 6 Global Step: 66810 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:25:03,116-Speed 129.55 samples/sec Loss 10.9122 Epoch: 6 Global Step: 66815 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:25:22,745-Speed 130.42 samples/sec Loss 10.8552 Epoch: 6 Global Step: 66820 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:25:42,967-Speed 126.60 samples/sec Loss 10.8955 Epoch: 6 Global Step: 66825 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:26:02,663-Speed 129.98 samples/sec Loss 10.6300 Epoch: 6 Global Step: 66830 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:26:22,178-Speed 131.19 samples/sec Loss 11.1370 Epoch: 6 Global Step: 66835 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:26:42,105-Speed 128.48 samples/sec Loss 11.0635 Epoch: 6 Global Step: 66840 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:27:01,892-Speed 129.38 samples/sec Loss 11.0819 Epoch: 6 Global Step: 66845 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:27:21,839-Speed 128.35 samples/sec Loss 11.0401 Epoch: 6 Global Step: 66850 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:27:41,389-Speed 130.95 samples/sec Loss 10.9429 Epoch: 6 Global Step: 66855 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:28:01,266-Speed 128.80 samples/sec Loss 10.6917 Epoch: 6 Global Step: 66860 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:28:20,856-Speed 130.69 samples/sec Loss 11.1223 Epoch: 6 Global Step: 66865 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:28:41,013-Speed 127.00 samples/sec Loss 11.1025 Epoch: 6 Global Step: 66870 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:29:00,692-Speed 130.10 samples/sec Loss 11.1369 Epoch: 6 Global Step: 66875 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:29:20,344-Speed 130.27 samples/sec Loss 10.8403 Epoch: 6 Global Step: 66880 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:29:40,194-Speed 128.98 samples/sec Loss 10.7917 Epoch: 6 Global Step: 66885 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:29:59,763-Speed 130.83 samples/sec Loss 10.9125 Epoch: 6 Global Step: 66890 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:30:19,562-Speed 129.31 samples/sec Loss 11.1196 Epoch: 6 Global Step: 66895 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:30:39,079-Speed 131.18 samples/sec Loss 11.0740 Epoch: 6 Global Step: 66900 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:30:59,015-Speed 128.42 samples/sec Loss 10.8846 Epoch: 6 Global Step: 66905 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:31:18,673-Speed 130.23 samples/sec Loss 10.9134 Epoch: 6 Global Step: 66910 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:31:38,839-Speed 126.95 samples/sec Loss 11.2307 Epoch: 6 Global Step: 66915 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:31:58,923-Speed 127.47 samples/sec Loss 11.1018 Epoch: 6 Global Step: 66920 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:32:18,547-Speed 130.45 samples/sec Loss 10.8923 Epoch: 6 Global Step: 66925 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:32:38,307-Speed 129.56 samples/sec Loss 10.8962 Epoch: 6 Global Step: 66930 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:32:57,882-Speed 130.79 samples/sec Loss 10.8330 Epoch: 6 Global Step: 66935 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:33:17,651-Speed 129.50 samples/sec Loss 10.8460 Epoch: 6 Global Step: 66940 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:33:37,357-Speed 129.92 samples/sec Loss 10.8286 Epoch: 6 Global Step: 66945 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:33:57,363-Speed 127.97 samples/sec Loss 10.9516 Epoch: 6 Global Step: 66950 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:34:16,984-Speed 130.48 samples/sec Loss 10.8275 Epoch: 6 Global Step: 66955 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:34:37,023-Speed 127.76 samples/sec Loss 10.8676 Epoch: 6 Global Step: 66960 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:34:56,939-Speed 128.55 samples/sec Loss 11.3453 Epoch: 6 Global Step: 66965 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:35:16,726-Speed 129.38 samples/sec Loss 11.1190 Epoch: 6 Global Step: 66970 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:35:36,504-Speed 129.44 samples/sec Loss 10.6256 Epoch: 6 Global Step: 66975 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:35:56,177-Speed 130.14 samples/sec Loss 10.7999 Epoch: 6 Global Step: 66980 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:36:15,951-Speed 129.47 samples/sec Loss 11.2576 Epoch: 6 Global Step: 66985 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:36:35,593-Speed 130.34 samples/sec Loss 11.2498 Epoch: 6 Global Step: 66990 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:36:55,327-Speed 129.74 samples/sec Loss 11.0576 Epoch: 6 Global Step: 66995 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:37:14,887-Speed 130.89 samples/sec Loss 10.8474 Epoch: 6 Global Step: 67000 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:37:34,933-Speed 127.71 samples/sec Loss 10.7766 Epoch: 6 Global Step: 67005 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:37:55,082-Speed 127.06 samples/sec Loss 10.8373 Epoch: 6 Global Step: 67010 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:38:14,588-Speed 131.25 samples/sec Loss 11.1173 Epoch: 6 Global Step: 67015 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:38:34,466-Speed 128.79 samples/sec Loss 11.1982 Epoch: 6 Global Step: 67020 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:38:54,067-Speed 130.62 samples/sec Loss 10.3398 Epoch: 6 Global Step: 67025 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:39:13,883-Speed 129.19 samples/sec Loss 10.9042 Epoch: 6 Global Step: 67030 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:39:33,550-Speed 130.17 samples/sec Loss 10.7918 Epoch: 6 Global Step: 67035 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:39:53,334-Speed 129.41 samples/sec Loss 10.8907 Epoch: 6 Global Step: 67040 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:40:12,994-Speed 130.22 samples/sec Loss 11.1395 Epoch: 6 Global Step: 67045 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:40:33,079-Speed 127.47 samples/sec Loss 10.9422 Epoch: 6 Global Step: 67050 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:40:53,051-Speed 128.18 samples/sec Loss 11.0800 Epoch: 6 Global Step: 67055 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:41:12,655-Speed 130.59 samples/sec Loss 10.7598 Epoch: 6 Global Step: 67060 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:41:32,655-Speed 128.01 samples/sec Loss 11.1275 Epoch: 6 Global Step: 67065 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:41:52,242-Speed 130.71 samples/sec Loss 11.2592 Epoch: 6 Global Step: 67070 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:42:12,236-Speed 128.04 samples/sec Loss 11.1173 Epoch: 6 Global Step: 67075 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:42:31,842-Speed 130.58 samples/sec Loss 11.0217 Epoch: 6 Global Step: 67080 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:42:51,632-Speed 129.37 samples/sec Loss 10.7888 Epoch: 6 Global Step: 67085 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:43:11,298-Speed 130.18 samples/sec Loss 10.9847 Epoch: 6 Global Step: 67090 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:43:31,455-Speed 127.01 samples/sec Loss 11.0637 Epoch: 6 Global Step: 67095 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:43:51,223-Speed 129.51 samples/sec Loss 10.8085 Epoch: 6 Global Step: 67100 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:44:10,750-Speed 131.10 samples/sec Loss 11.2168 Epoch: 6 Global Step: 67105 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:44:30,665-Speed 128.56 samples/sec Loss 10.8009 Epoch: 6 Global Step: 67110 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:44:50,256-Speed 130.68 samples/sec Loss 10.8637 Epoch: 6 Global Step: 67115 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:45:10,208-Speed 128.31 samples/sec Loss 11.1460 Epoch: 6 Global Step: 67120 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:45:29,819-Speed 130.55 samples/sec Loss 11.3581 Epoch: 6 Global Step: 67125 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:45:49,715-Speed 128.68 samples/sec Loss 10.6441 Epoch: 6 Global Step: 67130 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:46:09,552-Speed 129.05 samples/sec Loss 10.8150 Epoch: 6 Global Step: 67135 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:46:29,690-Speed 127.13 samples/sec Loss 11.3520 Epoch: 6 Global Step: 67140 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:46:49,353-Speed 130.20 samples/sec Loss 10.4628 Epoch: 6 Global Step: 67145 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:47:09,034-Speed 130.08 samples/sec Loss 10.8948 Epoch: 6 Global Step: 67150 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:47:28,687-Speed 130.27 samples/sec Loss 11.0359 Epoch: 6 Global Step: 67155 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:47:48,358-Speed 130.14 samples/sec Loss 10.8796 Epoch: 6 Global Step: 67160 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:48:08,206-Speed 128.99 samples/sec Loss 10.4652 Epoch: 6 Global Step: 67165 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:48:27,919-Speed 129.87 samples/sec Loss 11.0016 Epoch: 6 Global Step: 67170 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:48:47,869-Speed 128.33 samples/sec Loss 10.7272 Epoch: 6 Global Step: 67175 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:49:07,473-Speed 130.59 samples/sec Loss 10.9987 Epoch: 6 Global Step: 67180 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:49:27,709-Speed 126.51 samples/sec Loss 10.8179 Epoch: 6 Global Step: 67185 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:49:47,689-Speed 128.14 samples/sec Loss 11.2095 Epoch: 6 Global Step: 67190 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:50:07,301-Speed 130.54 samples/sec Loss 10.8619 Epoch: 6 Global Step: 67195 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:50:27,050-Speed 129.64 samples/sec Loss 10.8324 Epoch: 6 Global Step: 67200 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:50:46,712-Speed 130.20 samples/sec Loss 11.0595 Epoch: 6 Global Step: 67205 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:51:06,440-Speed 129.78 samples/sec Loss 10.8491 Epoch: 6 Global Step: 67210 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:51:26,098-Speed 130.23 samples/sec Loss 10.9011 Epoch: 6 Global Step: 67215 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:51:45,829-Speed 129.75 samples/sec Loss 11.1639 Epoch: 6 Global Step: 67220 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:52:05,422-Speed 130.66 samples/sec Loss 10.9834 Epoch: 6 Global Step: 67225 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:52:25,487-Speed 127.59 samples/sec Loss 10.9879 Epoch: 6 Global Step: 67230 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:52:45,544-Speed 127.64 samples/sec Loss 10.8975 Epoch: 6 Global Step: 67235 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:53:05,108-Speed 130.86 samples/sec Loss 10.6722 Epoch: 6 Global Step: 67240 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:53:25,237-Speed 127.19 samples/sec Loss 11.0043 Epoch: 6 Global Step: 67245 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:53:44,868-Speed 130.41 samples/sec Loss 10.9148 Epoch: 6 Global Step: 67250 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:54:04,916-Speed 127.71 samples/sec Loss 10.6472 Epoch: 6 Global Step: 67255 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:54:24,555-Speed 130.36 samples/sec Loss 10.8123 Epoch: 6 Global Step: 67260 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:54:44,363-Speed 129.25 samples/sec Loss 11.1170 Epoch: 6 Global Step: 67265 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:55:04,022-Speed 130.23 samples/sec Loss 11.0599 Epoch: 6 Global Step: 67270 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:55:24,185-Speed 126.97 samples/sec Loss 10.6524 Epoch: 6 Global Step: 67275 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:55:43,847-Speed 130.21 samples/sec Loss 11.0507 Epoch: 6 Global Step: 67280 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:56:03,453-Speed 130.58 samples/sec Loss 11.0219 Epoch: 6 Global Step: 67285 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:56:23,360-Speed 128.60 samples/sec Loss 11.0171 Epoch: 6 Global Step: 67290 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:56:42,949-Speed 130.70 samples/sec Loss 11.6875 Epoch: 6 Global Step: 67295 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:57:02,849-Speed 128.65 samples/sec Loss 10.9472 Epoch: 6 Global Step: 67300 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:57:22,335-Speed 131.39 samples/sec Loss 11.1288 Epoch: 6 Global Step: 67305 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:57:42,370-Speed 127.78 samples/sec Loss 10.9118 Epoch: 6 Global Step: 67310 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:58:01,905-Speed 131.05 samples/sec Loss 11.0112 Epoch: 6 Global Step: 67315 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:58:22,056-Speed 127.05 samples/sec Loss 11.2000 Epoch: 6 Global Step: 67320 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:58:42,602-Speed 124.60 samples/sec Loss 10.7509 Epoch: 6 Global Step: 67325 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:59:02,239-Speed 130.38 samples/sec Loss 11.1765 Epoch: 6 Global Step: 67330 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:59:22,014-Speed 129.46 samples/sec Loss 11.1837 Epoch: 6 Global Step: 67335 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 02:59:41,607-Speed 130.67 samples/sec Loss 11.0850 Epoch: 6 Global Step: 67340 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:00:01,382-Speed 129.47 samples/sec Loss 10.6296 Epoch: 6 Global Step: 67345 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:00:21,045-Speed 130.20 samples/sec Loss 11.1879 Epoch: 6 Global Step: 67350 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:00:40,899-Speed 128.95 samples/sec Loss 11.2822 Epoch: 6 Global Step: 67355 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:01:00,507-Speed 130.57 samples/sec Loss 11.0793 Epoch: 6 Global Step: 67360 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:01:20,596-Speed 127.44 samples/sec Loss 10.9724 Epoch: 6 Global Step: 67365 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:01:40,482-Speed 128.74 samples/sec Loss 10.9782 Epoch: 6 Global Step: 67370 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:02:00,112-Speed 130.42 samples/sec Loss 11.0698 Epoch: 6 Global Step: 67375 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:02:19,869-Speed 129.58 samples/sec Loss 10.9516 Epoch: 6 Global Step: 67380 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:02:39,513-Speed 130.32 samples/sec Loss 11.0359 Epoch: 6 Global Step: 67385 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:02:59,277-Speed 129.54 samples/sec Loss 10.7323 Epoch: 6 Global Step: 67390 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:03:18,932-Speed 130.25 samples/sec Loss 11.0422 Epoch: 6 Global Step: 67395 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:03:38,879-Speed 128.35 samples/sec Loss 11.0946 Epoch: 6 Global Step: 67400 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:03:58,491-Speed 130.54 samples/sec Loss 10.6020 Epoch: 6 Global Step: 67405 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:04:18,499-Speed 127.96 samples/sec Loss 10.6008 Epoch: 6 Global Step: 67410 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:04:38,467-Speed 128.21 samples/sec Loss 11.1584 Epoch: 6 Global Step: 67415 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:04:58,066-Speed 130.62 samples/sec Loss 10.6677 Epoch: 6 Global Step: 67420 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:05:17,735-Speed 130.16 samples/sec Loss 11.1519 Epoch: 6 Global Step: 67425 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:05:37,409-Speed 130.13 samples/sec Loss 11.0835 Epoch: 6 Global Step: 67430 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:05:57,288-Speed 128.78 samples/sec Loss 11.3423 Epoch: 6 Global Step: 67435 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:06:16,877-Speed 130.70 samples/sec Loss 11.0253 Epoch: 6 Global Step: 67440 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:06:36,701-Speed 129.14 samples/sec Loss 10.7429 Epoch: 6 Global Step: 67445 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:06:56,185-Speed 131.40 samples/sec Loss 11.1718 Epoch: 6 Global Step: 67450 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:07:16,212-Speed 127.83 samples/sec Loss 11.1070 Epoch: 6 Global Step: 67455 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:07:36,328-Speed 127.27 samples/sec Loss 10.7730 Epoch: 6 Global Step: 67460 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:07:55,913-Speed 130.72 samples/sec Loss 10.9585 Epoch: 6 Global Step: 67465 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:08:15,754-Speed 129.03 samples/sec Loss 10.8331 Epoch: 6 Global Step: 67470 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:08:35,351-Speed 130.64 samples/sec Loss 10.8487 Epoch: 6 Global Step: 67475 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:08:55,142-Speed 129.36 samples/sec Loss 10.9394 Epoch: 6 Global Step: 67480 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:09:14,781-Speed 130.36 samples/sec Loss 10.7923 Epoch: 6 Global Step: 67485 Fp16 Grad Scale: 8192 Required: 216 hours Training: 2021-06-10 03:09:34,436-Speed 130.26 samples/sec Loss 10.6228 Epoch: 6 Global Step: 67490 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:09:54,084-Speed 130.30 samples/sec Loss 10.8441 Epoch: 6 Global Step: 67495 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:10:14,171-Speed 127.45 samples/sec Loss 10.5253 Epoch: 6 Global Step: 67500 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:10:34,201-Speed 127.81 samples/sec Loss 10.5019 Epoch: 6 Global Step: 67505 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:10:53,729-Speed 131.10 samples/sec Loss 11.0230 Epoch: 6 Global Step: 67510 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:11:13,577-Speed 128.99 samples/sec Loss 11.2364 Epoch: 6 Global Step: 67515 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:11:33,233-Speed 130.25 samples/sec Loss 11.0368 Epoch: 6 Global Step: 67520 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:11:53,226-Speed 128.05 samples/sec Loss 11.0616 Epoch: 6 Global Step: 67525 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:12:12,826-Speed 130.63 samples/sec Loss 10.8000 Epoch: 6 Global Step: 67530 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:12:32,735-Speed 128.59 samples/sec Loss 10.7701 Epoch: 6 Global Step: 67535 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:12:52,347-Speed 130.54 samples/sec Loss 11.0662 Epoch: 6 Global Step: 67540 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:13:12,477-Speed 127.18 samples/sec Loss 10.8676 Epoch: 6 Global Step: 67545 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:13:32,269-Speed 129.35 samples/sec Loss 11.0382 Epoch: 6 Global Step: 67550 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:13:51,911-Speed 130.34 samples/sec Loss 10.9254 Epoch: 6 Global Step: 67555 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:14:11,702-Speed 129.36 samples/sec Loss 10.7384 Epoch: 6 Global Step: 67560 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:14:31,268-Speed 130.84 samples/sec Loss 10.9741 Epoch: 6 Global Step: 67565 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:14:51,213-Speed 128.36 samples/sec Loss 11.1543 Epoch: 6 Global Step: 67570 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:15:10,784-Speed 130.81 samples/sec Loss 10.8972 Epoch: 6 Global Step: 67575 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:15:30,645-Speed 128.90 samples/sec Loss 11.1449 Epoch: 6 Global Step: 67580 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:15:50,299-Speed 130.26 samples/sec Loss 10.8018 Epoch: 6 Global Step: 67585 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:16:10,379-Speed 127.50 samples/sec Loss 10.8408 Epoch: 6 Global Step: 67590 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:16:30,209-Speed 129.10 samples/sec Loss 11.1577 Epoch: 6 Global Step: 67595 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:16:49,858-Speed 130.29 samples/sec Loss 11.1132 Epoch: 6 Global Step: 67600 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:17:09,690-Speed 129.09 samples/sec Loss 10.7849 Epoch: 6 Global Step: 67605 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:17:29,283-Speed 130.66 samples/sec Loss 10.8247 Epoch: 6 Global Step: 67610 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:17:49,288-Speed 127.98 samples/sec Loss 11.1169 Epoch: 6 Global Step: 67615 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:18:08,778-Speed 131.35 samples/sec Loss 11.1643 Epoch: 6 Global Step: 67620 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:18:28,730-Speed 128.32 samples/sec Loss 10.8686 Epoch: 6 Global Step: 67625 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:18:48,397-Speed 130.17 samples/sec Loss 11.0435 Epoch: 6 Global Step: 67630 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:19:08,535-Speed 127.13 samples/sec Loss 10.8227 Epoch: 6 Global Step: 67635 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:19:28,366-Speed 129.10 samples/sec Loss 10.9435 Epoch: 6 Global Step: 67640 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:19:48,032-Speed 130.18 samples/sec Loss 11.1879 Epoch: 6 Global Step: 67645 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:20:08,007-Speed 128.17 samples/sec Loss 10.8278 Epoch: 6 Global Step: 67650 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:20:27,676-Speed 130.16 samples/sec Loss 10.8977 Epoch: 6 Global Step: 67655 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:20:47,325-Speed 130.29 samples/sec Loss 11.1498 Epoch: 6 Global Step: 67660 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:21:06,863-Speed 131.03 samples/sec Loss 11.0537 Epoch: 6 Global Step: 67665 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:21:26,847-Speed 128.11 samples/sec Loss 10.8850 Epoch: 6 Global Step: 67670 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:21:46,446-Speed 130.63 samples/sec Loss 10.6526 Epoch: 6 Global Step: 67675 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:22:06,564-Speed 127.26 samples/sec Loss 10.8911 Epoch: 6 Global Step: 67680 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:22:26,505-Speed 128.38 samples/sec Loss 11.2528 Epoch: 6 Global Step: 67685 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:22:46,109-Speed 130.59 samples/sec Loss 10.9633 Epoch: 6 Global Step: 67690 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:23:05,909-Speed 129.30 samples/sec Loss 10.6074 Epoch: 6 Global Step: 67695 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:23:25,529-Speed 130.49 samples/sec Loss 11.4353 Epoch: 6 Global Step: 67700 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:23:45,328-Speed 129.31 samples/sec Loss 11.0006 Epoch: 6 Global Step: 67705 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:24:04,992-Speed 130.19 samples/sec Loss 10.6355 Epoch: 6 Global Step: 67710 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:24:24,657-Speed 130.19 samples/sec Loss 10.6973 Epoch: 6 Global Step: 67715 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:24:44,289-Speed 130.41 samples/sec Loss 11.1028 Epoch: 6 Global Step: 67720 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:25:04,246-Speed 128.28 samples/sec Loss 10.7802 Epoch: 6 Global Step: 67725 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:25:24,283-Speed 127.77 samples/sec Loss 11.0704 Epoch: 6 Global Step: 67730 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:25:43,868-Speed 130.72 samples/sec Loss 10.6100 Epoch: 6 Global Step: 67735 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:26:03,799-Speed 128.45 samples/sec Loss 10.8332 Epoch: 6 Global Step: 67740 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:26:23,300-Speed 131.28 samples/sec Loss 10.8246 Epoch: 6 Global Step: 67745 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:26:43,176-Speed 128.80 samples/sec Loss 11.2310 Epoch: 6 Global Step: 67750 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:27:02,844-Speed 130.17 samples/sec Loss 10.9480 Epoch: 6 Global Step: 67755 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:27:22,626-Speed 129.42 samples/sec Loss 10.8340 Epoch: 6 Global Step: 67760 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:27:42,293-Speed 130.18 samples/sec Loss 11.4383 Epoch: 6 Global Step: 67765 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:28:02,270-Speed 128.15 samples/sec Loss 11.1124 Epoch: 6 Global Step: 67770 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:28:22,097-Speed 129.13 samples/sec Loss 10.9472 Epoch: 6 Global Step: 67775 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:28:41,737-Speed 130.35 samples/sec Loss 11.1620 Epoch: 6 Global Step: 67780 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:29:01,550-Speed 129.21 samples/sec Loss 10.9383 Epoch: 6 Global Step: 67785 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:29:21,136-Speed 130.71 samples/sec Loss 11.1317 Epoch: 6 Global Step: 67790 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:29:41,175-Speed 127.76 samples/sec Loss 11.1603 Epoch: 6 Global Step: 67795 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:30:00,773-Speed 130.63 samples/sec Loss 10.9655 Epoch: 6 Global Step: 67800 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:30:20,551-Speed 129.45 samples/sec Loss 10.9593 Epoch: 6 Global Step: 67805 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:30:40,159-Speed 130.56 samples/sec Loss 11.1401 Epoch: 6 Global Step: 67810 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:31:00,289-Speed 127.18 samples/sec Loss 10.8102 Epoch: 6 Global Step: 67815 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:31:20,074-Speed 129.40 samples/sec Loss 11.1407 Epoch: 6 Global Step: 67820 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:31:39,652-Speed 130.76 samples/sec Loss 11.3184 Epoch: 6 Global Step: 67825 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:31:59,463-Speed 129.23 samples/sec Loss 10.9406 Epoch: 6 Global Step: 67830 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:32:19,060-Speed 130.64 samples/sec Loss 10.9538 Epoch: 6 Global Step: 67835 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:32:39,021-Speed 128.25 samples/sec Loss 10.6226 Epoch: 6 Global Step: 67840 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:32:58,617-Speed 130.65 samples/sec Loss 10.7963 Epoch: 6 Global Step: 67845 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:33:18,322-Speed 129.92 samples/sec Loss 10.7883 Epoch: 6 Global Step: 67850 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:33:37,951-Speed 130.43 samples/sec Loss 10.8869 Epoch: 6 Global Step: 67855 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:33:58,192-Speed 126.48 samples/sec Loss 11.1601 Epoch: 6 Global Step: 67860 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:34:18,024-Speed 129.09 samples/sec Loss 11.0745 Epoch: 6 Global Step: 67865 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:34:37,799-Speed 129.46 samples/sec Loss 10.9932 Epoch: 6 Global Step: 67870 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:34:57,523-Speed 129.80 samples/sec Loss 10.9093 Epoch: 6 Global Step: 67875 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:35:17,177-Speed 130.26 samples/sec Loss 11.2412 Epoch: 6 Global Step: 67880 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:35:37,023-Speed 129.00 samples/sec Loss 10.9900 Epoch: 6 Global Step: 67885 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:35:56,663-Speed 130.35 samples/sec Loss 11.1588 Epoch: 6 Global Step: 67890 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:36:16,608-Speed 128.37 samples/sec Loss 11.0385 Epoch: 6 Global Step: 67895 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:36:36,279-Speed 130.15 samples/sec Loss 10.5631 Epoch: 6 Global Step: 67900 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:36:56,435-Speed 127.01 samples/sec Loss 10.7975 Epoch: 6 Global Step: 67905 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:37:16,359-Speed 128.50 samples/sec Loss 11.0310 Epoch: 6 Global Step: 67910 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:37:35,988-Speed 130.43 samples/sec Loss 10.9172 Epoch: 6 Global Step: 67915 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:37:55,818-Speed 129.10 samples/sec Loss 10.8551 Epoch: 6 Global Step: 67920 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:38:15,516-Speed 129.97 samples/sec Loss 11.2371 Epoch: 6 Global Step: 67925 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:38:35,310-Speed 129.34 samples/sec Loss 11.0433 Epoch: 6 Global Step: 67930 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:38:54,895-Speed 130.72 samples/sec Loss 10.9501 Epoch: 6 Global Step: 67935 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:39:14,700-Speed 129.27 samples/sec Loss 11.2988 Epoch: 6 Global Step: 67940 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:39:34,532-Speed 129.09 samples/sec Loss 10.8495 Epoch: 6 Global Step: 67945 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:39:54,545-Speed 127.92 samples/sec Loss 10.9131 Epoch: 6 Global Step: 67950 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:40:14,493-Speed 128.34 samples/sec Loss 11.1255 Epoch: 6 Global Step: 67955 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:40:34,065-Speed 130.81 samples/sec Loss 11.1810 Epoch: 6 Global Step: 67960 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:40:53,899-Speed 129.08 samples/sec Loss 11.0082 Epoch: 6 Global Step: 67965 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:41:13,539-Speed 130.36 samples/sec Loss 10.8827 Epoch: 6 Global Step: 67970 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:41:33,250-Speed 129.88 samples/sec Loss 10.9950 Epoch: 6 Global Step: 67975 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:41:52,880-Speed 130.42 samples/sec Loss 11.3497 Epoch: 6 Global Step: 67980 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:42:12,687-Speed 129.25 samples/sec Loss 10.9130 Epoch: 6 Global Step: 67985 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:42:32,301-Speed 130.53 samples/sec Loss 10.8535 Epoch: 6 Global Step: 67990 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:42:52,391-Speed 127.43 samples/sec Loss 11.0513 Epoch: 6 Global Step: 67995 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:43:12,238-Speed 128.99 samples/sec Loss 10.8052 Epoch: 6 Global Step: 68000 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:43:31,798-Speed 130.89 samples/sec Loss 10.5892 Epoch: 6 Global Step: 68005 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:43:51,713-Speed 128.55 samples/sec Loss 10.9825 Epoch: 6 Global Step: 68010 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:44:11,323-Speed 130.55 samples/sec Loss 11.1755 Epoch: 6 Global Step: 68015 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:44:31,024-Speed 129.95 samples/sec Loss 10.8297 Epoch: 6 Global Step: 68020 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:44:50,652-Speed 130.43 samples/sec Loss 10.7540 Epoch: 6 Global Step: 68025 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:45:10,437-Speed 129.40 samples/sec Loss 11.2094 Epoch: 6 Global Step: 68030 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:45:30,101-Speed 130.20 samples/sec Loss 11.1621 Epoch: 6 Global Step: 68035 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:45:50,100-Speed 128.01 samples/sec Loss 11.0884 Epoch: 6 Global Step: 68040 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:46:10,264-Speed 126.96 samples/sec Loss 11.1120 Epoch: 6 Global Step: 68045 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:46:29,893-Speed 130.42 samples/sec Loss 10.6185 Epoch: 6 Global Step: 68050 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:46:49,829-Speed 128.42 samples/sec Loss 10.9861 Epoch: 6 Global Step: 68055 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:47:09,435-Speed 130.58 samples/sec Loss 11.1497 Epoch: 6 Global Step: 68060 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:47:29,271-Speed 129.07 samples/sec Loss 11.1741 Epoch: 6 Global Step: 68065 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:47:48,927-Speed 130.25 samples/sec Loss 11.0793 Epoch: 6 Global Step: 68070 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:48:08,867-Speed 128.39 samples/sec Loss 10.9082 Epoch: 6 Global Step: 68075 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:48:28,450-Speed 130.74 samples/sec Loss 10.9902 Epoch: 6 Global Step: 68080 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:48:48,457-Speed 127.96 samples/sec Loss 10.7339 Epoch: 6 Global Step: 68085 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:49:08,215-Speed 129.57 samples/sec Loss 10.8482 Epoch: 6 Global Step: 68090 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:49:27,858-Speed 130.33 samples/sec Loss 11.0730 Epoch: 6 Global Step: 68095 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:49:47,664-Speed 129.26 samples/sec Loss 10.9327 Epoch: 6 Global Step: 68100 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:50:07,166-Speed 131.28 samples/sec Loss 10.6133 Epoch: 6 Global Step: 68105 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:50:27,088-Speed 128.51 samples/sec Loss 10.8872 Epoch: 6 Global Step: 68110 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:50:46,701-Speed 130.53 samples/sec Loss 10.3091 Epoch: 6 Global Step: 68115 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:51:06,628-Speed 128.47 samples/sec Loss 10.8722 Epoch: 6 Global Step: 68120 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:51:26,226-Speed 130.63 samples/sec Loss 11.2615 Epoch: 6 Global Step: 68125 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:51:46,246-Speed 127.88 samples/sec Loss 10.9003 Epoch: 6 Global Step: 68130 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:52:06,141-Speed 128.68 samples/sec Loss 11.2028 Epoch: 6 Global Step: 68135 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:52:25,844-Speed 129.93 samples/sec Loss 11.0858 Epoch: 6 Global Step: 68140 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:52:45,533-Speed 130.03 samples/sec Loss 10.8252 Epoch: 6 Global Step: 68145 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:53:05,360-Speed 129.13 samples/sec Loss 11.0637 Epoch: 6 Global Step: 68150 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:53:25,277-Speed 128.54 samples/sec Loss 10.8816 Epoch: 6 Global Step: 68155 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:53:44,797-Speed 131.16 samples/sec Loss 11.2340 Epoch: 6 Global Step: 68160 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:54:04,723-Speed 128.48 samples/sec Loss 11.2062 Epoch: 6 Global Step: 68165 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:54:24,204-Speed 131.42 samples/sec Loss 10.5765 Epoch: 6 Global Step: 68170 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:54:44,220-Speed 127.90 samples/sec Loss 10.9085 Epoch: 6 Global Step: 68175 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:55:04,209-Speed 128.08 samples/sec Loss 11.0178 Epoch: 6 Global Step: 68180 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:55:23,822-Speed 130.53 samples/sec Loss 11.3543 Epoch: 6 Global Step: 68185 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:55:43,482-Speed 130.22 samples/sec Loss 10.8610 Epoch: 6 Global Step: 68190 Fp16 Grad Scale: 8192 Required: 215 hours Training: 2021-06-10 03:56:03,164-Speed 130.08 samples/sec Loss 11.1702 Epoch: 6 Global Step: 68195 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:56:22,962-Speed 129.31 samples/sec Loss 11.1934 Epoch: 6 Global Step: 68200 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:56:42,721-Speed 129.57 samples/sec Loss 11.2299 Epoch: 6 Global Step: 68205 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:57:02,555-Speed 129.08 samples/sec Loss 10.7906 Epoch: 6 Global Step: 68210 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:57:22,069-Speed 131.19 samples/sec Loss 11.1892 Epoch: 6 Global Step: 68215 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:57:42,230-Speed 126.99 samples/sec Loss 11.3086 Epoch: 6 Global Step: 68220 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:58:02,303-Speed 127.54 samples/sec Loss 10.9698 Epoch: 6 Global Step: 68225 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:58:21,811-Speed 131.23 samples/sec Loss 11.2950 Epoch: 6 Global Step: 68230 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:58:41,755-Speed 128.37 samples/sec Loss 11.1356 Epoch: 6 Global Step: 68235 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:59:01,399-Speed 130.33 samples/sec Loss 10.9405 Epoch: 6 Global Step: 68240 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:59:21,119-Speed 129.83 samples/sec Loss 11.0930 Epoch: 6 Global Step: 68245 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 03:59:40,741-Speed 130.47 samples/sec Loss 10.5214 Epoch: 6 Global Step: 68250 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:00:00,398-Speed 130.24 samples/sec Loss 11.1761 Epoch: 6 Global Step: 68255 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:00:20,085-Speed 130.04 samples/sec Loss 11.0130 Epoch: 6 Global Step: 68260 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:00:40,147-Speed 127.61 samples/sec Loss 11.2684 Epoch: 6 Global Step: 68265 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:00:59,958-Speed 129.23 samples/sec Loss 11.0539 Epoch: 6 Global Step: 68270 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:01:19,525-Speed 130.84 samples/sec Loss 10.9306 Epoch: 6 Global Step: 68275 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:01:39,504-Speed 128.14 samples/sec Loss 10.6425 Epoch: 6 Global Step: 68280 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:01:59,101-Speed 130.64 samples/sec Loss 11.0075 Epoch: 6 Global Step: 68285 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:02:19,064-Speed 128.24 samples/sec Loss 11.1238 Epoch: 6 Global Step: 68290 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:02:38,713-Speed 130.29 samples/sec Loss 10.8020 Epoch: 6 Global Step: 68295 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:02:58,585-Speed 128.83 samples/sec Loss 11.0392 Epoch: 6 Global Step: 68300 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:03:18,228-Speed 130.34 samples/sec Loss 11.0207 Epoch: 6 Global Step: 68305 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:03:38,242-Speed 127.91 samples/sec Loss 11.2817 Epoch: 6 Global Step: 68310 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:03:57,917-Speed 130.12 samples/sec Loss 10.9796 Epoch: 6 Global Step: 68315 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:04:17,571-Speed 130.25 samples/sec Loss 10.9526 Epoch: 6 Global Step: 68320 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:04:37,494-Speed 128.50 samples/sec Loss 10.8035 Epoch: 6 Global Step: 68325 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:04:57,115-Speed 130.48 samples/sec Loss 11.2864 Epoch: 6 Global Step: 68330 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:05:16,949-Speed 129.08 samples/sec Loss 10.7751 Epoch: 6 Global Step: 68335 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:05:36,686-Speed 129.72 samples/sec Loss 11.0497 Epoch: 6 Global Step: 68340 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:05:56,521-Speed 129.07 samples/sec Loss 10.9616 Epoch: 6 Global Step: 68345 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:06:16,135-Speed 130.53 samples/sec Loss 10.9254 Epoch: 6 Global Step: 68350 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:06:36,255-Speed 127.24 samples/sec Loss 10.8910 Epoch: 6 Global Step: 68355 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:06:56,269-Speed 127.91 samples/sec Loss 10.9171 Epoch: 6 Global Step: 68360 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:07:15,996-Speed 129.78 samples/sec Loss 11.0994 Epoch: 6 Global Step: 68365 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:07:35,768-Speed 129.48 samples/sec Loss 11.1682 Epoch: 6 Global Step: 68370 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:07:55,386-Speed 130.50 samples/sec Loss 10.8768 Epoch: 6 Global Step: 68375 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:08:15,298-Speed 128.58 samples/sec Loss 11.0456 Epoch: 6 Global Step: 68380 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:08:34,952-Speed 130.26 samples/sec Loss 11.0052 Epoch: 6 Global Step: 68385 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:08:54,912-Speed 128.26 samples/sec Loss 10.8177 Epoch: 6 Global Step: 68390 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:09:14,493-Speed 130.75 samples/sec Loss 11.1397 Epoch: 6 Global Step: 68395 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:09:34,595-Speed 127.35 samples/sec Loss 11.2988 Epoch: 6 Global Step: 68400 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:09:54,431-Speed 129.07 samples/sec Loss 11.2854 Epoch: 6 Global Step: 68405 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:10:14,094-Speed 130.20 samples/sec Loss 11.0916 Epoch: 6 Global Step: 68410 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:10:33,850-Speed 129.59 samples/sec Loss 10.8878 Epoch: 6 Global Step: 68415 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:10:53,456-Speed 130.58 samples/sec Loss 11.0248 Epoch: 6 Global Step: 68420 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:11:13,358-Speed 128.64 samples/sec Loss 10.9879 Epoch: 6 Global Step: 68425 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:11:32,952-Speed 130.66 samples/sec Loss 11.0912 Epoch: 6 Global Step: 68430 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:11:52,926-Speed 128.17 samples/sec Loss 10.9823 Epoch: 6 Global Step: 68435 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:12:12,528-Speed 130.61 samples/sec Loss 10.9130 Epoch: 6 Global Step: 68440 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:12:32,662-Speed 127.15 samples/sec Loss 10.9367 Epoch: 6 Global Step: 68445 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:12:52,763-Speed 127.37 samples/sec Loss 10.8548 Epoch: 6 Global Step: 68450 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:13:12,337-Speed 130.79 samples/sec Loss 10.9696 Epoch: 6 Global Step: 68455 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:13:32,120-Speed 129.41 samples/sec Loss 11.1694 Epoch: 6 Global Step: 68460 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:13:51,752-Speed 130.40 samples/sec Loss 10.8792 Epoch: 6 Global Step: 68465 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:14:11,521-Speed 129.50 samples/sec Loss 10.8236 Epoch: 6 Global Step: 68470 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:14:31,732-Speed 126.67 samples/sec Loss 10.9885 Epoch: 6 Global Step: 68475 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:14:51,488-Speed 129.59 samples/sec Loss 10.9227 Epoch: 6 Global Step: 68480 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:15:11,073-Speed 130.72 samples/sec Loss 10.7025 Epoch: 6 Global Step: 68485 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:15:31,243-Speed 126.93 samples/sec Loss 10.8018 Epoch: 6 Global Step: 68490 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:15:51,237-Speed 128.05 samples/sec Loss 11.0014 Epoch: 6 Global Step: 68495 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:16:10,849-Speed 130.54 samples/sec Loss 11.0522 Epoch: 6 Global Step: 68500 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:16:30,787-Speed 128.41 samples/sec Loss 10.6874 Epoch: 6 Global Step: 68505 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:16:50,427-Speed 130.35 samples/sec Loss 10.6238 Epoch: 6 Global Step: 68510 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:17:10,222-Speed 129.33 samples/sec Loss 11.0387 Epoch: 6 Global Step: 68515 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:17:29,842-Speed 130.49 samples/sec Loss 10.9503 Epoch: 6 Global Step: 68520 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:17:49,551-Speed 129.89 samples/sec Loss 11.0926 Epoch: 6 Global Step: 68525 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:18:09,192-Speed 130.35 samples/sec Loss 10.8923 Epoch: 6 Global Step: 68530 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:18:29,135-Speed 128.37 samples/sec Loss 11.1602 Epoch: 6 Global Step: 68535 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:18:49,121-Speed 128.10 samples/sec Loss 10.8082 Epoch: 6 Global Step: 68540 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:19:08,724-Speed 130.60 samples/sec Loss 11.3341 Epoch: 6 Global Step: 68545 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:19:28,730-Speed 127.97 samples/sec Loss 10.6416 Epoch: 6 Global Step: 68550 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:19:48,225-Speed 131.32 samples/sec Loss 11.0953 Epoch: 6 Global Step: 68555 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:20:08,356-Speed 127.17 samples/sec Loss 11.1358 Epoch: 6 Global Step: 68560 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:20:28,082-Speed 129.79 samples/sec Loss 10.9205 Epoch: 6 Global Step: 68565 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:20:48,001-Speed 128.53 samples/sec Loss 11.0089 Epoch: 6 Global Step: 68570 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:21:07,645-Speed 130.33 samples/sec Loss 11.2206 Epoch: 6 Global Step: 68575 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:21:27,678-Speed 127.79 samples/sec Loss 10.5601 Epoch: 6 Global Step: 68580 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:21:47,488-Speed 129.24 samples/sec Loss 11.0264 Epoch: 6 Global Step: 68585 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:22:07,111-Speed 130.46 samples/sec Loss 10.8005 Epoch: 6 Global Step: 68590 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:22:27,030-Speed 128.53 samples/sec Loss 11.4927 Epoch: 6 Global Step: 68595 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:22:46,625-Speed 130.65 samples/sec Loss 11.1445 Epoch: 6 Global Step: 68600 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:23:06,639-Speed 127.92 samples/sec Loss 11.2347 Epoch: 6 Global Step: 68605 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:23:26,264-Speed 130.45 samples/sec Loss 10.7521 Epoch: 6 Global Step: 68610 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:23:46,293-Speed 127.82 samples/sec Loss 10.6811 Epoch: 6 Global Step: 68615 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:24:05,800-Speed 131.24 samples/sec Loss 11.0922 Epoch: 6 Global Step: 68620 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:24:25,975-Speed 126.89 samples/sec Loss 11.2238 Epoch: 6 Global Step: 68625 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:24:45,794-Speed 129.17 samples/sec Loss 10.6603 Epoch: 6 Global Step: 68630 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:25:05,423-Speed 130.43 samples/sec Loss 10.9590 Epoch: 6 Global Step: 68635 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:25:25,168-Speed 129.66 samples/sec Loss 10.7847 Epoch: 6 Global Step: 68640 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:25:44,806-Speed 130.37 samples/sec Loss 10.9242 Epoch: 6 Global Step: 68645 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:26:04,560-Speed 129.60 samples/sec Loss 10.7107 Epoch: 6 Global Step: 68650 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:26:24,099-Speed 131.03 samples/sec Loss 11.3108 Epoch: 6 Global Step: 68655 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:26:44,035-Speed 128.42 samples/sec Loss 10.9027 Epoch: 6 Global Step: 68660 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:27:03,628-Speed 130.66 samples/sec Loss 11.0279 Epoch: 6 Global Step: 68665 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:27:23,764-Speed 127.15 samples/sec Loss 10.4817 Epoch: 6 Global Step: 68670 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:27:44,479-Speed 123.59 samples/sec Loss 10.9694 Epoch: 6 Global Step: 68675 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:28:04,082-Speed 130.60 samples/sec Loss 11.0363 Epoch: 6 Global Step: 68680 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:28:23,859-Speed 129.45 samples/sec Loss 11.3073 Epoch: 6 Global Step: 68685 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:28:43,529-Speed 130.15 samples/sec Loss 10.9887 Epoch: 6 Global Step: 68690 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:29:03,230-Speed 129.95 samples/sec Loss 10.9802 Epoch: 6 Global Step: 68695 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:29:22,877-Speed 130.31 samples/sec Loss 11.1135 Epoch: 6 Global Step: 68700 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:29:42,653-Speed 129.45 samples/sec Loss 10.7955 Epoch: 6 Global Step: 68705 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:30:02,257-Speed 130.59 samples/sec Loss 10.9173 Epoch: 6 Global Step: 68710 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:30:22,349-Speed 127.42 samples/sec Loss 10.8553 Epoch: 6 Global Step: 68715 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:30:42,359-Speed 127.94 samples/sec Loss 10.9719 Epoch: 6 Global Step: 68720 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:31:01,896-Speed 131.03 samples/sec Loss 10.6874 Epoch: 6 Global Step: 68725 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:31:21,836-Speed 128.39 samples/sec Loss 11.0806 Epoch: 6 Global Step: 68730 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:31:41,424-Speed 130.70 samples/sec Loss 10.8855 Epoch: 6 Global Step: 68735 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:32:01,287-Speed 128.89 samples/sec Loss 10.9429 Epoch: 6 Global Step: 68740 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:32:20,963-Speed 130.12 samples/sec Loss 11.1431 Epoch: 6 Global Step: 68745 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:32:40,724-Speed 129.55 samples/sec Loss 10.8953 Epoch: 6 Global Step: 68750 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:33:00,377-Speed 130.27 samples/sec Loss 10.8453 Epoch: 6 Global Step: 68755 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:33:20,248-Speed 128.84 samples/sec Loss 10.8439 Epoch: 6 Global Step: 68760 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:33:40,298-Speed 127.68 samples/sec Loss 10.8684 Epoch: 6 Global Step: 68765 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:33:59,886-Speed 130.70 samples/sec Loss 11.1911 Epoch: 6 Global Step: 68770 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:34:19,868-Speed 128.12 samples/sec Loss 10.6525 Epoch: 6 Global Step: 68775 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:34:39,564-Speed 129.98 samples/sec Loss 10.8541 Epoch: 6 Global Step: 68780 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:34:59,495-Speed 128.45 samples/sec Loss 10.8279 Epoch: 6 Global Step: 68785 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:35:19,103-Speed 130.56 samples/sec Loss 11.0726 Epoch: 6 Global Step: 68790 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:35:39,047-Speed 128.37 samples/sec Loss 10.9900 Epoch: 6 Global Step: 68795 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:35:58,655-Speed 130.56 samples/sec Loss 11.1513 Epoch: 6 Global Step: 68800 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:36:18,858-Speed 126.72 samples/sec Loss 10.9840 Epoch: 6 Global Step: 68805 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:36:38,765-Speed 128.60 samples/sec Loss 10.7634 Epoch: 6 Global Step: 68810 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:36:58,344-Speed 130.76 samples/sec Loss 10.4510 Epoch: 6 Global Step: 68815 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:37:18,117-Speed 129.48 samples/sec Loss 11.3132 Epoch: 6 Global Step: 68820 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:37:37,788-Speed 130.14 samples/sec Loss 10.9874 Epoch: 6 Global Step: 68825 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:37:57,730-Speed 128.38 samples/sec Loss 11.1705 Epoch: 6 Global Step: 68830 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:38:17,319-Speed 130.69 samples/sec Loss 10.7910 Epoch: 6 Global Step: 68835 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:38:37,182-Speed 128.89 samples/sec Loss 10.5531 Epoch: 6 Global Step: 68840 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:38:56,831-Speed 130.30 samples/sec Loss 10.9172 Epoch: 6 Global Step: 68845 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:39:16,984-Speed 127.03 samples/sec Loss 11.0877 Epoch: 6 Global Step: 68850 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:39:36,747-Speed 129.55 samples/sec Loss 10.9030 Epoch: 6 Global Step: 68855 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:39:56,383-Speed 130.38 samples/sec Loss 11.0347 Epoch: 6 Global Step: 68860 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:40:16,273-Speed 128.72 samples/sec Loss 10.8076 Epoch: 6 Global Step: 68865 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:40:35,800-Speed 131.10 samples/sec Loss 10.8815 Epoch: 6 Global Step: 68870 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:40:55,855-Speed 127.66 samples/sec Loss 10.6664 Epoch: 6 Global Step: 68875 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:41:15,447-Speed 130.67 samples/sec Loss 10.9621 Epoch: 6 Global Step: 68880 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:41:35,394-Speed 128.35 samples/sec Loss 10.8742 Epoch: 6 Global Step: 68885 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:41:55,050-Speed 130.25 samples/sec Loss 10.8344 Epoch: 6 Global Step: 68890 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:42:15,099-Speed 127.69 samples/sec Loss 10.9142 Epoch: 6 Global Step: 68895 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:42:34,866-Speed 129.52 samples/sec Loss 11.1285 Epoch: 6 Global Step: 68900 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 04:42:54,532-Speed 130.18 samples/sec Loss 10.7256 Epoch: 6 Global Step: 68905 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:43:14,264-Speed 129.74 samples/sec Loss 10.6470 Epoch: 6 Global Step: 68910 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:43:33,855-Speed 130.68 samples/sec Loss 10.5485 Epoch: 6 Global Step: 68915 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:43:53,708-Speed 128.95 samples/sec Loss 10.7589 Epoch: 6 Global Step: 68920 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:44:13,308-Speed 130.62 samples/sec Loss 11.0180 Epoch: 6 Global Step: 68925 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:44:33,330-Speed 127.87 samples/sec Loss 10.6890 Epoch: 6 Global Step: 68930 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:44:53,069-Speed 129.70 samples/sec Loss 11.0747 Epoch: 6 Global Step: 68935 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:45:13,333-Speed 126.34 samples/sec Loss 10.9919 Epoch: 6 Global Step: 68940 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:45:33,227-Speed 128.69 samples/sec Loss 10.9810 Epoch: 6 Global Step: 68945 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:45:52,873-Speed 130.31 samples/sec Loss 11.0113 Epoch: 6 Global Step: 68950 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:46:12,561-Speed 130.04 samples/sec Loss 10.8615 Epoch: 6 Global Step: 68955 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:46:32,217-Speed 130.25 samples/sec Loss 10.8270 Epoch: 6 Global Step: 68960 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:46:51,995-Speed 129.45 samples/sec Loss 11.2416 Epoch: 6 Global Step: 68965 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:47:11,649-Speed 130.26 samples/sec Loss 10.9130 Epoch: 6 Global Step: 68970 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:47:31,326-Speed 130.11 samples/sec Loss 10.9028 Epoch: 6 Global Step: 68975 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:47:50,935-Speed 130.56 samples/sec Loss 10.8562 Epoch: 6 Global Step: 68980 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:48:10,982-Speed 127.71 samples/sec Loss 11.0929 Epoch: 6 Global Step: 68985 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:48:31,248-Speed 126.33 samples/sec Loss 11.0894 Epoch: 6 Global Step: 68990 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:48:50,720-Speed 131.48 samples/sec Loss 11.0119 Epoch: 6 Global Step: 68995 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:49:10,517-Speed 129.32 samples/sec Loss 11.0961 Epoch: 6 Global Step: 69000 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:49:30,135-Speed 130.50 samples/sec Loss 10.7768 Epoch: 6 Global Step: 69005 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:49:49,875-Speed 129.70 samples/sec Loss 10.8184 Epoch: 6 Global Step: 69010 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:50:09,536-Speed 130.21 samples/sec Loss 11.2576 Epoch: 6 Global Step: 69015 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:50:29,209-Speed 130.14 samples/sec Loss 10.8640 Epoch: 6 Global Step: 69020 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:50:48,812-Speed 130.60 samples/sec Loss 10.9357 Epoch: 6 Global Step: 69025 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:51:08,888-Speed 127.53 samples/sec Loss 10.7347 Epoch: 6 Global Step: 69030 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:51:29,002-Speed 127.28 samples/sec Loss 10.8834 Epoch: 6 Global Step: 69035 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:51:48,573-Speed 130.82 samples/sec Loss 10.9294 Epoch: 6 Global Step: 69040 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:52:08,378-Speed 129.27 samples/sec Loss 10.7367 Epoch: 6 Global Step: 69045 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:52:28,140-Speed 129.55 samples/sec Loss 11.0423 Epoch: 6 Global Step: 69050 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:52:47,961-Speed 129.16 samples/sec Loss 10.7632 Epoch: 6 Global Step: 69055 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:53:07,608-Speed 130.31 samples/sec Loss 10.8408 Epoch: 6 Global Step: 69060 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:53:27,354-Speed 129.65 samples/sec Loss 10.8266 Epoch: 6 Global Step: 69065 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:53:47,037-Speed 130.07 samples/sec Loss 10.9555 Epoch: 6 Global Step: 69070 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:54:07,103-Speed 127.59 samples/sec Loss 11.2289 Epoch: 6 Global Step: 69075 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:54:27,165-Speed 127.61 samples/sec Loss 10.7304 Epoch: 6 Global Step: 69080 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:54:46,656-Speed 131.34 samples/sec Loss 10.9580 Epoch: 6 Global Step: 69085 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:55:06,505-Speed 128.99 samples/sec Loss 10.8165 Epoch: 6 Global Step: 69090 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:55:26,162-Speed 130.24 samples/sec Loss 10.9713 Epoch: 6 Global Step: 69095 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:55:46,026-Speed 128.88 samples/sec Loss 10.9948 Epoch: 6 Global Step: 69100 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:56:05,647-Speed 130.48 samples/sec Loss 11.0172 Epoch: 6 Global Step: 69105 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:56:25,403-Speed 129.59 samples/sec Loss 10.8753 Epoch: 6 Global Step: 69110 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:56:45,101-Speed 129.97 samples/sec Loss 10.8753 Epoch: 6 Global Step: 69115 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:57:05,361-Speed 126.36 samples/sec Loss 10.9069 Epoch: 6 Global Step: 69120 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:57:25,300-Speed 128.40 samples/sec Loss 10.7534 Epoch: 6 Global Step: 69125 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:57:44,884-Speed 130.72 samples/sec Loss 11.0202 Epoch: 6 Global Step: 69130 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:58:04,834-Speed 128.33 samples/sec Loss 11.0732 Epoch: 6 Global Step: 69135 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:58:24,478-Speed 130.33 samples/sec Loss 10.6985 Epoch: 6 Global Step: 69140 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:58:44,401-Speed 128.50 samples/sec Loss 10.7716 Epoch: 6 Global Step: 69145 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:59:03,890-Speed 131.36 samples/sec Loss 10.7621 Epoch: 6 Global Step: 69150 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:59:23,688-Speed 129.31 samples/sec Loss 11.0355 Epoch: 6 Global Step: 69155 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 04:59:43,345-Speed 130.25 samples/sec Loss 11.0837 Epoch: 6 Global Step: 69160 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:00:03,374-Speed 127.81 samples/sec Loss 10.9105 Epoch: 6 Global Step: 69165 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:00:23,079-Speed 129.93 samples/sec Loss 11.2006 Epoch: 6 Global Step: 69170 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:00:42,732-Speed 130.27 samples/sec Loss 11.0985 Epoch: 6 Global Step: 69175 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:01:02,577-Speed 129.01 samples/sec Loss 11.1487 Epoch: 6 Global Step: 69180 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:01:22,108-Speed 131.08 samples/sec Loss 11.0728 Epoch: 6 Global Step: 69185 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:01:42,001-Speed 128.70 samples/sec Loss 10.9666 Epoch: 6 Global Step: 69190 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:02:01,589-Speed 130.70 samples/sec Loss 10.7289 Epoch: 6 Global Step: 69195 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:02:21,528-Speed 128.40 samples/sec Loss 11.0679 Epoch: 6 Global Step: 69200 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:02:41,186-Speed 130.23 samples/sec Loss 10.8025 Epoch: 6 Global Step: 69205 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:03:01,306-Speed 127.24 samples/sec Loss 10.8950 Epoch: 6 Global Step: 69210 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:03:21,111-Speed 129.27 samples/sec Loss 11.1479 Epoch: 6 Global Step: 69215 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:03:41,046-Speed 128.42 samples/sec Loss 10.9492 Epoch: 6 Global Step: 69220 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:04:00,739-Speed 130.00 samples/sec Loss 10.7456 Epoch: 6 Global Step: 69225 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:04:20,382-Speed 130.33 samples/sec Loss 11.2399 Epoch: 6 Global Step: 69230 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:04:40,049-Speed 130.18 samples/sec Loss 11.1782 Epoch: 6 Global Step: 69235 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:04:59,618-Speed 130.82 samples/sec Loss 10.9403 Epoch: 6 Global Step: 69240 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:05:19,666-Speed 127.70 samples/sec Loss 10.9763 Epoch: 6 Global Step: 69245 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:05:39,255-Speed 130.70 samples/sec Loss 10.7691 Epoch: 6 Global Step: 69250 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:05:59,271-Speed 127.90 samples/sec Loss 10.8175 Epoch: 6 Global Step: 69255 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:06:19,245-Speed 128.18 samples/sec Loss 11.0414 Epoch: 6 Global Step: 69260 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:06:38,884-Speed 130.35 samples/sec Loss 10.6213 Epoch: 6 Global Step: 69265 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:06:58,658-Speed 129.47 samples/sec Loss 10.5642 Epoch: 6 Global Step: 69270 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:07:18,300-Speed 130.34 samples/sec Loss 10.7229 Epoch: 6 Global Step: 69275 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:07:38,065-Speed 129.53 samples/sec Loss 10.8242 Epoch: 6 Global Step: 69280 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:07:57,690-Speed 130.45 samples/sec Loss 10.9420 Epoch: 6 Global Step: 69285 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:08:17,508-Speed 129.18 samples/sec Loss 10.6506 Epoch: 6 Global Step: 69290 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:08:37,134-Speed 130.45 samples/sec Loss 10.6869 Epoch: 6 Global Step: 69295 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:08:57,217-Speed 127.47 samples/sec Loss 10.5215 Epoch: 6 Global Step: 69300 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:09:17,265-Speed 127.70 samples/sec Loss 10.9248 Epoch: 6 Global Step: 69305 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:09:36,785-Speed 131.16 samples/sec Loss 10.8237 Epoch: 6 Global Step: 69310 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:09:56,532-Speed 129.65 samples/sec Loss 10.8146 Epoch: 6 Global Step: 69315 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:10:16,193-Speed 130.21 samples/sec Loss 11.1866 Epoch: 6 Global Step: 69320 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:10:35,925-Speed 129.74 samples/sec Loss 10.6171 Epoch: 6 Global Step: 69325 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:10:55,565-Speed 130.35 samples/sec Loss 10.6557 Epoch: 6 Global Step: 69330 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:11:15,744-Speed 126.88 samples/sec Loss 11.2770 Epoch: 6 Global Step: 69335 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:11:35,250-Speed 131.25 samples/sec Loss 10.8356 Epoch: 6 Global Step: 69340 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:11:55,374-Speed 127.21 samples/sec Loss 11.3662 Epoch: 6 Global Step: 69345 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:12:15,376-Speed 128.00 samples/sec Loss 11.1116 Epoch: 6 Global Step: 69350 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:12:34,961-Speed 130.71 samples/sec Loss 11.3513 Epoch: 6 Global Step: 69355 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:12:54,697-Speed 129.72 samples/sec Loss 10.5016 Epoch: 6 Global Step: 69360 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:13:14,293-Speed 130.65 samples/sec Loss 10.8630 Epoch: 6 Global Step: 69365 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:13:34,142-Speed 128.98 samples/sec Loss 10.6864 Epoch: 6 Global Step: 69370 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:13:53,749-Speed 130.58 samples/sec Loss 10.8169 Epoch: 6 Global Step: 69375 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:14:13,407-Speed 130.24 samples/sec Loss 11.0495 Epoch: 6 Global Step: 69380 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:14:33,560-Speed 127.04 samples/sec Loss 10.9832 Epoch: 6 Global Step: 69385 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:14:53,591-Speed 127.81 samples/sec Loss 10.9502 Epoch: 6 Global Step: 69390 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:15:13,487-Speed 128.67 samples/sec Loss 10.9518 Epoch: 6 Global Step: 69395 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:15:32,959-Speed 131.48 samples/sec Loss 11.0688 Epoch: 6 Global Step: 69400 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:15:52,904-Speed 128.36 samples/sec Loss 10.6173 Epoch: 6 Global Step: 69405 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:16:12,558-Speed 130.26 samples/sec Loss 10.6934 Epoch: 6 Global Step: 69410 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:16:32,371-Speed 129.21 samples/sec Loss 10.9453 Epoch: 6 Global Step: 69415 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:16:51,950-Speed 130.76 samples/sec Loss 11.1301 Epoch: 6 Global Step: 69420 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:17:11,636-Speed 130.05 samples/sec Loss 11.0541 Epoch: 6 Global Step: 69425 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:17:31,260-Speed 130.46 samples/sec Loss 11.0538 Epoch: 6 Global Step: 69430 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:17:51,335-Speed 127.53 samples/sec Loss 10.8513 Epoch: 6 Global Step: 69435 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:18:11,093-Speed 129.57 samples/sec Loss 11.0630 Epoch: 6 Global Step: 69440 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:18:30,662-Speed 130.82 samples/sec Loss 11.1136 Epoch: 6 Global Step: 69445 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:18:50,650-Speed 128.09 samples/sec Loss 11.1469 Epoch: 6 Global Step: 69450 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:19:10,212-Speed 130.87 samples/sec Loss 10.8357 Epoch: 6 Global Step: 69455 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:19:30,052-Speed 129.04 samples/sec Loss 10.9274 Epoch: 6 Global Step: 69460 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:19:49,554-Speed 131.28 samples/sec Loss 10.8134 Epoch: 6 Global Step: 69465 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:20:09,375-Speed 129.16 samples/sec Loss 11.1436 Epoch: 6 Global Step: 69470 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:20:29,038-Speed 130.21 samples/sec Loss 11.1590 Epoch: 6 Global Step: 69475 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:20:49,344-Speed 126.07 samples/sec Loss 11.1976 Epoch: 6 Global Step: 69480 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:21:09,031-Speed 130.04 samples/sec Loss 10.9421 Epoch: 6 Global Step: 69485 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:21:28,677-Speed 130.31 samples/sec Loss 10.5585 Epoch: 6 Global Step: 69490 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:21:48,612-Speed 128.42 samples/sec Loss 10.7331 Epoch: 6 Global Step: 69495 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:22:08,094-Speed 131.42 samples/sec Loss 11.0262 Epoch: 6 Global Step: 69500 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:22:27,913-Speed 129.18 samples/sec Loss 10.8794 Epoch: 6 Global Step: 69505 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:22:47,503-Speed 130.68 samples/sec Loss 11.3773 Epoch: 6 Global Step: 69510 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:23:07,500-Speed 128.02 samples/sec Loss 10.9348 Epoch: 6 Global Step: 69515 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:23:27,069-Speed 130.83 samples/sec Loss 10.9001 Epoch: 6 Global Step: 69520 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:23:47,098-Speed 127.82 samples/sec Loss 10.8873 Epoch: 6 Global Step: 69525 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:24:06,775-Speed 130.11 samples/sec Loss 11.0632 Epoch: 6 Global Step: 69530 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:24:26,475-Speed 129.96 samples/sec Loss 10.9847 Epoch: 6 Global Step: 69535 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:24:46,148-Speed 130.13 samples/sec Loss 10.9189 Epoch: 6 Global Step: 69540 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:25:05,660-Speed 131.21 samples/sec Loss 10.5303 Epoch: 6 Global Step: 69545 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:25:25,502-Speed 129.03 samples/sec Loss 10.7896 Epoch: 6 Global Step: 69550 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:25:45,156-Speed 130.27 samples/sec Loss 11.1092 Epoch: 6 Global Step: 69555 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:26:05,099-Speed 128.37 samples/sec Loss 10.9658 Epoch: 6 Global Step: 69560 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:26:24,584-Speed 131.39 samples/sec Loss 10.8934 Epoch: 6 Global Step: 69565 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:26:44,596-Speed 127.93 samples/sec Loss 10.8367 Epoch: 6 Global Step: 69570 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:27:04,583-Speed 128.09 samples/sec Loss 11.0956 Epoch: 6 Global Step: 69575 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:27:24,204-Speed 130.47 samples/sec Loss 10.9738 Epoch: 6 Global Step: 69580 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:27:43,863-Speed 130.23 samples/sec Loss 10.7754 Epoch: 6 Global Step: 69585 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:28:03,514-Speed 130.28 samples/sec Loss 11.1478 Epoch: 6 Global Step: 69590 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:28:23,209-Speed 129.99 samples/sec Loss 11.1688 Epoch: 6 Global Step: 69595 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:28:42,756-Speed 130.98 samples/sec Loss 11.0964 Epoch: 6 Global Step: 69600 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:29:02,421-Speed 130.19 samples/sec Loss 10.6629 Epoch: 6 Global Step: 69605 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:29:21,946-Speed 131.12 samples/sec Loss 10.9223 Epoch: 6 Global Step: 69610 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 05:29:42,034-Speed 127.45 samples/sec Loss 10.7120 Epoch: 6 Global Step: 69615 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:30:02,181-Speed 127.08 samples/sec Loss 11.0682 Epoch: 6 Global Step: 69620 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:30:21,689-Speed 131.23 samples/sec Loss 10.7557 Epoch: 6 Global Step: 69625 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:30:41,532-Speed 129.03 samples/sec Loss 10.6222 Epoch: 6 Global Step: 69630 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:31:01,081-Speed 130.96 samples/sec Loss 11.2039 Epoch: 6 Global Step: 69635 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:31:20,962-Speed 128.77 samples/sec Loss 11.1328 Epoch: 6 Global Step: 69640 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:31:40,585-Speed 130.47 samples/sec Loss 10.6980 Epoch: 6 Global Step: 69645 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:32:00,250-Speed 130.19 samples/sec Loss 10.7023 Epoch: 6 Global Step: 69650 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:32:19,899-Speed 130.30 samples/sec Loss 10.9373 Epoch: 6 Global Step: 69655 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:32:39,825-Speed 128.48 samples/sec Loss 11.4131 Epoch: 6 Global Step: 69660 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:32:59,746-Speed 128.51 samples/sec Loss 11.2418 Epoch: 6 Global Step: 69665 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:33:19,215-Speed 131.49 samples/sec Loss 10.9739 Epoch: 6 Global Step: 69670 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:33:39,189-Speed 128.18 samples/sec Loss 10.9820 Epoch: 6 Global Step: 69675 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:33:58,854-Speed 130.19 samples/sec Loss 10.9804 Epoch: 6 Global Step: 69680 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:34:18,771-Speed 128.54 samples/sec Loss 10.7452 Epoch: 6 Global Step: 69685 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:34:38,254-Speed 131.40 samples/sec Loss 11.2992 Epoch: 6 Global Step: 69690 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:34:58,099-Speed 129.01 samples/sec Loss 10.8639 Epoch: 6 Global Step: 69695 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:35:17,719-Speed 130.49 samples/sec Loss 10.9338 Epoch: 6 Global Step: 69700 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:35:37,910-Speed 126.80 samples/sec Loss 10.7351 Epoch: 6 Global Step: 69705 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:35:57,577-Speed 130.17 samples/sec Loss 11.1536 Epoch: 6 Global Step: 69710 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:36:17,162-Speed 130.72 samples/sec Loss 10.9188 Epoch: 6 Global Step: 69715 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:36:36,956-Speed 129.34 samples/sec Loss 11.2280 Epoch: 6 Global Step: 69720 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:36:56,751-Speed 129.33 samples/sec Loss 11.0630 Epoch: 6 Global Step: 69725 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:37:16,580-Speed 129.11 samples/sec Loss 10.9048 Epoch: 6 Global Step: 69730 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:37:36,153-Speed 130.80 samples/sec Loss 11.0135 Epoch: 6 Global Step: 69735 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:37:55,801-Speed 130.30 samples/sec Loss 10.7748 Epoch: 6 Global Step: 69740 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:38:15,386-Speed 130.72 samples/sec Loss 10.4586 Epoch: 6 Global Step: 69745 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:39:02,576-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-10 05:39:17,620-Speed 41.14 samples/sec Loss 10.8445 Epoch: 7 Global Step: 69750 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:39:38,020-Speed 125.49 samples/sec Loss 10.2655 Epoch: 7 Global Step: 69755 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:39:58,229-Speed 126.69 samples/sec Loss 10.2431 Epoch: 7 Global Step: 69760 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:40:18,541-Speed 126.04 samples/sec Loss 10.3835 Epoch: 7 Global Step: 69765 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:40:38,469-Speed 128.47 samples/sec Loss 9.9138 Epoch: 7 Global Step: 69770 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:40:58,319-Speed 128.97 samples/sec Loss 10.1150 Epoch: 7 Global Step: 69775 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:41:18,206-Speed 128.74 samples/sec Loss 10.5628 Epoch: 7 Global Step: 69780 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:41:37,990-Speed 129.40 samples/sec Loss 10.2108 Epoch: 7 Global Step: 69785 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:41:57,827-Speed 129.06 samples/sec Loss 10.1929 Epoch: 7 Global Step: 69790 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:42:17,939-Speed 127.29 samples/sec Loss 10.2723 Epoch: 7 Global Step: 69795 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:42:37,916-Speed 128.15 samples/sec Loss 10.5054 Epoch: 7 Global Step: 69800 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:42:58,026-Speed 127.31 samples/sec Loss 10.4584 Epoch: 7 Global Step: 69805 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:43:17,981-Speed 128.29 samples/sec Loss 10.2506 Epoch: 7 Global Step: 69810 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:43:37,832-Speed 128.97 samples/sec Loss 10.5479 Epoch: 7 Global Step: 69815 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:43:57,632-Speed 129.30 samples/sec Loss 10.2538 Epoch: 7 Global Step: 69820 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:44:17,552-Speed 128.52 samples/sec Loss 10.1571 Epoch: 7 Global Step: 69825 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:44:37,263-Speed 129.89 samples/sec Loss 10.0923 Epoch: 7 Global Step: 69830 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:44:57,305-Speed 127.74 samples/sec Loss 10.4602 Epoch: 7 Global Step: 69835 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:45:17,093-Speed 129.38 samples/sec Loss 10.2751 Epoch: 7 Global Step: 69840 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:45:37,097-Speed 127.98 samples/sec Loss 10.5834 Epoch: 7 Global Step: 69845 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:45:57,550-Speed 125.17 samples/sec Loss 10.5261 Epoch: 7 Global Step: 69850 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:46:17,456-Speed 128.61 samples/sec Loss 10.3133 Epoch: 7 Global Step: 69855 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:46:37,135-Speed 130.09 samples/sec Loss 10.0867 Epoch: 7 Global Step: 69860 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:46:57,082-Speed 128.35 samples/sec Loss 10.5241 Epoch: 7 Global Step: 69865 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:47:16,763-Speed 130.08 samples/sec Loss 10.4259 Epoch: 7 Global Step: 69870 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:47:36,648-Speed 128.74 samples/sec Loss 10.3238 Epoch: 7 Global Step: 69875 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:47:56,614-Speed 128.22 samples/sec Loss 10.8417 Epoch: 7 Global Step: 69880 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:48:16,398-Speed 129.41 samples/sec Loss 10.7855 Epoch: 7 Global Step: 69885 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:48:36,553-Speed 127.02 samples/sec Loss 10.3755 Epoch: 7 Global Step: 69890 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:48:56,679-Speed 127.20 samples/sec Loss 10.4751 Epoch: 7 Global Step: 69895 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:49:16,483-Speed 129.27 samples/sec Loss 10.5373 Epoch: 7 Global Step: 69900 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:49:36,295-Speed 129.23 samples/sec Loss 10.5690 Epoch: 7 Global Step: 69905 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:49:56,100-Speed 129.27 samples/sec Loss 10.1577 Epoch: 7 Global Step: 69910 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:50:15,917-Speed 129.19 samples/sec Loss 10.6256 Epoch: 7 Global Step: 69915 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:50:35,893-Speed 128.16 samples/sec Loss 10.8314 Epoch: 7 Global Step: 69920 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:50:55,591-Speed 129.97 samples/sec Loss 10.5898 Epoch: 7 Global Step: 69925 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:51:15,634-Speed 127.73 samples/sec Loss 10.7737 Epoch: 7 Global Step: 69930 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:51:35,821-Speed 126.82 samples/sec Loss 10.3395 Epoch: 7 Global Step: 69935 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:51:55,988-Speed 126.95 samples/sec Loss 10.7142 Epoch: 7 Global Step: 69940 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:52:15,649-Speed 130.22 samples/sec Loss 10.3038 Epoch: 7 Global Step: 69945 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:52:35,494-Speed 129.00 samples/sec Loss 10.0882 Epoch: 7 Global Step: 69950 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:52:55,306-Speed 129.22 samples/sec Loss 10.2814 Epoch: 7 Global Step: 69955 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:53:15,131-Speed 129.14 samples/sec Loss 10.1674 Epoch: 7 Global Step: 69960 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:53:34,962-Speed 129.10 samples/sec Loss 10.3420 Epoch: 7 Global Step: 69965 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:53:54,761-Speed 129.31 samples/sec Loss 10.4511 Epoch: 7 Global Step: 69970 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:54:14,590-Speed 129.11 samples/sec Loss 10.6038 Epoch: 7 Global Step: 69975 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:54:34,841-Speed 126.42 samples/sec Loss 10.2891 Epoch: 7 Global Step: 69980 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:54:54,905-Speed 127.60 samples/sec Loss 10.4558 Epoch: 7 Global Step: 69985 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:55:14,887-Speed 128.12 samples/sec Loss 10.7088 Epoch: 7 Global Step: 69990 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:55:34,708-Speed 129.16 samples/sec Loss 10.1742 Epoch: 7 Global Step: 69995 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 05:55:54,517-Speed 129.24 samples/sec Loss 10.5497 Epoch: 7 Global Step: 70000 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 06:10:44,137-[lfw][70000]XNorm: 23.095985 Training: 2021-06-10 06:10:44,137-[lfw][70000]Accuracy-Flip: 0.99750+-0.00214 Training: 2021-06-10 06:10:44,137-[lfw][70000]Accuracy-Highest: 0.99750 Training: 2021-06-10 06:27:59,668-[cfp_fp][70000]XNorm: 20.673745 Training: 2021-06-10 06:27:59,668-[cfp_fp][70000]Accuracy-Flip: 0.97429+-0.00670 Training: 2021-06-10 06:27:59,668-[cfp_fp][70000]Accuracy-Highest: 0.97514 Training: 2021-06-10 06:42:49,179-[agedb_30][70000]XNorm: 22.728381 Training: 2021-06-10 06:42:49,179-[agedb_30][70000]Accuracy-Flip: 0.97183+-0.00905 Training: 2021-06-10 06:42:49,179-[agedb_30][70000]Accuracy-Highest: 0.97183 Training: 2021-06-10 06:43:09,299-Speed 0.90 samples/sec Loss 10.3091 Epoch: 7 Global Step: 70005 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:43:29,079-Speed 129.43 samples/sec Loss 10.6182 Epoch: 7 Global Step: 70010 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:43:48,695-Speed 130.51 samples/sec Loss 10.1767 Epoch: 7 Global Step: 70015 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:44:08,735-Speed 127.75 samples/sec Loss 10.1659 Epoch: 7 Global Step: 70020 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:44:28,598-Speed 128.89 samples/sec Loss 10.7208 Epoch: 7 Global Step: 70025 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:44:48,308-Speed 129.89 samples/sec Loss 10.0489 Epoch: 7 Global Step: 70030 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:45:08,168-Speed 128.91 samples/sec Loss 10.5247 Epoch: 7 Global Step: 70035 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:45:27,788-Speed 130.48 samples/sec Loss 10.6092 Epoch: 7 Global Step: 70040 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:45:47,524-Speed 129.72 samples/sec Loss 10.8878 Epoch: 7 Global Step: 70045 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:46:07,183-Speed 130.23 samples/sec Loss 10.3726 Epoch: 7 Global Step: 70050 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:46:27,066-Speed 128.76 samples/sec Loss 10.6505 Epoch: 7 Global Step: 70055 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:46:46,735-Speed 130.17 samples/sec Loss 10.8004 Epoch: 7 Global Step: 70060 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:47:06,600-Speed 128.87 samples/sec Loss 10.4220 Epoch: 7 Global Step: 70065 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:47:26,467-Speed 128.87 samples/sec Loss 10.5957 Epoch: 7 Global Step: 70070 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:47:46,074-Speed 130.57 samples/sec Loss 10.6787 Epoch: 7 Global Step: 70075 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:48:06,086-Speed 127.93 samples/sec Loss 10.3505 Epoch: 7 Global Step: 70080 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:48:25,687-Speed 130.61 samples/sec Loss 10.5913 Epoch: 7 Global Step: 70085 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:48:45,438-Speed 129.62 samples/sec Loss 10.8807 Epoch: 7 Global Step: 70090 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:49:05,029-Speed 130.68 samples/sec Loss 10.5827 Epoch: 7 Global Step: 70095 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:49:24,801-Speed 129.49 samples/sec Loss 10.8383 Epoch: 7 Global Step: 70100 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:49:44,476-Speed 130.12 samples/sec Loss 10.4728 Epoch: 7 Global Step: 70105 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:50:04,424-Speed 128.34 samples/sec Loss 10.3400 Epoch: 7 Global Step: 70110 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:50:24,325-Speed 128.64 samples/sec Loss 10.3041 Epoch: 7 Global Step: 70115 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:50:43,925-Speed 130.62 samples/sec Loss 10.6081 Epoch: 7 Global Step: 70120 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:51:03,950-Speed 127.85 samples/sec Loss 10.2481 Epoch: 7 Global Step: 70125 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:51:23,442-Speed 131.35 samples/sec Loss 10.5812 Epoch: 7 Global Step: 70130 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:51:43,383-Speed 128.39 samples/sec Loss 10.7989 Epoch: 7 Global Step: 70135 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:52:02,988-Speed 130.58 samples/sec Loss 10.4041 Epoch: 7 Global Step: 70140 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:52:23,025-Speed 127.77 samples/sec Loss 10.6563 Epoch: 7 Global Step: 70145 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:52:42,616-Speed 130.68 samples/sec Loss 10.6510 Epoch: 7 Global Step: 70150 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:53:02,638-Speed 127.87 samples/sec Loss 10.7536 Epoch: 7 Global Step: 70155 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:53:22,391-Speed 129.61 samples/sec Loss 10.6280 Epoch: 7 Global Step: 70160 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:53:42,043-Speed 130.27 samples/sec Loss 10.7806 Epoch: 7 Global Step: 70165 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:54:01,721-Speed 130.10 samples/sec Loss 10.9208 Epoch: 7 Global Step: 70170 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:54:21,246-Speed 131.12 samples/sec Loss 10.5558 Epoch: 7 Global Step: 70175 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:54:41,143-Speed 128.67 samples/sec Loss 10.6032 Epoch: 7 Global Step: 70180 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:55:00,734-Speed 130.68 samples/sec Loss 10.7760 Epoch: 7 Global Step: 70185 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:55:20,832-Speed 127.39 samples/sec Loss 10.4445 Epoch: 7 Global Step: 70190 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:55:40,414-Speed 130.74 samples/sec Loss 10.6471 Epoch: 7 Global Step: 70195 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:56:00,625-Speed 126.67 samples/sec Loss 10.6073 Epoch: 7 Global Step: 70200 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:56:20,767-Speed 127.10 samples/sec Loss 10.4860 Epoch: 7 Global Step: 70205 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:56:40,344-Speed 130.77 samples/sec Loss 10.7718 Epoch: 7 Global Step: 70210 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:57:00,034-Speed 130.02 samples/sec Loss 10.7327 Epoch: 7 Global Step: 70215 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:57:19,643-Speed 130.57 samples/sec Loss 10.7503 Epoch: 7 Global Step: 70220 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:57:39,545-Speed 128.64 samples/sec Loss 10.2993 Epoch: 7 Global Step: 70225 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:57:59,185-Speed 130.35 samples/sec Loss 10.6764 Epoch: 7 Global Step: 70230 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:58:18,858-Speed 130.13 samples/sec Loss 10.8845 Epoch: 7 Global Step: 70235 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:58:38,444-Speed 130.72 samples/sec Loss 10.8532 Epoch: 7 Global Step: 70240 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:58:58,670-Speed 126.57 samples/sec Loss 10.7702 Epoch: 7 Global Step: 70245 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:59:18,789-Speed 127.25 samples/sec Loss 10.3780 Epoch: 7 Global Step: 70250 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:59:38,261-Speed 131.48 samples/sec Loss 10.9441 Epoch: 7 Global Step: 70255 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 06:59:58,094-Speed 129.08 samples/sec Loss 10.5232 Epoch: 7 Global Step: 70260 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:00:17,699-Speed 130.58 samples/sec Loss 10.6039 Epoch: 7 Global Step: 70265 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:00:37,517-Speed 129.19 samples/sec Loss 10.8350 Epoch: 7 Global Step: 70270 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:00:57,180-Speed 130.20 samples/sec Loss 10.5680 Epoch: 7 Global Step: 70275 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:01:16,825-Speed 130.32 samples/sec Loss 10.7194 Epoch: 7 Global Step: 70280 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:01:36,463-Speed 130.37 samples/sec Loss 10.8171 Epoch: 7 Global Step: 70285 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:01:56,488-Speed 127.84 samples/sec Loss 10.5888 Epoch: 7 Global Step: 70290 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:02:16,449-Speed 128.26 samples/sec Loss 10.9280 Epoch: 7 Global Step: 70295 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:02:35,968-Speed 131.16 samples/sec Loss 10.6217 Epoch: 7 Global Step: 70300 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:02:55,910-Speed 128.38 samples/sec Loss 10.7495 Epoch: 7 Global Step: 70305 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:03:15,512-Speed 130.60 samples/sec Loss 11.0821 Epoch: 7 Global Step: 70310 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:03:35,531-Speed 127.89 samples/sec Loss 10.9068 Epoch: 7 Global Step: 70315 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:03:55,040-Speed 131.23 samples/sec Loss 10.8302 Epoch: 7 Global Step: 70320 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:04:14,754-Speed 129.86 samples/sec Loss 10.6404 Epoch: 7 Global Step: 70325 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:04:34,395-Speed 130.35 samples/sec Loss 10.7672 Epoch: 7 Global Step: 70330 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:04:54,491-Speed 127.40 samples/sec Loss 10.9731 Epoch: 7 Global Step: 70335 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:05:14,143-Speed 130.27 samples/sec Loss 10.7100 Epoch: 7 Global Step: 70340 Fp16 Grad Scale: 8192 Required: 214 hours Training: 2021-06-10 07:05:33,708-Speed 130.85 samples/sec Loss 10.9010 Epoch: 7 Global Step: 70345 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:05:53,582-Speed 128.82 samples/sec Loss 10.3884 Epoch: 7 Global Step: 70350 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:06:13,259-Speed 130.10 samples/sec Loss 10.5917 Epoch: 7 Global Step: 70355 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:06:33,185-Speed 128.49 samples/sec Loss 10.4413 Epoch: 7 Global Step: 70360 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:06:52,666-Speed 131.42 samples/sec Loss 10.4329 Epoch: 7 Global Step: 70365 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:07:12,634-Speed 128.21 samples/sec Loss 10.5907 Epoch: 7 Global Step: 70370 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:07:32,236-Speed 130.61 samples/sec Loss 11.0594 Epoch: 7 Global Step: 70375 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:07:52,467-Speed 126.55 samples/sec Loss 10.9231 Epoch: 7 Global Step: 70380 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:08:12,119-Speed 130.27 samples/sec Loss 10.3599 Epoch: 7 Global Step: 70385 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:08:31,767-Speed 130.30 samples/sec Loss 10.7909 Epoch: 7 Global Step: 70390 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:08:51,538-Speed 129.49 samples/sec Loss 10.7478 Epoch: 7 Global Step: 70395 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:09:11,200-Speed 130.21 samples/sec Loss 10.7738 Epoch: 7 Global Step: 70400 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:09:30,996-Speed 129.33 samples/sec Loss 10.5748 Epoch: 7 Global Step: 70405 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:09:50,481-Speed 131.39 samples/sec Loss 10.7446 Epoch: 7 Global Step: 70410 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:10:10,420-Speed 128.40 samples/sec Loss 10.5268 Epoch: 7 Global Step: 70415 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:10:29,970-Speed 130.95 samples/sec Loss 10.5986 Epoch: 7 Global Step: 70420 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:10:50,174-Speed 126.71 samples/sec Loss 10.8957 Epoch: 7 Global Step: 70425 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:11:09,828-Speed 130.26 samples/sec Loss 10.6434 Epoch: 7 Global Step: 70430 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:11:29,488-Speed 130.22 samples/sec Loss 11.0215 Epoch: 7 Global Step: 70435 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:11:49,324-Speed 129.07 samples/sec Loss 10.8470 Epoch: 7 Global Step: 70440 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:12:08,912-Speed 130.70 samples/sec Loss 10.8667 Epoch: 7 Global Step: 70445 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:12:28,674-Speed 129.55 samples/sec Loss 10.8734 Epoch: 7 Global Step: 70450 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:12:48,263-Speed 130.69 samples/sec Loss 10.8159 Epoch: 7 Global Step: 70455 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:13:08,202-Speed 128.40 samples/sec Loss 10.6453 Epoch: 7 Global Step: 70460 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:13:27,910-Speed 129.91 samples/sec Loss 10.8035 Epoch: 7 Global Step: 70465 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:13:48,322-Speed 125.42 samples/sec Loss 11.0108 Epoch: 7 Global Step: 70470 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:14:08,097-Speed 129.46 samples/sec Loss 11.0350 Epoch: 7 Global Step: 70475 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:14:27,757-Speed 130.22 samples/sec Loss 10.9438 Epoch: 7 Global Step: 70480 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:14:47,509-Speed 129.62 samples/sec Loss 10.9383 Epoch: 7 Global Step: 70485 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:15:07,167-Speed 130.23 samples/sec Loss 10.6060 Epoch: 7 Global Step: 70490 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:15:26,926-Speed 129.57 samples/sec Loss 10.7737 Epoch: 7 Global Step: 70495 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:15:46,527-Speed 130.61 samples/sec Loss 10.6504 Epoch: 7 Global Step: 70500 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:16:06,490-Speed 128.24 samples/sec Loss 10.5167 Epoch: 7 Global Step: 70505 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:16:26,089-Speed 130.62 samples/sec Loss 11.0693 Epoch: 7 Global Step: 70510 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:16:46,119-Speed 127.81 samples/sec Loss 11.0290 Epoch: 7 Global Step: 70515 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:17:06,037-Speed 128.54 samples/sec Loss 10.5326 Epoch: 7 Global Step: 70520 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:17:25,696-Speed 130.23 samples/sec Loss 10.9249 Epoch: 7 Global Step: 70525 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:17:45,488-Speed 129.35 samples/sec Loss 10.5813 Epoch: 7 Global Step: 70530 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:18:05,145-Speed 130.24 samples/sec Loss 10.8142 Epoch: 7 Global Step: 70535 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:18:24,912-Speed 129.51 samples/sec Loss 10.9719 Epoch: 7 Global Step: 70540 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:18:44,500-Speed 130.70 samples/sec Loss 11.0138 Epoch: 7 Global Step: 70545 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:19:04,431-Speed 128.45 samples/sec Loss 10.9074 Epoch: 7 Global Step: 70550 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:19:23,984-Speed 130.93 samples/sec Loss 10.8500 Epoch: 7 Global Step: 70555 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:19:44,121-Speed 127.14 samples/sec Loss 10.5637 Epoch: 7 Global Step: 70560 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:20:04,322-Speed 126.73 samples/sec Loss 10.9231 Epoch: 7 Global Step: 70565 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:20:23,931-Speed 130.56 samples/sec Loss 10.8848 Epoch: 7 Global Step: 70570 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:20:43,567-Speed 130.38 samples/sec Loss 10.9051 Epoch: 7 Global Step: 70575 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:21:03,218-Speed 130.28 samples/sec Loss 10.5888 Epoch: 7 Global Step: 70580 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:21:22,985-Speed 129.51 samples/sec Loss 10.6999 Epoch: 7 Global Step: 70585 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:21:42,666-Speed 130.09 samples/sec Loss 10.6927 Epoch: 7 Global Step: 70590 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:22:02,508-Speed 129.02 samples/sec Loss 10.7175 Epoch: 7 Global Step: 70595 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:22:22,088-Speed 130.76 samples/sec Loss 10.7925 Epoch: 7 Global Step: 70600 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:22:42,267-Speed 126.87 samples/sec Loss 11.0380 Epoch: 7 Global Step: 70605 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:23:02,381-Speed 127.28 samples/sec Loss 11.0100 Epoch: 7 Global Step: 70610 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:23:21,959-Speed 130.76 samples/sec Loss 10.7706 Epoch: 7 Global Step: 70615 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:23:41,755-Speed 129.33 samples/sec Loss 10.3044 Epoch: 7 Global Step: 70620 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:24:01,400-Speed 130.32 samples/sec Loss 10.7393 Epoch: 7 Global Step: 70625 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:24:21,239-Speed 129.05 samples/sec Loss 10.8514 Epoch: 7 Global Step: 70630 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:24:40,889-Speed 130.29 samples/sec Loss 10.8851 Epoch: 7 Global Step: 70635 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:25:00,765-Speed 128.80 samples/sec Loss 11.1810 Epoch: 7 Global Step: 70640 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:25:20,355-Speed 130.69 samples/sec Loss 10.5835 Epoch: 7 Global Step: 70645 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:25:41,871-Speed 118.99 samples/sec Loss 10.9701 Epoch: 7 Global Step: 70650 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:26:01,973-Speed 127.36 samples/sec Loss 10.6667 Epoch: 7 Global Step: 70655 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:26:21,546-Speed 130.80 samples/sec Loss 10.7727 Epoch: 7 Global Step: 70660 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:26:41,422-Speed 128.81 samples/sec Loss 10.8701 Epoch: 7 Global Step: 70665 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:27:01,072-Speed 130.29 samples/sec Loss 10.8540 Epoch: 7 Global Step: 70670 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:27:20,806-Speed 129.73 samples/sec Loss 10.7479 Epoch: 7 Global Step: 70675 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:27:40,441-Speed 130.39 samples/sec Loss 10.4676 Epoch: 7 Global Step: 70680 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:28:00,094-Speed 130.26 samples/sec Loss 10.7809 Epoch: 7 Global Step: 70685 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:28:19,750-Speed 130.25 samples/sec Loss 10.5114 Epoch: 7 Global Step: 70690 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:28:39,735-Speed 128.11 samples/sec Loss 10.4509 Epoch: 7 Global Step: 70695 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:28:59,717-Speed 128.12 samples/sec Loss 10.9176 Epoch: 7 Global Step: 70700 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:29:19,280-Speed 130.86 samples/sec Loss 10.7079 Epoch: 7 Global Step: 70705 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:29:39,298-Speed 127.90 samples/sec Loss 10.7572 Epoch: 7 Global Step: 70710 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:29:58,788-Speed 131.35 samples/sec Loss 10.7861 Epoch: 7 Global Step: 70715 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:30:18,722-Speed 128.43 samples/sec Loss 10.6767 Epoch: 7 Global Step: 70720 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:30:38,290-Speed 130.83 samples/sec Loss 11.0596 Epoch: 7 Global Step: 70725 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:30:58,141-Speed 128.97 samples/sec Loss 10.8727 Epoch: 7 Global Step: 70730 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:31:17,802-Speed 130.22 samples/sec Loss 10.5363 Epoch: 7 Global Step: 70735 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:31:37,916-Speed 127.28 samples/sec Loss 11.0394 Epoch: 7 Global Step: 70740 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:31:57,566-Speed 130.28 samples/sec Loss 10.7642 Epoch: 7 Global Step: 70745 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:32:17,209-Speed 130.33 samples/sec Loss 10.9692 Epoch: 7 Global Step: 70750 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:32:37,053-Speed 129.01 samples/sec Loss 10.8593 Epoch: 7 Global Step: 70755 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:32:56,796-Speed 129.68 samples/sec Loss 10.6011 Epoch: 7 Global Step: 70760 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:33:16,700-Speed 128.62 samples/sec Loss 10.7324 Epoch: 7 Global Step: 70765 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:33:36,250-Speed 130.95 samples/sec Loss 11.1146 Epoch: 7 Global Step: 70770 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:33:56,159-Speed 128.60 samples/sec Loss 10.7212 Epoch: 7 Global Step: 70775 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:34:15,762-Speed 130.60 samples/sec Loss 10.9688 Epoch: 7 Global Step: 70780 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:34:35,904-Speed 127.10 samples/sec Loss 10.6687 Epoch: 7 Global Step: 70785 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:34:55,708-Speed 129.27 samples/sec Loss 10.3493 Epoch: 7 Global Step: 70790 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:35:15,369-Speed 130.21 samples/sec Loss 11.0919 Epoch: 7 Global Step: 70795 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:35:35,084-Speed 129.86 samples/sec Loss 10.6933 Epoch: 7 Global Step: 70800 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:35:54,703-Speed 130.49 samples/sec Loss 10.7484 Epoch: 7 Global Step: 70805 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:36:14,383-Speed 130.09 samples/sec Loss 10.9791 Epoch: 7 Global Step: 70810 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:36:34,030-Speed 130.31 samples/sec Loss 10.7886 Epoch: 7 Global Step: 70815 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:36:53,812-Speed 129.42 samples/sec Loss 10.8096 Epoch: 7 Global Step: 70820 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:37:13,491-Speed 130.09 samples/sec Loss 10.9592 Epoch: 7 Global Step: 70825 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:37:33,500-Speed 127.96 samples/sec Loss 10.6879 Epoch: 7 Global Step: 70830 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:37:53,534-Speed 127.78 samples/sec Loss 10.9442 Epoch: 7 Global Step: 70835 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:38:13,118-Speed 130.73 samples/sec Loss 10.8499 Epoch: 7 Global Step: 70840 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:38:33,070-Speed 128.32 samples/sec Loss 10.3376 Epoch: 7 Global Step: 70845 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:38:52,544-Speed 131.46 samples/sec Loss 10.6605 Epoch: 7 Global Step: 70850 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:39:12,485-Speed 128.39 samples/sec Loss 10.7942 Epoch: 7 Global Step: 70855 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:39:32,154-Speed 130.16 samples/sec Loss 10.8365 Epoch: 7 Global Step: 70860 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:39:52,156-Speed 128.00 samples/sec Loss 10.7781 Epoch: 7 Global Step: 70865 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:40:12,897-Speed 123.43 samples/sec Loss 10.7993 Epoch: 7 Global Step: 70870 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:40:33,008-Speed 127.30 samples/sec Loss 11.0147 Epoch: 7 Global Step: 70875 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:40:52,763-Speed 129.59 samples/sec Loss 10.7752 Epoch: 7 Global Step: 70880 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:41:12,338-Speed 130.79 samples/sec Loss 10.5684 Epoch: 7 Global Step: 70885 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:41:32,182-Speed 129.01 samples/sec Loss 10.7235 Epoch: 7 Global Step: 70890 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:41:51,781-Speed 130.63 samples/sec Loss 11.2075 Epoch: 7 Global Step: 70895 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:42:11,742-Speed 128.25 samples/sec Loss 11.2347 Epoch: 7 Global Step: 70900 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:42:31,346-Speed 130.59 samples/sec Loss 10.8818 Epoch: 7 Global Step: 70905 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:42:51,146-Speed 129.30 samples/sec Loss 10.6740 Epoch: 7 Global Step: 70910 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:43:10,751-Speed 130.59 samples/sec Loss 10.9883 Epoch: 7 Global Step: 70915 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:43:30,845-Speed 127.41 samples/sec Loss 10.7928 Epoch: 7 Global Step: 70920 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:43:50,621-Speed 129.46 samples/sec Loss 10.9264 Epoch: 7 Global Step: 70925 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:44:10,221-Speed 130.62 samples/sec Loss 11.0468 Epoch: 7 Global Step: 70930 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:44:30,177-Speed 128.29 samples/sec Loss 10.9807 Epoch: 7 Global Step: 70935 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:44:49,811-Speed 130.39 samples/sec Loss 10.5953 Epoch: 7 Global Step: 70940 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:45:09,667-Speed 128.94 samples/sec Loss 10.8124 Epoch: 7 Global Step: 70945 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:45:29,269-Speed 130.61 samples/sec Loss 10.9343 Epoch: 7 Global Step: 70950 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:45:48,918-Speed 130.29 samples/sec Loss 10.7416 Epoch: 7 Global Step: 70955 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:46:08,603-Speed 130.06 samples/sec Loss 11.2862 Epoch: 7 Global Step: 70960 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:46:28,800-Speed 126.76 samples/sec Loss 10.7757 Epoch: 7 Global Step: 70965 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:46:48,555-Speed 129.59 samples/sec Loss 10.7745 Epoch: 7 Global Step: 70970 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:47:08,115-Speed 130.88 samples/sec Loss 11.0143 Epoch: 7 Global Step: 70975 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:47:27,992-Speed 128.80 samples/sec Loss 10.9681 Epoch: 7 Global Step: 70980 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:47:47,585-Speed 130.67 samples/sec Loss 10.7658 Epoch: 7 Global Step: 70985 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:48:07,530-Speed 128.36 samples/sec Loss 10.9550 Epoch: 7 Global Step: 70990 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:48:27,118-Speed 130.69 samples/sec Loss 10.3736 Epoch: 7 Global Step: 70995 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:48:47,053-Speed 128.43 samples/sec Loss 11.0081 Epoch: 7 Global Step: 71000 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:49:06,712-Speed 130.23 samples/sec Loss 11.0342 Epoch: 7 Global Step: 71005 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:49:26,739-Speed 127.83 samples/sec Loss 11.1386 Epoch: 7 Global Step: 71010 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:49:46,387-Speed 130.30 samples/sec Loss 11.3769 Epoch: 7 Global Step: 71015 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:50:05,961-Speed 130.80 samples/sec Loss 10.8474 Epoch: 7 Global Step: 71020 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:50:25,822-Speed 128.90 samples/sec Loss 10.9916 Epoch: 7 Global Step: 71025 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:50:45,401-Speed 130.76 samples/sec Loss 10.8521 Epoch: 7 Global Step: 71030 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:51:05,317-Speed 128.55 samples/sec Loss 10.6042 Epoch: 7 Global Step: 71035 Fp16 Grad Scale: 8192 Required: 213 hours Training: 2021-06-10 07:51:24,801-Speed 131.40 samples/sec Loss 10.6412 Epoch: 7 Global Step: 71040 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:51:44,936-Speed 127.15 samples/sec Loss 10.9502 Epoch: 7 Global Step: 71045 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:52:04,521-Speed 130.71 samples/sec Loss 10.8467 Epoch: 7 Global Step: 71050 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:52:24,670-Speed 127.06 samples/sec Loss 11.0525 Epoch: 7 Global Step: 71055 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:52:44,562-Speed 128.70 samples/sec Loss 11.2164 Epoch: 7 Global Step: 71060 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:53:04,217-Speed 130.25 samples/sec Loss 11.0744 Epoch: 7 Global Step: 71065 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:53:23,884-Speed 130.17 samples/sec Loss 10.9225 Epoch: 7 Global Step: 71070 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:53:43,532-Speed 130.30 samples/sec Loss 11.0556 Epoch: 7 Global Step: 71075 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:54:03,193-Speed 130.21 samples/sec Loss 11.1004 Epoch: 7 Global Step: 71080 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:54:22,826-Speed 130.41 samples/sec Loss 11.0104 Epoch: 7 Global Step: 71085 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:54:42,752-Speed 128.48 samples/sec Loss 10.7327 Epoch: 7 Global Step: 71090 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:55:02,352-Speed 130.62 samples/sec Loss 10.7777 Epoch: 7 Global Step: 71095 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:55:22,472-Speed 127.24 samples/sec Loss 10.8476 Epoch: 7 Global Step: 71100 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:55:42,531-Speed 127.63 samples/sec Loss 11.0083 Epoch: 7 Global Step: 71105 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:56:02,101-Speed 130.82 samples/sec Loss 10.6235 Epoch: 7 Global Step: 71110 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:56:21,872-Speed 129.49 samples/sec Loss 11.0342 Epoch: 7 Global Step: 71115 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:56:41,523-Speed 130.28 samples/sec Loss 11.1917 Epoch: 7 Global Step: 71120 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:57:01,278-Speed 129.60 samples/sec Loss 10.8980 Epoch: 7 Global Step: 71125 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:57:20,914-Speed 130.38 samples/sec Loss 10.8986 Epoch: 7 Global Step: 71130 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:57:40,773-Speed 128.92 samples/sec Loss 10.7729 Epoch: 7 Global Step: 71135 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:58:00,577-Speed 129.27 samples/sec Loss 10.8436 Epoch: 7 Global Step: 71140 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:58:20,460-Speed 128.77 samples/sec Loss 10.9735 Epoch: 7 Global Step: 71145 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:58:40,589-Speed 127.19 samples/sec Loss 10.9443 Epoch: 7 Global Step: 71150 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:59:00,160-Speed 130.81 samples/sec Loss 10.9642 Epoch: 7 Global Step: 71155 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:59:20,089-Speed 128.46 samples/sec Loss 10.4085 Epoch: 7 Global Step: 71160 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:59:39,596-Speed 131.24 samples/sec Loss 10.5656 Epoch: 7 Global Step: 71165 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 07:59:59,461-Speed 128.88 samples/sec Loss 10.7725 Epoch: 7 Global Step: 71170 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:00:19,069-Speed 130.57 samples/sec Loss 10.8270 Epoch: 7 Global Step: 71175 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:00:38,836-Speed 129.51 samples/sec Loss 10.7156 Epoch: 7 Global Step: 71180 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:00:58,505-Speed 130.17 samples/sec Loss 10.8985 Epoch: 7 Global Step: 71185 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:01:18,441-Speed 128.42 samples/sec Loss 11.1025 Epoch: 7 Global Step: 71190 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:01:38,406-Speed 128.23 samples/sec Loss 10.7959 Epoch: 7 Global Step: 71195 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:01:58,008-Speed 130.60 samples/sec Loss 10.9475 Epoch: 7 Global Step: 71200 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:02:18,082-Speed 127.54 samples/sec Loss 10.6058 Epoch: 7 Global Step: 71205 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:02:37,748-Speed 130.18 samples/sec Loss 10.7260 Epoch: 7 Global Step: 71210 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:02:57,588-Speed 129.05 samples/sec Loss 10.8108 Epoch: 7 Global Step: 71215 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:03:17,180-Speed 130.67 samples/sec Loss 10.6447 Epoch: 7 Global Step: 71220 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:03:37,035-Speed 128.94 samples/sec Loss 10.8517 Epoch: 7 Global Step: 71225 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:03:56,679-Speed 130.33 samples/sec Loss 10.9118 Epoch: 7 Global Step: 71230 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:04:16,886-Speed 126.69 samples/sec Loss 10.7803 Epoch: 7 Global Step: 71235 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:04:36,649-Speed 129.54 samples/sec Loss 10.5275 Epoch: 7 Global Step: 71240 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:04:56,322-Speed 130.13 samples/sec Loss 10.7777 Epoch: 7 Global Step: 71245 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:05:16,139-Speed 129.20 samples/sec Loss 10.9541 Epoch: 7 Global Step: 71250 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:05:35,691-Speed 130.94 samples/sec Loss 10.4945 Epoch: 7 Global Step: 71255 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:05:55,549-Speed 128.92 samples/sec Loss 11.1474 Epoch: 7 Global Step: 71260 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:06:15,146-Speed 130.64 samples/sec Loss 10.7702 Epoch: 7 Global Step: 71265 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:06:35,053-Speed 128.60 samples/sec Loss 10.3434 Epoch: 7 Global Step: 71270 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:06:54,696-Speed 130.33 samples/sec Loss 10.7665 Epoch: 7 Global Step: 71275 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:07:14,735-Speed 127.76 samples/sec Loss 11.2474 Epoch: 7 Global Step: 71280 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:07:34,537-Speed 129.29 samples/sec Loss 10.8488 Epoch: 7 Global Step: 71285 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:07:54,150-Speed 130.53 samples/sec Loss 11.2245 Epoch: 7 Global Step: 71290 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:08:13,888-Speed 129.70 samples/sec Loss 10.6076 Epoch: 7 Global Step: 71295 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:08:33,554-Speed 130.19 samples/sec Loss 10.9015 Epoch: 7 Global Step: 71300 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:08:53,299-Speed 129.66 samples/sec Loss 11.0716 Epoch: 7 Global Step: 71305 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:09:12,881-Speed 130.74 samples/sec Loss 10.7816 Epoch: 7 Global Step: 71310 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:09:32,832-Speed 128.32 samples/sec Loss 10.9776 Epoch: 7 Global Step: 71315 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:09:52,461-Speed 130.43 samples/sec Loss 10.8092 Epoch: 7 Global Step: 71320 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:10:12,592-Speed 127.17 samples/sec Loss 11.0238 Epoch: 7 Global Step: 71325 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:10:32,643-Speed 127.68 samples/sec Loss 11.2346 Epoch: 7 Global Step: 71330 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:10:52,417-Speed 129.47 samples/sec Loss 11.1319 Epoch: 7 Global Step: 71335 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:11:12,215-Speed 129.31 samples/sec Loss 10.9892 Epoch: 7 Global Step: 71340 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:11:31,852-Speed 130.38 samples/sec Loss 10.7346 Epoch: 7 Global Step: 71345 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:11:51,688-Speed 129.07 samples/sec Loss 10.5964 Epoch: 7 Global Step: 71350 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:12:11,317-Speed 130.43 samples/sec Loss 10.8832 Epoch: 7 Global Step: 71355 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:12:31,112-Speed 129.33 samples/sec Loss 10.8180 Epoch: 7 Global Step: 71360 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:12:50,709-Speed 130.64 samples/sec Loss 10.7189 Epoch: 7 Global Step: 71365 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:13:10,798-Speed 127.44 samples/sec Loss 11.1333 Epoch: 7 Global Step: 71370 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:13:30,918-Speed 127.24 samples/sec Loss 11.0022 Epoch: 7 Global Step: 71375 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:13:50,488-Speed 130.82 samples/sec Loss 10.8288 Epoch: 7 Global Step: 71380 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:14:10,428-Speed 128.39 samples/sec Loss 10.9226 Epoch: 7 Global Step: 71385 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:14:30,020-Speed 130.67 samples/sec Loss 10.9357 Epoch: 7 Global Step: 71390 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:14:49,797-Speed 129.45 samples/sec Loss 10.6778 Epoch: 7 Global Step: 71395 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:15:09,455-Speed 130.24 samples/sec Loss 10.8625 Epoch: 7 Global Step: 71400 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:15:29,278-Speed 129.15 samples/sec Loss 10.7440 Epoch: 7 Global Step: 71405 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:15:48,940-Speed 130.20 samples/sec Loss 10.9719 Epoch: 7 Global Step: 71410 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:16:09,074-Speed 127.16 samples/sec Loss 10.8124 Epoch: 7 Global Step: 71415 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:16:28,847-Speed 129.48 samples/sec Loss 11.1471 Epoch: 7 Global Step: 71420 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:16:48,352-Speed 131.25 samples/sec Loss 11.0601 Epoch: 7 Global Step: 71425 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:17:08,372-Speed 127.88 samples/sec Loss 10.7004 Epoch: 7 Global Step: 71430 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:17:27,951-Speed 130.76 samples/sec Loss 11.0267 Epoch: 7 Global Step: 71435 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:17:48,008-Speed 127.64 samples/sec Loss 10.5976 Epoch: 7 Global Step: 71440 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:18:07,607-Speed 130.63 samples/sec Loss 10.8043 Epoch: 7 Global Step: 71445 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:18:27,518-Speed 128.58 samples/sec Loss 11.0115 Epoch: 7 Global Step: 71450 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:18:47,113-Speed 130.65 samples/sec Loss 11.0705 Epoch: 7 Global Step: 71455 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:19:07,231-Speed 127.25 samples/sec Loss 11.1396 Epoch: 7 Global Step: 71460 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:19:26,858-Speed 130.44 samples/sec Loss 10.6883 Epoch: 7 Global Step: 71465 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:19:46,510-Speed 130.27 samples/sec Loss 11.2331 Epoch: 7 Global Step: 71470 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:20:06,309-Speed 129.30 samples/sec Loss 11.1976 Epoch: 7 Global Step: 71475 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:20:26,079-Speed 129.49 samples/sec Loss 10.7053 Epoch: 7 Global Step: 71480 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:20:45,986-Speed 128.60 samples/sec Loss 10.8135 Epoch: 7 Global Step: 71485 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:21:05,502-Speed 131.19 samples/sec Loss 11.2537 Epoch: 7 Global Step: 71490 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:21:25,411-Speed 128.59 samples/sec Loss 11.0956 Epoch: 7 Global Step: 71495 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:21:45,038-Speed 130.44 samples/sec Loss 10.9055 Epoch: 7 Global Step: 71500 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:22:05,219-Speed 126.86 samples/sec Loss 10.6698 Epoch: 7 Global Step: 71505 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:22:24,999-Speed 129.43 samples/sec Loss 10.7500 Epoch: 7 Global Step: 71510 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:22:44,576-Speed 130.77 samples/sec Loss 10.7676 Epoch: 7 Global Step: 71515 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:23:04,330-Speed 129.61 samples/sec Loss 11.1212 Epoch: 7 Global Step: 71520 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:23:23,993-Speed 130.20 samples/sec Loss 10.9387 Epoch: 7 Global Step: 71525 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:23:43,635-Speed 130.34 samples/sec Loss 10.9719 Epoch: 7 Global Step: 71530 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:24:03,211-Speed 130.78 samples/sec Loss 11.1545 Epoch: 7 Global Step: 71535 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:24:23,080-Speed 128.85 samples/sec Loss 10.7135 Epoch: 7 Global Step: 71540 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:24:42,740-Speed 130.22 samples/sec Loss 11.1784 Epoch: 7 Global Step: 71545 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:25:02,904-Speed 126.97 samples/sec Loss 10.8794 Epoch: 7 Global Step: 71550 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:25:22,831-Speed 128.47 samples/sec Loss 10.6600 Epoch: 7 Global Step: 71555 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:25:42,409-Speed 130.77 samples/sec Loss 11.2131 Epoch: 7 Global Step: 71560 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:26:02,281-Speed 128.83 samples/sec Loss 10.9567 Epoch: 7 Global Step: 71565 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:26:21,964-Speed 130.07 samples/sec Loss 10.6066 Epoch: 7 Global Step: 71570 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:26:41,608-Speed 130.33 samples/sec Loss 10.8076 Epoch: 7 Global Step: 71575 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:27:01,238-Speed 130.42 samples/sec Loss 10.8020 Epoch: 7 Global Step: 71580 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:27:20,971-Speed 129.74 samples/sec Loss 11.1013 Epoch: 7 Global Step: 71585 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:27:40,640-Speed 130.16 samples/sec Loss 10.7537 Epoch: 7 Global Step: 71590 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:28:00,507-Speed 128.87 samples/sec Loss 11.1216 Epoch: 7 Global Step: 71595 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:28:20,493-Speed 128.10 samples/sec Loss 10.9469 Epoch: 7 Global Step: 71600 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:28:40,048-Speed 130.92 samples/sec Loss 10.9440 Epoch: 7 Global Step: 71605 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:28:59,987-Speed 128.40 samples/sec Loss 11.1471 Epoch: 7 Global Step: 71610 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:29:19,460-Speed 131.47 samples/sec Loss 10.7438 Epoch: 7 Global Step: 71615 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:29:39,432-Speed 128.19 samples/sec Loss 10.8144 Epoch: 7 Global Step: 71620 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:29:59,025-Speed 130.66 samples/sec Loss 10.9525 Epoch: 7 Global Step: 71625 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:30:18,850-Speed 129.14 samples/sec Loss 11.1805 Epoch: 7 Global Step: 71630 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:30:38,969-Speed 127.25 samples/sec Loss 10.9489 Epoch: 7 Global Step: 71635 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:30:58,980-Speed 127.93 samples/sec Loss 11.0475 Epoch: 7 Global Step: 71640 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:31:19,598-Speed 124.17 samples/sec Loss 10.9390 Epoch: 7 Global Step: 71645 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:31:39,196-Speed 130.63 samples/sec Loss 11.1411 Epoch: 7 Global Step: 71650 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:31:58,991-Speed 129.33 samples/sec Loss 10.9702 Epoch: 7 Global Step: 71655 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:32:18,607-Speed 130.52 samples/sec Loss 10.9890 Epoch: 7 Global Step: 71660 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:32:38,538-Speed 128.44 samples/sec Loss 10.4347 Epoch: 7 Global Step: 71665 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:32:58,200-Speed 130.21 samples/sec Loss 10.8374 Epoch: 7 Global Step: 71670 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:33:18,160-Speed 128.26 samples/sec Loss 10.9517 Epoch: 7 Global Step: 71675 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:33:37,711-Speed 130.95 samples/sec Loss 10.9347 Epoch: 7 Global Step: 71680 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:33:57,767-Speed 127.64 samples/sec Loss 11.1721 Epoch: 7 Global Step: 71685 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:34:17,635-Speed 128.86 samples/sec Loss 10.8850 Epoch: 7 Global Step: 71690 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:34:37,644-Speed 127.95 samples/sec Loss 11.1265 Epoch: 7 Global Step: 71695 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:34:57,443-Speed 129.30 samples/sec Loss 10.9338 Epoch: 7 Global Step: 71700 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:35:17,063-Speed 130.49 samples/sec Loss 11.1432 Epoch: 7 Global Step: 71705 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:35:37,007-Speed 128.37 samples/sec Loss 10.8082 Epoch: 7 Global Step: 71710 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:35:56,647-Speed 130.35 samples/sec Loss 10.9335 Epoch: 7 Global Step: 71715 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:36:16,479-Speed 129.09 samples/sec Loss 11.0425 Epoch: 7 Global Step: 71720 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:36:36,065-Speed 130.71 samples/sec Loss 10.9653 Epoch: 7 Global Step: 71725 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:36:56,214-Speed 127.06 samples/sec Loss 10.8387 Epoch: 7 Global Step: 71730 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:37:16,173-Speed 128.27 samples/sec Loss 10.7630 Epoch: 7 Global Step: 71735 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:37:35,794-Speed 130.48 samples/sec Loss 10.6725 Epoch: 7 Global Step: 71740 Fp16 Grad Scale: 8192 Required: 212 hours Training: 2021-06-10 08:37:55,544-Speed 129.63 samples/sec Loss 11.0070 Epoch: 7 Global Step: 71745 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:38:15,207-Speed 130.20 samples/sec Loss 11.0063 Epoch: 7 Global Step: 71750 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:38:34,954-Speed 129.65 samples/sec Loss 10.8578 Epoch: 7 Global Step: 71755 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:38:54,500-Speed 130.98 samples/sec Loss 10.9401 Epoch: 7 Global Step: 71760 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:39:14,286-Speed 129.39 samples/sec Loss 10.9375 Epoch: 7 Global Step: 71765 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:39:33,880-Speed 130.66 samples/sec Loss 11.0401 Epoch: 7 Global Step: 71770 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:39:54,083-Speed 126.72 samples/sec Loss 10.9200 Epoch: 7 Global Step: 71775 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:40:14,019-Speed 128.42 samples/sec Loss 11.0191 Epoch: 7 Global Step: 71780 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:40:33,612-Speed 130.66 samples/sec Loss 10.8177 Epoch: 7 Global Step: 71785 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:40:53,690-Speed 127.52 samples/sec Loss 10.9050 Epoch: 7 Global Step: 71790 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:41:13,322-Speed 130.40 samples/sec Loss 10.7516 Epoch: 7 Global Step: 71795 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:41:33,082-Speed 129.56 samples/sec Loss 11.1171 Epoch: 7 Global Step: 71800 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:41:52,729-Speed 130.31 samples/sec Loss 10.8673 Epoch: 7 Global Step: 71805 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:42:12,485-Speed 129.59 samples/sec Loss 10.8110 Epoch: 7 Global Step: 71810 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:42:32,147-Speed 130.20 samples/sec Loss 10.7669 Epoch: 7 Global Step: 71815 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:42:52,109-Speed 128.25 samples/sec Loss 11.0112 Epoch: 7 Global Step: 71820 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:43:12,141-Speed 127.80 samples/sec Loss 10.8651 Epoch: 7 Global Step: 71825 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:43:31,630-Speed 131.36 samples/sec Loss 10.9100 Epoch: 7 Global Step: 71830 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:43:51,628-Speed 128.02 samples/sec Loss 10.9482 Epoch: 7 Global Step: 71835 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:44:11,198-Speed 130.82 samples/sec Loss 10.7322 Epoch: 7 Global Step: 71840 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:44:30,921-Speed 129.80 samples/sec Loss 10.9450 Epoch: 7 Global Step: 71845 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:44:50,529-Speed 130.57 samples/sec Loss 10.9677 Epoch: 7 Global Step: 71850 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:45:10,298-Speed 129.50 samples/sec Loss 10.9873 Epoch: 7 Global Step: 71855 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:45:29,947-Speed 130.30 samples/sec Loss 11.1448 Epoch: 7 Global Step: 71860 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:45:49,956-Speed 127.95 samples/sec Loss 10.6623 Epoch: 7 Global Step: 71865 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:46:09,775-Speed 129.18 samples/sec Loss 11.0726 Epoch: 7 Global Step: 71870 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:46:29,360-Speed 130.72 samples/sec Loss 11.0295 Epoch: 7 Global Step: 71875 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:46:49,371-Speed 127.94 samples/sec Loss 10.9623 Epoch: 7 Global Step: 71880 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:47:08,946-Speed 130.79 samples/sec Loss 11.2512 Epoch: 7 Global Step: 71885 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:47:28,876-Speed 128.46 samples/sec Loss 11.0403 Epoch: 7 Global Step: 71890 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:47:48,362-Speed 131.39 samples/sec Loss 11.1799 Epoch: 7 Global Step: 71895 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:48:08,188-Speed 129.13 samples/sec Loss 10.7824 Epoch: 7 Global Step: 71900 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:48:27,797-Speed 130.56 samples/sec Loss 10.8162 Epoch: 7 Global Step: 71905 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:48:47,925-Speed 127.19 samples/sec Loss 10.6525 Epoch: 7 Global Step: 71910 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:49:07,670-Speed 129.66 samples/sec Loss 10.7218 Epoch: 7 Global Step: 71915 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:49:27,433-Speed 129.54 samples/sec Loss 11.0919 Epoch: 7 Global Step: 71920 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:49:47,242-Speed 129.24 samples/sec Loss 10.7745 Epoch: 7 Global Step: 71925 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:50:06,861-Speed 130.49 samples/sec Loss 11.0401 Epoch: 7 Global Step: 71930 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:50:26,759-Speed 128.67 samples/sec Loss 10.7212 Epoch: 7 Global Step: 71935 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:50:46,336-Speed 130.77 samples/sec Loss 11.2073 Epoch: 7 Global Step: 71940 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:51:06,343-Speed 127.96 samples/sec Loss 10.8729 Epoch: 7 Global Step: 71945 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:51:25,923-Speed 130.75 samples/sec Loss 11.2316 Epoch: 7 Global Step: 71950 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:51:46,027-Speed 127.34 samples/sec Loss 11.0511 Epoch: 7 Global Step: 71955 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:52:05,825-Speed 129.31 samples/sec Loss 10.8666 Epoch: 7 Global Step: 71960 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:52:25,524-Speed 129.96 samples/sec Loss 11.1916 Epoch: 7 Global Step: 71965 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:52:45,285-Speed 129.56 samples/sec Loss 10.9709 Epoch: 7 Global Step: 71970 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:53:04,872-Speed 130.70 samples/sec Loss 10.6983 Epoch: 7 Global Step: 71975 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:53:24,794-Speed 128.51 samples/sec Loss 11.1446 Epoch: 7 Global Step: 71980 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:53:44,451-Speed 130.24 samples/sec Loss 10.6567 Epoch: 7 Global Step: 71985 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:54:04,305-Speed 128.95 samples/sec Loss 10.8244 Epoch: 7 Global Step: 71990 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:54:23,906-Speed 130.61 samples/sec Loss 11.1580 Epoch: 7 Global Step: 71995 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:54:43,988-Speed 127.49 samples/sec Loss 11.1979 Epoch: 7 Global Step: 72000 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:55:03,892-Speed 128.62 samples/sec Loss 11.3001 Epoch: 7 Global Step: 72005 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:55:23,496-Speed 130.59 samples/sec Loss 10.9947 Epoch: 7 Global Step: 72010 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:55:43,238-Speed 129.67 samples/sec Loss 11.0372 Epoch: 7 Global Step: 72015 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:56:02,880-Speed 130.34 samples/sec Loss 10.9343 Epoch: 7 Global Step: 72020 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:56:22,679-Speed 129.30 samples/sec Loss 10.8901 Epoch: 7 Global Step: 72025 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:56:42,240-Speed 130.88 samples/sec Loss 10.6966 Epoch: 7 Global Step: 72030 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:57:02,193-Speed 128.31 samples/sec Loss 10.7656 Epoch: 7 Global Step: 72035 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:57:21,733-Speed 131.02 samples/sec Loss 10.8400 Epoch: 7 Global Step: 72040 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:57:41,777-Speed 127.72 samples/sec Loss 10.8706 Epoch: 7 Global Step: 72045 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:58:01,770-Speed 128.05 samples/sec Loss 10.7439 Epoch: 7 Global Step: 72050 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:58:21,357-Speed 130.71 samples/sec Loss 11.0287 Epoch: 7 Global Step: 72055 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:58:41,160-Speed 129.28 samples/sec Loss 10.9863 Epoch: 7 Global Step: 72060 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:59:00,782-Speed 130.47 samples/sec Loss 10.7010 Epoch: 7 Global Step: 72065 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:59:20,545-Speed 129.55 samples/sec Loss 11.0246 Epoch: 7 Global Step: 72070 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:59:40,134-Speed 130.69 samples/sec Loss 10.5918 Epoch: 7 Global Step: 72075 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 08:59:59,961-Speed 129.12 samples/sec Loss 10.7275 Epoch: 7 Global Step: 72080 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:00:19,480-Speed 131.16 samples/sec Loss 11.4194 Epoch: 7 Global Step: 72085 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:00:39,680-Speed 126.74 samples/sec Loss 11.1807 Epoch: 7 Global Step: 72090 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:00:59,709-Speed 127.82 samples/sec Loss 11.1648 Epoch: 7 Global Step: 72095 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:01:19,301-Speed 130.67 samples/sec Loss 10.7608 Epoch: 7 Global Step: 72100 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:01:39,038-Speed 129.71 samples/sec Loss 10.7144 Epoch: 7 Global Step: 72105 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:01:58,693-Speed 130.26 samples/sec Loss 10.9785 Epoch: 7 Global Step: 72110 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:02:18,432-Speed 129.70 samples/sec Loss 10.9015 Epoch: 7 Global Step: 72115 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:02:38,073-Speed 130.34 samples/sec Loss 11.1246 Epoch: 7 Global Step: 72120 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:02:57,818-Speed 129.66 samples/sec Loss 10.8158 Epoch: 7 Global Step: 72125 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:03:17,372-Speed 130.93 samples/sec Loss 10.8281 Epoch: 7 Global Step: 72130 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:03:37,494-Speed 127.23 samples/sec Loss 11.2553 Epoch: 7 Global Step: 72135 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:03:57,539-Speed 127.72 samples/sec Loss 11.0517 Epoch: 7 Global Step: 72140 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:04:17,234-Speed 129.99 samples/sec Loss 11.4126 Epoch: 7 Global Step: 72145 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:04:37,157-Speed 128.50 samples/sec Loss 11.0377 Epoch: 7 Global Step: 72150 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:04:56,765-Speed 130.56 samples/sec Loss 11.0103 Epoch: 7 Global Step: 72155 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:05:16,568-Speed 129.28 samples/sec Loss 11.1110 Epoch: 7 Global Step: 72160 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:05:36,194-Speed 130.45 samples/sec Loss 11.0325 Epoch: 7 Global Step: 72165 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:05:55,837-Speed 130.34 samples/sec Loss 11.0193 Epoch: 7 Global Step: 72170 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:06:15,499-Speed 130.21 samples/sec Loss 10.8081 Epoch: 7 Global Step: 72175 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:06:35,638-Speed 127.12 samples/sec Loss 11.0117 Epoch: 7 Global Step: 72180 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:06:55,294-Speed 130.25 samples/sec Loss 10.7700 Epoch: 7 Global Step: 72185 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:07:14,796-Speed 131.27 samples/sec Loss 11.0514 Epoch: 7 Global Step: 72190 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:07:34,754-Speed 128.27 samples/sec Loss 11.0246 Epoch: 7 Global Step: 72195 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:07:54,354-Speed 130.62 samples/sec Loss 10.8621 Epoch: 7 Global Step: 72200 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:08:14,312-Speed 128.27 samples/sec Loss 11.1867 Epoch: 7 Global Step: 72205 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:08:33,911-Speed 130.63 samples/sec Loss 11.2587 Epoch: 7 Global Step: 72210 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:08:53,851-Speed 128.39 samples/sec Loss 10.7181 Epoch: 7 Global Step: 72215 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:09:13,447-Speed 130.64 samples/sec Loss 10.6491 Epoch: 7 Global Step: 72220 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:09:33,588-Speed 127.11 samples/sec Loss 10.6929 Epoch: 7 Global Step: 72225 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:09:53,280-Speed 130.01 samples/sec Loss 10.8424 Epoch: 7 Global Step: 72230 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:10:12,928-Speed 130.30 samples/sec Loss 10.6710 Epoch: 7 Global Step: 72235 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:10:32,696-Speed 129.51 samples/sec Loss 10.9586 Epoch: 7 Global Step: 72240 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:10:52,347-Speed 130.28 samples/sec Loss 10.5601 Epoch: 7 Global Step: 72245 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:11:11,993-Speed 130.31 samples/sec Loss 10.8432 Epoch: 7 Global Step: 72250 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:11:31,520-Speed 131.11 samples/sec Loss 10.7925 Epoch: 7 Global Step: 72255 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:11:51,447-Speed 128.48 samples/sec Loss 11.0615 Epoch: 7 Global Step: 72260 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:12:11,063-Speed 130.51 samples/sec Loss 10.6526 Epoch: 7 Global Step: 72265 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:12:31,199-Speed 127.14 samples/sec Loss 11.1744 Epoch: 7 Global Step: 72270 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:12:51,187-Speed 128.08 samples/sec Loss 11.1284 Epoch: 7 Global Step: 72275 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:13:10,767-Speed 130.75 samples/sec Loss 11.0225 Epoch: 7 Global Step: 72280 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:13:30,570-Speed 129.29 samples/sec Loss 10.9215 Epoch: 7 Global Step: 72285 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:13:50,184-Speed 130.53 samples/sec Loss 10.9154 Epoch: 7 Global Step: 72290 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:14:09,833-Speed 130.29 samples/sec Loss 10.9822 Epoch: 7 Global Step: 72295 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:14:29,485-Speed 130.28 samples/sec Loss 10.6512 Epoch: 7 Global Step: 72300 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:14:49,200-Speed 129.86 samples/sec Loss 10.9383 Epoch: 7 Global Step: 72305 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:15:08,781-Speed 130.75 samples/sec Loss 10.7235 Epoch: 7 Global Step: 72310 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:15:28,717-Speed 128.42 samples/sec Loss 11.0887 Epoch: 7 Global Step: 72315 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:15:48,726-Speed 127.95 samples/sec Loss 10.8063 Epoch: 7 Global Step: 72320 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:16:08,392-Speed 130.18 samples/sec Loss 10.6576 Epoch: 7 Global Step: 72325 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:16:28,419-Speed 127.83 samples/sec Loss 10.6633 Epoch: 7 Global Step: 72330 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:16:47,894-Speed 131.46 samples/sec Loss 10.6432 Epoch: 7 Global Step: 72335 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:17:07,755-Speed 128.91 samples/sec Loss 11.1412 Epoch: 7 Global Step: 72340 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:17:27,558-Speed 129.28 samples/sec Loss 10.5629 Epoch: 7 Global Step: 72345 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:17:47,338-Speed 129.43 samples/sec Loss 10.8727 Epoch: 7 Global Step: 72350 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:18:06,993-Speed 130.26 samples/sec Loss 10.7992 Epoch: 7 Global Step: 72355 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:18:26,945-Speed 128.31 samples/sec Loss 11.0097 Epoch: 7 Global Step: 72360 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:18:46,880-Speed 128.42 samples/sec Loss 10.9660 Epoch: 7 Global Step: 72365 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:19:06,473-Speed 130.67 samples/sec Loss 10.4663 Epoch: 7 Global Step: 72370 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:19:26,413-Speed 128.39 samples/sec Loss 10.5164 Epoch: 7 Global Step: 72375 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:19:45,983-Speed 130.82 samples/sec Loss 10.7243 Epoch: 7 Global Step: 72380 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:20:05,995-Speed 127.93 samples/sec Loss 10.6545 Epoch: 7 Global Step: 72385 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:20:25,558-Speed 130.87 samples/sec Loss 10.9192 Epoch: 7 Global Step: 72390 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:20:45,304-Speed 129.65 samples/sec Loss 10.8437 Epoch: 7 Global Step: 72395 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:21:04,910-Speed 130.58 samples/sec Loss 11.0422 Epoch: 7 Global Step: 72400 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:21:25,087-Speed 126.88 samples/sec Loss 10.6407 Epoch: 7 Global Step: 72405 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:21:44,877-Speed 129.36 samples/sec Loss 10.6786 Epoch: 7 Global Step: 72410 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:22:04,522-Speed 130.32 samples/sec Loss 10.8932 Epoch: 7 Global Step: 72415 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:22:24,180-Speed 130.24 samples/sec Loss 10.8436 Epoch: 7 Global Step: 72420 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:22:43,848-Speed 130.17 samples/sec Loss 10.9586 Epoch: 7 Global Step: 72425 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:23:03,657-Speed 129.24 samples/sec Loss 11.1222 Epoch: 7 Global Step: 72430 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:23:23,268-Speed 130.55 samples/sec Loss 10.8557 Epoch: 7 Global Step: 72435 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:23:43,188-Speed 128.52 samples/sec Loss 10.8930 Epoch: 7 Global Step: 72440 Fp16 Grad Scale: 8192 Required: 211 hours Training: 2021-06-10 09:24:02,776-Speed 130.70 samples/sec Loss 11.1414 Epoch: 7 Global Step: 72445 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:24:22,987-Speed 126.67 samples/sec Loss 10.8081 Epoch: 7 Global Step: 72450 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:24:42,839-Speed 128.96 samples/sec Loss 10.7468 Epoch: 7 Global Step: 72455 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:25:02,484-Speed 130.32 samples/sec Loss 10.7602 Epoch: 7 Global Step: 72460 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:25:22,295-Speed 129.23 samples/sec Loss 10.7521 Epoch: 7 Global Step: 72465 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:25:41,952-Speed 130.24 samples/sec Loss 11.2054 Epoch: 7 Global Step: 72470 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:26:01,709-Speed 129.58 samples/sec Loss 10.5369 Epoch: 7 Global Step: 72475 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:26:21,296-Speed 130.70 samples/sec Loss 10.6628 Epoch: 7 Global Step: 72480 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:26:41,098-Speed 129.29 samples/sec Loss 11.0563 Epoch: 7 Global Step: 72485 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:27:00,742-Speed 130.33 samples/sec Loss 10.9386 Epoch: 7 Global Step: 72490 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:27:20,744-Speed 127.99 samples/sec Loss 10.9035 Epoch: 7 Global Step: 72495 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:27:40,804-Speed 127.62 samples/sec Loss 10.9724 Epoch: 7 Global Step: 72500 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:28:00,397-Speed 130.66 samples/sec Loss 11.1241 Epoch: 7 Global Step: 72505 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:28:20,525-Speed 127.19 samples/sec Loss 10.3120 Epoch: 7 Global Step: 72510 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:28:40,098-Speed 130.80 samples/sec Loss 10.8574 Epoch: 7 Global Step: 72515 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:28:59,878-Speed 129.43 samples/sec Loss 11.0177 Epoch: 7 Global Step: 72520 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:29:19,501-Speed 130.47 samples/sec Loss 11.2847 Epoch: 7 Global Step: 72525 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:29:39,207-Speed 129.91 samples/sec Loss 10.4975 Epoch: 7 Global Step: 72530 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:29:58,881-Speed 130.13 samples/sec Loss 10.8864 Epoch: 7 Global Step: 72535 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:30:19,002-Speed 127.24 samples/sec Loss 10.8277 Epoch: 7 Global Step: 72540 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:30:38,754-Speed 129.61 samples/sec Loss 10.5914 Epoch: 7 Global Step: 72545 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:30:58,405-Speed 130.29 samples/sec Loss 11.0225 Epoch: 7 Global Step: 72550 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:31:18,308-Speed 128.63 samples/sec Loss 10.8310 Epoch: 7 Global Step: 72555 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:31:37,898-Speed 130.68 samples/sec Loss 11.0441 Epoch: 7 Global Step: 72560 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:31:57,815-Speed 128.55 samples/sec Loss 10.9063 Epoch: 7 Global Step: 72565 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:32:17,400-Speed 130.72 samples/sec Loss 10.7133 Epoch: 7 Global Step: 72570 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:32:37,414-Speed 127.92 samples/sec Loss 11.0225 Epoch: 7 Global Step: 72575 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:32:56,995-Speed 130.75 samples/sec Loss 11.0553 Epoch: 7 Global Step: 72580 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:33:17,148-Speed 127.03 samples/sec Loss 11.1130 Epoch: 7 Global Step: 72585 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:33:36,907-Speed 129.57 samples/sec Loss 10.7084 Epoch: 7 Global Step: 72590 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:33:56,559-Speed 130.27 samples/sec Loss 10.7108 Epoch: 7 Global Step: 72595 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:34:16,431-Speed 128.83 samples/sec Loss 11.1016 Epoch: 7 Global Step: 72600 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:34:37,142-Speed 123.62 samples/sec Loss 10.7751 Epoch: 7 Global Step: 72605 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:34:56,918-Speed 129.46 samples/sec Loss 11.1152 Epoch: 7 Global Step: 72610 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:35:16,567-Speed 130.29 samples/sec Loss 10.9634 Epoch: 7 Global Step: 72615 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:35:36,508-Speed 128.39 samples/sec Loss 10.9205 Epoch: 7 Global Step: 72620 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:35:56,114-Speed 130.58 samples/sec Loss 10.5801 Epoch: 7 Global Step: 72625 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:36:16,242-Speed 127.19 samples/sec Loss 11.0214 Epoch: 7 Global Step: 72630 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:36:36,116-Speed 128.82 samples/sec Loss 11.1138 Epoch: 7 Global Step: 72635 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:36:55,729-Speed 130.53 samples/sec Loss 10.7538 Epoch: 7 Global Step: 72640 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:37:15,471-Speed 129.68 samples/sec Loss 10.5716 Epoch: 7 Global Step: 72645 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:37:35,120-Speed 130.29 samples/sec Loss 10.8090 Epoch: 7 Global Step: 72650 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:37:54,847-Speed 129.78 samples/sec Loss 10.8653 Epoch: 7 Global Step: 72655 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:38:14,451-Speed 130.60 samples/sec Loss 10.9827 Epoch: 7 Global Step: 72660 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:38:34,346-Speed 128.68 samples/sec Loss 10.9532 Epoch: 7 Global Step: 72665 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:38:53,962-Speed 130.51 samples/sec Loss 10.8771 Epoch: 7 Global Step: 72670 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:39:13,976-Speed 127.91 samples/sec Loss 11.0325 Epoch: 7 Global Step: 72675 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:39:33,924-Speed 128.35 samples/sec Loss 11.2195 Epoch: 7 Global Step: 72680 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:39:53,543-Speed 130.49 samples/sec Loss 10.8738 Epoch: 7 Global Step: 72685 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:40:13,300-Speed 129.58 samples/sec Loss 10.9668 Epoch: 7 Global Step: 72690 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:40:32,950-Speed 130.29 samples/sec Loss 10.8308 Epoch: 7 Global Step: 72695 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:40:52,766-Speed 129.19 samples/sec Loss 11.0610 Epoch: 7 Global Step: 72700 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:41:12,329-Speed 130.87 samples/sec Loss 11.2275 Epoch: 7 Global Step: 72705 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:41:32,220-Speed 128.71 samples/sec Loss 10.8084 Epoch: 7 Global Step: 72710 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:41:51,769-Speed 130.96 samples/sec Loss 10.9442 Epoch: 7 Global Step: 72715 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:42:11,894-Speed 127.21 samples/sec Loss 10.8555 Epoch: 7 Global Step: 72720 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:42:31,998-Speed 127.34 samples/sec Loss 10.7846 Epoch: 7 Global Step: 72725 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:42:51,570-Speed 130.80 samples/sec Loss 10.9452 Epoch: 7 Global Step: 72730 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:43:11,378-Speed 129.25 samples/sec Loss 10.8069 Epoch: 7 Global Step: 72735 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:43:31,038-Speed 130.22 samples/sec Loss 10.9934 Epoch: 7 Global Step: 72740 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:43:50,729-Speed 130.02 samples/sec Loss 10.7413 Epoch: 7 Global Step: 72745 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:44:10,338-Speed 130.56 samples/sec Loss 11.2205 Epoch: 7 Global Step: 72750 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:44:30,099-Speed 129.56 samples/sec Loss 10.7703 Epoch: 7 Global Step: 72755 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:44:49,745-Speed 130.31 samples/sec Loss 10.6929 Epoch: 7 Global Step: 72760 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:45:09,793-Speed 127.70 samples/sec Loss 10.7390 Epoch: 7 Global Step: 72765 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:45:29,835-Speed 127.74 samples/sec Loss 11.0042 Epoch: 7 Global Step: 72770 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:45:49,416-Speed 130.74 samples/sec Loss 10.9765 Epoch: 7 Global Step: 72775 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:46:09,290-Speed 128.82 samples/sec Loss 10.9402 Epoch: 7 Global Step: 72780 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:46:28,902-Speed 130.54 samples/sec Loss 10.7964 Epoch: 7 Global Step: 72785 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:46:48,840-Speed 128.41 samples/sec Loss 10.7585 Epoch: 7 Global Step: 72790 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:47:08,429-Speed 130.69 samples/sec Loss 10.8921 Epoch: 7 Global Step: 72795 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:47:28,242-Speed 129.22 samples/sec Loss 10.9886 Epoch: 7 Global Step: 72800 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:47:47,917-Speed 130.12 samples/sec Loss 10.9817 Epoch: 7 Global Step: 72805 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:48:07,941-Speed 127.85 samples/sec Loss 10.7332 Epoch: 7 Global Step: 72810 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:48:27,765-Speed 129.14 samples/sec Loss 10.8118 Epoch: 7 Global Step: 72815 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:48:47,354-Speed 130.69 samples/sec Loss 10.8953 Epoch: 7 Global Step: 72820 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:49:07,357-Speed 127.99 samples/sec Loss 10.6950 Epoch: 7 Global Step: 72825 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:49:26,873-Speed 131.18 samples/sec Loss 10.9358 Epoch: 7 Global Step: 72830 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:49:46,798-Speed 128.49 samples/sec Loss 10.9469 Epoch: 7 Global Step: 72835 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:50:06,398-Speed 130.62 samples/sec Loss 10.6577 Epoch: 7 Global Step: 72840 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:50:26,249-Speed 128.97 samples/sec Loss 11.2391 Epoch: 7 Global Step: 72845 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:50:45,820-Speed 130.81 samples/sec Loss 10.5911 Epoch: 7 Global Step: 72850 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:51:05,827-Speed 127.96 samples/sec Loss 10.7303 Epoch: 7 Global Step: 72855 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:51:25,603-Speed 129.46 samples/sec Loss 10.9274 Epoch: 7 Global Step: 72860 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:51:45,227-Speed 130.46 samples/sec Loss 10.9410 Epoch: 7 Global Step: 72865 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:52:05,024-Speed 129.32 samples/sec Loss 10.8359 Epoch: 7 Global Step: 72870 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:52:24,619-Speed 130.65 samples/sec Loss 10.8117 Epoch: 7 Global Step: 72875 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:52:44,560-Speed 128.39 samples/sec Loss 11.0742 Epoch: 7 Global Step: 72880 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:53:04,160-Speed 130.62 samples/sec Loss 11.0078 Epoch: 7 Global Step: 72885 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:53:24,142-Speed 128.12 samples/sec Loss 10.9329 Epoch: 7 Global Step: 72890 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:53:43,747-Speed 130.58 samples/sec Loss 10.8194 Epoch: 7 Global Step: 72895 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:54:03,917-Speed 126.93 samples/sec Loss 10.5556 Epoch: 7 Global Step: 72900 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:54:23,709-Speed 129.35 samples/sec Loss 10.8391 Epoch: 7 Global Step: 72905 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:54:43,356-Speed 130.30 samples/sec Loss 10.6352 Epoch: 7 Global Step: 72910 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:55:03,112-Speed 129.59 samples/sec Loss 10.7298 Epoch: 7 Global Step: 72915 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:55:22,764-Speed 130.28 samples/sec Loss 10.9658 Epoch: 7 Global Step: 72920 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:55:42,531-Speed 129.51 samples/sec Loss 10.6882 Epoch: 7 Global Step: 72925 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:56:02,042-Speed 131.22 samples/sec Loss 10.6858 Epoch: 7 Global Step: 72930 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:56:22,020-Speed 128.15 samples/sec Loss 10.9925 Epoch: 7 Global Step: 72935 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:56:41,621-Speed 130.61 samples/sec Loss 11.1441 Epoch: 7 Global Step: 72940 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:57:01,759-Speed 127.13 samples/sec Loss 10.9413 Epoch: 7 Global Step: 72945 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:57:21,844-Speed 127.46 samples/sec Loss 11.3206 Epoch: 7 Global Step: 72950 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:57:41,402-Speed 130.90 samples/sec Loss 11.0165 Epoch: 7 Global Step: 72955 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:58:01,171-Speed 129.50 samples/sec Loss 10.7615 Epoch: 7 Global Step: 72960 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:58:20,788-Speed 130.50 samples/sec Loss 10.7366 Epoch: 7 Global Step: 72965 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:58:40,477-Speed 130.03 samples/sec Loss 10.9703 Epoch: 7 Global Step: 72970 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:59:00,122-Speed 130.32 samples/sec Loss 11.0037 Epoch: 7 Global Step: 72975 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:59:19,797-Speed 130.12 samples/sec Loss 10.8737 Epoch: 7 Global Step: 72980 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:59:39,376-Speed 130.76 samples/sec Loss 10.7640 Epoch: 7 Global Step: 72985 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 09:59:59,629-Speed 126.41 samples/sec Loss 11.1131 Epoch: 7 Global Step: 72990 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:00:19,746-Speed 127.26 samples/sec Loss 11.2338 Epoch: 7 Global Step: 72995 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:00:39,268-Speed 131.14 samples/sec Loss 11.0009 Epoch: 7 Global Step: 73000 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:00:59,208-Speed 128.39 samples/sec Loss 10.6182 Epoch: 7 Global Step: 73005 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:01:18,794-Speed 130.72 samples/sec Loss 10.8883 Epoch: 7 Global Step: 73010 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:01:38,479-Speed 130.05 samples/sec Loss 11.0901 Epoch: 7 Global Step: 73015 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:01:58,117-Speed 130.37 samples/sec Loss 10.9253 Epoch: 7 Global Step: 73020 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:02:17,861-Speed 129.67 samples/sec Loss 10.9773 Epoch: 7 Global Step: 73025 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:02:37,516-Speed 130.25 samples/sec Loss 11.0008 Epoch: 7 Global Step: 73030 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:02:57,495-Speed 128.14 samples/sec Loss 10.3637 Epoch: 7 Global Step: 73035 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:03:17,501-Speed 127.97 samples/sec Loss 10.8625 Epoch: 7 Global Step: 73040 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:03:37,080-Speed 130.76 samples/sec Loss 10.8261 Epoch: 7 Global Step: 73045 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:03:57,038-Speed 128.28 samples/sec Loss 10.8343 Epoch: 7 Global Step: 73050 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:04:16,529-Speed 131.35 samples/sec Loss 10.9111 Epoch: 7 Global Step: 73055 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:04:36,535-Speed 127.97 samples/sec Loss 10.9776 Epoch: 7 Global Step: 73060 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:04:56,016-Speed 131.41 samples/sec Loss 11.0691 Epoch: 7 Global Step: 73065 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:05:15,783-Speed 129.52 samples/sec Loss 11.0597 Epoch: 7 Global Step: 73070 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:05:35,403-Speed 130.49 samples/sec Loss 10.9493 Epoch: 7 Global Step: 73075 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:05:55,593-Speed 126.80 samples/sec Loss 10.8952 Epoch: 7 Global Step: 73080 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:06:15,254-Speed 130.21 samples/sec Loss 10.5798 Epoch: 7 Global Step: 73085 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:06:34,904-Speed 130.29 samples/sec Loss 10.7467 Epoch: 7 Global Step: 73090 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:06:54,670-Speed 129.52 samples/sec Loss 10.9044 Epoch: 7 Global Step: 73095 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:07:14,718-Speed 127.70 samples/sec Loss 11.0234 Epoch: 7 Global Step: 73100 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:07:34,533-Speed 129.20 samples/sec Loss 10.5055 Epoch: 7 Global Step: 73105 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:07:54,761-Speed 126.57 samples/sec Loss 10.8481 Epoch: 7 Global Step: 73110 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:08:14,708-Speed 128.35 samples/sec Loss 10.6896 Epoch: 7 Global Step: 73115 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:08:34,262-Speed 130.92 samples/sec Loss 11.2702 Epoch: 7 Global Step: 73120 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:08:54,413-Speed 127.05 samples/sec Loss 10.6678 Epoch: 7 Global Step: 73125 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:09:14,279-Speed 128.87 samples/sec Loss 10.7864 Epoch: 7 Global Step: 73130 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:09:33,852-Speed 130.80 samples/sec Loss 11.0603 Epoch: 7 Global Step: 73135 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:09:53,670-Speed 129.18 samples/sec Loss 11.1109 Epoch: 7 Global Step: 73140 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:10:13,323-Speed 130.26 samples/sec Loss 10.6806 Epoch: 7 Global Step: 73145 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:10:32,954-Speed 130.42 samples/sec Loss 10.9440 Epoch: 7 Global Step: 73150 Fp16 Grad Scale: 8192 Required: 210 hours Training: 2021-06-10 10:10:52,597-Speed 130.34 samples/sec Loss 10.9038 Epoch: 7 Global Step: 73155 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:11:12,492-Speed 128.68 samples/sec Loss 11.0155 Epoch: 7 Global Step: 73160 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:11:32,105-Speed 130.53 samples/sec Loss 10.7171 Epoch: 7 Global Step: 73165 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:11:52,080-Speed 128.17 samples/sec Loss 11.2049 Epoch: 7 Global Step: 73170 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:12:12,130-Speed 127.69 samples/sec Loss 10.8601 Epoch: 7 Global Step: 73175 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:12:31,758-Speed 130.43 samples/sec Loss 10.9470 Epoch: 7 Global Step: 73180 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:12:51,455-Speed 129.98 samples/sec Loss 10.9492 Epoch: 7 Global Step: 73185 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:13:11,060-Speed 130.58 samples/sec Loss 10.8270 Epoch: 7 Global Step: 73190 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:13:30,826-Speed 129.52 samples/sec Loss 11.1412 Epoch: 7 Global Step: 73195 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:13:50,484-Speed 130.24 samples/sec Loss 11.2022 Epoch: 7 Global Step: 73200 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:14:10,290-Speed 129.26 samples/sec Loss 10.7911 Epoch: 7 Global Step: 73205 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:14:29,826-Speed 131.05 samples/sec Loss 10.9447 Epoch: 7 Global Step: 73210 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:14:49,763-Speed 128.41 samples/sec Loss 10.5794 Epoch: 7 Global Step: 73215 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:15:09,850-Speed 127.45 samples/sec Loss 10.8722 Epoch: 7 Global Step: 73220 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:15:29,424-Speed 130.79 samples/sec Loss 10.8142 Epoch: 7 Global Step: 73225 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:15:49,255-Speed 129.10 samples/sec Loss 10.5702 Epoch: 7 Global Step: 73230 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:16:09,743-Speed 124.95 samples/sec Loss 11.3893 Epoch: 7 Global Step: 73235 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:16:29,510-Speed 129.52 samples/sec Loss 10.9516 Epoch: 7 Global Step: 73240 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:16:49,152-Speed 130.34 samples/sec Loss 10.6271 Epoch: 7 Global Step: 73245 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:17:08,818-Speed 130.18 samples/sec Loss 10.9992 Epoch: 7 Global Step: 73250 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:17:28,435-Speed 130.50 samples/sec Loss 10.7561 Epoch: 7 Global Step: 73255 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:17:48,424-Speed 128.08 samples/sec Loss 11.1261 Epoch: 7 Global Step: 73260 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:18:08,452-Speed 127.83 samples/sec Loss 10.7173 Epoch: 7 Global Step: 73265 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:18:27,927-Speed 131.46 samples/sec Loss 10.5262 Epoch: 7 Global Step: 73270 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:18:47,793-Speed 128.87 samples/sec Loss 10.6663 Epoch: 7 Global Step: 73275 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:19:07,333-Speed 131.02 samples/sec Loss 10.7822 Epoch: 7 Global Step: 73280 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:19:27,330-Speed 128.03 samples/sec Loss 10.9596 Epoch: 7 Global Step: 73285 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:19:46,833-Speed 131.27 samples/sec Loss 10.9794 Epoch: 7 Global Step: 73290 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:20:06,656-Speed 129.16 samples/sec Loss 10.9728 Epoch: 7 Global Step: 73295 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:20:26,271-Speed 130.52 samples/sec Loss 10.7519 Epoch: 7 Global Step: 73300 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:20:46,308-Speed 127.77 samples/sec Loss 10.8105 Epoch: 7 Global Step: 73305 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:21:05,979-Speed 130.15 samples/sec Loss 10.7745 Epoch: 7 Global Step: 73310 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:21:25,699-Speed 129.82 samples/sec Loss 10.8650 Epoch: 7 Global Step: 73315 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:21:45,456-Speed 129.58 samples/sec Loss 10.7704 Epoch: 7 Global Step: 73320 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:22:05,114-Speed 130.23 samples/sec Loss 10.7746 Epoch: 7 Global Step: 73325 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:22:24,861-Speed 129.65 samples/sec Loss 11.0329 Epoch: 7 Global Step: 73330 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:22:44,379-Speed 131.17 samples/sec Loss 11.0407 Epoch: 7 Global Step: 73335 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:23:04,288-Speed 128.59 samples/sec Loss 11.1020 Epoch: 7 Global Step: 73340 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:23:23,993-Speed 129.92 samples/sec Loss 11.3278 Epoch: 7 Global Step: 73345 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:23:44,411-Speed 125.39 samples/sec Loss 11.0496 Epoch: 7 Global Step: 73350 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:24:04,135-Speed 129.80 samples/sec Loss 10.8989 Epoch: 7 Global Step: 73355 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:24:23,695-Speed 130.89 samples/sec Loss 11.3646 Epoch: 7 Global Step: 73360 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:24:43,374-Speed 130.10 samples/sec Loss 11.1563 Epoch: 7 Global Step: 73365 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:25:03,018-Speed 130.32 samples/sec Loss 11.1106 Epoch: 7 Global Step: 73370 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:25:22,671-Speed 130.27 samples/sec Loss 10.8270 Epoch: 7 Global Step: 73375 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:25:42,313-Speed 130.34 samples/sec Loss 10.6884 Epoch: 7 Global Step: 73380 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:26:02,038-Speed 129.79 samples/sec Loss 10.7593 Epoch: 7 Global Step: 73385 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:26:21,665-Speed 130.44 samples/sec Loss 10.8233 Epoch: 7 Global Step: 73390 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:26:41,824-Speed 126.99 samples/sec Loss 10.6554 Epoch: 7 Global Step: 73395 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:27:01,763-Speed 128.40 samples/sec Loss 10.9684 Epoch: 7 Global Step: 73400 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:27:21,335-Speed 130.80 samples/sec Loss 10.9775 Epoch: 7 Global Step: 73405 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:27:41,321-Speed 128.10 samples/sec Loss 10.7464 Epoch: 7 Global Step: 73410 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:28:00,877-Speed 130.92 samples/sec Loss 10.8364 Epoch: 7 Global Step: 73415 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:28:20,543-Speed 130.18 samples/sec Loss 10.7076 Epoch: 7 Global Step: 73420 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:28:40,174-Speed 130.41 samples/sec Loss 11.2113 Epoch: 7 Global Step: 73425 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:28:59,926-Speed 129.61 samples/sec Loss 10.7411 Epoch: 7 Global Step: 73430 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:29:19,702-Speed 129.46 samples/sec Loss 10.7613 Epoch: 7 Global Step: 73435 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:29:39,864-Speed 126.98 samples/sec Loss 10.3053 Epoch: 7 Global Step: 73440 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:29:59,788-Speed 128.49 samples/sec Loss 11.0741 Epoch: 7 Global Step: 73445 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:30:19,432-Speed 130.33 samples/sec Loss 11.1662 Epoch: 7 Global Step: 73450 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:30:39,407-Speed 128.16 samples/sec Loss 10.6128 Epoch: 7 Global Step: 73455 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:30:58,892-Speed 131.39 samples/sec Loss 10.9840 Epoch: 7 Global Step: 73460 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:31:18,593-Speed 129.95 samples/sec Loss 10.8414 Epoch: 7 Global Step: 73465 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:31:38,199-Speed 130.58 samples/sec Loss 11.0801 Epoch: 7 Global Step: 73470 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:31:58,154-Speed 128.30 samples/sec Loss 11.0498 Epoch: 7 Global Step: 73475 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:32:17,930-Speed 129.45 samples/sec Loss 10.7202 Epoch: 7 Global Step: 73480 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:32:37,874-Speed 128.37 samples/sec Loss 10.9819 Epoch: 7 Global Step: 73485 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:32:57,697-Speed 129.15 samples/sec Loss 10.6994 Epoch: 7 Global Step: 73490 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:33:17,276-Speed 130.75 samples/sec Loss 10.4781 Epoch: 7 Global Step: 73495 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:33:37,169-Speed 128.70 samples/sec Loss 10.9477 Epoch: 7 Global Step: 73500 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:33:56,751-Speed 130.74 samples/sec Loss 10.6453 Epoch: 7 Global Step: 73505 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:34:16,574-Speed 129.15 samples/sec Loss 10.6678 Epoch: 7 Global Step: 73510 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:34:36,229-Speed 130.25 samples/sec Loss 10.8469 Epoch: 7 Global Step: 73515 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:34:56,158-Speed 128.47 samples/sec Loss 10.9371 Epoch: 7 Global Step: 73520 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:35:15,723-Speed 130.86 samples/sec Loss 10.9280 Epoch: 7 Global Step: 73525 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:35:35,861-Speed 127.12 samples/sec Loss 11.0726 Epoch: 7 Global Step: 73530 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:35:55,636-Speed 129.47 samples/sec Loss 10.8265 Epoch: 7 Global Step: 73535 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:36:15,288-Speed 130.27 samples/sec Loss 10.8454 Epoch: 7 Global Step: 73540 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:36:35,026-Speed 129.71 samples/sec Loss 11.1160 Epoch: 7 Global Step: 73545 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:36:54,664-Speed 130.37 samples/sec Loss 10.9295 Epoch: 7 Global Step: 73550 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:37:14,494-Speed 129.10 samples/sec Loss 10.9814 Epoch: 7 Global Step: 73555 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:37:34,118-Speed 130.46 samples/sec Loss 10.6895 Epoch: 7 Global Step: 73560 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:37:53,993-Speed 128.81 samples/sec Loss 10.7427 Epoch: 7 Global Step: 73565 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:38:13,579-Speed 130.72 samples/sec Loss 10.9255 Epoch: 7 Global Step: 73570 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:38:33,595-Speed 127.90 samples/sec Loss 10.8563 Epoch: 7 Global Step: 73575 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:38:53,667-Speed 127.55 samples/sec Loss 10.3918 Epoch: 7 Global Step: 73580 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:39:13,273-Speed 130.58 samples/sec Loss 11.1985 Epoch: 7 Global Step: 73585 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:39:32,963-Speed 130.02 samples/sec Loss 10.9486 Epoch: 7 Global Step: 73590 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:39:52,571-Speed 130.59 samples/sec Loss 11.1811 Epoch: 7 Global Step: 73595 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:40:12,335-Speed 129.54 samples/sec Loss 10.9218 Epoch: 7 Global Step: 73600 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:40:31,978-Speed 130.33 samples/sec Loss 10.7212 Epoch: 7 Global Step: 73605 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:40:51,810-Speed 129.09 samples/sec Loss 10.8141 Epoch: 7 Global Step: 73610 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:41:11,305-Speed 131.32 samples/sec Loss 10.9608 Epoch: 7 Global Step: 73615 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:41:31,434-Speed 127.19 samples/sec Loss 11.0596 Epoch: 7 Global Step: 73620 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:41:51,492-Speed 127.64 samples/sec Loss 10.6667 Epoch: 7 Global Step: 73625 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:42:11,019-Speed 131.11 samples/sec Loss 10.8748 Epoch: 7 Global Step: 73630 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:42:30,699-Speed 130.09 samples/sec Loss 10.9929 Epoch: 7 Global Step: 73635 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:42:50,337-Speed 130.37 samples/sec Loss 10.7448 Epoch: 7 Global Step: 73640 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:43:10,103-Speed 129.52 samples/sec Loss 10.8508 Epoch: 7 Global Step: 73645 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:43:29,761-Speed 130.23 samples/sec Loss 10.6016 Epoch: 7 Global Step: 73650 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:43:49,402-Speed 130.35 samples/sec Loss 11.0138 Epoch: 7 Global Step: 73655 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:44:08,983-Speed 130.75 samples/sec Loss 10.8403 Epoch: 7 Global Step: 73660 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:44:29,051-Speed 127.57 samples/sec Loss 11.0783 Epoch: 7 Global Step: 73665 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:44:49,118-Speed 127.58 samples/sec Loss 11.0204 Epoch: 7 Global Step: 73670 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:45:08,592-Speed 131.46 samples/sec Loss 11.0554 Epoch: 7 Global Step: 73675 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:45:28,607-Speed 127.91 samples/sec Loss 10.9117 Epoch: 7 Global Step: 73680 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:45:48,185-Speed 130.77 samples/sec Loss 10.9926 Epoch: 7 Global Step: 73685 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:46:07,973-Speed 129.38 samples/sec Loss 10.9834 Epoch: 7 Global Step: 73690 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:46:27,579-Speed 130.58 samples/sec Loss 11.2474 Epoch: 7 Global Step: 73695 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:46:47,329-Speed 129.63 samples/sec Loss 10.8184 Epoch: 7 Global Step: 73700 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:47:06,980-Speed 130.28 samples/sec Loss 10.8116 Epoch: 7 Global Step: 73705 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:47:27,155-Speed 126.90 samples/sec Loss 11.1368 Epoch: 7 Global Step: 73710 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:47:46,872-Speed 129.85 samples/sec Loss 10.8450 Epoch: 7 Global Step: 73715 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:48:06,512-Speed 130.35 samples/sec Loss 10.7676 Epoch: 7 Global Step: 73720 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:48:26,345-Speed 129.08 samples/sec Loss 10.7178 Epoch: 7 Global Step: 73725 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:48:45,945-Speed 130.62 samples/sec Loss 10.9806 Epoch: 7 Global Step: 73730 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:49:05,876-Speed 128.45 samples/sec Loss 10.9415 Epoch: 7 Global Step: 73735 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:49:25,451-Speed 130.79 samples/sec Loss 11.2841 Epoch: 7 Global Step: 73740 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:49:45,355-Speed 128.62 samples/sec Loss 11.0942 Epoch: 7 Global Step: 73745 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:50:04,939-Speed 130.73 samples/sec Loss 10.9721 Epoch: 7 Global Step: 73750 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:50:25,257-Speed 126.00 samples/sec Loss 11.0620 Epoch: 7 Global Step: 73755 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:50:44,998-Speed 129.69 samples/sec Loss 10.7903 Epoch: 7 Global Step: 73760 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:51:04,658-Speed 130.22 samples/sec Loss 10.6289 Epoch: 7 Global Step: 73765 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:51:24,444-Speed 129.39 samples/sec Loss 10.8098 Epoch: 7 Global Step: 73770 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:51:44,038-Speed 130.66 samples/sec Loss 10.8093 Epoch: 7 Global Step: 73775 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:52:03,886-Speed 128.99 samples/sec Loss 10.8761 Epoch: 7 Global Step: 73780 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:52:23,504-Speed 130.50 samples/sec Loss 10.8136 Epoch: 7 Global Step: 73785 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:52:43,346-Speed 129.03 samples/sec Loss 10.8090 Epoch: 7 Global Step: 73790 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:53:02,928-Speed 130.74 samples/sec Loss 10.7014 Epoch: 7 Global Step: 73795 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:53:23,031-Speed 127.35 samples/sec Loss 11.0567 Epoch: 7 Global Step: 73800 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:53:43,052-Speed 127.87 samples/sec Loss 10.6835 Epoch: 7 Global Step: 73805 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:54:02,662-Speed 130.55 samples/sec Loss 11.0287 Epoch: 7 Global Step: 73810 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:54:22,426-Speed 129.54 samples/sec Loss 10.8228 Epoch: 7 Global Step: 73815 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:54:42,119-Speed 130.00 samples/sec Loss 11.1129 Epoch: 7 Global Step: 73820 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:55:01,872-Speed 129.61 samples/sec Loss 10.6326 Epoch: 7 Global Step: 73825 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:55:21,421-Speed 130.96 samples/sec Loss 10.7944 Epoch: 7 Global Step: 73830 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:55:41,303-Speed 128.77 samples/sec Loss 10.7152 Epoch: 7 Global Step: 73835 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:56:00,905-Speed 130.60 samples/sec Loss 10.7603 Epoch: 7 Global Step: 73840 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:56:20,910-Speed 127.98 samples/sec Loss 10.7286 Epoch: 7 Global Step: 73845 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:56:40,771-Speed 128.90 samples/sec Loss 11.0387 Epoch: 7 Global Step: 73850 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:57:00,375-Speed 130.59 samples/sec Loss 10.7058 Epoch: 7 Global Step: 73855 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:57:20,156-Speed 129.42 samples/sec Loss 11.0650 Epoch: 7 Global Step: 73860 Fp16 Grad Scale: 8192 Required: 209 hours Training: 2021-06-10 10:57:39,803-Speed 130.31 samples/sec Loss 10.8340 Epoch: 7 Global Step: 73865 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 10:57:59,561-Speed 129.57 samples/sec Loss 10.4409 Epoch: 7 Global Step: 73870 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 10:58:19,071-Speed 131.23 samples/sec Loss 10.7022 Epoch: 7 Global Step: 73875 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 10:58:38,984-Speed 128.57 samples/sec Loss 10.6028 Epoch: 7 Global Step: 73880 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 10:58:58,568-Speed 130.73 samples/sec Loss 10.4900 Epoch: 7 Global Step: 73885 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 10:59:18,706-Speed 127.13 samples/sec Loss 10.7391 Epoch: 7 Global Step: 73890 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 10:59:38,761-Speed 127.66 samples/sec Loss 10.6657 Epoch: 7 Global Step: 73895 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 10:59:58,365-Speed 130.59 samples/sec Loss 10.7940 Epoch: 7 Global Step: 73900 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:00:18,151-Speed 129.39 samples/sec Loss 11.0118 Epoch: 7 Global Step: 73905 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:00:37,946-Speed 129.33 samples/sec Loss 10.7481 Epoch: 7 Global Step: 73910 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:00:57,607-Speed 130.21 samples/sec Loss 10.8670 Epoch: 7 Global Step: 73915 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:01:17,267-Speed 130.22 samples/sec Loss 11.0409 Epoch: 7 Global Step: 73920 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:01:37,031-Speed 129.53 samples/sec Loss 10.9395 Epoch: 7 Global Step: 73925 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:01:56,651-Speed 130.49 samples/sec Loss 11.0231 Epoch: 7 Global Step: 73930 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:02:16,808-Speed 127.01 samples/sec Loss 10.7515 Epoch: 7 Global Step: 73935 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:02:36,869-Speed 127.62 samples/sec Loss 11.1557 Epoch: 7 Global Step: 73940 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:02:56,405-Speed 131.05 samples/sec Loss 10.5192 Epoch: 7 Global Step: 73945 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:03:16,250-Speed 129.01 samples/sec Loss 11.1353 Epoch: 7 Global Step: 73950 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:03:35,849-Speed 130.62 samples/sec Loss 10.8246 Epoch: 7 Global Step: 73955 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:03:55,570-Speed 129.82 samples/sec Loss 10.6950 Epoch: 7 Global Step: 73960 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:04:15,184-Speed 130.53 samples/sec Loss 10.8226 Epoch: 7 Global Step: 73965 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:04:34,942-Speed 129.58 samples/sec Loss 10.7583 Epoch: 7 Global Step: 73970 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:04:54,571-Speed 130.42 samples/sec Loss 10.8104 Epoch: 7 Global Step: 73975 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:05:14,595-Speed 127.85 samples/sec Loss 10.9178 Epoch: 7 Global Step: 73980 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:05:34,639-Speed 127.72 samples/sec Loss 10.7919 Epoch: 7 Global Step: 73985 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:05:54,234-Speed 130.66 samples/sec Loss 10.8661 Epoch: 7 Global Step: 73990 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:06:14,307-Speed 127.54 samples/sec Loss 10.7072 Epoch: 7 Global Step: 73995 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:06:33,870-Speed 130.87 samples/sec Loss 11.1010 Epoch: 7 Global Step: 74000 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:06:53,657-Speed 129.39 samples/sec Loss 10.7612 Epoch: 7 Global Step: 74005 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:07:13,266-Speed 130.56 samples/sec Loss 10.9470 Epoch: 7 Global Step: 74010 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:07:33,028-Speed 129.54 samples/sec Loss 10.7729 Epoch: 7 Global Step: 74015 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:07:52,674-Speed 130.32 samples/sec Loss 10.9940 Epoch: 7 Global Step: 74020 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:08:12,570-Speed 128.67 samples/sec Loss 10.7535 Epoch: 7 Global Step: 74025 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:08:32,558-Speed 128.08 samples/sec Loss 11.1906 Epoch: 7 Global Step: 74030 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:08:52,201-Speed 130.33 samples/sec Loss 10.6344 Epoch: 7 Global Step: 74035 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:09:12,112-Speed 128.58 samples/sec Loss 10.8279 Epoch: 7 Global Step: 74040 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:09:31,581-Speed 131.50 samples/sec Loss 11.0256 Epoch: 7 Global Step: 74045 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:09:51,524-Speed 128.37 samples/sec Loss 10.8540 Epoch: 7 Global Step: 74050 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:10:11,090-Speed 130.84 samples/sec Loss 10.8781 Epoch: 7 Global Step: 74055 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:10:30,952-Speed 128.90 samples/sec Loss 10.5826 Epoch: 7 Global Step: 74060 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:10:50,558-Speed 130.58 samples/sec Loss 10.8643 Epoch: 7 Global Step: 74065 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:11:10,670-Speed 127.30 samples/sec Loss 10.6733 Epoch: 7 Global Step: 74070 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:11:30,419-Speed 129.63 samples/sec Loss 10.9241 Epoch: 7 Global Step: 74075 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:11:50,029-Speed 130.55 samples/sec Loss 10.7419 Epoch: 7 Global Step: 74080 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:12:09,803-Speed 129.47 samples/sec Loss 10.9122 Epoch: 7 Global Step: 74085 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:12:29,398-Speed 130.65 samples/sec Loss 11.0139 Epoch: 7 Global Step: 74090 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:12:49,300-Speed 128.64 samples/sec Loss 10.8784 Epoch: 7 Global Step: 74095 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:13:08,961-Speed 130.22 samples/sec Loss 10.7442 Epoch: 7 Global Step: 74100 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:13:28,889-Speed 128.47 samples/sec Loss 11.0868 Epoch: 7 Global Step: 74105 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:13:48,388-Speed 131.29 samples/sec Loss 10.7471 Epoch: 7 Global Step: 74110 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:14:08,529-Speed 127.11 samples/sec Loss 10.5047 Epoch: 7 Global Step: 74115 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:14:28,350-Speed 129.16 samples/sec Loss 10.8541 Epoch: 7 Global Step: 74120 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:14:47,981-Speed 130.41 samples/sec Loss 11.2848 Epoch: 7 Global Step: 74125 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:15:07,632-Speed 130.28 samples/sec Loss 11.0292 Epoch: 7 Global Step: 74130 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:15:27,214-Speed 130.74 samples/sec Loss 11.1082 Epoch: 7 Global Step: 74135 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:15:47,100-Speed 128.74 samples/sec Loss 10.6468 Epoch: 7 Global Step: 74140 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:16:06,714-Speed 130.52 samples/sec Loss 10.8573 Epoch: 7 Global Step: 74145 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:16:26,527-Speed 129.22 samples/sec Loss 10.8875 Epoch: 7 Global Step: 74150 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:16:46,124-Speed 130.64 samples/sec Loss 10.7403 Epoch: 7 Global Step: 74155 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:17:06,272-Speed 127.06 samples/sec Loss 10.8433 Epoch: 7 Global Step: 74160 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:17:26,196-Speed 128.50 samples/sec Loss 10.9363 Epoch: 7 Global Step: 74165 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:17:45,830-Speed 130.39 samples/sec Loss 10.6507 Epoch: 7 Global Step: 74170 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:18:05,714-Speed 128.75 samples/sec Loss 10.9216 Epoch: 7 Global Step: 74175 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:18:25,365-Speed 130.28 samples/sec Loss 10.8197 Epoch: 7 Global Step: 74180 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:18:45,170-Speed 129.27 samples/sec Loss 10.8382 Epoch: 7 Global Step: 74185 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:19:04,764-Speed 130.66 samples/sec Loss 10.9156 Epoch: 7 Global Step: 74190 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:19:24,546-Speed 129.42 samples/sec Loss 10.8245 Epoch: 7 Global Step: 74195 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:19:44,137-Speed 130.68 samples/sec Loss 10.7629 Epoch: 7 Global Step: 74200 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:20:04,157-Speed 127.87 samples/sec Loss 11.0563 Epoch: 7 Global Step: 74205 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:20:24,077-Speed 128.52 samples/sec Loss 10.8630 Epoch: 7 Global Step: 74210 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:20:43,655-Speed 130.77 samples/sec Loss 11.1044 Epoch: 7 Global Step: 74215 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:21:03,378-Speed 129.80 samples/sec Loss 11.1276 Epoch: 7 Global Step: 74220 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:21:23,023-Speed 130.33 samples/sec Loss 10.7077 Epoch: 7 Global Step: 74225 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:21:42,762-Speed 129.69 samples/sec Loss 10.8510 Epoch: 7 Global Step: 74230 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:22:02,275-Speed 131.21 samples/sec Loss 10.7750 Epoch: 7 Global Step: 74235 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:22:22,169-Speed 128.69 samples/sec Loss 10.4581 Epoch: 7 Global Step: 74240 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:22:41,745-Speed 130.78 samples/sec Loss 10.9709 Epoch: 7 Global Step: 74245 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:23:01,937-Speed 126.79 samples/sec Loss 10.9829 Epoch: 7 Global Step: 74250 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:23:21,816-Speed 128.79 samples/sec Loss 11.0118 Epoch: 7 Global Step: 74255 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:23:41,414-Speed 130.63 samples/sec Loss 10.5753 Epoch: 7 Global Step: 74260 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:24:01,090-Speed 130.12 samples/sec Loss 10.7212 Epoch: 7 Global Step: 74265 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:24:20,963-Speed 128.82 samples/sec Loss 10.8502 Epoch: 7 Global Step: 74270 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:24:40,622-Speed 130.23 samples/sec Loss 11.3143 Epoch: 7 Global Step: 74275 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:25:00,388-Speed 129.52 samples/sec Loss 11.2944 Epoch: 7 Global Step: 74280 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:25:20,149-Speed 129.56 samples/sec Loss 10.9647 Epoch: 7 Global Step: 74285 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:25:39,736-Speed 130.70 samples/sec Loss 10.9081 Epoch: 7 Global Step: 74290 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:25:59,743-Speed 127.96 samples/sec Loss 11.2309 Epoch: 7 Global Step: 74295 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:26:19,662-Speed 128.53 samples/sec Loss 10.8274 Epoch: 7 Global Step: 74300 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:26:39,241-Speed 130.76 samples/sec Loss 11.0555 Epoch: 7 Global Step: 74305 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:26:59,238-Speed 128.03 samples/sec Loss 10.6434 Epoch: 7 Global Step: 74310 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:27:18,812-Speed 130.79 samples/sec Loss 10.7142 Epoch: 7 Global Step: 74315 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:27:38,636-Speed 129.15 samples/sec Loss 10.7660 Epoch: 7 Global Step: 74320 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:27:58,223-Speed 130.71 samples/sec Loss 10.9149 Epoch: 7 Global Step: 74325 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:28:17,892-Speed 130.16 samples/sec Loss 10.9015 Epoch: 7 Global Step: 74330 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:28:37,536-Speed 130.33 samples/sec Loss 11.1683 Epoch: 7 Global Step: 74335 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:28:57,548-Speed 127.93 samples/sec Loss 10.6113 Epoch: 7 Global Step: 74340 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:29:17,226-Speed 130.10 samples/sec Loss 10.6980 Epoch: 7 Global Step: 74345 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:29:36,771-Speed 130.99 samples/sec Loss 11.1033 Epoch: 7 Global Step: 74350 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:29:56,645-Speed 128.82 samples/sec Loss 11.0640 Epoch: 7 Global Step: 74355 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:30:16,253-Speed 130.56 samples/sec Loss 10.8517 Epoch: 7 Global Step: 74360 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:30:36,198-Speed 128.36 samples/sec Loss 11.2233 Epoch: 7 Global Step: 74365 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:30:55,782-Speed 130.72 samples/sec Loss 10.8040 Epoch: 7 Global Step: 74370 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:31:15,771-Speed 128.08 samples/sec Loss 10.8558 Epoch: 7 Global Step: 74375 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:31:35,352-Speed 130.75 samples/sec Loss 10.8719 Epoch: 7 Global Step: 74380 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:31:55,455-Speed 127.35 samples/sec Loss 10.9301 Epoch: 7 Global Step: 74385 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:32:15,180-Speed 129.79 samples/sec Loss 10.8708 Epoch: 7 Global Step: 74390 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:32:34,829-Speed 130.29 samples/sec Loss 11.0098 Epoch: 7 Global Step: 74395 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:32:54,565-Speed 129.72 samples/sec Loss 10.7675 Epoch: 7 Global Step: 74400 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:33:14,074-Speed 131.23 samples/sec Loss 10.9181 Epoch: 7 Global Step: 74405 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:33:33,982-Speed 128.60 samples/sec Loss 10.5799 Epoch: 7 Global Step: 74410 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:33:53,612-Speed 130.42 samples/sec Loss 10.9497 Epoch: 7 Global Step: 74415 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:34:13,464-Speed 128.96 samples/sec Loss 10.8899 Epoch: 7 Global Step: 74420 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:34:33,025-Speed 130.87 samples/sec Loss 10.8596 Epoch: 7 Global Step: 74425 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:34:53,062-Speed 127.77 samples/sec Loss 11.1927 Epoch: 7 Global Step: 74430 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:35:12,928-Speed 128.87 samples/sec Loss 11.0183 Epoch: 7 Global Step: 74435 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:35:32,817-Speed 128.72 samples/sec Loss 10.5502 Epoch: 7 Global Step: 74440 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:35:52,560-Speed 129.67 samples/sec Loss 10.8182 Epoch: 7 Global Step: 74445 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:36:12,132-Speed 130.81 samples/sec Loss 10.7691 Epoch: 7 Global Step: 74450 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:36:32,008-Speed 128.80 samples/sec Loss 11.0526 Epoch: 7 Global Step: 74455 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:36:51,911-Speed 128.63 samples/sec Loss 11.0004 Epoch: 7 Global Step: 74460 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:37:11,759-Speed 128.99 samples/sec Loss 10.7867 Epoch: 7 Global Step: 74465 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:37:31,248-Speed 131.37 samples/sec Loss 10.7628 Epoch: 7 Global Step: 74470 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:37:51,388-Speed 127.11 samples/sec Loss 10.9073 Epoch: 7 Global Step: 74475 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:38:11,405-Speed 127.90 samples/sec Loss 11.0825 Epoch: 7 Global Step: 74480 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:38:30,986-Speed 130.75 samples/sec Loss 11.0634 Epoch: 7 Global Step: 74485 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:38:50,639-Speed 130.27 samples/sec Loss 10.9645 Epoch: 7 Global Step: 74490 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:39:10,285-Speed 130.31 samples/sec Loss 11.0604 Epoch: 7 Global Step: 74495 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:39:30,111-Speed 129.13 samples/sec Loss 10.7739 Epoch: 7 Global Step: 74500 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:39:50,048-Speed 128.41 samples/sec Loss 10.8364 Epoch: 7 Global Step: 74505 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:40:09,886-Speed 129.06 samples/sec Loss 10.7453 Epoch: 7 Global Step: 74510 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:40:29,478-Speed 130.67 samples/sec Loss 11.1114 Epoch: 7 Global Step: 74515 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:40:49,618-Speed 127.12 samples/sec Loss 10.9010 Epoch: 7 Global Step: 74520 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:41:09,737-Speed 127.25 samples/sec Loss 10.6112 Epoch: 7 Global Step: 74525 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:41:29,211-Speed 131.46 samples/sec Loss 10.9142 Epoch: 7 Global Step: 74530 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:41:49,002-Speed 129.36 samples/sec Loss 10.8221 Epoch: 7 Global Step: 74535 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:42:08,627-Speed 130.45 samples/sec Loss 11.0039 Epoch: 7 Global Step: 74540 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:42:28,295-Speed 130.17 samples/sec Loss 10.7960 Epoch: 7 Global Step: 74545 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:42:47,949-Speed 130.26 samples/sec Loss 11.1088 Epoch: 7 Global Step: 74550 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:43:07,591-Speed 130.34 samples/sec Loss 10.9909 Epoch: 7 Global Step: 74555 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:43:27,235-Speed 130.33 samples/sec Loss 10.8410 Epoch: 7 Global Step: 74560 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:43:47,483-Speed 126.44 samples/sec Loss 10.9443 Epoch: 7 Global Step: 74565 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:44:07,320-Speed 129.06 samples/sec Loss 10.8890 Epoch: 7 Global Step: 74570 Fp16 Grad Scale: 8192 Required: 208 hours Training: 2021-06-10 11:44:26,890-Speed 130.82 samples/sec Loss 10.7265 Epoch: 7 Global Step: 74575 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:44:46,788-Speed 128.66 samples/sec Loss 10.7189 Epoch: 7 Global Step: 74580 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:45:06,423-Speed 130.38 samples/sec Loss 10.6725 Epoch: 7 Global Step: 74585 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:45:26,235-Speed 129.23 samples/sec Loss 10.7966 Epoch: 7 Global Step: 74590 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:45:45,839-Speed 130.59 samples/sec Loss 10.8468 Epoch: 7 Global Step: 74595 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:46:05,599-Speed 129.56 samples/sec Loss 10.8772 Epoch: 7 Global Step: 74600 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:46:25,424-Speed 129.14 samples/sec Loss 10.9131 Epoch: 7 Global Step: 74605 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:46:45,131-Speed 129.92 samples/sec Loss 10.9884 Epoch: 7 Global Step: 74610 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:47:04,982-Speed 128.97 samples/sec Loss 11.0140 Epoch: 7 Global Step: 74615 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:47:24,558-Speed 130.77 samples/sec Loss 10.7692 Epoch: 7 Global Step: 74620 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:47:44,940-Speed 125.61 samples/sec Loss 10.4112 Epoch: 7 Global Step: 74625 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:48:04,501-Speed 130.88 samples/sec Loss 10.8027 Epoch: 7 Global Step: 74630 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:48:24,178-Speed 130.11 samples/sec Loss 11.2715 Epoch: 7 Global Step: 74635 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:48:43,778-Speed 130.62 samples/sec Loss 10.5631 Epoch: 7 Global Step: 74640 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:49:03,566-Speed 129.37 samples/sec Loss 10.7964 Epoch: 7 Global Step: 74645 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:49:23,185-Speed 130.50 samples/sec Loss 11.0798 Epoch: 7 Global Step: 74650 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:49:43,312-Speed 127.19 samples/sec Loss 10.9740 Epoch: 7 Global Step: 74655 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:50:03,073-Speed 129.55 samples/sec Loss 10.6011 Epoch: 7 Global Step: 74660 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:50:22,718-Speed 130.32 samples/sec Loss 11.2307 Epoch: 7 Global Step: 74665 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:50:42,410-Speed 130.01 samples/sec Loss 10.8286 Epoch: 7 Global Step: 74670 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:51:01,987-Speed 130.78 samples/sec Loss 10.9048 Epoch: 7 Global Step: 74675 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:51:21,819-Speed 129.09 samples/sec Loss 10.8980 Epoch: 7 Global Step: 74680 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:51:41,405-Speed 130.71 samples/sec Loss 10.8294 Epoch: 7 Global Step: 74685 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:52:01,373-Speed 128.21 samples/sec Loss 10.9958 Epoch: 7 Global Step: 74690 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:52:20,970-Speed 130.64 samples/sec Loss 10.8088 Epoch: 7 Global Step: 74695 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:52:40,985-Speed 127.91 samples/sec Loss 10.6018 Epoch: 7 Global Step: 74700 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:53:00,754-Speed 129.50 samples/sec Loss 10.7746 Epoch: 7 Global Step: 74705 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:53:20,327-Speed 130.80 samples/sec Loss 10.9032 Epoch: 7 Global Step: 74710 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:53:40,144-Speed 129.19 samples/sec Loss 10.8586 Epoch: 7 Global Step: 74715 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:53:59,638-Speed 131.33 samples/sec Loss 10.9045 Epoch: 7 Global Step: 74720 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:54:19,752-Speed 127.28 samples/sec Loss 10.4969 Epoch: 7 Global Step: 74725 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:54:39,402-Speed 130.28 samples/sec Loss 11.0905 Epoch: 7 Global Step: 74730 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:54:59,326-Speed 128.50 samples/sec Loss 10.8256 Epoch: 7 Global Step: 74735 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:55:18,902-Speed 130.78 samples/sec Loss 10.8454 Epoch: 7 Global Step: 74740 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:55:38,918-Speed 127.90 samples/sec Loss 10.8454 Epoch: 7 Global Step: 74745 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:55:58,752-Speed 129.08 samples/sec Loss 10.8460 Epoch: 7 Global Step: 74750 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:56:18,396-Speed 130.32 samples/sec Loss 10.7529 Epoch: 7 Global Step: 74755 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:56:38,147-Speed 129.63 samples/sec Loss 10.8621 Epoch: 7 Global Step: 74760 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:56:57,797-Speed 130.28 samples/sec Loss 10.9941 Epoch: 7 Global Step: 74765 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:57:17,560-Speed 129.54 samples/sec Loss 10.8754 Epoch: 7 Global Step: 74770 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:57:37,137-Speed 130.77 samples/sec Loss 10.5804 Epoch: 7 Global Step: 74775 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:57:56,912-Speed 129.46 samples/sec Loss 11.0861 Epoch: 7 Global Step: 74780 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:58:16,478-Speed 130.85 samples/sec Loss 10.9995 Epoch: 7 Global Step: 74785 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:58:36,608-Speed 127.18 samples/sec Loss 10.6324 Epoch: 7 Global Step: 74790 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:58:56,728-Speed 127.24 samples/sec Loss 11.1860 Epoch: 7 Global Step: 74795 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:59:16,237-Speed 131.23 samples/sec Loss 11.0538 Epoch: 7 Global Step: 74800 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:59:35,971-Speed 129.73 samples/sec Loss 10.6328 Epoch: 7 Global Step: 74805 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 11:59:55,549-Speed 130.77 samples/sec Loss 11.0180 Epoch: 7 Global Step: 74810 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:00:15,377-Speed 129.12 samples/sec Loss 10.7506 Epoch: 7 Global Step: 74815 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:00:35,026-Speed 130.29 samples/sec Loss 11.0533 Epoch: 7 Global Step: 74820 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:00:54,744-Speed 129.84 samples/sec Loss 10.5579 Epoch: 7 Global Step: 74825 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:01:14,308-Speed 130.86 samples/sec Loss 10.8600 Epoch: 7 Global Step: 74830 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:01:34,421-Speed 127.29 samples/sec Loss 10.8839 Epoch: 7 Global Step: 74835 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:01:54,332-Speed 128.57 samples/sec Loss 10.7525 Epoch: 7 Global Step: 74840 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:02:13,887-Speed 130.92 samples/sec Loss 10.5575 Epoch: 7 Global Step: 74845 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:02:33,811-Speed 128.50 samples/sec Loss 10.9693 Epoch: 7 Global Step: 74850 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:02:53,466-Speed 130.25 samples/sec Loss 10.7150 Epoch: 7 Global Step: 74855 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:03:13,250-Speed 129.41 samples/sec Loss 10.4540 Epoch: 7 Global Step: 74860 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:03:32,860-Speed 130.56 samples/sec Loss 10.7768 Epoch: 7 Global Step: 74865 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:03:52,552-Speed 130.01 samples/sec Loss 10.7741 Epoch: 7 Global Step: 74870 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:04:12,204-Speed 130.27 samples/sec Loss 10.4204 Epoch: 7 Global Step: 74875 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:04:32,244-Speed 127.76 samples/sec Loss 10.9820 Epoch: 7 Global Step: 74880 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:04:52,213-Speed 128.20 samples/sec Loss 10.6682 Epoch: 7 Global Step: 74885 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:05:11,703-Speed 131.36 samples/sec Loss 11.0767 Epoch: 7 Global Step: 74890 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:05:31,657-Speed 128.30 samples/sec Loss 11.2029 Epoch: 7 Global Step: 74895 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:05:51,246-Speed 130.69 samples/sec Loss 10.7871 Epoch: 7 Global Step: 74900 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:06:10,939-Speed 130.00 samples/sec Loss 10.8973 Epoch: 7 Global Step: 74905 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:06:30,551-Speed 130.54 samples/sec Loss 10.6790 Epoch: 7 Global Step: 74910 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:06:50,272-Speed 129.82 samples/sec Loss 10.9473 Epoch: 7 Global Step: 74915 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:07:09,975-Speed 129.93 samples/sec Loss 10.7256 Epoch: 7 Global Step: 74920 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:07:29,999-Speed 127.85 samples/sec Loss 10.9574 Epoch: 7 Global Step: 74925 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:07:49,847-Speed 128.99 samples/sec Loss 11.2860 Epoch: 7 Global Step: 74930 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:08:09,428-Speed 130.74 samples/sec Loss 10.9329 Epoch: 7 Global Step: 74935 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:08:29,370-Speed 128.38 samples/sec Loss 10.7031 Epoch: 7 Global Step: 74940 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:08:48,818-Speed 131.65 samples/sec Loss 10.6490 Epoch: 7 Global Step: 74945 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:09:09,331-Speed 124.80 samples/sec Loss 10.7751 Epoch: 7 Global Step: 74950 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:09:28,890-Speed 130.90 samples/sec Loss 10.9931 Epoch: 7 Global Step: 74955 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:09:48,755-Speed 128.88 samples/sec Loss 10.8679 Epoch: 7 Global Step: 74960 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:10:08,370-Speed 130.51 samples/sec Loss 10.8766 Epoch: 7 Global Step: 74965 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:10:28,375-Speed 127.98 samples/sec Loss 10.7075 Epoch: 7 Global Step: 74970 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:10:48,060-Speed 130.05 samples/sec Loss 10.6151 Epoch: 7 Global Step: 74975 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:11:07,666-Speed 130.58 samples/sec Loss 11.0249 Epoch: 7 Global Step: 74980 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:11:27,509-Speed 129.02 samples/sec Loss 11.0997 Epoch: 7 Global Step: 74985 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:11:47,114-Speed 130.59 samples/sec Loss 11.1051 Epoch: 7 Global Step: 74990 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:12:07,126-Speed 127.93 samples/sec Loss 10.6970 Epoch: 7 Global Step: 74995 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:12:26,789-Speed 130.20 samples/sec Loss 10.7891 Epoch: 7 Global Step: 75000 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:12:46,726-Speed 128.41 samples/sec Loss 11.0704 Epoch: 7 Global Step: 75005 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:13:06,199-Speed 131.47 samples/sec Loss 10.7645 Epoch: 7 Global Step: 75010 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:13:26,226-Speed 127.83 samples/sec Loss 10.5846 Epoch: 7 Global Step: 75015 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:13:46,042-Speed 129.20 samples/sec Loss 11.2960 Epoch: 7 Global Step: 75020 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:14:05,709-Speed 130.17 samples/sec Loss 10.8046 Epoch: 7 Global Step: 75025 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:14:25,367-Speed 130.24 samples/sec Loss 10.8725 Epoch: 7 Global Step: 75030 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:14:45,013-Speed 130.31 samples/sec Loss 10.6399 Epoch: 7 Global Step: 75035 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:15:04,779-Speed 129.52 samples/sec Loss 11.0053 Epoch: 7 Global Step: 75040 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:15:24,442-Speed 130.20 samples/sec Loss 10.8659 Epoch: 7 Global Step: 75045 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:15:44,340-Speed 128.66 samples/sec Loss 10.9579 Epoch: 7 Global Step: 75050 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:16:03,823-Speed 131.40 samples/sec Loss 10.8578 Epoch: 7 Global Step: 75055 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:16:23,953-Speed 127.18 samples/sec Loss 10.8910 Epoch: 7 Global Step: 75060 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:16:43,907-Speed 128.30 samples/sec Loss 10.9838 Epoch: 7 Global Step: 75065 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:17:03,529-Speed 130.47 samples/sec Loss 10.9259 Epoch: 7 Global Step: 75070 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:17:23,227-Speed 129.97 samples/sec Loss 10.7025 Epoch: 7 Global Step: 75075 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:17:42,837-Speed 130.55 samples/sec Loss 10.7816 Epoch: 7 Global Step: 75080 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:18:02,666-Speed 129.11 samples/sec Loss 10.8646 Epoch: 7 Global Step: 75085 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:18:22,320-Speed 130.26 samples/sec Loss 10.7694 Epoch: 7 Global Step: 75090 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:18:42,081-Speed 129.56 samples/sec Loss 10.8112 Epoch: 7 Global Step: 75095 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:19:01,640-Speed 130.89 samples/sec Loss 10.6320 Epoch: 7 Global Step: 75100 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:19:21,721-Speed 127.50 samples/sec Loss 10.6617 Epoch: 7 Global Step: 75105 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:19:41,861-Speed 127.11 samples/sec Loss 10.7131 Epoch: 7 Global Step: 75110 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:20:01,323-Speed 131.55 samples/sec Loss 10.8747 Epoch: 7 Global Step: 75115 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:20:21,184-Speed 128.91 samples/sec Loss 11.1918 Epoch: 7 Global Step: 75120 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:20:40,814-Speed 130.42 samples/sec Loss 10.8460 Epoch: 7 Global Step: 75125 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:21:00,747-Speed 128.44 samples/sec Loss 10.8899 Epoch: 7 Global Step: 75130 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:21:20,356-Speed 130.56 samples/sec Loss 10.6770 Epoch: 7 Global Step: 75135 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:21:39,986-Speed 130.42 samples/sec Loss 10.9981 Epoch: 7 Global Step: 75140 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:21:59,640-Speed 130.26 samples/sec Loss 11.0175 Epoch: 7 Global Step: 75145 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:22:19,658-Speed 127.89 samples/sec Loss 10.6307 Epoch: 7 Global Step: 75150 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:22:39,600-Speed 128.38 samples/sec Loss 10.9846 Epoch: 7 Global Step: 75155 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:22:59,060-Speed 131.56 samples/sec Loss 10.8104 Epoch: 7 Global Step: 75160 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:23:18,978-Speed 128.53 samples/sec Loss 10.7656 Epoch: 7 Global Step: 75165 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:23:38,561-Speed 130.73 samples/sec Loss 11.0221 Epoch: 7 Global Step: 75170 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:23:58,592-Speed 127.81 samples/sec Loss 10.8535 Epoch: 7 Global Step: 75175 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:24:18,071-Speed 131.43 samples/sec Loss 11.0028 Epoch: 7 Global Step: 75180 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:24:37,769-Speed 129.97 samples/sec Loss 10.5082 Epoch: 7 Global Step: 75185 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:24:57,388-Speed 130.50 samples/sec Loss 10.9191 Epoch: 7 Global Step: 75190 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:25:17,534-Speed 127.07 samples/sec Loss 10.8907 Epoch: 7 Global Step: 75195 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:25:37,177-Speed 130.33 samples/sec Loss 10.7474 Epoch: 7 Global Step: 75200 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:25:56,670-Speed 131.34 samples/sec Loss 10.6885 Epoch: 7 Global Step: 75205 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:26:16,623-Speed 128.31 samples/sec Loss 10.3986 Epoch: 7 Global Step: 75210 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:26:36,207-Speed 130.73 samples/sec Loss 10.9057 Epoch: 7 Global Step: 75215 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:26:56,021-Speed 129.21 samples/sec Loss 10.7543 Epoch: 7 Global Step: 75220 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:27:15,499-Speed 131.44 samples/sec Loss 10.6370 Epoch: 7 Global Step: 75225 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:27:35,440-Speed 128.39 samples/sec Loss 10.6740 Epoch: 7 Global Step: 75230 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:27:55,103-Speed 130.20 samples/sec Loss 10.8088 Epoch: 7 Global Step: 75235 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:28:15,222-Speed 127.25 samples/sec Loss 10.7268 Epoch: 7 Global Step: 75240 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:28:34,862-Speed 130.35 samples/sec Loss 11.2712 Epoch: 7 Global Step: 75245 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:28:54,516-Speed 130.26 samples/sec Loss 10.8840 Epoch: 7 Global Step: 75250 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:29:14,274-Speed 129.57 samples/sec Loss 10.7435 Epoch: 7 Global Step: 75255 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:29:33,913-Speed 130.36 samples/sec Loss 10.6306 Epoch: 7 Global Step: 75260 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:29:53,601-Speed 130.04 samples/sec Loss 10.6059 Epoch: 7 Global Step: 75265 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:30:13,305-Speed 129.93 samples/sec Loss 11.1274 Epoch: 7 Global Step: 75270 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:30:33,295-Speed 128.07 samples/sec Loss 11.2042 Epoch: 7 Global Step: 75275 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:30:52,881-Speed 130.72 samples/sec Loss 10.9229 Epoch: 7 Global Step: 75280 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:31:12,936-Speed 127.65 samples/sec Loss 10.5195 Epoch: 7 Global Step: 75285 Fp16 Grad Scale: 8192 Required: 207 hours Training: 2021-06-10 12:31:32,741-Speed 129.27 samples/sec Loss 11.0525 Epoch: 7 Global Step: 75290 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:31:52,370-Speed 130.43 samples/sec Loss 11.0393 Epoch: 7 Global Step: 75295 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:32:12,145-Speed 129.47 samples/sec Loss 11.1358 Epoch: 7 Global Step: 75300 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:32:31,798-Speed 130.27 samples/sec Loss 10.8793 Epoch: 7 Global Step: 75305 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:32:51,460-Speed 130.21 samples/sec Loss 10.8043 Epoch: 7 Global Step: 75310 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:33:11,096-Speed 130.38 samples/sec Loss 11.1492 Epoch: 7 Global Step: 75315 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:33:30,867-Speed 129.49 samples/sec Loss 10.7643 Epoch: 7 Global Step: 75320 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:33:50,441-Speed 130.80 samples/sec Loss 10.9676 Epoch: 7 Global Step: 75325 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:34:10,419-Speed 128.14 samples/sec Loss 10.7873 Epoch: 7 Global Step: 75330 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:34:30,541-Speed 127.23 samples/sec Loss 10.6697 Epoch: 7 Global Step: 75335 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:34:50,115-Speed 130.80 samples/sec Loss 10.8762 Epoch: 7 Global Step: 75340 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:35:10,001-Speed 128.74 samples/sec Loss 10.5486 Epoch: 7 Global Step: 75345 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:35:29,464-Speed 131.54 samples/sec Loss 10.9064 Epoch: 7 Global Step: 75350 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:35:49,267-Speed 129.28 samples/sec Loss 10.7970 Epoch: 7 Global Step: 75355 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:36:08,905-Speed 130.37 samples/sec Loss 10.8711 Epoch: 7 Global Step: 75360 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:36:28,662-Speed 129.58 samples/sec Loss 10.7507 Epoch: 7 Global Step: 75365 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:36:48,283-Speed 130.48 samples/sec Loss 10.9696 Epoch: 7 Global Step: 75370 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:37:08,338-Speed 127.65 samples/sec Loss 10.7178 Epoch: 7 Global Step: 75375 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:37:28,082-Speed 129.66 samples/sec Loss 11.0437 Epoch: 7 Global Step: 75380 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:37:47,674-Speed 130.68 samples/sec Loss 10.9214 Epoch: 7 Global Step: 75385 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:38:07,461-Speed 129.38 samples/sec Loss 10.9208 Epoch: 7 Global Step: 75390 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:38:27,036-Speed 130.79 samples/sec Loss 11.1466 Epoch: 7 Global Step: 75395 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:38:47,019-Speed 128.11 samples/sec Loss 10.5408 Epoch: 7 Global Step: 75400 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:39:06,586-Speed 130.84 samples/sec Loss 10.6615 Epoch: 7 Global Step: 75405 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:39:26,530-Speed 128.37 samples/sec Loss 10.7193 Epoch: 7 Global Step: 75410 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:39:46,019-Speed 131.37 samples/sec Loss 10.8343 Epoch: 7 Global Step: 75415 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:40:06,088-Speed 127.57 samples/sec Loss 10.7226 Epoch: 7 Global Step: 75420 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:40:25,869-Speed 129.42 samples/sec Loss 10.9948 Epoch: 7 Global Step: 75425 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:40:45,645-Speed 129.45 samples/sec Loss 10.5479 Epoch: 7 Global Step: 75430 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:41:05,380-Speed 129.73 samples/sec Loss 11.0132 Epoch: 7 Global Step: 75435 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:41:24,887-Speed 131.24 samples/sec Loss 11.0002 Epoch: 7 Global Step: 75440 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:41:44,857-Speed 128.20 samples/sec Loss 11.0071 Epoch: 7 Global Step: 75445 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:42:04,489-Speed 130.41 samples/sec Loss 10.8814 Epoch: 7 Global Step: 75450 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:42:24,552-Speed 127.61 samples/sec Loss 10.8098 Epoch: 7 Global Step: 75455 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:42:44,144-Speed 130.67 samples/sec Loss 10.9216 Epoch: 7 Global Step: 75460 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:43:04,178-Speed 127.79 samples/sec Loss 10.9504 Epoch: 7 Global Step: 75465 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:43:24,102-Speed 128.49 samples/sec Loss 10.6556 Epoch: 7 Global Step: 75470 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:43:43,698-Speed 130.64 samples/sec Loss 11.1005 Epoch: 7 Global Step: 75475 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:44:03,410-Speed 129.88 samples/sec Loss 10.7220 Epoch: 7 Global Step: 75480 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:44:23,060-Speed 130.29 samples/sec Loss 10.6477 Epoch: 7 Global Step: 75485 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:44:42,910-Speed 128.97 samples/sec Loss 11.1127 Epoch: 7 Global Step: 75490 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:45:02,552-Speed 130.34 samples/sec Loss 11.0129 Epoch: 7 Global Step: 75495 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:45:22,296-Speed 129.67 samples/sec Loss 11.1148 Epoch: 7 Global Step: 75500 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:45:41,803-Speed 131.25 samples/sec Loss 10.5684 Epoch: 7 Global Step: 75505 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:46:01,960-Speed 127.00 samples/sec Loss 10.6836 Epoch: 7 Global Step: 75510 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:46:21,923-Speed 128.25 samples/sec Loss 10.9689 Epoch: 7 Global Step: 75515 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:46:41,498-Speed 130.78 samples/sec Loss 10.5205 Epoch: 7 Global Step: 75520 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:47:01,315-Speed 129.19 samples/sec Loss 10.6749 Epoch: 7 Global Step: 75525 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:47:20,952-Speed 130.37 samples/sec Loss 10.7846 Epoch: 7 Global Step: 75530 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:47:40,678-Speed 129.79 samples/sec Loss 10.9778 Epoch: 7 Global Step: 75535 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:48:00,301-Speed 130.46 samples/sec Loss 10.5019 Epoch: 7 Global Step: 75540 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:48:20,046-Speed 129.66 samples/sec Loss 10.7080 Epoch: 7 Global Step: 75545 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:48:39,681-Speed 130.39 samples/sec Loss 10.8060 Epoch: 7 Global Step: 75550 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:48:59,703-Speed 127.87 samples/sec Loss 10.9389 Epoch: 7 Global Step: 75555 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:49:19,583-Speed 128.78 samples/sec Loss 10.9807 Epoch: 7 Global Step: 75560 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:49:39,053-Speed 131.49 samples/sec Loss 10.8401 Epoch: 7 Global Step: 75565 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:49:59,041-Speed 128.09 samples/sec Loss 10.9312 Epoch: 7 Global Step: 75570 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:50:18,746-Speed 129.92 samples/sec Loss 10.9820 Epoch: 7 Global Step: 75575 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:50:38,580-Speed 129.08 samples/sec Loss 11.1631 Epoch: 7 Global Step: 75580 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:50:58,152-Speed 130.80 samples/sec Loss 10.8726 Epoch: 7 Global Step: 75585 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:51:17,933-Speed 129.42 samples/sec Loss 10.7483 Epoch: 7 Global Step: 75590 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:51:37,556-Speed 130.47 samples/sec Loss 10.8792 Epoch: 7 Global Step: 75595 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:51:57,680-Speed 127.22 samples/sec Loss 10.6514 Epoch: 7 Global Step: 75600 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:52:17,571-Speed 128.70 samples/sec Loss 10.7481 Epoch: 7 Global Step: 75605 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:52:37,190-Speed 130.49 samples/sec Loss 10.8162 Epoch: 7 Global Step: 75610 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:52:57,090-Speed 128.65 samples/sec Loss 10.9037 Epoch: 7 Global Step: 75615 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:53:16,668-Speed 130.76 samples/sec Loss 10.5067 Epoch: 7 Global Step: 75620 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:53:36,543-Speed 128.82 samples/sec Loss 10.6626 Epoch: 7 Global Step: 75625 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:53:56,075-Speed 131.07 samples/sec Loss 10.8237 Epoch: 7 Global Step: 75630 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:54:16,061-Speed 128.10 samples/sec Loss 10.8910 Epoch: 7 Global Step: 75635 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:54:35,649-Speed 130.70 samples/sec Loss 11.4721 Epoch: 7 Global Step: 75640 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:54:55,774-Speed 127.21 samples/sec Loss 10.8795 Epoch: 7 Global Step: 75645 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:55:15,413-Speed 130.36 samples/sec Loss 10.9865 Epoch: 7 Global Step: 75650 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:55:35,075-Speed 130.20 samples/sec Loss 10.7291 Epoch: 7 Global Step: 75655 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:55:54,826-Speed 129.63 samples/sec Loss 10.7170 Epoch: 7 Global Step: 75660 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:56:14,369-Speed 131.00 samples/sec Loss 10.4447 Epoch: 7 Global Step: 75665 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:56:34,311-Speed 128.38 samples/sec Loss 10.8020 Epoch: 7 Global Step: 75670 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:56:53,839-Speed 131.11 samples/sec Loss 10.8049 Epoch: 7 Global Step: 75675 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:57:13,779-Speed 128.40 samples/sec Loss 11.3632 Epoch: 7 Global Step: 75680 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:57:33,363-Speed 130.72 samples/sec Loss 10.6569 Epoch: 7 Global Step: 75685 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:57:53,510-Speed 127.07 samples/sec Loss 10.8114 Epoch: 7 Global Step: 75690 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:58:13,451-Speed 128.39 samples/sec Loss 11.0619 Epoch: 7 Global Step: 75695 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:58:33,061-Speed 130.55 samples/sec Loss 11.1668 Epoch: 7 Global Step: 75700 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:58:53,647-Speed 124.37 samples/sec Loss 10.8045 Epoch: 7 Global Step: 75705 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:59:13,290-Speed 130.33 samples/sec Loss 10.7297 Epoch: 7 Global Step: 75710 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:59:32,937-Speed 130.30 samples/sec Loss 10.7422 Epoch: 7 Global Step: 75715 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 12:59:52,576-Speed 130.36 samples/sec Loss 11.0140 Epoch: 7 Global Step: 75720 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:00:12,425-Speed 128.98 samples/sec Loss 10.9714 Epoch: 7 Global Step: 75725 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:00:32,014-Speed 130.69 samples/sec Loss 11.1839 Epoch: 7 Global Step: 75730 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:00:52,087-Speed 127.54 samples/sec Loss 11.0969 Epoch: 7 Global Step: 75735 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:01:12,107-Speed 127.88 samples/sec Loss 10.8797 Epoch: 7 Global Step: 75740 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:01:31,686-Speed 130.76 samples/sec Loss 10.9153 Epoch: 7 Global Step: 75745 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:01:51,548-Speed 128.90 samples/sec Loss 10.8508 Epoch: 7 Global Step: 75750 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:02:11,031-Speed 131.40 samples/sec Loss 11.0020 Epoch: 7 Global Step: 75755 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:02:30,867-Speed 129.07 samples/sec Loss 10.9338 Epoch: 7 Global Step: 75760 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:02:50,466-Speed 130.63 samples/sec Loss 10.7811 Epoch: 7 Global Step: 75765 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:03:10,232-Speed 129.52 samples/sec Loss 10.7838 Epoch: 7 Global Step: 75770 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:03:29,867-Speed 130.38 samples/sec Loss 11.0681 Epoch: 7 Global Step: 75775 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:03:49,721-Speed 128.95 samples/sec Loss 10.8567 Epoch: 7 Global Step: 75780 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:04:09,744-Speed 127.86 samples/sec Loss 10.8931 Epoch: 7 Global Step: 75785 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:04:29,325-Speed 130.75 samples/sec Loss 11.2161 Epoch: 7 Global Step: 75790 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:04:49,148-Speed 129.15 samples/sec Loss 10.9719 Epoch: 7 Global Step: 75795 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:05:08,692-Speed 131.00 samples/sec Loss 10.8973 Epoch: 7 Global Step: 75800 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:05:28,462-Speed 129.49 samples/sec Loss 10.6935 Epoch: 7 Global Step: 75805 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:05:48,067-Speed 130.59 samples/sec Loss 10.9691 Epoch: 7 Global Step: 75810 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:06:07,780-Speed 129.87 samples/sec Loss 11.0047 Epoch: 7 Global Step: 75815 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:06:27,426-Speed 130.31 samples/sec Loss 10.7339 Epoch: 7 Global Step: 75820 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:06:47,470-Speed 127.73 samples/sec Loss 10.8908 Epoch: 7 Global Step: 75825 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:07:07,255-Speed 129.40 samples/sec Loss 10.4161 Epoch: 7 Global Step: 75830 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:07:26,829-Speed 130.79 samples/sec Loss 10.4065 Epoch: 7 Global Step: 75835 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:07:46,660-Speed 129.10 samples/sec Loss 10.7080 Epoch: 7 Global Step: 75840 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:08:06,301-Speed 130.35 samples/sec Loss 10.7857 Epoch: 7 Global Step: 75845 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:08:26,227-Speed 128.48 samples/sec Loss 10.8382 Epoch: 7 Global Step: 75850 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:08:45,835-Speed 130.57 samples/sec Loss 10.9572 Epoch: 7 Global Step: 75855 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:09:05,658-Speed 129.15 samples/sec Loss 11.0560 Epoch: 7 Global Step: 75860 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:09:25,210-Speed 130.94 samples/sec Loss 10.9141 Epoch: 7 Global Step: 75865 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:09:45,352-Speed 127.10 samples/sec Loss 10.5622 Epoch: 7 Global Step: 75870 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:10:05,101-Speed 129.63 samples/sec Loss 11.0306 Epoch: 7 Global Step: 75875 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:10:24,730-Speed 130.42 samples/sec Loss 10.8384 Epoch: 7 Global Step: 75880 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:10:44,471-Speed 129.68 samples/sec Loss 10.8422 Epoch: 7 Global Step: 75885 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:11:04,022-Speed 130.95 samples/sec Loss 10.9933 Epoch: 7 Global Step: 75890 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:11:23,926-Speed 128.62 samples/sec Loss 10.7218 Epoch: 7 Global Step: 75895 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:11:43,533-Speed 130.58 samples/sec Loss 10.7855 Epoch: 7 Global Step: 75900 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:12:03,416-Speed 128.77 samples/sec Loss 10.7256 Epoch: 7 Global Step: 75905 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:12:23,054-Speed 130.36 samples/sec Loss 10.6856 Epoch: 7 Global Step: 75910 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:12:43,063-Speed 127.95 samples/sec Loss 10.6624 Epoch: 7 Global Step: 75915 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:13:02,859-Speed 129.33 samples/sec Loss 10.6076 Epoch: 7 Global Step: 75920 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:13:22,447-Speed 130.70 samples/sec Loss 10.8360 Epoch: 7 Global Step: 75925 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:13:42,183-Speed 129.72 samples/sec Loss 11.2083 Epoch: 7 Global Step: 75930 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:14:01,720-Speed 131.04 samples/sec Loss 10.8871 Epoch: 7 Global Step: 75935 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:14:21,583-Speed 128.90 samples/sec Loss 10.8113 Epoch: 7 Global Step: 75940 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:14:41,046-Speed 131.54 samples/sec Loss 10.6884 Epoch: 7 Global Step: 75945 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:15:01,023-Speed 128.15 samples/sec Loss 10.5299 Epoch: 7 Global Step: 75950 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:15:20,584-Speed 130.88 samples/sec Loss 10.9264 Epoch: 7 Global Step: 75955 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:15:40,662-Speed 127.51 samples/sec Loss 10.5972 Epoch: 7 Global Step: 75960 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:16:00,471-Speed 129.24 samples/sec Loss 10.9066 Epoch: 7 Global Step: 75965 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:16:20,096-Speed 130.45 samples/sec Loss 10.8597 Epoch: 7 Global Step: 75970 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:16:39,831-Speed 129.73 samples/sec Loss 11.0643 Epoch: 7 Global Step: 75975 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:16:59,475-Speed 130.32 samples/sec Loss 10.8224 Epoch: 7 Global Step: 75980 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:17:19,202-Speed 129.78 samples/sec Loss 10.8187 Epoch: 7 Global Step: 75985 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:17:38,881-Speed 130.09 samples/sec Loss 10.8225 Epoch: 7 Global Step: 75990 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:17:58,622-Speed 129.69 samples/sec Loss 10.8166 Epoch: 7 Global Step: 75995 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:18:18,197-Speed 130.79 samples/sec Loss 11.1546 Epoch: 7 Global Step: 76000 Fp16 Grad Scale: 8192 Required: 206 hours Training: 2021-06-10 13:18:38,271-Speed 127.53 samples/sec Loss 11.2660 Epoch: 7 Global Step: 76005 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:18:58,311-Speed 127.75 samples/sec Loss 10.9791 Epoch: 7 Global Step: 76010 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:19:17,860-Speed 130.96 samples/sec Loss 10.9199 Epoch: 7 Global Step: 76015 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:19:37,687-Speed 129.13 samples/sec Loss 10.5836 Epoch: 7 Global Step: 76020 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:19:57,272-Speed 130.72 samples/sec Loss 11.1762 Epoch: 7 Global Step: 76025 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:20:16,988-Speed 129.85 samples/sec Loss 10.8623 Epoch: 7 Global Step: 76030 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:20:36,617-Speed 130.43 samples/sec Loss 11.3480 Epoch: 7 Global Step: 76035 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:20:56,441-Speed 129.15 samples/sec Loss 10.9478 Epoch: 7 Global Step: 76040 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:21:16,022-Speed 130.75 samples/sec Loss 10.8677 Epoch: 7 Global Step: 76045 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:21:36,039-Speed 127.90 samples/sec Loss 10.7489 Epoch: 7 Global Step: 76050 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:21:56,241-Speed 126.73 samples/sec Loss 10.7517 Epoch: 7 Global Step: 76055 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:22:15,802-Speed 130.88 samples/sec Loss 10.9833 Epoch: 7 Global Step: 76060 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:22:36,746-Speed 122.24 samples/sec Loss 10.8021 Epoch: 7 Global Step: 76065 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:22:56,209-Speed 131.54 samples/sec Loss 10.7828 Epoch: 7 Global Step: 76070 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:23:16,207-Speed 128.02 samples/sec Loss 10.9744 Epoch: 7 Global Step: 76075 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:23:35,766-Speed 130.89 samples/sec Loss 10.9162 Epoch: 7 Global Step: 76080 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:23:55,546-Speed 129.43 samples/sec Loss 11.2302 Epoch: 7 Global Step: 76085 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:24:15,146-Speed 130.62 samples/sec Loss 10.8929 Epoch: 7 Global Step: 76090 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:24:35,165-Speed 127.88 samples/sec Loss 10.8900 Epoch: 7 Global Step: 76095 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:24:54,912-Speed 129.65 samples/sec Loss 10.9272 Epoch: 7 Global Step: 76100 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:25:14,482-Speed 130.82 samples/sec Loss 10.7120 Epoch: 7 Global Step: 76105 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:25:34,309-Speed 129.12 samples/sec Loss 10.7854 Epoch: 7 Global Step: 76110 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:25:53,857-Speed 130.97 samples/sec Loss 10.5231 Epoch: 7 Global Step: 76115 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:26:13,744-Speed 128.73 samples/sec Loss 10.7720 Epoch: 7 Global Step: 76120 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:26:33,319-Speed 130.79 samples/sec Loss 10.9426 Epoch: 7 Global Step: 76125 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:26:53,439-Speed 127.24 samples/sec Loss 10.7395 Epoch: 7 Global Step: 76130 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:27:12,906-Speed 131.52 samples/sec Loss 11.1983 Epoch: 7 Global Step: 76135 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:27:33,002-Speed 127.39 samples/sec Loss 10.7324 Epoch: 7 Global Step: 76140 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:27:52,669-Speed 130.17 samples/sec Loss 10.7926 Epoch: 7 Global Step: 76145 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:28:12,503-Speed 129.08 samples/sec Loss 11.0810 Epoch: 7 Global Step: 76150 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:28:32,138-Speed 130.39 samples/sec Loss 11.0354 Epoch: 7 Global Step: 76155 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:28:51,774-Speed 130.37 samples/sec Loss 10.9205 Epoch: 7 Global Step: 76160 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:29:11,601-Speed 129.13 samples/sec Loss 10.8664 Epoch: 7 Global Step: 76165 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:29:31,160-Speed 130.89 samples/sec Loss 10.7170 Epoch: 7 Global Step: 76170 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:29:51,055-Speed 128.68 samples/sec Loss 10.8149 Epoch: 7 Global Step: 76175 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:30:10,511-Speed 131.58 samples/sec Loss 10.9861 Epoch: 7 Global Step: 76180 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:30:30,817-Speed 126.08 samples/sec Loss 11.3469 Epoch: 7 Global Step: 76185 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:30:50,750-Speed 128.44 samples/sec Loss 10.8208 Epoch: 7 Global Step: 76190 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:31:10,314-Speed 130.86 samples/sec Loss 11.0058 Epoch: 7 Global Step: 76195 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:31:30,134-Speed 129.17 samples/sec Loss 11.0506 Epoch: 7 Global Step: 76200 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:31:49,773-Speed 130.36 samples/sec Loss 10.8747 Epoch: 7 Global Step: 76205 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:32:09,419-Speed 130.31 samples/sec Loss 10.6372 Epoch: 7 Global Step: 76210 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:32:29,058-Speed 130.36 samples/sec Loss 10.8433 Epoch: 7 Global Step: 76215 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:32:48,691-Speed 130.40 samples/sec Loss 11.0578 Epoch: 7 Global Step: 76220 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:33:08,342-Speed 130.28 samples/sec Loss 10.7374 Epoch: 7 Global Step: 76225 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:33:28,433-Speed 127.43 samples/sec Loss 10.8600 Epoch: 7 Global Step: 76230 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:33:48,466-Speed 127.80 samples/sec Loss 10.8405 Epoch: 7 Global Step: 76235 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:34:08,003-Speed 131.03 samples/sec Loss 10.8402 Epoch: 7 Global Step: 76240 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:34:28,458-Speed 125.16 samples/sec Loss 10.8380 Epoch: 7 Global Step: 76245 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:34:48,030-Speed 130.81 samples/sec Loss 10.5774 Epoch: 7 Global Step: 76250 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:35:07,692-Speed 130.21 samples/sec Loss 10.7937 Epoch: 7 Global Step: 76255 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:35:27,327-Speed 130.39 samples/sec Loss 11.0221 Epoch: 7 Global Step: 76260 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:35:47,067-Speed 129.69 samples/sec Loss 10.8342 Epoch: 7 Global Step: 76265 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:36:06,636-Speed 130.82 samples/sec Loss 10.9968 Epoch: 7 Global Step: 76270 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:36:26,706-Speed 127.56 samples/sec Loss 11.0794 Epoch: 7 Global Step: 76275 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:36:46,633-Speed 128.48 samples/sec Loss 11.0819 Epoch: 7 Global Step: 76280 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:37:06,268-Speed 130.38 samples/sec Loss 11.0773 Epoch: 7 Global Step: 76285 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:37:25,981-Speed 129.87 samples/sec Loss 10.7443 Epoch: 7 Global Step: 76290 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:37:45,588-Speed 130.58 samples/sec Loss 11.0160 Epoch: 7 Global Step: 76295 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:38:05,336-Speed 129.64 samples/sec Loss 11.0621 Epoch: 7 Global Step: 76300 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:38:24,969-Speed 130.40 samples/sec Loss 10.8577 Epoch: 7 Global Step: 76305 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:38:44,738-Speed 129.50 samples/sec Loss 11.0249 Epoch: 7 Global Step: 76310 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:39:04,152-Speed 131.87 samples/sec Loss 10.6470 Epoch: 7 Global Step: 76315 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:39:24,298-Speed 127.08 samples/sec Loss 10.5543 Epoch: 7 Global Step: 76320 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:39:44,330-Speed 127.80 samples/sec Loss 10.9838 Epoch: 7 Global Step: 76325 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:40:03,920-Speed 130.68 samples/sec Loss 10.7098 Epoch: 7 Global Step: 76330 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:40:23,865-Speed 128.36 samples/sec Loss 10.6369 Epoch: 7 Global Step: 76335 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:40:43,536-Speed 130.15 samples/sec Loss 10.7997 Epoch: 7 Global Step: 76340 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:41:03,373-Speed 129.05 samples/sec Loss 10.6307 Epoch: 7 Global Step: 76345 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:41:23,010-Speed 130.37 samples/sec Loss 10.8740 Epoch: 7 Global Step: 76350 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:41:42,654-Speed 130.33 samples/sec Loss 10.7532 Epoch: 7 Global Step: 76355 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:42:02,303-Speed 130.29 samples/sec Loss 10.4845 Epoch: 7 Global Step: 76360 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:42:22,427-Speed 127.22 samples/sec Loss 10.7902 Epoch: 7 Global Step: 76365 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:42:42,244-Speed 129.19 samples/sec Loss 10.8605 Epoch: 7 Global Step: 76370 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:43:01,830-Speed 130.71 samples/sec Loss 11.0974 Epoch: 7 Global Step: 76375 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:43:21,712-Speed 128.77 samples/sec Loss 10.8708 Epoch: 7 Global Step: 76380 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:43:41,823-Speed 127.30 samples/sec Loss 10.8154 Epoch: 7 Global Step: 76385 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:44:01,666-Speed 129.02 samples/sec Loss 10.7587 Epoch: 7 Global Step: 76390 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:44:21,301-Speed 130.39 samples/sec Loss 10.8433 Epoch: 7 Global Step: 76395 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:44:41,212-Speed 128.58 samples/sec Loss 10.7557 Epoch: 7 Global Step: 76400 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:45:00,744-Speed 131.08 samples/sec Loss 10.9831 Epoch: 7 Global Step: 76405 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:45:20,888-Speed 127.09 samples/sec Loss 11.0025 Epoch: 7 Global Step: 76410 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:45:40,545-Speed 130.23 samples/sec Loss 10.8272 Epoch: 7 Global Step: 76415 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:46:00,199-Speed 130.27 samples/sec Loss 10.8781 Epoch: 7 Global Step: 76420 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:46:19,879-Speed 130.08 samples/sec Loss 10.5798 Epoch: 7 Global Step: 76425 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:46:39,461-Speed 130.74 samples/sec Loss 11.0654 Epoch: 7 Global Step: 76430 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:46:59,414-Speed 128.31 samples/sec Loss 10.7610 Epoch: 7 Global Step: 76435 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:47:18,991-Speed 130.77 samples/sec Loss 11.0681 Epoch: 7 Global Step: 76440 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:47:38,923-Speed 128.44 samples/sec Loss 11.0324 Epoch: 7 Global Step: 76445 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:47:58,500-Speed 130.77 samples/sec Loss 10.9376 Epoch: 7 Global Step: 76450 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:48:18,477-Speed 128.16 samples/sec Loss 10.7999 Epoch: 7 Global Step: 76455 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:48:38,377-Speed 128.64 samples/sec Loss 10.6580 Epoch: 7 Global Step: 76460 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:48:58,032-Speed 130.26 samples/sec Loss 11.0927 Epoch: 7 Global Step: 76465 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:49:17,773-Speed 129.69 samples/sec Loss 10.9680 Epoch: 7 Global Step: 76470 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:49:37,405-Speed 130.40 samples/sec Loss 10.7230 Epoch: 7 Global Step: 76475 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:49:57,119-Speed 129.87 samples/sec Loss 10.8592 Epoch: 7 Global Step: 76480 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:50:16,700-Speed 130.74 samples/sec Loss 10.9219 Epoch: 7 Global Step: 76485 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:50:36,518-Speed 129.19 samples/sec Loss 10.8327 Epoch: 7 Global Step: 76490 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:50:56,137-Speed 130.49 samples/sec Loss 11.0351 Epoch: 7 Global Step: 76495 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:51:16,329-Speed 126.78 samples/sec Loss 10.8665 Epoch: 7 Global Step: 76500 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:51:36,266-Speed 128.41 samples/sec Loss 10.6226 Epoch: 7 Global Step: 76505 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:51:55,828-Speed 130.87 samples/sec Loss 10.8792 Epoch: 7 Global Step: 76510 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:52:15,698-Speed 128.84 samples/sec Loss 10.7860 Epoch: 7 Global Step: 76515 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:52:35,380-Speed 130.08 samples/sec Loss 11.0496 Epoch: 7 Global Step: 76520 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:52:55,140-Speed 129.56 samples/sec Loss 11.0837 Epoch: 7 Global Step: 76525 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:53:14,790-Speed 130.28 samples/sec Loss 10.8881 Epoch: 7 Global Step: 76530 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:53:34,538-Speed 129.64 samples/sec Loss 11.0508 Epoch: 7 Global Step: 76535 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:53:54,171-Speed 130.40 samples/sec Loss 10.6682 Epoch: 7 Global Step: 76540 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:54:15,048-Speed 122.63 samples/sec Loss 10.6052 Epoch: 7 Global Step: 76545 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:54:34,979-Speed 128.44 samples/sec Loss 10.9873 Epoch: 7 Global Step: 76550 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:54:54,575-Speed 130.65 samples/sec Loss 10.6919 Epoch: 7 Global Step: 76555 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:55:14,583-Speed 127.95 samples/sec Loss 11.0575 Epoch: 7 Global Step: 76560 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:55:34,158-Speed 130.79 samples/sec Loss 11.0779 Epoch: 7 Global Step: 76565 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:55:53,995-Speed 129.06 samples/sec Loss 10.7337 Epoch: 7 Global Step: 76570 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:56:13,455-Speed 131.56 samples/sec Loss 10.6854 Epoch: 7 Global Step: 76575 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:56:33,241-Speed 129.39 samples/sec Loss 10.9126 Epoch: 7 Global Step: 76580 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:56:52,866-Speed 130.45 samples/sec Loss 11.0013 Epoch: 7 Global Step: 76585 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:57:12,893-Speed 127.84 samples/sec Loss 10.7843 Epoch: 7 Global Step: 76590 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:57:32,564-Speed 130.14 samples/sec Loss 10.6602 Epoch: 7 Global Step: 76595 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:57:52,122-Speed 130.90 samples/sec Loss 10.6158 Epoch: 7 Global Step: 76600 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:58:12,078-Speed 128.29 samples/sec Loss 10.6596 Epoch: 7 Global Step: 76605 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:58:31,540-Speed 131.55 samples/sec Loss 10.8620 Epoch: 7 Global Step: 76610 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:58:51,885-Speed 125.84 samples/sec Loss 10.8347 Epoch: 7 Global Step: 76615 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:59:11,467-Speed 130.74 samples/sec Loss 10.6213 Epoch: 7 Global Step: 76620 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:59:31,317-Speed 128.97 samples/sec Loss 10.7025 Epoch: 7 Global Step: 76625 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 13:59:50,859-Speed 131.01 samples/sec Loss 10.6929 Epoch: 7 Global Step: 76630 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:00:10,842-Speed 128.11 samples/sec Loss 10.7018 Epoch: 7 Global Step: 76635 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:00:30,608-Speed 129.52 samples/sec Loss 10.7777 Epoch: 7 Global Step: 76640 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:00:50,239-Speed 130.41 samples/sec Loss 11.1281 Epoch: 7 Global Step: 76645 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:01:09,889-Speed 130.29 samples/sec Loss 10.9324 Epoch: 7 Global Step: 76650 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:01:29,456-Speed 130.84 samples/sec Loss 10.6558 Epoch: 7 Global Step: 76655 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:01:49,277-Speed 129.16 samples/sec Loss 10.8019 Epoch: 7 Global Step: 76660 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:02:08,868-Speed 130.68 samples/sec Loss 10.8732 Epoch: 7 Global Step: 76665 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:02:28,866-Speed 128.02 samples/sec Loss 10.8188 Epoch: 7 Global Step: 76670 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:02:48,429-Speed 130.86 samples/sec Loss 10.8543 Epoch: 7 Global Step: 76675 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:03:08,549-Speed 127.25 samples/sec Loss 10.9584 Epoch: 7 Global Step: 76680 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:03:28,304-Speed 129.59 samples/sec Loss 10.9924 Epoch: 7 Global Step: 76685 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:03:47,936-Speed 130.41 samples/sec Loss 10.8441 Epoch: 7 Global Step: 76690 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:04:07,576-Speed 130.35 samples/sec Loss 10.9622 Epoch: 7 Global Step: 76695 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:04:27,138-Speed 130.88 samples/sec Loss 10.9473 Epoch: 7 Global Step: 76700 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:04:47,045-Speed 128.60 samples/sec Loss 11.1383 Epoch: 7 Global Step: 76705 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:05:06,686-Speed 130.35 samples/sec Loss 10.7020 Epoch: 7 Global Step: 76710 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:05:26,612-Speed 128.48 samples/sec Loss 10.9944 Epoch: 7 Global Step: 76715 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:05:46,182-Speed 130.82 samples/sec Loss 10.7710 Epoch: 7 Global Step: 76720 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:06:06,188-Speed 127.96 samples/sec Loss 10.7769 Epoch: 7 Global Step: 76725 Fp16 Grad Scale: 8192 Required: 205 hours Training: 2021-06-10 14:06:26,155-Speed 128.22 samples/sec Loss 10.8692 Epoch: 7 Global Step: 76730 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:06:45,767-Speed 130.53 samples/sec Loss 10.9153 Epoch: 7 Global Step: 76735 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:07:05,545-Speed 129.45 samples/sec Loss 10.6267 Epoch: 7 Global Step: 76740 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:07:25,178-Speed 130.40 samples/sec Loss 10.9387 Epoch: 7 Global Step: 76745 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:07:44,911-Speed 129.74 samples/sec Loss 10.7590 Epoch: 7 Global Step: 76750 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:08:04,509-Speed 130.63 samples/sec Loss 11.0044 Epoch: 7 Global Step: 76755 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:08:24,317-Speed 129.25 samples/sec Loss 10.9173 Epoch: 7 Global Step: 76760 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:08:43,894-Speed 130.77 samples/sec Loss 10.8141 Epoch: 7 Global Step: 76765 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:09:04,028-Speed 127.16 samples/sec Loss 10.7887 Epoch: 7 Global Step: 76770 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:09:23,853-Speed 129.14 samples/sec Loss 10.9929 Epoch: 7 Global Step: 76775 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:09:43,573-Speed 129.82 samples/sec Loss 10.7638 Epoch: 7 Global Step: 76780 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:10:03,310-Speed 129.72 samples/sec Loss 10.9828 Epoch: 7 Global Step: 76785 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:10:22,944-Speed 130.39 samples/sec Loss 10.5731 Epoch: 7 Global Step: 76790 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:10:42,747-Speed 129.28 samples/sec Loss 10.7238 Epoch: 7 Global Step: 76795 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:11:02,249-Speed 131.28 samples/sec Loss 10.7629 Epoch: 7 Global Step: 76800 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:11:22,066-Speed 129.19 samples/sec Loss 10.6478 Epoch: 7 Global Step: 76805 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:11:41,655-Speed 130.69 samples/sec Loss 11.0274 Epoch: 7 Global Step: 76810 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:12:01,765-Speed 127.31 samples/sec Loss 11.0277 Epoch: 7 Global Step: 76815 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:12:21,661-Speed 128.68 samples/sec Loss 11.0447 Epoch: 7 Global Step: 76820 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:12:41,228-Speed 130.83 samples/sec Loss 10.7813 Epoch: 7 Global Step: 76825 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:13:01,136-Speed 128.60 samples/sec Loss 11.1006 Epoch: 7 Global Step: 76830 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:13:20,819-Speed 130.07 samples/sec Loss 10.6743 Epoch: 7 Global Step: 76835 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:13:40,644-Speed 129.14 samples/sec Loss 10.4305 Epoch: 7 Global Step: 76840 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:14:00,130-Speed 131.39 samples/sec Loss 10.8116 Epoch: 7 Global Step: 76845 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:14:20,023-Speed 128.71 samples/sec Loss 10.9174 Epoch: 7 Global Step: 76850 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:14:39,626-Speed 130.60 samples/sec Loss 10.4964 Epoch: 7 Global Step: 76855 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:14:59,730-Speed 127.35 samples/sec Loss 10.9629 Epoch: 7 Global Step: 76860 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:15:19,652-Speed 128.51 samples/sec Loss 10.8676 Epoch: 7 Global Step: 76865 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:15:39,298-Speed 130.31 samples/sec Loss 10.8576 Epoch: 7 Global Step: 76870 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:15:59,044-Speed 129.66 samples/sec Loss 11.0108 Epoch: 7 Global Step: 76875 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:16:18,689-Speed 130.32 samples/sec Loss 11.0210 Epoch: 7 Global Step: 76880 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:16:38,329-Speed 130.35 samples/sec Loss 10.8491 Epoch: 7 Global Step: 76885 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:16:57,970-Speed 130.35 samples/sec Loss 10.3062 Epoch: 7 Global Step: 76890 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:17:17,717-Speed 129.65 samples/sec Loss 11.0466 Epoch: 7 Global Step: 76895 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:17:37,279-Speed 130.87 samples/sec Loss 10.9378 Epoch: 7 Global Step: 76900 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:17:57,368-Speed 127.44 samples/sec Loss 10.9023 Epoch: 7 Global Step: 76905 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:18:17,294-Speed 128.48 samples/sec Loss 11.0431 Epoch: 7 Global Step: 76910 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:18:36,874-Speed 130.75 samples/sec Loss 10.8904 Epoch: 7 Global Step: 76915 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:18:56,800-Speed 128.48 samples/sec Loss 10.8083 Epoch: 7 Global Step: 76920 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:19:16,344-Speed 130.99 samples/sec Loss 10.7692 Epoch: 7 Global Step: 76925 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:19:36,087-Speed 129.68 samples/sec Loss 10.6022 Epoch: 7 Global Step: 76930 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:19:55,673-Speed 130.71 samples/sec Loss 10.4642 Epoch: 7 Global Step: 76935 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:20:15,445-Speed 129.49 samples/sec Loss 10.8521 Epoch: 7 Global Step: 76940 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:20:35,080-Speed 130.39 samples/sec Loss 10.9225 Epoch: 7 Global Step: 76945 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:20:55,077-Speed 128.03 samples/sec Loss 10.8798 Epoch: 7 Global Step: 76950 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:21:14,832-Speed 129.59 samples/sec Loss 10.8293 Epoch: 7 Global Step: 76955 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:21:34,371-Speed 131.03 samples/sec Loss 10.5504 Epoch: 7 Global Step: 76960 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:21:54,349-Speed 128.15 samples/sec Loss 10.7620 Epoch: 7 Global Step: 76965 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:22:13,815-Speed 131.52 samples/sec Loss 11.0683 Epoch: 7 Global Step: 76970 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:22:33,751-Speed 128.42 samples/sec Loss 10.6551 Epoch: 7 Global Step: 76975 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:22:53,349-Speed 130.63 samples/sec Loss 10.5690 Epoch: 7 Global Step: 76980 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:23:13,117-Speed 129.51 samples/sec Loss 10.8951 Epoch: 7 Global Step: 76985 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:23:32,719-Speed 130.61 samples/sec Loss 10.6890 Epoch: 7 Global Step: 76990 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:23:52,752-Speed 127.79 samples/sec Loss 10.9340 Epoch: 7 Global Step: 76995 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:24:12,511-Speed 129.57 samples/sec Loss 10.6273 Epoch: 7 Global Step: 77000 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:24:32,153-Speed 130.34 samples/sec Loss 10.7146 Epoch: 7 Global Step: 77005 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:24:51,916-Speed 129.54 samples/sec Loss 11.0047 Epoch: 7 Global Step: 77010 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:25:11,498-Speed 130.74 samples/sec Loss 11.1621 Epoch: 7 Global Step: 77015 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:25:31,436-Speed 128.41 samples/sec Loss 11.0076 Epoch: 7 Global Step: 77020 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:25:51,029-Speed 130.66 samples/sec Loss 11.0274 Epoch: 7 Global Step: 77025 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:26:11,020-Speed 128.07 samples/sec Loss 10.9361 Epoch: 7 Global Step: 77030 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:26:30,516-Speed 131.32 samples/sec Loss 10.8330 Epoch: 7 Global Step: 77035 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:26:50,543-Speed 127.83 samples/sec Loss 10.9533 Epoch: 7 Global Step: 77040 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:27:10,373-Speed 129.11 samples/sec Loss 10.7074 Epoch: 7 Global Step: 77045 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:27:29,972-Speed 130.62 samples/sec Loss 10.4612 Epoch: 7 Global Step: 77050 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:27:49,710-Speed 129.71 samples/sec Loss 11.0524 Epoch: 7 Global Step: 77055 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:28:09,361-Speed 130.28 samples/sec Loss 10.6072 Epoch: 7 Global Step: 77060 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:28:29,131-Speed 129.50 samples/sec Loss 11.0693 Epoch: 7 Global Step: 77065 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:28:48,765-Speed 130.39 samples/sec Loss 10.9731 Epoch: 7 Global Step: 77070 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:29:08,547-Speed 129.42 samples/sec Loss 10.9506 Epoch: 7 Global Step: 77075 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:29:28,077-Speed 131.09 samples/sec Loss 10.8431 Epoch: 7 Global Step: 77080 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:29:48,084-Speed 127.96 samples/sec Loss 10.9466 Epoch: 7 Global Step: 77085 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:30:07,941-Speed 128.93 samples/sec Loss 11.0901 Epoch: 7 Global Step: 77090 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:30:27,537-Speed 130.64 samples/sec Loss 10.6335 Epoch: 7 Global Step: 77095 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:30:47,193-Speed 130.24 samples/sec Loss 10.8443 Epoch: 7 Global Step: 77100 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:31:06,832-Speed 130.36 samples/sec Loss 10.7513 Epoch: 7 Global Step: 77105 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:31:26,640-Speed 129.25 samples/sec Loss 10.7176 Epoch: 7 Global Step: 77110 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:31:46,270-Speed 130.42 samples/sec Loss 10.7585 Epoch: 7 Global Step: 77115 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:32:06,035-Speed 129.53 samples/sec Loss 10.8250 Epoch: 7 Global Step: 77120 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:32:25,522-Speed 131.38 samples/sec Loss 11.0092 Epoch: 7 Global Step: 77125 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:32:45,678-Speed 127.02 samples/sec Loss 11.2172 Epoch: 7 Global Step: 77130 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:33:05,722-Speed 127.73 samples/sec Loss 10.8532 Epoch: 7 Global Step: 77135 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:33:25,209-Speed 131.37 samples/sec Loss 10.8128 Epoch: 7 Global Step: 77140 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:33:44,981-Speed 129.49 samples/sec Loss 11.0972 Epoch: 7 Global Step: 77145 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:34:04,615-Speed 130.40 samples/sec Loss 10.6536 Epoch: 7 Global Step: 77150 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:34:24,305-Speed 130.02 samples/sec Loss 10.7026 Epoch: 7 Global Step: 77155 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:34:43,952-Speed 130.30 samples/sec Loss 10.8397 Epoch: 7 Global Step: 77160 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:35:03,599-Speed 130.31 samples/sec Loss 11.1633 Epoch: 7 Global Step: 77165 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:35:23,241-Speed 130.34 samples/sec Loss 10.9267 Epoch: 7 Global Step: 77170 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:35:43,289-Speed 127.70 samples/sec Loss 10.7299 Epoch: 7 Global Step: 77175 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:36:03,199-Speed 128.59 samples/sec Loss 10.6948 Epoch: 7 Global Step: 77180 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:36:22,661-Speed 131.54 samples/sec Loss 10.8664 Epoch: 7 Global Step: 77185 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:36:42,583-Speed 128.51 samples/sec Loss 10.8625 Epoch: 7 Global Step: 77190 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:37:02,221-Speed 130.37 samples/sec Loss 10.9443 Epoch: 7 Global Step: 77195 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:37:22,132-Speed 128.57 samples/sec Loss 11.0788 Epoch: 7 Global Step: 77200 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:37:41,594-Speed 131.55 samples/sec Loss 11.0755 Epoch: 7 Global Step: 77205 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:38:01,266-Speed 130.14 samples/sec Loss 10.5060 Epoch: 7 Global Step: 77210 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:38:20,910-Speed 130.33 samples/sec Loss 11.0377 Epoch: 7 Global Step: 77215 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:38:41,130-Speed 126.61 samples/sec Loss 10.9448 Epoch: 7 Global Step: 77220 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:39:00,878-Speed 129.64 samples/sec Loss 10.8401 Epoch: 7 Global Step: 77225 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:39:20,648-Speed 129.50 samples/sec Loss 10.4964 Epoch: 7 Global Step: 77230 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:39:40,733-Speed 127.47 samples/sec Loss 10.8118 Epoch: 7 Global Step: 77235 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:40:00,298-Speed 130.85 samples/sec Loss 10.9673 Epoch: 7 Global Step: 77240 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:40:20,222-Speed 128.49 samples/sec Loss 10.7274 Epoch: 7 Global Step: 77245 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:40:39,683-Speed 131.56 samples/sec Loss 10.6087 Epoch: 7 Global Step: 77250 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:40:59,535-Speed 128.96 samples/sec Loss 10.9656 Epoch: 7 Global Step: 77255 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:41:19,181-Speed 130.32 samples/sec Loss 10.7357 Epoch: 7 Global Step: 77260 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:41:39,300-Speed 127.24 samples/sec Loss 10.8501 Epoch: 7 Global Step: 77265 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:41:58,935-Speed 130.39 samples/sec Loss 11.1396 Epoch: 7 Global Step: 77270 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:42:18,580-Speed 130.32 samples/sec Loss 10.8312 Epoch: 7 Global Step: 77275 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:42:38,337-Speed 129.58 samples/sec Loss 10.7364 Epoch: 7 Global Step: 77280 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:42:57,953-Speed 130.52 samples/sec Loss 11.0259 Epoch: 7 Global Step: 77285 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:43:17,601-Speed 130.30 samples/sec Loss 10.9081 Epoch: 7 Global Step: 77290 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:43:37,167-Speed 130.84 samples/sec Loss 10.8731 Epoch: 7 Global Step: 77295 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:43:57,122-Speed 128.30 samples/sec Loss 10.6847 Epoch: 7 Global Step: 77300 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:44:16,721-Speed 130.63 samples/sec Loss 10.8374 Epoch: 7 Global Step: 77305 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:44:36,731-Speed 127.94 samples/sec Loss 11.0706 Epoch: 7 Global Step: 77310 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:44:56,825-Speed 127.40 samples/sec Loss 10.9341 Epoch: 7 Global Step: 77315 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:45:16,459-Speed 130.39 samples/sec Loss 10.6220 Epoch: 7 Global Step: 77320 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:45:36,198-Speed 129.70 samples/sec Loss 11.1035 Epoch: 7 Global Step: 77325 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:45:55,821-Speed 130.47 samples/sec Loss 10.3014 Epoch: 7 Global Step: 77330 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:46:15,566-Speed 129.66 samples/sec Loss 10.6800 Epoch: 7 Global Step: 77335 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:46:35,197-Speed 130.41 samples/sec Loss 10.8657 Epoch: 7 Global Step: 77340 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:46:54,969-Speed 129.49 samples/sec Loss 10.8866 Epoch: 7 Global Step: 77345 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:47:14,540-Speed 130.81 samples/sec Loss 10.6604 Epoch: 7 Global Step: 77350 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:47:34,537-Speed 128.02 samples/sec Loss 10.4889 Epoch: 7 Global Step: 77355 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:47:54,595-Speed 127.64 samples/sec Loss 10.6530 Epoch: 7 Global Step: 77360 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:48:14,183-Speed 130.70 samples/sec Loss 10.9132 Epoch: 7 Global Step: 77365 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:48:33,869-Speed 130.05 samples/sec Loss 11.1266 Epoch: 7 Global Step: 77370 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:48:53,469-Speed 130.62 samples/sec Loss 10.5296 Epoch: 7 Global Step: 77375 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:49:13,240-Speed 129.49 samples/sec Loss 11.3140 Epoch: 7 Global Step: 77380 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:49:32,891-Speed 130.28 samples/sec Loss 11.1899 Epoch: 7 Global Step: 77385 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:49:52,814-Speed 128.50 samples/sec Loss 10.9010 Epoch: 7 Global Step: 77390 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:50:12,331-Speed 131.18 samples/sec Loss 11.0709 Epoch: 7 Global Step: 77395 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:50:32,301-Speed 128.20 samples/sec Loss 10.5203 Epoch: 7 Global Step: 77400 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:50:52,328-Speed 127.83 samples/sec Loss 11.0068 Epoch: 7 Global Step: 77405 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:51:11,967-Speed 130.36 samples/sec Loss 11.1749 Epoch: 7 Global Step: 77410 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:51:31,818-Speed 128.97 samples/sec Loss 10.9850 Epoch: 7 Global Step: 77415 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:51:51,413-Speed 130.66 samples/sec Loss 10.9425 Epoch: 7 Global Step: 77420 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:52:11,321-Speed 128.60 samples/sec Loss 10.7934 Epoch: 7 Global Step: 77425 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:52:30,912-Speed 130.68 samples/sec Loss 10.6445 Epoch: 7 Global Step: 77430 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:52:50,553-Speed 130.34 samples/sec Loss 10.6850 Epoch: 7 Global Step: 77435 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:53:10,302-Speed 129.64 samples/sec Loss 11.0050 Epoch: 7 Global Step: 77440 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:53:30,366-Speed 127.60 samples/sec Loss 11.0630 Epoch: 7 Global Step: 77445 Fp16 Grad Scale: 8192 Required: 204 hours Training: 2021-06-10 14:53:50,247-Speed 128.77 samples/sec Loss 10.8017 Epoch: 7 Global Step: 77450 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:54:09,718-Speed 131.49 samples/sec Loss 10.8497 Epoch: 7 Global Step: 77455 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:54:29,541-Speed 129.15 samples/sec Loss 10.5399 Epoch: 7 Global Step: 77460 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:54:49,178-Speed 130.37 samples/sec Loss 10.9547 Epoch: 7 Global Step: 77465 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:55:09,101-Speed 128.50 samples/sec Loss 11.1338 Epoch: 7 Global Step: 77470 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:55:28,622-Speed 131.15 samples/sec Loss 10.9441 Epoch: 7 Global Step: 77475 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:55:48,462-Speed 129.04 samples/sec Loss 10.8317 Epoch: 7 Global Step: 77480 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:56:08,175-Speed 129.87 samples/sec Loss 10.9354 Epoch: 7 Global Step: 77485 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:56:28,428-Speed 126.41 samples/sec Loss 10.6970 Epoch: 7 Global Step: 77490 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:56:48,071-Speed 130.33 samples/sec Loss 10.4530 Epoch: 7 Global Step: 77495 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:57:07,552-Speed 131.41 samples/sec Loss 10.7292 Epoch: 7 Global Step: 77500 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:57:27,455-Speed 128.63 samples/sec Loss 10.8828 Epoch: 7 Global Step: 77505 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:57:47,082-Speed 130.44 samples/sec Loss 10.4823 Epoch: 7 Global Step: 77510 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:58:07,001-Speed 128.52 samples/sec Loss 10.8784 Epoch: 7 Global Step: 77515 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:58:26,465-Speed 131.54 samples/sec Loss 11.2028 Epoch: 7 Global Step: 77520 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:58:46,194-Speed 129.76 samples/sec Loss 10.6316 Epoch: 7 Global Step: 77525 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:59:05,854-Speed 130.23 samples/sec Loss 11.2493 Epoch: 7 Global Step: 77530 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:59:25,998-Speed 127.09 samples/sec Loss 10.8133 Epoch: 7 Global Step: 77535 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 14:59:45,632-Speed 130.39 samples/sec Loss 10.6936 Epoch: 7 Global Step: 77540 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:00:05,176-Speed 130.99 samples/sec Loss 10.8523 Epoch: 7 Global Step: 77545 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:00:25,087-Speed 128.58 samples/sec Loss 10.8908 Epoch: 7 Global Step: 77550 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:00:44,644-Speed 130.91 samples/sec Loss 10.9543 Epoch: 7 Global Step: 77555 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:01:04,440-Speed 129.33 samples/sec Loss 10.8868 Epoch: 7 Global Step: 77560 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:01:23,973-Speed 131.06 samples/sec Loss 10.7588 Epoch: 7 Global Step: 77565 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:01:43,899-Speed 128.49 samples/sec Loss 10.9090 Epoch: 7 Global Step: 77570 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:02:03,537-Speed 130.37 samples/sec Loss 10.9125 Epoch: 7 Global Step: 77575 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:02:23,647-Speed 127.30 samples/sec Loss 10.7568 Epoch: 7 Global Step: 77580 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:02:43,491-Speed 129.02 samples/sec Loss 10.6419 Epoch: 7 Global Step: 77585 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:03:03,113-Speed 130.47 samples/sec Loss 10.6691 Epoch: 7 Global Step: 77590 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:03:22,869-Speed 129.59 samples/sec Loss 10.9376 Epoch: 7 Global Step: 77595 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:03:42,513-Speed 130.32 samples/sec Loss 10.8465 Epoch: 7 Global Step: 77600 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:04:02,140-Speed 130.44 samples/sec Loss 10.8568 Epoch: 7 Global Step: 77605 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:04:21,792-Speed 130.27 samples/sec Loss 10.8743 Epoch: 7 Global Step: 77610 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:04:41,599-Speed 129.25 samples/sec Loss 11.1887 Epoch: 7 Global Step: 77615 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:05:01,197-Speed 130.64 samples/sec Loss 10.9602 Epoch: 7 Global Step: 77620 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:05:21,233-Speed 127.77 samples/sec Loss 11.0061 Epoch: 7 Global Step: 77625 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:05:41,159-Speed 128.48 samples/sec Loss 10.7749 Epoch: 7 Global Step: 77630 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:06:00,788-Speed 130.42 samples/sec Loss 10.9042 Epoch: 7 Global Step: 77635 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:06:20,609-Speed 129.16 samples/sec Loss 10.6126 Epoch: 7 Global Step: 77640 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:06:40,165-Speed 130.91 samples/sec Loss 11.0281 Epoch: 7 Global Step: 77645 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:06:59,942-Speed 129.45 samples/sec Loss 10.9547 Epoch: 7 Global Step: 77650 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:07:19,573-Speed 130.41 samples/sec Loss 10.7466 Epoch: 7 Global Step: 77655 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:07:39,309-Speed 129.72 samples/sec Loss 10.8920 Epoch: 7 Global Step: 77660 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:07:59,046-Speed 129.72 samples/sec Loss 10.9225 Epoch: 7 Global Step: 77665 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:08:18,914-Speed 128.85 samples/sec Loss 10.7624 Epoch: 7 Global Step: 77670 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:08:38,869-Speed 128.30 samples/sec Loss 10.6320 Epoch: 7 Global Step: 77675 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:08:58,620-Speed 129.61 samples/sec Loss 10.9189 Epoch: 7 Global Step: 77680 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:09:18,555-Speed 128.43 samples/sec Loss 11.1261 Epoch: 7 Global Step: 77685 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:09:38,022-Speed 131.53 samples/sec Loss 10.7068 Epoch: 7 Global Step: 77690 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:09:57,764-Speed 129.68 samples/sec Loss 11.0711 Epoch: 7 Global Step: 77695 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:10:17,355-Speed 130.68 samples/sec Loss 10.7452 Epoch: 7 Global Step: 77700 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:10:37,110-Speed 129.60 samples/sec Loss 10.9906 Epoch: 7 Global Step: 77705 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:10:57,441-Speed 125.93 samples/sec Loss 11.0498 Epoch: 7 Global Step: 77710 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:11:17,366-Speed 128.48 samples/sec Loss 11.0412 Epoch: 7 Global Step: 77715 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:11:37,234-Speed 128.86 samples/sec Loss 10.8131 Epoch: 7 Global Step: 77720 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:11:56,809-Speed 130.78 samples/sec Loss 11.0111 Epoch: 7 Global Step: 77725 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:12:16,631-Speed 129.16 samples/sec Loss 10.5725 Epoch: 7 Global Step: 77730 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:12:36,149-Speed 131.17 samples/sec Loss 10.6044 Epoch: 7 Global Step: 77735 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:12:56,220-Speed 127.56 samples/sec Loss 11.2180 Epoch: 7 Global Step: 77740 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:13:15,779-Speed 130.89 samples/sec Loss 10.5613 Epoch: 7 Global Step: 77745 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:13:35,694-Speed 128.55 samples/sec Loss 10.8280 Epoch: 7 Global Step: 77750 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:13:55,167-Speed 131.47 samples/sec Loss 10.7143 Epoch: 7 Global Step: 77755 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:14:15,281-Speed 127.28 samples/sec Loss 11.1366 Epoch: 7 Global Step: 77760 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:14:35,037-Speed 129.59 samples/sec Loss 10.7557 Epoch: 7 Global Step: 77765 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:14:54,666-Speed 130.42 samples/sec Loss 10.8572 Epoch: 7 Global Step: 77770 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:15:14,392-Speed 129.79 samples/sec Loss 10.9203 Epoch: 7 Global Step: 77775 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:15:34,024-Speed 130.41 samples/sec Loss 10.8929 Epoch: 7 Global Step: 77780 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:15:53,903-Speed 128.79 samples/sec Loss 11.1066 Epoch: 7 Global Step: 77785 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:16:13,475-Speed 130.81 samples/sec Loss 10.8663 Epoch: 7 Global Step: 77790 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:16:33,519-Speed 127.73 samples/sec Loss 10.6510 Epoch: 7 Global Step: 77795 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:16:53,232-Speed 129.87 samples/sec Loss 10.6212 Epoch: 7 Global Step: 77800 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:17:13,375-Speed 127.10 samples/sec Loss 10.8902 Epoch: 7 Global Step: 77805 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:17:33,346-Speed 128.19 samples/sec Loss 10.9716 Epoch: 7 Global Step: 77810 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:17:52,970-Speed 130.46 samples/sec Loss 10.6575 Epoch: 7 Global Step: 77815 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:18:12,697-Speed 129.78 samples/sec Loss 10.7204 Epoch: 7 Global Step: 77820 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:18:32,251-Speed 130.93 samples/sec Loss 10.7935 Epoch: 7 Global Step: 77825 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:18:52,120-Speed 128.85 samples/sec Loss 11.0910 Epoch: 7 Global Step: 77830 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:19:11,868-Speed 129.64 samples/sec Loss 10.8082 Epoch: 7 Global Step: 77835 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:19:31,734-Speed 128.87 samples/sec Loss 10.7572 Epoch: 7 Global Step: 77840 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:19:51,276-Speed 131.00 samples/sec Loss 10.5929 Epoch: 7 Global Step: 77845 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:20:11,530-Speed 126.40 samples/sec Loss 11.0729 Epoch: 7 Global Step: 77850 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:20:31,339-Speed 129.24 samples/sec Loss 10.9225 Epoch: 7 Global Step: 77855 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:20:50,925-Speed 130.71 samples/sec Loss 10.8738 Epoch: 7 Global Step: 77860 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:21:10,746-Speed 129.17 samples/sec Loss 10.8557 Epoch: 7 Global Step: 77865 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:21:30,441-Speed 129.99 samples/sec Loss 10.8507 Epoch: 7 Global Step: 77870 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:21:50,172-Speed 129.76 samples/sec Loss 10.9917 Epoch: 7 Global Step: 77875 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:22:09,687-Speed 131.19 samples/sec Loss 11.1581 Epoch: 7 Global Step: 77880 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:22:29,538-Speed 128.97 samples/sec Loss 10.8578 Epoch: 7 Global Step: 77885 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:22:49,195-Speed 130.25 samples/sec Loss 10.7341 Epoch: 7 Global Step: 77890 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:23:09,448-Speed 126.40 samples/sec Loss 10.8727 Epoch: 7 Global Step: 77895 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:23:29,204-Speed 129.59 samples/sec Loss 10.4842 Epoch: 7 Global Step: 77900 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:23:48,803-Speed 130.63 samples/sec Loss 11.1452 Epoch: 7 Global Step: 77905 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:24:08,533-Speed 129.76 samples/sec Loss 10.8436 Epoch: 7 Global Step: 77910 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:24:28,170-Speed 130.37 samples/sec Loss 10.9117 Epoch: 7 Global Step: 77915 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:24:47,854-Speed 130.06 samples/sec Loss 10.6860 Epoch: 7 Global Step: 77920 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:25:07,373-Speed 131.16 samples/sec Loss 10.6451 Epoch: 7 Global Step: 77925 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:25:27,325-Speed 128.33 samples/sec Loss 10.4966 Epoch: 7 Global Step: 77930 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:25:46,905-Speed 130.75 samples/sec Loss 10.7492 Epoch: 7 Global Step: 77935 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:26:07,013-Speed 127.31 samples/sec Loss 10.6762 Epoch: 7 Global Step: 77940 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:26:27,159-Speed 127.08 samples/sec Loss 11.0132 Epoch: 7 Global Step: 77945 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:26:46,748-Speed 130.69 samples/sec Loss 10.5844 Epoch: 7 Global Step: 77950 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:27:06,556-Speed 129.25 samples/sec Loss 10.6679 Epoch: 7 Global Step: 77955 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:27:26,148-Speed 130.67 samples/sec Loss 10.7269 Epoch: 7 Global Step: 77960 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:27:45,778-Speed 130.42 samples/sec Loss 10.6574 Epoch: 7 Global Step: 77965 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:28:05,415-Speed 130.37 samples/sec Loss 10.8106 Epoch: 7 Global Step: 77970 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:28:25,044-Speed 130.42 samples/sec Loss 10.4542 Epoch: 7 Global Step: 77975 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:28:44,720-Speed 130.12 samples/sec Loss 10.7710 Epoch: 7 Global Step: 77980 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:29:04,805-Speed 127.46 samples/sec Loss 10.8151 Epoch: 7 Global Step: 77985 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:29:24,886-Speed 127.49 samples/sec Loss 10.8527 Epoch: 7 Global Step: 77990 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:29:44,449-Speed 130.87 samples/sec Loss 10.7110 Epoch: 7 Global Step: 77995 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:30:04,218-Speed 129.50 samples/sec Loss 10.9518 Epoch: 7 Global Step: 78000 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:30:23,806-Speed 130.70 samples/sec Loss 11.0422 Epoch: 7 Global Step: 78005 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:30:43,571-Speed 129.53 samples/sec Loss 11.0505 Epoch: 7 Global Step: 78010 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:31:03,207-Speed 130.38 samples/sec Loss 11.0268 Epoch: 7 Global Step: 78015 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:31:22,962-Speed 129.60 samples/sec Loss 10.7271 Epoch: 7 Global Step: 78020 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:31:42,579-Speed 130.51 samples/sec Loss 10.8952 Epoch: 7 Global Step: 78025 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:32:02,444-Speed 128.87 samples/sec Loss 10.6872 Epoch: 7 Global Step: 78030 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:32:22,372-Speed 128.47 samples/sec Loss 10.3522 Epoch: 7 Global Step: 78035 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:32:41,952-Speed 130.75 samples/sec Loss 10.7769 Epoch: 7 Global Step: 78040 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:33:01,854-Speed 128.64 samples/sec Loss 10.6612 Epoch: 7 Global Step: 78045 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:33:21,353-Speed 131.29 samples/sec Loss 10.7401 Epoch: 7 Global Step: 78050 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:33:41,344-Speed 128.06 samples/sec Loss 10.8258 Epoch: 7 Global Step: 78055 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:34:00,886-Speed 131.01 samples/sec Loss 10.8010 Epoch: 7 Global Step: 78060 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:34:20,718-Speed 129.09 samples/sec Loss 10.9511 Epoch: 7 Global Step: 78065 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:34:40,280-Speed 130.88 samples/sec Loss 10.6463 Epoch: 7 Global Step: 78070 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:35:00,390-Speed 127.30 samples/sec Loss 10.8458 Epoch: 7 Global Step: 78075 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:35:20,107-Speed 129.84 samples/sec Loss 10.6884 Epoch: 7 Global Step: 78080 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:35:39,740-Speed 130.40 samples/sec Loss 11.2625 Epoch: 7 Global Step: 78085 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:35:59,412-Speed 130.14 samples/sec Loss 10.7003 Epoch: 7 Global Step: 78090 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:36:19,051-Speed 130.36 samples/sec Loss 10.7345 Epoch: 7 Global Step: 78095 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:36:38,838-Speed 129.38 samples/sec Loss 10.9617 Epoch: 7 Global Step: 78100 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:36:58,453-Speed 130.52 samples/sec Loss 11.3706 Epoch: 7 Global Step: 78105 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:37:18,354-Speed 128.64 samples/sec Loss 11.0000 Epoch: 7 Global Step: 78110 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:37:37,966-Speed 130.54 samples/sec Loss 11.0207 Epoch: 7 Global Step: 78115 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:37:58,153-Speed 126.82 samples/sec Loss 10.3555 Epoch: 7 Global Step: 78120 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:38:18,147-Speed 128.04 samples/sec Loss 10.6844 Epoch: 7 Global Step: 78125 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:38:37,787-Speed 130.35 samples/sec Loss 11.0879 Epoch: 7 Global Step: 78130 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:38:57,527-Speed 129.69 samples/sec Loss 11.1577 Epoch: 7 Global Step: 78135 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:39:17,186-Speed 130.23 samples/sec Loss 11.0169 Epoch: 7 Global Step: 78140 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:39:36,930-Speed 129.67 samples/sec Loss 10.6172 Epoch: 7 Global Step: 78145 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:39:56,517-Speed 130.70 samples/sec Loss 10.7255 Epoch: 7 Global Step: 78150 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:40:16,524-Speed 127.96 samples/sec Loss 10.6910 Epoch: 7 Global Step: 78155 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:40:36,155-Speed 130.42 samples/sec Loss 10.9861 Epoch: 7 Global Step: 78160 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:40:56,167-Speed 127.93 samples/sec Loss 10.7878 Epoch: 7 Global Step: 78165 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:41:16,103-Speed 128.42 samples/sec Loss 10.8393 Epoch: 7 Global Step: 78170 Fp16 Grad Scale: 8192 Required: 203 hours Training: 2021-06-10 15:41:35,661-Speed 130.90 samples/sec Loss 10.8535 Epoch: 7 Global Step: 78175 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:41:55,593-Speed 128.44 samples/sec Loss 11.0297 Epoch: 7 Global Step: 78180 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:42:15,156-Speed 130.87 samples/sec Loss 10.7208 Epoch: 7 Global Step: 78185 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:42:34,972-Speed 129.20 samples/sec Loss 10.4632 Epoch: 7 Global Step: 78190 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:42:54,588-Speed 130.51 samples/sec Loss 10.7732 Epoch: 7 Global Step: 78195 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:43:14,336-Speed 129.64 samples/sec Loss 10.4588 Epoch: 7 Global Step: 78200 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:43:33,892-Speed 130.91 samples/sec Loss 10.7933 Epoch: 7 Global Step: 78205 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:43:53,869-Speed 128.15 samples/sec Loss 10.7201 Epoch: 7 Global Step: 78210 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:44:13,885-Speed 127.91 samples/sec Loss 10.7560 Epoch: 7 Global Step: 78215 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:44:33,490-Speed 130.58 samples/sec Loss 10.9516 Epoch: 7 Global Step: 78220 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:44:53,332-Speed 129.02 samples/sec Loss 11.0341 Epoch: 7 Global Step: 78225 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:45:12,931-Speed 130.63 samples/sec Loss 10.9554 Epoch: 7 Global Step: 78230 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:45:32,592-Speed 130.21 samples/sec Loss 10.8244 Epoch: 7 Global Step: 78235 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:45:52,179-Speed 130.70 samples/sec Loss 10.5155 Epoch: 7 Global Step: 78240 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:46:11,991-Speed 129.23 samples/sec Loss 10.7694 Epoch: 7 Global Step: 78245 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:46:31,629-Speed 130.37 samples/sec Loss 10.7017 Epoch: 7 Global Step: 78250 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:46:51,657-Speed 127.83 samples/sec Loss 10.7532 Epoch: 7 Global Step: 78255 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:47:11,533-Speed 128.81 samples/sec Loss 10.4959 Epoch: 7 Global Step: 78260 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:47:31,171-Speed 130.36 samples/sec Loss 10.5341 Epoch: 7 Global Step: 78265 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:47:51,031-Speed 128.91 samples/sec Loss 10.6548 Epoch: 7 Global Step: 78270 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:48:10,600-Speed 130.82 samples/sec Loss 10.7691 Epoch: 7 Global Step: 78275 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:48:30,519-Speed 128.53 samples/sec Loss 10.8568 Epoch: 7 Global Step: 78280 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:48:50,070-Speed 130.95 samples/sec Loss 10.4160 Epoch: 7 Global Step: 78285 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:49:09,867-Speed 129.32 samples/sec Loss 10.6621 Epoch: 7 Global Step: 78290 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:49:29,464-Speed 130.64 samples/sec Loss 11.0177 Epoch: 7 Global Step: 78295 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:49:49,538-Speed 127.53 samples/sec Loss 10.8289 Epoch: 7 Global Step: 78300 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:50:09,340-Speed 129.29 samples/sec Loss 10.7483 Epoch: 7 Global Step: 78305 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:50:28,816-Speed 131.45 samples/sec Loss 10.8789 Epoch: 7 Global Step: 78310 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:50:48,696-Speed 128.78 samples/sec Loss 10.8975 Epoch: 7 Global Step: 78315 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:51:08,185-Speed 131.37 samples/sec Loss 10.8811 Epoch: 7 Global Step: 78320 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:51:28,098-Speed 128.56 samples/sec Loss 10.6604 Epoch: 7 Global Step: 78325 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:51:47,709-Speed 130.54 samples/sec Loss 11.1177 Epoch: 7 Global Step: 78330 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:52:07,667-Speed 128.28 samples/sec Loss 10.8750 Epoch: 7 Global Step: 78335 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:52:27,213-Speed 130.98 samples/sec Loss 11.0121 Epoch: 7 Global Step: 78340 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:52:47,374-Speed 126.98 samples/sec Loss 10.7344 Epoch: 7 Global Step: 78345 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:53:07,060-Speed 130.05 samples/sec Loss 10.8435 Epoch: 7 Global Step: 78350 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:53:26,700-Speed 130.35 samples/sec Loss 10.7335 Epoch: 7 Global Step: 78355 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:53:46,442-Speed 129.68 samples/sec Loss 10.7039 Epoch: 7 Global Step: 78360 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:54:06,023-Speed 130.75 samples/sec Loss 10.5314 Epoch: 7 Global Step: 78365 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:54:26,019-Speed 128.04 samples/sec Loss 11.0959 Epoch: 7 Global Step: 78370 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:54:45,586-Speed 130.84 samples/sec Loss 10.6728 Epoch: 7 Global Step: 78375 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:55:05,520-Speed 128.43 samples/sec Loss 10.9509 Epoch: 7 Global Step: 78380 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:55:25,077-Speed 130.91 samples/sec Loss 11.0085 Epoch: 7 Global Step: 78385 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:55:45,175-Speed 127.38 samples/sec Loss 10.6505 Epoch: 7 Global Step: 78390 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:56:05,145-Speed 128.20 samples/sec Loss 10.9348 Epoch: 7 Global Step: 78395 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:56:24,747-Speed 130.60 samples/sec Loss 10.8697 Epoch: 7 Global Step: 78400 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:56:44,696-Speed 128.34 samples/sec Loss 10.7014 Epoch: 7 Global Step: 78405 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:57:04,356-Speed 130.23 samples/sec Loss 10.6177 Epoch: 7 Global Step: 78410 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:57:24,137-Speed 129.42 samples/sec Loss 10.5773 Epoch: 7 Global Step: 78415 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:57:43,735-Speed 130.63 samples/sec Loss 10.8096 Epoch: 7 Global Step: 78420 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:58:03,607-Speed 128.84 samples/sec Loss 10.5265 Epoch: 7 Global Step: 78425 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:58:23,189-Speed 130.74 samples/sec Loss 10.8376 Epoch: 7 Global Step: 78430 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:58:43,427-Speed 126.50 samples/sec Loss 10.8052 Epoch: 7 Global Step: 78435 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:59:03,416-Speed 128.08 samples/sec Loss 11.1151 Epoch: 7 Global Step: 78440 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:59:22,961-Speed 130.98 samples/sec Loss 10.8505 Epoch: 7 Global Step: 78445 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 15:59:42,875-Speed 128.56 samples/sec Loss 10.7695 Epoch: 7 Global Step: 78450 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:00:02,512-Speed 130.37 samples/sec Loss 10.8369 Epoch: 7 Global Step: 78455 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:00:22,352-Speed 129.04 samples/sec Loss 11.0601 Epoch: 7 Global Step: 78460 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:00:41,936-Speed 130.73 samples/sec Loss 10.9523 Epoch: 7 Global Step: 78465 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:01:01,697-Speed 129.56 samples/sec Loss 10.6584 Epoch: 7 Global Step: 78470 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:01:21,333-Speed 130.38 samples/sec Loss 10.8083 Epoch: 7 Global Step: 78475 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:01:41,259-Speed 128.48 samples/sec Loss 11.0496 Epoch: 7 Global Step: 78480 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:02:00,998-Speed 129.70 samples/sec Loss 11.0198 Epoch: 7 Global Step: 78485 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:02:20,555-Speed 130.91 samples/sec Loss 10.7933 Epoch: 7 Global Step: 78490 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:02:40,549-Speed 128.04 samples/sec Loss 10.9053 Epoch: 7 Global Step: 78495 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:03:00,030-Speed 131.42 samples/sec Loss 10.8753 Epoch: 7 Global Step: 78500 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:03:19,946-Speed 128.55 samples/sec Loss 11.0223 Epoch: 7 Global Step: 78505 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:03:39,512-Speed 130.85 samples/sec Loss 10.8867 Epoch: 7 Global Step: 78510 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:03:59,284-Speed 129.48 samples/sec Loss 10.8557 Epoch: 7 Global Step: 78515 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:04:18,922-Speed 130.37 samples/sec Loss 11.0548 Epoch: 7 Global Step: 78520 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:04:39,034-Speed 127.30 samples/sec Loss 10.8145 Epoch: 7 Global Step: 78525 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:04:58,942-Speed 128.60 samples/sec Loss 10.9483 Epoch: 7 Global Step: 78530 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:05:18,559-Speed 130.50 samples/sec Loss 11.4221 Epoch: 7 Global Step: 78535 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:05:38,355-Speed 129.33 samples/sec Loss 10.9199 Epoch: 7 Global Step: 78540 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:05:57,948-Speed 130.67 samples/sec Loss 10.9723 Epoch: 7 Global Step: 78545 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:06:17,774-Speed 129.13 samples/sec Loss 10.5230 Epoch: 7 Global Step: 78550 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:06:37,360-Speed 130.72 samples/sec Loss 10.5256 Epoch: 7 Global Step: 78555 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:06:58,288-Speed 122.33 samples/sec Loss 11.0262 Epoch: 7 Global Step: 78560 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:07:17,751-Speed 131.54 samples/sec Loss 10.7222 Epoch: 7 Global Step: 78565 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:07:37,866-Speed 127.27 samples/sec Loss 10.5405 Epoch: 7 Global Step: 78570 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:07:57,593-Speed 129.78 samples/sec Loss 10.6472 Epoch: 7 Global Step: 78575 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:08:17,217-Speed 130.46 samples/sec Loss 10.8381 Epoch: 7 Global Step: 78580 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:08:36,839-Speed 130.48 samples/sec Loss 10.7335 Epoch: 7 Global Step: 78585 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:08:56,410-Speed 130.81 samples/sec Loss 11.0220 Epoch: 7 Global Step: 78590 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:09:16,245-Speed 129.08 samples/sec Loss 10.9260 Epoch: 7 Global Step: 78595 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:09:35,880-Speed 130.39 samples/sec Loss 10.8374 Epoch: 7 Global Step: 78600 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:09:55,665-Speed 129.40 samples/sec Loss 10.9224 Epoch: 7 Global Step: 78605 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:10:15,180-Speed 131.18 samples/sec Loss 10.6948 Epoch: 7 Global Step: 78610 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:10:35,246-Speed 127.58 samples/sec Loss 10.9541 Epoch: 7 Global Step: 78615 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:10:55,350-Speed 127.34 samples/sec Loss 10.3990 Epoch: 7 Global Step: 78620 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:11:14,794-Speed 131.67 samples/sec Loss 10.9814 Epoch: 7 Global Step: 78625 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:11:34,577-Speed 129.41 samples/sec Loss 10.7781 Epoch: 7 Global Step: 78630 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:11:54,192-Speed 130.52 samples/sec Loss 10.8374 Epoch: 7 Global Step: 78635 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:12:13,929-Speed 129.72 samples/sec Loss 10.5531 Epoch: 7 Global Step: 78640 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:12:33,556-Speed 130.44 samples/sec Loss 10.7107 Epoch: 7 Global Step: 78645 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:12:53,194-Speed 130.36 samples/sec Loss 10.8061 Epoch: 7 Global Step: 78650 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:13:12,762-Speed 130.84 samples/sec Loss 10.6876 Epoch: 7 Global Step: 78655 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:13:32,834-Speed 127.55 samples/sec Loss 10.9132 Epoch: 7 Global Step: 78660 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:13:52,830-Speed 128.03 samples/sec Loss 10.9813 Epoch: 7 Global Step: 78665 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:14:12,294-Speed 131.53 samples/sec Loss 10.6211 Epoch: 7 Global Step: 78670 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:14:32,125-Speed 129.10 samples/sec Loss 10.4525 Epoch: 7 Global Step: 78675 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:14:51,685-Speed 130.88 samples/sec Loss 10.8460 Epoch: 7 Global Step: 78680 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:15:11,518-Speed 129.09 samples/sec Loss 10.5805 Epoch: 7 Global Step: 78685 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:15:31,142-Speed 130.46 samples/sec Loss 10.9895 Epoch: 7 Global Step: 78690 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:15:50,868-Speed 129.79 samples/sec Loss 10.5478 Epoch: 7 Global Step: 78695 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:16:10,507-Speed 130.36 samples/sec Loss 10.6903 Epoch: 7 Global Step: 78700 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:16:30,457-Speed 128.33 samples/sec Loss 10.9360 Epoch: 7 Global Step: 78705 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:16:50,335-Speed 128.79 samples/sec Loss 11.0318 Epoch: 7 Global Step: 78710 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:17:09,911-Speed 130.78 samples/sec Loss 10.7987 Epoch: 7 Global Step: 78715 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:17:29,856-Speed 128.37 samples/sec Loss 10.8214 Epoch: 7 Global Step: 78720 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:17:49,477-Speed 130.48 samples/sec Loss 10.7355 Epoch: 7 Global Step: 78725 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:18:09,557-Speed 127.50 samples/sec Loss 10.8637 Epoch: 7 Global Step: 78730 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:18:29,014-Speed 131.57 samples/sec Loss 10.9747 Epoch: 7 Global Step: 78735 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:18:48,799-Speed 129.40 samples/sec Loss 11.0718 Epoch: 7 Global Step: 78740 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:19:08,406-Speed 130.57 samples/sec Loss 10.5374 Epoch: 7 Global Step: 78745 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:19:28,584-Speed 126.87 samples/sec Loss 10.1666 Epoch: 7 Global Step: 78750 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:19:48,239-Speed 130.25 samples/sec Loss 10.8233 Epoch: 7 Global Step: 78755 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:20:07,811-Speed 130.81 samples/sec Loss 10.8423 Epoch: 7 Global Step: 78760 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:20:27,700-Speed 128.72 samples/sec Loss 10.7708 Epoch: 7 Global Step: 78765 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:20:47,298-Speed 130.63 samples/sec Loss 10.4520 Epoch: 7 Global Step: 78770 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:21:07,172-Speed 128.82 samples/sec Loss 10.7583 Epoch: 7 Global Step: 78775 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:21:26,734-Speed 130.87 samples/sec Loss 10.9472 Epoch: 7 Global Step: 78780 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:21:46,572-Speed 129.05 samples/sec Loss 10.9798 Epoch: 7 Global Step: 78785 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:22:06,218-Speed 130.31 samples/sec Loss 10.7344 Epoch: 7 Global Step: 78790 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:22:26,280-Speed 127.61 samples/sec Loss 10.7804 Epoch: 7 Global Step: 78795 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:22:46,036-Speed 129.59 samples/sec Loss 10.9691 Epoch: 7 Global Step: 78800 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:23:05,663-Speed 130.44 samples/sec Loss 10.8906 Epoch: 7 Global Step: 78805 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:23:25,465-Speed 129.29 samples/sec Loss 10.8421 Epoch: 7 Global Step: 78810 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:23:44,952-Speed 131.38 samples/sec Loss 10.9813 Epoch: 7 Global Step: 78815 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:24:04,817-Speed 128.88 samples/sec Loss 10.5089 Epoch: 7 Global Step: 78820 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:24:24,401-Speed 130.73 samples/sec Loss 10.7246 Epoch: 7 Global Step: 78825 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:24:44,309-Speed 128.60 samples/sec Loss 10.8645 Epoch: 7 Global Step: 78830 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:25:03,884-Speed 130.78 samples/sec Loss 10.8832 Epoch: 7 Global Step: 78835 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:25:23,881-Speed 128.03 samples/sec Loss 11.0289 Epoch: 7 Global Step: 78840 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:25:43,762-Speed 128.77 samples/sec Loss 10.9476 Epoch: 7 Global Step: 78845 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:26:03,523-Speed 129.55 samples/sec Loss 10.8547 Epoch: 7 Global Step: 78850 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:26:23,230-Speed 129.91 samples/sec Loss 10.8462 Epoch: 7 Global Step: 78855 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:26:42,844-Speed 130.53 samples/sec Loss 10.3963 Epoch: 7 Global Step: 78860 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:27:02,608-Speed 129.54 samples/sec Loss 10.7988 Epoch: 7 Global Step: 78865 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:27:22,348-Speed 129.70 samples/sec Loss 10.9052 Epoch: 7 Global Step: 78870 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:27:42,309-Speed 128.26 samples/sec Loss 11.0885 Epoch: 7 Global Step: 78875 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:28:01,775-Speed 131.52 samples/sec Loss 11.0064 Epoch: 7 Global Step: 78880 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:28:21,868-Speed 127.41 samples/sec Loss 11.0204 Epoch: 7 Global Step: 78885 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:28:41,830-Speed 128.25 samples/sec Loss 10.7754 Epoch: 7 Global Step: 78890 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:29:01,409-Speed 130.76 samples/sec Loss 10.7776 Epoch: 7 Global Step: 78895 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:29:21,093-Speed 130.06 samples/sec Loss 11.1012 Epoch: 7 Global Step: 78900 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 16:29:40,695-Speed 130.61 samples/sec Loss 10.7397 Epoch: 7 Global Step: 78905 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:30:00,450-Speed 129.60 samples/sec Loss 10.6399 Epoch: 7 Global Step: 78910 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:30:20,086-Speed 130.38 samples/sec Loss 10.7941 Epoch: 7 Global Step: 78915 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:30:39,833-Speed 129.65 samples/sec Loss 10.7615 Epoch: 7 Global Step: 78920 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:30:59,446-Speed 130.53 samples/sec Loss 10.7803 Epoch: 7 Global Step: 78925 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:31:19,399-Speed 128.31 samples/sec Loss 10.8076 Epoch: 7 Global Step: 78930 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:31:39,427-Speed 127.82 samples/sec Loss 10.9610 Epoch: 7 Global Step: 78935 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:31:58,886-Speed 131.56 samples/sec Loss 10.6068 Epoch: 7 Global Step: 78940 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:32:18,708-Speed 129.16 samples/sec Loss 10.9534 Epoch: 7 Global Step: 78945 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:32:38,283-Speed 130.79 samples/sec Loss 10.6909 Epoch: 7 Global Step: 78950 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:32:58,009-Speed 129.78 samples/sec Loss 10.5913 Epoch: 7 Global Step: 78955 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:33:17,638-Speed 130.43 samples/sec Loss 10.7020 Epoch: 7 Global Step: 78960 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:33:37,283-Speed 130.32 samples/sec Loss 10.8640 Epoch: 7 Global Step: 78965 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:33:56,911-Speed 130.44 samples/sec Loss 10.8380 Epoch: 7 Global Step: 78970 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:34:17,024-Speed 127.29 samples/sec Loss 10.6464 Epoch: 7 Global Step: 78975 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:34:36,761-Speed 129.72 samples/sec Loss 11.0478 Epoch: 7 Global Step: 78980 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:34:56,295-Speed 131.06 samples/sec Loss 11.0299 Epoch: 7 Global Step: 78985 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:35:16,170-Speed 128.81 samples/sec Loss 11.1628 Epoch: 7 Global Step: 78990 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:35:35,742-Speed 130.81 samples/sec Loss 10.8214 Epoch: 7 Global Step: 78995 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:35:55,715-Speed 128.17 samples/sec Loss 10.9836 Epoch: 7 Global Step: 79000 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:36:15,198-Speed 131.41 samples/sec Loss 10.9019 Epoch: 7 Global Step: 79005 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:36:34,929-Speed 129.75 samples/sec Loss 10.9384 Epoch: 7 Global Step: 79010 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:36:54,559-Speed 130.42 samples/sec Loss 10.7077 Epoch: 7 Global Step: 79015 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:37:14,567-Speed 127.95 samples/sec Loss 10.5778 Epoch: 7 Global Step: 79020 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:37:34,192-Speed 130.45 samples/sec Loss 10.2916 Epoch: 7 Global Step: 79025 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:37:53,732-Speed 131.02 samples/sec Loss 10.6284 Epoch: 7 Global Step: 79030 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:38:13,629-Speed 128.67 samples/sec Loss 10.8657 Epoch: 7 Global Step: 79035 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:38:33,129-Speed 131.29 samples/sec Loss 10.8782 Epoch: 7 Global Step: 79040 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:38:53,075-Speed 128.35 samples/sec Loss 11.0058 Epoch: 7 Global Step: 79045 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:39:12,528-Speed 131.61 samples/sec Loss 10.9494 Epoch: 7 Global Step: 79050 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:39:32,436-Speed 128.60 samples/sec Loss 10.8464 Epoch: 7 Global Step: 79055 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:39:52,022-Speed 130.71 samples/sec Loss 11.0176 Epoch: 7 Global Step: 79060 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:40:12,157-Speed 127.15 samples/sec Loss 10.4460 Epoch: 7 Global Step: 79065 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:40:31,829-Speed 130.14 samples/sec Loss 10.8324 Epoch: 7 Global Step: 79070 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:40:51,410-Speed 130.75 samples/sec Loss 10.7673 Epoch: 7 Global Step: 79075 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:41:11,222-Speed 129.22 samples/sec Loss 10.7782 Epoch: 7 Global Step: 79080 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:41:30,859-Speed 130.38 samples/sec Loss 11.1440 Epoch: 7 Global Step: 79085 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:41:50,683-Speed 129.14 samples/sec Loss 10.9342 Epoch: 7 Global Step: 79090 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:42:10,258-Speed 130.78 samples/sec Loss 10.7108 Epoch: 7 Global Step: 79095 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:42:30,177-Speed 128.53 samples/sec Loss 11.0961 Epoch: 7 Global Step: 79100 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:42:49,712-Speed 131.05 samples/sec Loss 11.0690 Epoch: 7 Global Step: 79105 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:43:09,825-Speed 127.29 samples/sec Loss 10.9978 Epoch: 7 Global Step: 79110 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:43:29,599-Speed 129.47 samples/sec Loss 10.8476 Epoch: 7 Global Step: 79115 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:43:49,179-Speed 130.75 samples/sec Loss 10.9755 Epoch: 7 Global Step: 79120 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:44:09,054-Speed 128.82 samples/sec Loss 10.9864 Epoch: 7 Global Step: 79125 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:44:28,610-Speed 130.91 samples/sec Loss 10.8027 Epoch: 7 Global Step: 79130 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:44:48,501-Speed 128.71 samples/sec Loss 11.1355 Epoch: 7 Global Step: 79135 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:45:08,064-Speed 130.86 samples/sec Loss 10.9141 Epoch: 7 Global Step: 79140 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:45:27,991-Speed 128.48 samples/sec Loss 10.8608 Epoch: 7 Global Step: 79145 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:45:47,590-Speed 130.63 samples/sec Loss 10.8928 Epoch: 7 Global Step: 79150 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:46:07,815-Speed 126.58 samples/sec Loss 10.7087 Epoch: 7 Global Step: 79155 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:46:27,560-Speed 129.66 samples/sec Loss 10.7510 Epoch: 7 Global Step: 79160 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:46:47,192-Speed 130.40 samples/sec Loss 10.8408 Epoch: 7 Global Step: 79165 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:47:06,993-Speed 129.29 samples/sec Loss 10.6974 Epoch: 7 Global Step: 79170 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:47:26,617-Speed 130.46 samples/sec Loss 10.9693 Epoch: 7 Global Step: 79175 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:47:46,245-Speed 130.43 samples/sec Loss 11.1256 Epoch: 7 Global Step: 79180 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:48:05,812-Speed 130.84 samples/sec Loss 10.7405 Epoch: 7 Global Step: 79185 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:48:25,789-Speed 128.16 samples/sec Loss 10.6106 Epoch: 7 Global Step: 79190 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:48:45,431-Speed 130.34 samples/sec Loss 10.6237 Epoch: 7 Global Step: 79195 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:49:05,502-Speed 127.55 samples/sec Loss 10.7366 Epoch: 7 Global Step: 79200 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:49:25,445-Speed 128.37 samples/sec Loss 10.7141 Epoch: 7 Global Step: 79205 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:49:45,052-Speed 130.57 samples/sec Loss 11.1555 Epoch: 7 Global Step: 79210 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:50:04,781-Speed 129.76 samples/sec Loss 11.1873 Epoch: 7 Global Step: 79215 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:50:24,409-Speed 130.43 samples/sec Loss 11.0240 Epoch: 7 Global Step: 79220 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:50:44,255-Speed 129.00 samples/sec Loss 10.8945 Epoch: 7 Global Step: 79225 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:51:03,848-Speed 130.67 samples/sec Loss 10.9736 Epoch: 7 Global Step: 79230 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:51:23,796-Speed 128.34 samples/sec Loss 10.8097 Epoch: 7 Global Step: 79235 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:51:43,376-Speed 130.76 samples/sec Loss 11.0670 Epoch: 7 Global Step: 79240 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:52:03,437-Speed 127.61 samples/sec Loss 10.6225 Epoch: 7 Global Step: 79245 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:52:23,431-Speed 128.05 samples/sec Loss 10.8728 Epoch: 7 Global Step: 79250 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:52:43,008-Speed 130.77 samples/sec Loss 10.7329 Epoch: 7 Global Step: 79255 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:53:02,798-Speed 129.37 samples/sec Loss 10.9799 Epoch: 7 Global Step: 79260 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:53:22,390-Speed 130.67 samples/sec Loss 10.8967 Epoch: 7 Global Step: 79265 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:53:42,254-Speed 128.88 samples/sec Loss 10.4980 Epoch: 7 Global Step: 79270 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:54:01,885-Speed 130.41 samples/sec Loss 10.7550 Epoch: 7 Global Step: 79275 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:54:21,590-Speed 129.93 samples/sec Loss 10.9174 Epoch: 7 Global Step: 79280 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:54:41,121-Speed 131.08 samples/sec Loss 10.8767 Epoch: 7 Global Step: 79285 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:55:01,220-Speed 127.37 samples/sec Loss 10.9862 Epoch: 7 Global Step: 79290 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:55:21,298-Speed 127.51 samples/sec Loss 10.5570 Epoch: 7 Global Step: 79295 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:55:40,842-Speed 130.99 samples/sec Loss 10.5527 Epoch: 7 Global Step: 79300 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:56:00,633-Speed 129.36 samples/sec Loss 10.9941 Epoch: 7 Global Step: 79305 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:56:20,235-Speed 130.61 samples/sec Loss 10.3533 Epoch: 7 Global Step: 79310 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:56:40,015-Speed 129.43 samples/sec Loss 11.1562 Epoch: 7 Global Step: 79315 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:56:59,676-Speed 130.21 samples/sec Loss 10.9297 Epoch: 7 Global Step: 79320 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:57:19,309-Speed 130.40 samples/sec Loss 10.6594 Epoch: 7 Global Step: 79325 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:57:38,874-Speed 130.85 samples/sec Loss 10.6072 Epoch: 7 Global Step: 79330 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:57:58,847-Speed 128.18 samples/sec Loss 10.9718 Epoch: 7 Global Step: 79335 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:58:18,959-Speed 127.30 samples/sec Loss 10.8591 Epoch: 7 Global Step: 79340 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:58:38,427-Speed 131.50 samples/sec Loss 11.1448 Epoch: 7 Global Step: 79345 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:58:58,374-Speed 128.35 samples/sec Loss 10.7724 Epoch: 7 Global Step: 79350 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:59:17,920-Speed 130.99 samples/sec Loss 11.5290 Epoch: 7 Global Step: 79355 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:59:37,806-Speed 128.74 samples/sec Loss 10.8397 Epoch: 7 Global Step: 79360 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 16:59:57,373-Speed 130.84 samples/sec Loss 10.8830 Epoch: 7 Global Step: 79365 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:00:17,124-Speed 129.62 samples/sec Loss 10.8744 Epoch: 7 Global Step: 79370 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:00:36,744-Speed 130.49 samples/sec Loss 10.9641 Epoch: 7 Global Step: 79375 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:00:56,837-Speed 127.42 samples/sec Loss 10.6404 Epoch: 7 Global Step: 79380 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:01:16,583-Speed 129.65 samples/sec Loss 10.6604 Epoch: 7 Global Step: 79385 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:01:36,143-Speed 130.90 samples/sec Loss 11.2616 Epoch: 7 Global Step: 79390 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:01:55,914-Speed 129.49 samples/sec Loss 10.7381 Epoch: 7 Global Step: 79395 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:02:15,543-Speed 130.42 samples/sec Loss 10.4517 Epoch: 7 Global Step: 79400 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:02:35,469-Speed 128.49 samples/sec Loss 10.9395 Epoch: 7 Global Step: 79405 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:02:55,641-Speed 126.92 samples/sec Loss 10.7471 Epoch: 7 Global Step: 79410 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:03:15,477-Speed 129.07 samples/sec Loss 10.7682 Epoch: 7 Global Step: 79415 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:03:35,101-Speed 130.46 samples/sec Loss 10.9702 Epoch: 7 Global Step: 79420 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:03:55,294-Speed 126.78 samples/sec Loss 11.0670 Epoch: 7 Global Step: 79425 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:04:15,035-Speed 129.69 samples/sec Loss 10.8338 Epoch: 7 Global Step: 79430 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:04:34,655-Speed 130.48 samples/sec Loss 10.4454 Epoch: 7 Global Step: 79435 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:04:54,400-Speed 129.66 samples/sec Loss 10.2041 Epoch: 7 Global Step: 79440 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:05:14,040-Speed 130.36 samples/sec Loss 10.7384 Epoch: 7 Global Step: 79445 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:05:33,893-Speed 128.95 samples/sec Loss 10.8246 Epoch: 7 Global Step: 79450 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:05:53,477-Speed 130.72 samples/sec Loss 10.7056 Epoch: 7 Global Step: 79455 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:06:13,376-Speed 128.66 samples/sec Loss 10.8281 Epoch: 7 Global Step: 79460 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:06:32,955-Speed 130.76 samples/sec Loss 10.5077 Epoch: 7 Global Step: 79465 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:06:53,057-Speed 127.36 samples/sec Loss 10.7282 Epoch: 7 Global Step: 79470 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:07:12,862-Speed 129.26 samples/sec Loss 10.4503 Epoch: 7 Global Step: 79475 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:07:32,455-Speed 130.66 samples/sec Loss 10.5653 Epoch: 7 Global Step: 79480 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:07:52,254-Speed 129.30 samples/sec Loss 10.5471 Epoch: 7 Global Step: 79485 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:08:11,879-Speed 130.46 samples/sec Loss 10.9118 Epoch: 7 Global Step: 79490 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:08:31,625-Speed 129.65 samples/sec Loss 10.9988 Epoch: 7 Global Step: 79495 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:08:51,207-Speed 130.74 samples/sec Loss 10.8574 Epoch: 7 Global Step: 79500 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:09:11,133-Speed 128.48 samples/sec Loss 10.7050 Epoch: 7 Global Step: 79505 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:09:30,791-Speed 130.23 samples/sec Loss 10.7218 Epoch: 7 Global Step: 79510 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:09:50,908-Speed 127.26 samples/sec Loss 10.7608 Epoch: 7 Global Step: 79515 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:10:10,785-Speed 128.80 samples/sec Loss 11.1330 Epoch: 7 Global Step: 79520 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:10:30,380-Speed 130.65 samples/sec Loss 11.0237 Epoch: 7 Global Step: 79525 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:10:50,105-Speed 129.79 samples/sec Loss 10.7331 Epoch: 7 Global Step: 79530 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:11:09,734-Speed 130.43 samples/sec Loss 10.7914 Epoch: 7 Global Step: 79535 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:11:29,428-Speed 129.99 samples/sec Loss 10.8307 Epoch: 7 Global Step: 79540 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:11:49,048-Speed 130.48 samples/sec Loss 10.9611 Epoch: 7 Global Step: 79545 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:12:08,780-Speed 129.75 samples/sec Loss 10.5324 Epoch: 7 Global Step: 79550 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:12:28,348-Speed 130.83 samples/sec Loss 10.6257 Epoch: 7 Global Step: 79555 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:12:48,356-Speed 127.96 samples/sec Loss 10.5161 Epoch: 7 Global Step: 79560 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:13:08,335-Speed 128.14 samples/sec Loss 10.6917 Epoch: 7 Global Step: 79565 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:13:27,947-Speed 130.54 samples/sec Loss 10.9733 Epoch: 7 Global Step: 79570 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:13:47,727-Speed 129.43 samples/sec Loss 10.8994 Epoch: 7 Global Step: 79575 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:14:07,317-Speed 130.69 samples/sec Loss 10.8563 Epoch: 7 Global Step: 79580 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:14:27,054-Speed 129.72 samples/sec Loss 10.7766 Epoch: 7 Global Step: 79585 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:14:46,712-Speed 130.23 samples/sec Loss 10.5935 Epoch: 7 Global Step: 79590 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:15:06,488-Speed 129.46 samples/sec Loss 11.0174 Epoch: 7 Global Step: 79595 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:15:26,064-Speed 130.78 samples/sec Loss 10.5629 Epoch: 7 Global Step: 79600 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:15:46,146-Speed 127.49 samples/sec Loss 10.9978 Epoch: 7 Global Step: 79605 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:16:06,158-Speed 127.93 samples/sec Loss 11.1932 Epoch: 7 Global Step: 79610 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:16:25,721-Speed 130.87 samples/sec Loss 11.0849 Epoch: 7 Global Step: 79615 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:16:45,615-Speed 128.69 samples/sec Loss 11.0293 Epoch: 7 Global Step: 79620 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:17:05,189-Speed 130.79 samples/sec Loss 11.0441 Epoch: 7 Global Step: 79625 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:17:25,032-Speed 129.02 samples/sec Loss 10.5000 Epoch: 7 Global Step: 79630 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 17:17:44,673-Speed 130.35 samples/sec Loss 10.6922 Epoch: 7 Global Step: 79635 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:18:04,432-Speed 129.57 samples/sec Loss 10.4568 Epoch: 7 Global Step: 79640 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:18:24,069-Speed 130.37 samples/sec Loss 10.4577 Epoch: 7 Global Step: 79645 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:18:44,144-Speed 127.53 samples/sec Loss 10.8308 Epoch: 7 Global Step: 79650 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:19:04,074-Speed 128.45 samples/sec Loss 10.9655 Epoch: 7 Global Step: 79655 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:19:23,655-Speed 130.75 samples/sec Loss 10.9380 Epoch: 7 Global Step: 79660 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:19:43,588-Speed 128.44 samples/sec Loss 10.5372 Epoch: 7 Global Step: 79665 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:20:03,154-Speed 130.84 samples/sec Loss 10.9461 Epoch: 7 Global Step: 79670 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:20:23,002-Speed 128.99 samples/sec Loss 10.6889 Epoch: 7 Global Step: 79675 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:20:42,616-Speed 130.53 samples/sec Loss 11.0120 Epoch: 7 Global Step: 79680 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:21:02,334-Speed 129.84 samples/sec Loss 11.0179 Epoch: 7 Global Step: 79685 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:21:22,039-Speed 129.92 samples/sec Loss 10.4414 Epoch: 7 Global Step: 79690 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:21:42,165-Speed 127.21 samples/sec Loss 10.7589 Epoch: 7 Global Step: 79695 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:22:01,931-Speed 129.51 samples/sec Loss 10.8364 Epoch: 7 Global Step: 79700 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:22:21,442-Speed 131.22 samples/sec Loss 11.1030 Epoch: 7 Global Step: 79705 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:22:41,235-Speed 129.34 samples/sec Loss 10.9927 Epoch: 7 Global Step: 79710 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:23:23,981-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-10 17:23:43,276-Speed 41.26 samples/sec Loss 10.6063 Epoch: 8 Global Step: 79715 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:24:02,729-Speed 131.60 samples/sec Loss 10.0975 Epoch: 8 Global Step: 79720 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:24:22,371-Speed 130.34 samples/sec Loss 10.1339 Epoch: 8 Global Step: 79725 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:24:42,004-Speed 130.40 samples/sec Loss 9.8695 Epoch: 8 Global Step: 79730 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:25:01,789-Speed 129.40 samples/sec Loss 10.0016 Epoch: 8 Global Step: 79735 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:25:21,595-Speed 129.26 samples/sec Loss 9.9678 Epoch: 8 Global Step: 79740 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:25:41,103-Speed 131.24 samples/sec Loss 10.4186 Epoch: 8 Global Step: 79745 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:26:00,754-Speed 130.28 samples/sec Loss 10.2970 Epoch: 8 Global Step: 79750 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:26:20,255-Speed 131.28 samples/sec Loss 10.2109 Epoch: 8 Global Step: 79755 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:26:39,895-Speed 130.36 samples/sec Loss 10.0393 Epoch: 8 Global Step: 79760 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:26:59,506-Speed 130.54 samples/sec Loss 10.3338 Epoch: 8 Global Step: 79765 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:27:18,979-Speed 131.47 samples/sec Loss 10.3816 Epoch: 8 Global Step: 79770 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:27:38,610-Speed 130.42 samples/sec Loss 9.9679 Epoch: 8 Global Step: 79775 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:27:58,554-Speed 128.36 samples/sec Loss 10.1602 Epoch: 8 Global Step: 79780 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:28:18,019-Speed 131.52 samples/sec Loss 10.2095 Epoch: 8 Global Step: 79785 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:28:37,617-Speed 130.63 samples/sec Loss 10.0142 Epoch: 8 Global Step: 79790 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:28:57,249-Speed 130.40 samples/sec Loss 10.3780 Epoch: 8 Global Step: 79795 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:29:16,836-Speed 130.71 samples/sec Loss 10.1838 Epoch: 8 Global Step: 79800 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:29:36,469-Speed 130.40 samples/sec Loss 10.1646 Epoch: 8 Global Step: 79805 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:29:55,927-Speed 131.58 samples/sec Loss 10.3124 Epoch: 8 Global Step: 79810 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:30:15,750-Speed 129.15 samples/sec Loss 9.8279 Epoch: 8 Global Step: 79815 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:30:35,274-Speed 131.13 samples/sec Loss 10.1600 Epoch: 8 Global Step: 79820 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:30:55,250-Speed 128.16 samples/sec Loss 10.4476 Epoch: 8 Global Step: 79825 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:31:14,702-Speed 131.61 samples/sec Loss 10.3397 Epoch: 8 Global Step: 79830 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:31:34,431-Speed 129.77 samples/sec Loss 9.9147 Epoch: 8 Global Step: 79835 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:31:53,989-Speed 130.90 samples/sec Loss 10.3816 Epoch: 8 Global Step: 79840 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:32:13,608-Speed 130.50 samples/sec Loss 10.2755 Epoch: 8 Global Step: 79845 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:32:33,284-Speed 130.11 samples/sec Loss 10.4688 Epoch: 8 Global Step: 79850 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:32:52,856-Speed 130.81 samples/sec Loss 10.1318 Epoch: 8 Global Step: 79855 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:33:12,477-Speed 130.48 samples/sec Loss 10.3104 Epoch: 8 Global Step: 79860 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:33:32,049-Speed 130.80 samples/sec Loss 9.9650 Epoch: 8 Global Step: 79865 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:33:51,870-Speed 129.17 samples/sec Loss 10.3964 Epoch: 8 Global Step: 79870 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:34:11,475-Speed 130.58 samples/sec Loss 10.5158 Epoch: 8 Global Step: 79875 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:34:31,165-Speed 130.02 samples/sec Loss 10.4547 Epoch: 8 Global Step: 79880 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:34:50,719-Speed 130.92 samples/sec Loss 10.3551 Epoch: 8 Global Step: 79885 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:35:10,369-Speed 130.29 samples/sec Loss 10.4663 Epoch: 8 Global Step: 79890 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:35:29,969-Speed 130.62 samples/sec Loss 10.1941 Epoch: 8 Global Step: 79895 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:35:49,529-Speed 130.89 samples/sec Loss 10.0704 Epoch: 8 Global Step: 79900 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:36:09,474-Speed 128.36 samples/sec Loss 10.3727 Epoch: 8 Global Step: 79905 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:36:29,053-Speed 130.76 samples/sec Loss 10.0948 Epoch: 8 Global Step: 79910 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:36:48,861-Speed 129.25 samples/sec Loss 10.1661 Epoch: 8 Global Step: 79915 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:37:08,634-Speed 129.47 samples/sec Loss 10.2146 Epoch: 8 Global Step: 79920 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:37:28,152-Speed 131.17 samples/sec Loss 10.6778 Epoch: 8 Global Step: 79925 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:37:47,765-Speed 130.53 samples/sec Loss 10.5546 Epoch: 8 Global Step: 79930 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:38:07,385-Speed 130.49 samples/sec Loss 10.4220 Epoch: 8 Global Step: 79935 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:38:26,863-Speed 131.44 samples/sec Loss 10.4781 Epoch: 8 Global Step: 79940 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:38:46,511-Speed 130.30 samples/sec Loss 10.5503 Epoch: 8 Global Step: 79945 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:39:06,059-Speed 130.97 samples/sec Loss 10.3386 Epoch: 8 Global Step: 79950 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:39:25,723-Speed 130.20 samples/sec Loss 10.2743 Epoch: 8 Global Step: 79955 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:39:45,526-Speed 129.28 samples/sec Loss 10.4965 Epoch: 8 Global Step: 79960 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:40:05,334-Speed 129.25 samples/sec Loss 10.2551 Epoch: 8 Global Step: 79965 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:40:24,906-Speed 130.82 samples/sec Loss 10.4767 Epoch: 8 Global Step: 79970 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:40:44,537-Speed 130.41 samples/sec Loss 10.3152 Epoch: 8 Global Step: 79975 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:41:04,017-Speed 131.42 samples/sec Loss 10.4456 Epoch: 8 Global Step: 79980 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:41:23,581-Speed 130.86 samples/sec Loss 10.4780 Epoch: 8 Global Step: 79985 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:41:43,175-Speed 130.66 samples/sec Loss 10.7760 Epoch: 8 Global Step: 79990 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:42:02,731-Speed 130.92 samples/sec Loss 10.3568 Epoch: 8 Global Step: 79995 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:42:22,299-Speed 130.83 samples/sec Loss 10.6831 Epoch: 8 Global Step: 80000 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 17:57:10,144-[lfw][80000]XNorm: 22.293777 Training: 2021-06-10 17:57:10,144-[lfw][80000]Accuracy-Flip: 0.99650+-0.00353 Training: 2021-06-10 17:57:10,144-[lfw][80000]Accuracy-Highest: 0.99750 Training: 2021-06-10 18:14:29,611-[cfp_fp][80000]XNorm: 20.815130 Training: 2021-06-10 18:14:29,611-[cfp_fp][80000]Accuracy-Flip: 0.97386+-0.00564 Training: 2021-06-10 18:14:29,611-[cfp_fp][80000]Accuracy-Highest: 0.97514 Training: 2021-06-10 18:29:19,818-[agedb_30][80000]XNorm: 22.143496 Training: 2021-06-10 18:29:19,818-[agedb_30][80000]Accuracy-Flip: 0.97233+-0.00837 Training: 2021-06-10 18:29:19,819-[agedb_30][80000]Accuracy-Highest: 0.97233 Training: 2021-06-10 18:29:39,496-Speed 0.90 samples/sec Loss 10.5743 Epoch: 8 Global Step: 80005 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:29:59,529-Speed 127.80 samples/sec Loss 10.6144 Epoch: 8 Global Step: 80010 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:30:19,368-Speed 129.05 samples/sec Loss 10.7022 Epoch: 8 Global Step: 80015 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:30:38,997-Speed 130.42 samples/sec Loss 10.7942 Epoch: 8 Global Step: 80020 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:30:58,647-Speed 130.29 samples/sec Loss 10.2912 Epoch: 8 Global Step: 80025 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:31:18,124-Speed 131.44 samples/sec Loss 10.1759 Epoch: 8 Global Step: 80030 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:31:38,047-Speed 128.50 samples/sec Loss 10.0549 Epoch: 8 Global Step: 80035 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:31:57,573-Speed 131.11 samples/sec Loss 10.5820 Epoch: 8 Global Step: 80040 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:32:17,457-Speed 128.76 samples/sec Loss 10.4167 Epoch: 8 Global Step: 80045 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:32:36,955-Speed 131.30 samples/sec Loss 10.4207 Epoch: 8 Global Step: 80050 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:32:57,015-Speed 127.62 samples/sec Loss 10.5095 Epoch: 8 Global Step: 80055 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:33:16,753-Speed 129.70 samples/sec Loss 10.3627 Epoch: 8 Global Step: 80060 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:33:36,373-Speed 130.48 samples/sec Loss 10.1280 Epoch: 8 Global Step: 80065 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:33:56,004-Speed 130.41 samples/sec Loss 10.5037 Epoch: 8 Global Step: 80070 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:34:15,514-Speed 131.22 samples/sec Loss 10.6134 Epoch: 8 Global Step: 80075 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:34:35,525-Speed 127.94 samples/sec Loss 10.3130 Epoch: 8 Global Step: 80080 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:34:55,164-Speed 130.36 samples/sec Loss 10.4679 Epoch: 8 Global Step: 80085 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:35:15,043-Speed 128.78 samples/sec Loss 10.7068 Epoch: 8 Global Step: 80090 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:35:34,516-Speed 131.47 samples/sec Loss 10.1634 Epoch: 8 Global Step: 80095 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:35:54,614-Speed 127.38 samples/sec Loss 10.3498 Epoch: 8 Global Step: 80100 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:36:14,461-Speed 129.00 samples/sec Loss 10.6478 Epoch: 8 Global Step: 80105 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:36:34,068-Speed 130.57 samples/sec Loss 10.0817 Epoch: 8 Global Step: 80110 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:36:53,713-Speed 130.32 samples/sec Loss 10.4175 Epoch: 8 Global Step: 80115 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:37:13,420-Speed 129.92 samples/sec Loss 10.5439 Epoch: 8 Global Step: 80120 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:37:33,367-Speed 128.35 samples/sec Loss 10.6666 Epoch: 8 Global Step: 80125 Fp16 Grad Scale: 8192 Required: 202 hours Training: 2021-06-10 18:37:52,939-Speed 130.81 samples/sec Loss 10.5164 Epoch: 8 Global Step: 80130 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:38:12,751-Speed 129.22 samples/sec Loss 10.8610 Epoch: 8 Global Step: 80135 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:38:32,315-Speed 130.86 samples/sec Loss 10.3365 Epoch: 8 Global Step: 80140 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:38:52,379-Speed 127.60 samples/sec Loss 10.8821 Epoch: 8 Global Step: 80145 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:39:12,280-Speed 128.64 samples/sec Loss 10.7978 Epoch: 8 Global Step: 80150 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:39:31,924-Speed 130.32 samples/sec Loss 10.5436 Epoch: 8 Global Step: 80155 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:39:51,674-Speed 129.63 samples/sec Loss 10.4736 Epoch: 8 Global Step: 80160 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:40:11,236-Speed 130.88 samples/sec Loss 10.4137 Epoch: 8 Global Step: 80165 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:40:31,100-Speed 128.88 samples/sec Loss 10.5488 Epoch: 8 Global Step: 80170 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:40:50,666-Speed 130.85 samples/sec Loss 10.3825 Epoch: 8 Global Step: 80175 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:41:10,551-Speed 128.75 samples/sec Loss 10.4599 Epoch: 8 Global Step: 80180 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:41:30,145-Speed 130.65 samples/sec Loss 10.5313 Epoch: 8 Global Step: 80185 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:41:50,158-Speed 127.93 samples/sec Loss 10.7610 Epoch: 8 Global Step: 80190 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:42:10,177-Speed 127.88 samples/sec Loss 10.4789 Epoch: 8 Global Step: 80195 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:42:29,736-Speed 130.90 samples/sec Loss 10.3879 Epoch: 8 Global Step: 80200 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:42:49,513-Speed 129.45 samples/sec Loss 10.4497 Epoch: 8 Global Step: 80205 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:43:09,147-Speed 130.39 samples/sec Loss 10.5285 Epoch: 8 Global Step: 80210 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:43:28,913-Speed 129.52 samples/sec Loss 10.3926 Epoch: 8 Global Step: 80215 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:43:48,556-Speed 130.33 samples/sec Loss 10.2485 Epoch: 8 Global Step: 80220 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:44:08,295-Speed 129.70 samples/sec Loss 10.7635 Epoch: 8 Global Step: 80225 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:44:27,878-Speed 130.73 samples/sec Loss 10.8955 Epoch: 8 Global Step: 80230 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:44:47,952-Speed 127.54 samples/sec Loss 10.5363 Epoch: 8 Global Step: 80235 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:45:07,990-Speed 127.76 samples/sec Loss 10.8101 Epoch: 8 Global Step: 80240 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:45:27,544-Speed 130.93 samples/sec Loss 10.8844 Epoch: 8 Global Step: 80245 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:45:47,358-Speed 129.20 samples/sec Loss 10.5195 Epoch: 8 Global Step: 80250 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:46:07,002-Speed 130.33 samples/sec Loss 10.4004 Epoch: 8 Global Step: 80255 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:46:26,825-Speed 129.15 samples/sec Loss 10.3686 Epoch: 8 Global Step: 80260 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:46:46,381-Speed 130.91 samples/sec Loss 10.5697 Epoch: 8 Global Step: 80265 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:47:06,135-Speed 129.60 samples/sec Loss 10.5821 Epoch: 8 Global Step: 80270 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:47:25,766-Speed 130.42 samples/sec Loss 10.5978 Epoch: 8 Global Step: 80275 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:47:45,864-Speed 127.37 samples/sec Loss 10.1166 Epoch: 8 Global Step: 80280 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:48:05,878-Speed 127.92 samples/sec Loss 10.5556 Epoch: 8 Global Step: 80285 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:48:25,394-Speed 131.18 samples/sec Loss 10.4665 Epoch: 8 Global Step: 80290 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:48:45,320-Speed 128.48 samples/sec Loss 10.2807 Epoch: 8 Global Step: 80295 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:49:04,976-Speed 130.25 samples/sec Loss 10.6646 Epoch: 8 Global Step: 80300 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:49:24,887-Speed 128.58 samples/sec Loss 10.5949 Epoch: 8 Global Step: 80305 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:49:44,454-Speed 130.84 samples/sec Loss 10.6475 Epoch: 8 Global Step: 80310 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:50:04,190-Speed 129.72 samples/sec Loss 10.5152 Epoch: 8 Global Step: 80315 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:50:23,784-Speed 130.66 samples/sec Loss 10.2542 Epoch: 8 Global Step: 80320 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:50:43,980-Speed 126.76 samples/sec Loss 10.6141 Epoch: 8 Global Step: 80325 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:51:03,732-Speed 129.61 samples/sec Loss 10.7153 Epoch: 8 Global Step: 80330 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:51:23,364-Speed 130.40 samples/sec Loss 10.6904 Epoch: 8 Global Step: 80335 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:51:43,142-Speed 129.44 samples/sec Loss 10.7743 Epoch: 8 Global Step: 80340 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:52:02,848-Speed 129.92 samples/sec Loss 11.0337 Epoch: 8 Global Step: 80345 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:52:22,725-Speed 128.80 samples/sec Loss 10.6210 Epoch: 8 Global Step: 80350 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:52:42,314-Speed 130.69 samples/sec Loss 10.5049 Epoch: 8 Global Step: 80355 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:53:02,137-Speed 129.15 samples/sec Loss 10.4578 Epoch: 8 Global Step: 80360 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:53:21,700-Speed 130.87 samples/sec Loss 10.8508 Epoch: 8 Global Step: 80365 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:53:41,804-Speed 127.34 samples/sec Loss 10.4648 Epoch: 8 Global Step: 80370 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:54:01,723-Speed 128.53 samples/sec Loss 10.6751 Epoch: 8 Global Step: 80375 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:54:21,294-Speed 130.81 samples/sec Loss 10.5893 Epoch: 8 Global Step: 80380 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:54:41,106-Speed 129.22 samples/sec Loss 10.4126 Epoch: 8 Global Step: 80385 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:55:00,737-Speed 130.41 samples/sec Loss 10.9584 Epoch: 8 Global Step: 80390 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:55:20,492-Speed 129.59 samples/sec Loss 10.5640 Epoch: 8 Global Step: 80395 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:55:40,131-Speed 130.36 samples/sec Loss 10.7315 Epoch: 8 Global Step: 80400 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:55:59,760-Speed 130.43 samples/sec Loss 10.8206 Epoch: 8 Global Step: 80405 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:56:19,415-Speed 130.26 samples/sec Loss 10.8332 Epoch: 8 Global Step: 80410 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:56:39,501-Speed 127.46 samples/sec Loss 10.4430 Epoch: 8 Global Step: 80415 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:56:59,457-Speed 128.29 samples/sec Loss 10.9027 Epoch: 8 Global Step: 80420 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:57:18,903-Speed 131.65 samples/sec Loss 10.6388 Epoch: 8 Global Step: 80425 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:57:38,902-Speed 128.02 samples/sec Loss 10.4515 Epoch: 8 Global Step: 80430 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:57:58,448-Speed 130.98 samples/sec Loss 10.5670 Epoch: 8 Global Step: 80435 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:58:18,229-Speed 129.43 samples/sec Loss 10.5249 Epoch: 8 Global Step: 80440 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:58:37,811-Speed 130.74 samples/sec Loss 10.6736 Epoch: 8 Global Step: 80445 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:58:57,441-Speed 130.42 samples/sec Loss 10.5914 Epoch: 8 Global Step: 80450 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:59:17,118-Speed 130.11 samples/sec Loss 10.6798 Epoch: 8 Global Step: 80455 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:59:37,115-Speed 128.02 samples/sec Loss 10.7641 Epoch: 8 Global Step: 80460 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 18:59:56,884-Speed 129.50 samples/sec Loss 10.6342 Epoch: 8 Global Step: 80465 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:00:16,436-Speed 130.94 samples/sec Loss 10.5194 Epoch: 8 Global Step: 80470 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:00:36,445-Speed 127.95 samples/sec Loss 10.9929 Epoch: 8 Global Step: 80475 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:00:56,007-Speed 130.88 samples/sec Loss 10.8804 Epoch: 8 Global Step: 80480 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:01:15,823-Speed 129.19 samples/sec Loss 10.8881 Epoch: 8 Global Step: 80485 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:01:35,282-Speed 131.56 samples/sec Loss 10.9180 Epoch: 8 Global Step: 80490 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:01:55,253-Speed 128.20 samples/sec Loss 10.6566 Epoch: 8 Global Step: 80495 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:02:14,824-Speed 130.81 samples/sec Loss 10.9306 Epoch: 8 Global Step: 80500 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:02:34,833-Speed 127.95 samples/sec Loss 10.6358 Epoch: 8 Global Step: 80505 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:02:54,457-Speed 130.46 samples/sec Loss 10.5831 Epoch: 8 Global Step: 80510 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:03:14,092-Speed 130.38 samples/sec Loss 10.4304 Epoch: 8 Global Step: 80515 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:03:33,879-Speed 129.39 samples/sec Loss 10.3666 Epoch: 8 Global Step: 80520 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:03:53,370-Speed 131.35 samples/sec Loss 10.5382 Epoch: 8 Global Step: 80525 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:04:13,174-Speed 129.27 samples/sec Loss 10.7281 Epoch: 8 Global Step: 80530 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:04:32,743-Speed 130.82 samples/sec Loss 10.3521 Epoch: 8 Global Step: 80535 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:04:52,737-Speed 128.04 samples/sec Loss 10.6166 Epoch: 8 Global Step: 80540 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:05:12,320-Speed 130.74 samples/sec Loss 10.7239 Epoch: 8 Global Step: 80545 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:05:32,306-Speed 128.10 samples/sec Loss 10.6928 Epoch: 8 Global Step: 80550 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:05:53,366-Speed 121.56 samples/sec Loss 10.7407 Epoch: 8 Global Step: 80555 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:06:12,986-Speed 130.48 samples/sec Loss 10.5729 Epoch: 8 Global Step: 80560 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:06:32,645-Speed 130.22 samples/sec Loss 10.4802 Epoch: 8 Global Step: 80565 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:06:52,362-Speed 129.84 samples/sec Loss 10.8101 Epoch: 8 Global Step: 80570 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:07:12,047-Speed 130.06 samples/sec Loss 10.8333 Epoch: 8 Global Step: 80575 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:07:31,634-Speed 130.71 samples/sec Loss 10.7316 Epoch: 8 Global Step: 80580 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:07:51,523-Speed 128.72 samples/sec Loss 10.4205 Epoch: 8 Global Step: 80585 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:08:10,991-Speed 131.50 samples/sec Loss 10.6599 Epoch: 8 Global Step: 80590 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:08:30,995-Speed 127.98 samples/sec Loss 10.5130 Epoch: 8 Global Step: 80595 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:08:51,057-Speed 127.61 samples/sec Loss 10.9795 Epoch: 8 Global Step: 80600 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:09:10,640-Speed 130.73 samples/sec Loss 10.7624 Epoch: 8 Global Step: 80605 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:09:30,354-Speed 129.87 samples/sec Loss 10.8902 Epoch: 8 Global Step: 80610 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:09:49,911-Speed 130.90 samples/sec Loss 10.5010 Epoch: 8 Global Step: 80615 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:10:09,646-Speed 129.73 samples/sec Loss 10.5086 Epoch: 8 Global Step: 80620 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:10:29,278-Speed 130.41 samples/sec Loss 10.5513 Epoch: 8 Global Step: 80625 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:10:48,940-Speed 130.21 samples/sec Loss 10.6395 Epoch: 8 Global Step: 80630 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:11:08,426-Speed 131.38 samples/sec Loss 10.3611 Epoch: 8 Global Step: 80635 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:11:28,387-Speed 128.26 samples/sec Loss 10.4627 Epoch: 8 Global Step: 80640 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:11:48,496-Speed 127.31 samples/sec Loss 10.6990 Epoch: 8 Global Step: 80645 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:12:07,981-Speed 131.39 samples/sec Loss 10.6603 Epoch: 8 Global Step: 80650 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:12:27,801-Speed 129.17 samples/sec Loss 10.4619 Epoch: 8 Global Step: 80655 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:12:47,294-Speed 131.34 samples/sec Loss 10.7528 Epoch: 8 Global Step: 80660 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:13:07,075-Speed 129.43 samples/sec Loss 10.4915 Epoch: 8 Global Step: 80665 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:13:26,715-Speed 130.36 samples/sec Loss 10.5788 Epoch: 8 Global Step: 80670 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:13:46,364-Speed 130.29 samples/sec Loss 10.3783 Epoch: 8 Global Step: 80675 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:14:05,997-Speed 130.40 samples/sec Loss 11.0881 Epoch: 8 Global Step: 80680 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:14:25,929-Speed 128.45 samples/sec Loss 11.1211 Epoch: 8 Global Step: 80685 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:14:46,027-Speed 127.38 samples/sec Loss 10.7477 Epoch: 8 Global Step: 80690 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:15:05,474-Speed 131.65 samples/sec Loss 10.8719 Epoch: 8 Global Step: 80695 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:15:25,310-Speed 129.06 samples/sec Loss 10.6317 Epoch: 8 Global Step: 80700 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:15:44,862-Speed 130.95 samples/sec Loss 10.5341 Epoch: 8 Global Step: 80705 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:16:04,759-Speed 128.67 samples/sec Loss 10.7487 Epoch: 8 Global Step: 80710 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:16:24,361-Speed 130.60 samples/sec Loss 10.9693 Epoch: 8 Global Step: 80715 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:16:43,990-Speed 130.43 samples/sec Loss 10.5201 Epoch: 8 Global Step: 80720 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:17:03,629-Speed 130.36 samples/sec Loss 10.6201 Epoch: 8 Global Step: 80725 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:17:23,596-Speed 128.21 samples/sec Loss 10.4585 Epoch: 8 Global Step: 80730 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:17:43,594-Speed 128.02 samples/sec Loss 10.4656 Epoch: 8 Global Step: 80735 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:18:03,556-Speed 128.25 samples/sec Loss 11.1693 Epoch: 8 Global Step: 80740 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:18:23,445-Speed 128.72 samples/sec Loss 10.7818 Epoch: 8 Global Step: 80745 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:18:43,009-Speed 130.86 samples/sec Loss 10.9665 Epoch: 8 Global Step: 80750 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:19:02,989-Speed 128.14 samples/sec Loss 10.7313 Epoch: 8 Global Step: 80755 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:19:22,459-Speed 131.49 samples/sec Loss 10.7920 Epoch: 8 Global Step: 80760 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:19:42,136-Speed 130.11 samples/sec Loss 10.7191 Epoch: 8 Global Step: 80765 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:20:01,833-Speed 129.98 samples/sec Loss 10.9069 Epoch: 8 Global Step: 80770 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:20:21,929-Speed 127.39 samples/sec Loss 11.0878 Epoch: 8 Global Step: 80775 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:20:41,568-Speed 130.36 samples/sec Loss 10.7180 Epoch: 8 Global Step: 80780 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:21:01,101-Speed 131.06 samples/sec Loss 10.7588 Epoch: 8 Global Step: 80785 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:21:21,082-Speed 128.13 samples/sec Loss 10.5847 Epoch: 8 Global Step: 80790 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:21:40,661-Speed 130.76 samples/sec Loss 10.6688 Epoch: 8 Global Step: 80795 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:22:00,616-Speed 128.30 samples/sec Loss 10.5090 Epoch: 8 Global Step: 80800 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:22:20,081-Speed 131.53 samples/sec Loss 11.2613 Epoch: 8 Global Step: 80805 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:22:39,871-Speed 129.37 samples/sec Loss 10.6498 Epoch: 8 Global Step: 80810 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:22:59,418-Speed 130.97 samples/sec Loss 10.5946 Epoch: 8 Global Step: 80815 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:23:19,532-Speed 127.28 samples/sec Loss 10.9657 Epoch: 8 Global Step: 80820 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:23:39,174-Speed 130.34 samples/sec Loss 10.5925 Epoch: 8 Global Step: 80825 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:23:58,790-Speed 130.51 samples/sec Loss 10.9065 Epoch: 8 Global Step: 80830 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:24:18,667-Speed 128.80 samples/sec Loss 10.4511 Epoch: 8 Global Step: 80835 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:24:38,316-Speed 130.29 samples/sec Loss 11.1557 Epoch: 8 Global Step: 80840 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:24:58,194-Speed 128.79 samples/sec Loss 10.8287 Epoch: 8 Global Step: 80845 Fp16 Grad Scale: 8192 Required: 201 hours Training: 2021-06-10 19:25:17,656-Speed 131.55 samples/sec Loss 10.8497 Epoch: 8 Global Step: 80850 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:25:37,553-Speed 128.66 samples/sec Loss 11.1647 Epoch: 8 Global Step: 80855 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:25:57,125-Speed 130.81 samples/sec Loss 10.7825 Epoch: 8 Global Step: 80860 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:26:17,294-Speed 126.93 samples/sec Loss 10.5347 Epoch: 8 Global Step: 80865 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:26:37,027-Speed 129.74 samples/sec Loss 10.9144 Epoch: 8 Global Step: 80870 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:26:56,722-Speed 129.99 samples/sec Loss 10.1281 Epoch: 8 Global Step: 80875 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:27:16,478-Speed 129.59 samples/sec Loss 10.5941 Epoch: 8 Global Step: 80880 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:27:36,115-Speed 130.38 samples/sec Loss 10.9677 Epoch: 8 Global Step: 80885 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:27:55,919-Speed 129.28 samples/sec Loss 10.6678 Epoch: 8 Global Step: 80890 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:28:15,408-Speed 131.36 samples/sec Loss 10.6791 Epoch: 8 Global Step: 80895 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:28:35,337-Speed 128.47 samples/sec Loss 10.2473 Epoch: 8 Global Step: 80900 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:28:55,278-Speed 128.39 samples/sec Loss 10.5675 Epoch: 8 Global Step: 80905 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:29:15,750-Speed 125.05 samples/sec Loss 10.9692 Epoch: 8 Global Step: 80910 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:29:35,555-Speed 129.27 samples/sec Loss 10.9058 Epoch: 8 Global Step: 80915 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:29:55,165-Speed 130.55 samples/sec Loss 10.8313 Epoch: 8 Global Step: 80920 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:30:14,874-Speed 129.89 samples/sec Loss 10.9460 Epoch: 8 Global Step: 80925 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:30:34,506-Speed 130.41 samples/sec Loss 10.9073 Epoch: 8 Global Step: 80930 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:30:54,187-Speed 130.08 samples/sec Loss 10.7526 Epoch: 8 Global Step: 80935 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:31:13,776-Speed 130.69 samples/sec Loss 10.7253 Epoch: 8 Global Step: 80940 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:31:33,655-Speed 128.79 samples/sec Loss 10.6979 Epoch: 8 Global Step: 80945 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:31:53,304-Speed 130.30 samples/sec Loss 10.7736 Epoch: 8 Global Step: 80950 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:32:13,412-Speed 127.32 samples/sec Loss 10.6119 Epoch: 8 Global Step: 80955 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:32:33,278-Speed 128.87 samples/sec Loss 10.8704 Epoch: 8 Global Step: 80960 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:32:52,861-Speed 130.73 samples/sec Loss 10.7429 Epoch: 8 Global Step: 80965 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:33:12,680-Speed 129.18 samples/sec Loss 10.8305 Epoch: 8 Global Step: 80970 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:33:32,292-Speed 130.54 samples/sec Loss 10.8346 Epoch: 8 Global Step: 80975 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:33:51,956-Speed 130.20 samples/sec Loss 10.6241 Epoch: 8 Global Step: 80980 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:34:11,575-Speed 130.49 samples/sec Loss 11.0519 Epoch: 8 Global Step: 80985 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:34:31,261-Speed 130.05 samples/sec Loss 11.0196 Epoch: 8 Global Step: 80990 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:34:51,003-Speed 129.69 samples/sec Loss 10.5904 Epoch: 8 Global Step: 80995 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:35:11,065-Speed 127.61 samples/sec Loss 10.8112 Epoch: 8 Global Step: 81000 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:35:31,093-Speed 127.83 samples/sec Loss 10.9373 Epoch: 8 Global Step: 81005 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:35:50,537-Speed 131.67 samples/sec Loss 10.6271 Epoch: 8 Global Step: 81010 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:36:10,412-Speed 128.81 samples/sec Loss 11.1628 Epoch: 8 Global Step: 81015 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:36:30,030-Speed 130.51 samples/sec Loss 10.9183 Epoch: 8 Global Step: 81020 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:36:49,734-Speed 129.93 samples/sec Loss 10.9662 Epoch: 8 Global Step: 81025 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:37:09,378-Speed 130.32 samples/sec Loss 10.7737 Epoch: 8 Global Step: 81030 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:37:29,119-Speed 129.69 samples/sec Loss 10.7869 Epoch: 8 Global Step: 81035 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:37:48,754-Speed 130.38 samples/sec Loss 11.1198 Epoch: 8 Global Step: 81040 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:38:08,825-Speed 127.56 samples/sec Loss 10.5364 Epoch: 8 Global Step: 81045 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:38:28,773-Speed 128.34 samples/sec Loss 10.6447 Epoch: 8 Global Step: 81050 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:38:48,332-Speed 130.89 samples/sec Loss 10.6635 Epoch: 8 Global Step: 81055 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:39:08,291-Speed 128.27 samples/sec Loss 11.1716 Epoch: 8 Global Step: 81060 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:39:27,845-Speed 130.92 samples/sec Loss 10.5964 Epoch: 8 Global Step: 81065 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:39:47,660-Speed 129.21 samples/sec Loss 10.8373 Epoch: 8 Global Step: 81070 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:40:07,253-Speed 130.67 samples/sec Loss 10.7732 Epoch: 8 Global Step: 81075 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:40:27,003-Speed 129.62 samples/sec Loss 10.6845 Epoch: 8 Global Step: 81080 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:40:46,599-Speed 130.65 samples/sec Loss 10.7721 Epoch: 8 Global Step: 81085 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:41:06,727-Speed 127.19 samples/sec Loss 10.6674 Epoch: 8 Global Step: 81090 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:41:26,296-Speed 130.82 samples/sec Loss 10.5183 Epoch: 8 Global Step: 81095 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:41:45,929-Speed 130.40 samples/sec Loss 10.8421 Epoch: 8 Global Step: 81100 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:42:05,811-Speed 128.77 samples/sec Loss 10.7011 Epoch: 8 Global Step: 81105 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:42:25,386-Speed 130.79 samples/sec Loss 10.4864 Epoch: 8 Global Step: 81110 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:42:45,259-Speed 128.82 samples/sec Loss 10.9493 Epoch: 8 Global Step: 81115 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:43:04,763-Speed 131.26 samples/sec Loss 10.8825 Epoch: 8 Global Step: 81120 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:43:24,716-Speed 128.31 samples/sec Loss 10.7090 Epoch: 8 Global Step: 81125 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:43:44,306-Speed 130.68 samples/sec Loss 11.0502 Epoch: 8 Global Step: 81130 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:44:04,411-Speed 127.34 samples/sec Loss 11.0095 Epoch: 8 Global Step: 81135 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:44:24,120-Speed 129.90 samples/sec Loss 10.5884 Epoch: 8 Global Step: 81140 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:44:43,752-Speed 130.40 samples/sec Loss 10.8628 Epoch: 8 Global Step: 81145 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:45:03,471-Speed 129.83 samples/sec Loss 10.8045 Epoch: 8 Global Step: 81150 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:45:23,041-Speed 130.82 samples/sec Loss 10.6799 Epoch: 8 Global Step: 81155 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:45:42,802-Speed 129.56 samples/sec Loss 10.7650 Epoch: 8 Global Step: 81160 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:46:02,428-Speed 130.44 samples/sec Loss 10.7824 Epoch: 8 Global Step: 81165 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:46:22,350-Speed 128.51 samples/sec Loss 10.7201 Epoch: 8 Global Step: 81170 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:46:41,911-Speed 130.88 samples/sec Loss 10.4951 Epoch: 8 Global Step: 81175 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:47:02,029-Speed 127.26 samples/sec Loss 10.7066 Epoch: 8 Global Step: 81180 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:47:21,872-Speed 129.02 samples/sec Loss 10.6313 Epoch: 8 Global Step: 81185 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:47:41,469-Speed 130.64 samples/sec Loss 10.9577 Epoch: 8 Global Step: 81190 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:48:01,199-Speed 129.75 samples/sec Loss 10.8372 Epoch: 8 Global Step: 81195 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:48:20,838-Speed 130.36 samples/sec Loss 10.9373 Epoch: 8 Global Step: 81200 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:48:40,473-Speed 130.39 samples/sec Loss 10.8174 Epoch: 8 Global Step: 81205 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:49:00,111-Speed 130.37 samples/sec Loss 10.5878 Epoch: 8 Global Step: 81210 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:49:19,917-Speed 129.26 samples/sec Loss 10.9785 Epoch: 8 Global Step: 81215 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:49:39,502-Speed 130.72 samples/sec Loss 10.7736 Epoch: 8 Global Step: 81220 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:49:59,545-Speed 127.73 samples/sec Loss 10.6946 Epoch: 8 Global Step: 81225 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:50:19,546-Speed 128.01 samples/sec Loss 10.7805 Epoch: 8 Global Step: 81230 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:50:39,086-Speed 131.01 samples/sec Loss 11.0562 Epoch: 8 Global Step: 81235 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:50:58,936-Speed 128.98 samples/sec Loss 10.8674 Epoch: 8 Global Step: 81240 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:51:18,454-Speed 131.17 samples/sec Loss 10.5933 Epoch: 8 Global Step: 81245 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:51:38,130-Speed 130.12 samples/sec Loss 10.3471 Epoch: 8 Global Step: 81250 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:51:57,750-Speed 130.48 samples/sec Loss 10.8762 Epoch: 8 Global Step: 81255 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:52:17,615-Speed 128.88 samples/sec Loss 11.0435 Epoch: 8 Global Step: 81260 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:52:37,229-Speed 130.52 samples/sec Loss 11.0208 Epoch: 8 Global Step: 81265 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:52:57,075-Speed 129.00 samples/sec Loss 10.7165 Epoch: 8 Global Step: 81270 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:53:17,069-Speed 128.04 samples/sec Loss 10.5119 Epoch: 8 Global Step: 81275 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:53:36,617-Speed 130.96 samples/sec Loss 10.7329 Epoch: 8 Global Step: 81280 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:53:56,482-Speed 128.88 samples/sec Loss 10.9805 Epoch: 8 Global Step: 81285 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:54:15,925-Speed 131.67 samples/sec Loss 10.6366 Epoch: 8 Global Step: 81290 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:54:35,767-Speed 129.02 samples/sec Loss 11.2251 Epoch: 8 Global Step: 81295 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:54:55,348-Speed 130.75 samples/sec Loss 10.6323 Epoch: 8 Global Step: 81300 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:55:15,133-Speed 129.40 samples/sec Loss 10.5495 Epoch: 8 Global Step: 81305 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:55:34,850-Speed 129.85 samples/sec Loss 10.7129 Epoch: 8 Global Step: 81310 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:55:54,716-Speed 128.86 samples/sec Loss 10.8746 Epoch: 8 Global Step: 81315 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:56:14,627-Speed 128.58 samples/sec Loss 11.1873 Epoch: 8 Global Step: 81320 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:56:34,191-Speed 130.86 samples/sec Loss 10.8086 Epoch: 8 Global Step: 81325 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:56:54,057-Speed 128.87 samples/sec Loss 10.8790 Epoch: 8 Global Step: 81330 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:57:13,608-Speed 130.95 samples/sec Loss 11.0331 Epoch: 8 Global Step: 81335 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:57:33,617-Speed 127.95 samples/sec Loss 10.8456 Epoch: 8 Global Step: 81340 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:57:53,158-Speed 131.01 samples/sec Loss 10.4735 Epoch: 8 Global Step: 81345 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:58:12,945-Speed 129.38 samples/sec Loss 10.7808 Epoch: 8 Global Step: 81350 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:58:32,548-Speed 130.60 samples/sec Loss 10.7039 Epoch: 8 Global Step: 81355 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:58:52,646-Speed 127.38 samples/sec Loss 10.6228 Epoch: 8 Global Step: 81360 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:59:12,347-Speed 129.95 samples/sec Loss 11.0155 Epoch: 8 Global Step: 81365 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:59:31,872-Speed 131.12 samples/sec Loss 10.7814 Epoch: 8 Global Step: 81370 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 19:59:51,651-Speed 129.44 samples/sec Loss 11.1388 Epoch: 8 Global Step: 81375 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:00:11,207-Speed 130.91 samples/sec Loss 10.3944 Epoch: 8 Global Step: 81380 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:00:31,129-Speed 128.51 samples/sec Loss 10.7526 Epoch: 8 Global Step: 81385 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:00:50,686-Speed 130.91 samples/sec Loss 10.8043 Epoch: 8 Global Step: 81390 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:01:10,421-Speed 129.72 samples/sec Loss 11.1497 Epoch: 8 Global Step: 81395 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:01:29,982-Speed 130.88 samples/sec Loss 10.8236 Epoch: 8 Global Step: 81400 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:01:50,089-Speed 127.32 samples/sec Loss 10.8662 Epoch: 8 Global Step: 81405 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:02:09,840-Speed 129.62 samples/sec Loss 10.4072 Epoch: 8 Global Step: 81410 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:02:29,407-Speed 130.84 samples/sec Loss 10.5716 Epoch: 8 Global Step: 81415 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:02:49,274-Speed 128.87 samples/sec Loss 10.5024 Epoch: 8 Global Step: 81420 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:03:08,941-Speed 130.18 samples/sec Loss 10.7724 Epoch: 8 Global Step: 81425 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:03:28,934-Speed 128.05 samples/sec Loss 10.7526 Epoch: 8 Global Step: 81430 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:03:48,510-Speed 130.78 samples/sec Loss 11.0580 Epoch: 8 Global Step: 81435 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:04:08,327-Speed 129.19 samples/sec Loss 11.4295 Epoch: 8 Global Step: 81440 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:04:27,947-Speed 130.49 samples/sec Loss 10.7483 Epoch: 8 Global Step: 81445 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:04:48,165-Speed 126.62 samples/sec Loss 10.8511 Epoch: 8 Global Step: 81450 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:05:07,785-Speed 130.49 samples/sec Loss 10.8699 Epoch: 8 Global Step: 81455 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:05:28,133-Speed 125.82 samples/sec Loss 10.8609 Epoch: 8 Global Step: 81460 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:05:47,863-Speed 129.76 samples/sec Loss 10.6005 Epoch: 8 Global Step: 81465 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:06:07,501-Speed 130.37 samples/sec Loss 10.7237 Epoch: 8 Global Step: 81470 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:06:27,254-Speed 129.61 samples/sec Loss 10.8456 Epoch: 8 Global Step: 81475 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:06:46,748-Speed 131.33 samples/sec Loss 10.9313 Epoch: 8 Global Step: 81480 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:07:06,665-Speed 128.54 samples/sec Loss 10.7926 Epoch: 8 Global Step: 81485 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:07:26,365-Speed 129.96 samples/sec Loss 10.5316 Epoch: 8 Global Step: 81490 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:07:46,504-Speed 127.12 samples/sec Loss 10.9413 Epoch: 8 Global Step: 81495 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:08:06,419-Speed 128.55 samples/sec Loss 10.8954 Epoch: 8 Global Step: 81500 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:08:25,970-Speed 130.95 samples/sec Loss 10.7113 Epoch: 8 Global Step: 81505 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:08:45,717-Speed 129.64 samples/sec Loss 10.3495 Epoch: 8 Global Step: 81510 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:09:05,614-Speed 128.67 samples/sec Loss 10.9993 Epoch: 8 Global Step: 81515 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:09:25,328-Speed 129.86 samples/sec Loss 10.5264 Epoch: 8 Global Step: 81520 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:09:44,868-Speed 131.02 samples/sec Loss 10.8417 Epoch: 8 Global Step: 81525 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:10:04,722-Speed 128.95 samples/sec Loss 11.1173 Epoch: 8 Global Step: 81530 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:10:24,352-Speed 130.42 samples/sec Loss 10.8940 Epoch: 8 Global Step: 81535 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:10:44,462-Speed 127.31 samples/sec Loss 11.0459 Epoch: 8 Global Step: 81540 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:11:04,420-Speed 128.28 samples/sec Loss 10.8823 Epoch: 8 Global Step: 81545 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:11:23,991-Speed 130.81 samples/sec Loss 11.1300 Epoch: 8 Global Step: 81550 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:11:43,847-Speed 128.94 samples/sec Loss 10.8845 Epoch: 8 Global Step: 81555 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:12:03,448-Speed 130.61 samples/sec Loss 10.8834 Epoch: 8 Global Step: 81560 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:12:23,105-Speed 130.24 samples/sec Loss 10.6508 Epoch: 8 Global Step: 81565 Fp16 Grad Scale: 8192 Required: 200 hours Training: 2021-06-10 20:12:42,721-Speed 130.51 samples/sec Loss 10.8555 Epoch: 8 Global Step: 81570 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:13:02,455-Speed 129.74 samples/sec Loss 10.8565 Epoch: 8 Global Step: 81575 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:13:22,113-Speed 130.23 samples/sec Loss 10.8173 Epoch: 8 Global Step: 81580 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:13:42,114-Speed 128.00 samples/sec Loss 10.7655 Epoch: 8 Global Step: 81585 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:14:02,159-Speed 127.72 samples/sec Loss 11.2258 Epoch: 8 Global Step: 81590 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:14:21,778-Speed 130.50 samples/sec Loss 10.9695 Epoch: 8 Global Step: 81595 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:14:41,705-Speed 128.47 samples/sec Loss 10.7611 Epoch: 8 Global Step: 81600 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:15:01,224-Speed 131.16 samples/sec Loss 10.6122 Epoch: 8 Global Step: 81605 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:15:21,001-Speed 129.45 samples/sec Loss 10.8594 Epoch: 8 Global Step: 81610 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:15:40,592-Speed 130.68 samples/sec Loss 11.0511 Epoch: 8 Global Step: 81615 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:16:00,407-Speed 129.20 samples/sec Loss 10.9119 Epoch: 8 Global Step: 81620 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:16:19,987-Speed 130.75 samples/sec Loss 10.9566 Epoch: 8 Global Step: 81625 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:16:39,960-Speed 128.18 samples/sec Loss 10.5385 Epoch: 8 Global Step: 81630 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:16:59,921-Speed 128.26 samples/sec Loss 10.5369 Epoch: 8 Global Step: 81635 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:17:19,547-Speed 130.45 samples/sec Loss 10.9417 Epoch: 8 Global Step: 81640 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:17:39,490-Speed 128.37 samples/sec Loss 10.7952 Epoch: 8 Global Step: 81645 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:17:59,062-Speed 130.80 samples/sec Loss 10.9172 Epoch: 8 Global Step: 81650 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:18:18,992-Speed 128.46 samples/sec Loss 10.5915 Epoch: 8 Global Step: 81655 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:18:38,548-Speed 130.91 samples/sec Loss 10.9087 Epoch: 8 Global Step: 81660 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:18:58,313-Speed 129.53 samples/sec Loss 10.5142 Epoch: 8 Global Step: 81665 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:19:17,926-Speed 130.53 samples/sec Loss 11.1213 Epoch: 8 Global Step: 81670 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:19:38,044-Speed 127.25 samples/sec Loss 10.8411 Epoch: 8 Global Step: 81675 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:19:57,848-Speed 129.28 samples/sec Loss 10.8271 Epoch: 8 Global Step: 81680 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:20:17,420-Speed 130.80 samples/sec Loss 10.8550 Epoch: 8 Global Step: 81685 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:20:37,201-Speed 129.43 samples/sec Loss 10.9377 Epoch: 8 Global Step: 81690 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:20:56,757-Speed 130.92 samples/sec Loss 10.6491 Epoch: 8 Global Step: 81695 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:21:16,694-Speed 128.41 samples/sec Loss 10.3596 Epoch: 8 Global Step: 81700 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:21:36,314-Speed 130.49 samples/sec Loss 10.9693 Epoch: 8 Global Step: 81705 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:21:56,244-Speed 128.46 samples/sec Loss 10.9599 Epoch: 8 Global Step: 81710 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:22:15,804-Speed 130.89 samples/sec Loss 10.5028 Epoch: 8 Global Step: 81715 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:22:35,919-Speed 127.28 samples/sec Loss 10.4342 Epoch: 8 Global Step: 81720 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:22:55,638-Speed 129.83 samples/sec Loss 10.9569 Epoch: 8 Global Step: 81725 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:23:15,288-Speed 130.29 samples/sec Loss 10.8676 Epoch: 8 Global Step: 81730 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:23:35,019-Speed 129.75 samples/sec Loss 10.6439 Epoch: 8 Global Step: 81735 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:23:54,599-Speed 130.76 samples/sec Loss 10.8153 Epoch: 8 Global Step: 81740 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:24:14,519-Speed 128.52 samples/sec Loss 10.8415 Epoch: 8 Global Step: 81745 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:24:34,203-Speed 130.06 samples/sec Loss 10.8924 Epoch: 8 Global Step: 81750 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:24:54,045-Speed 129.03 samples/sec Loss 10.9903 Epoch: 8 Global Step: 81755 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:25:13,608-Speed 130.87 samples/sec Loss 10.8905 Epoch: 8 Global Step: 81760 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:25:33,708-Speed 127.37 samples/sec Loss 10.5716 Epoch: 8 Global Step: 81765 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:25:53,642-Speed 128.43 samples/sec Loss 10.6412 Epoch: 8 Global Step: 81770 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:26:13,347-Speed 129.92 samples/sec Loss 10.7550 Epoch: 8 Global Step: 81775 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:26:33,007-Speed 130.21 samples/sec Loss 11.0740 Epoch: 8 Global Step: 81780 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:26:52,748-Speed 129.69 samples/sec Loss 10.5965 Epoch: 8 Global Step: 81785 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:27:12,701-Speed 128.31 samples/sec Loss 10.8940 Epoch: 8 Global Step: 81790 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:27:32,287-Speed 130.71 samples/sec Loss 10.6029 Epoch: 8 Global Step: 81795 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:27:52,194-Speed 128.61 samples/sec Loss 10.7064 Epoch: 8 Global Step: 81800 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:28:11,809-Speed 130.52 samples/sec Loss 10.8870 Epoch: 8 Global Step: 81805 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:28:31,918-Speed 127.31 samples/sec Loss 10.4232 Epoch: 8 Global Step: 81810 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:28:51,863-Speed 128.36 samples/sec Loss 10.4277 Epoch: 8 Global Step: 81815 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:29:11,452-Speed 130.70 samples/sec Loss 10.7207 Epoch: 8 Global Step: 81820 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:29:31,194-Speed 129.68 samples/sec Loss 10.6430 Epoch: 8 Global Step: 81825 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:29:50,841-Speed 130.31 samples/sec Loss 10.7054 Epoch: 8 Global Step: 81830 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:30:10,650-Speed 129.24 samples/sec Loss 10.8641 Epoch: 8 Global Step: 81835 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:30:30,330-Speed 130.09 samples/sec Loss 10.8599 Epoch: 8 Global Step: 81840 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:30:50,120-Speed 129.37 samples/sec Loss 10.6439 Epoch: 8 Global Step: 81845 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:31:09,709-Speed 130.69 samples/sec Loss 10.7484 Epoch: 8 Global Step: 81850 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:31:29,788-Speed 127.50 samples/sec Loss 10.5286 Epoch: 8 Global Step: 81855 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:31:49,886-Speed 127.39 samples/sec Loss 10.9919 Epoch: 8 Global Step: 81860 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:32:09,546-Speed 130.22 samples/sec Loss 10.9461 Epoch: 8 Global Step: 81865 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:32:29,251-Speed 129.92 samples/sec Loss 11.1046 Epoch: 8 Global Step: 81870 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:32:48,838-Speed 130.71 samples/sec Loss 10.5738 Epoch: 8 Global Step: 81875 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:33:08,662-Speed 129.14 samples/sec Loss 11.0310 Epoch: 8 Global Step: 81880 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:33:28,299-Speed 130.38 samples/sec Loss 11.0726 Epoch: 8 Global Step: 81885 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:33:47,996-Speed 129.97 samples/sec Loss 10.8641 Epoch: 8 Global Step: 81890 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:34:07,616-Speed 130.49 samples/sec Loss 10.5524 Epoch: 8 Global Step: 81895 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:34:27,605-Speed 128.07 samples/sec Loss 10.2990 Epoch: 8 Global Step: 81900 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:34:47,513-Speed 128.60 samples/sec Loss 10.7456 Epoch: 8 Global Step: 81905 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:35:06,982-Speed 131.49 samples/sec Loss 10.9380 Epoch: 8 Global Step: 81910 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:35:26,873-Speed 128.71 samples/sec Loss 10.9435 Epoch: 8 Global Step: 81915 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:35:46,487-Speed 130.52 samples/sec Loss 10.6331 Epoch: 8 Global Step: 81920 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:36:06,413-Speed 128.48 samples/sec Loss 10.4383 Epoch: 8 Global Step: 81925 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:36:25,975-Speed 130.87 samples/sec Loss 10.7661 Epoch: 8 Global Step: 81930 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:36:45,645-Speed 130.15 samples/sec Loss 10.4877 Epoch: 8 Global Step: 81935 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:37:05,252-Speed 130.58 samples/sec Loss 10.8098 Epoch: 8 Global Step: 81940 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:37:25,403-Speed 127.05 samples/sec Loss 10.8513 Epoch: 8 Global Step: 81945 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:37:45,123-Speed 129.82 samples/sec Loss 10.6782 Epoch: 8 Global Step: 81950 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:38:04,799-Speed 130.11 samples/sec Loss 10.7896 Epoch: 8 Global Step: 81955 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:38:24,620-Speed 129.16 samples/sec Loss 10.6276 Epoch: 8 Global Step: 81960 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:38:44,247-Speed 130.45 samples/sec Loss 10.4572 Epoch: 8 Global Step: 81965 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:39:04,052-Speed 129.26 samples/sec Loss 10.6097 Epoch: 8 Global Step: 81970 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:39:23,578-Speed 131.12 samples/sec Loss 10.6784 Epoch: 8 Global Step: 81975 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:39:43,501-Speed 128.50 samples/sec Loss 10.8696 Epoch: 8 Global Step: 81980 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:40:03,061-Speed 130.89 samples/sec Loss 10.5859 Epoch: 8 Global Step: 81985 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:40:23,251-Speed 126.80 samples/sec Loss 10.7631 Epoch: 8 Global Step: 81990 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:40:42,892-Speed 130.35 samples/sec Loss 10.4682 Epoch: 8 Global Step: 81995 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:41:02,575-Speed 130.07 samples/sec Loss 10.9075 Epoch: 8 Global Step: 82000 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:41:22,329-Speed 129.60 samples/sec Loss 11.2761 Epoch: 8 Global Step: 82005 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:41:41,923-Speed 130.66 samples/sec Loss 10.8701 Epoch: 8 Global Step: 82010 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:42:01,732-Speed 129.24 samples/sec Loss 10.4057 Epoch: 8 Global Step: 82015 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:42:21,278-Speed 130.98 samples/sec Loss 10.7156 Epoch: 8 Global Step: 82020 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:42:41,126-Speed 128.99 samples/sec Loss 10.8664 Epoch: 8 Global Step: 82025 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:43:00,710-Speed 130.73 samples/sec Loss 10.9966 Epoch: 8 Global Step: 82030 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:43:20,889-Speed 126.87 samples/sec Loss 10.6982 Epoch: 8 Global Step: 82035 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:43:40,640-Speed 129.63 samples/sec Loss 10.6062 Epoch: 8 Global Step: 82040 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:44:00,230-Speed 130.68 samples/sec Loss 10.6621 Epoch: 8 Global Step: 82045 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:44:19,987-Speed 129.58 samples/sec Loss 10.8095 Epoch: 8 Global Step: 82050 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:44:39,670-Speed 130.07 samples/sec Loss 10.8714 Epoch: 8 Global Step: 82055 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:44:59,324-Speed 130.26 samples/sec Loss 10.5785 Epoch: 8 Global Step: 82060 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:45:18,947-Speed 130.47 samples/sec Loss 10.5645 Epoch: 8 Global Step: 82065 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:45:38,696-Speed 129.63 samples/sec Loss 10.6626 Epoch: 8 Global Step: 82070 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:45:58,346-Speed 130.29 samples/sec Loss 11.3359 Epoch: 8 Global Step: 82075 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:46:18,333-Speed 128.09 samples/sec Loss 10.4720 Epoch: 8 Global Step: 82080 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:46:38,277-Speed 128.36 samples/sec Loss 11.2871 Epoch: 8 Global Step: 82085 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:46:57,846-Speed 130.82 samples/sec Loss 10.5476 Epoch: 8 Global Step: 82090 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:47:17,846-Speed 128.01 samples/sec Loss 10.7947 Epoch: 8 Global Step: 82095 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:47:37,309-Speed 131.54 samples/sec Loss 10.3645 Epoch: 8 Global Step: 82100 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:47:57,022-Speed 129.87 samples/sec Loss 10.6312 Epoch: 8 Global Step: 82105 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:48:16,699-Speed 130.11 samples/sec Loss 10.9334 Epoch: 8 Global Step: 82110 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:48:36,471-Speed 129.48 samples/sec Loss 10.8805 Epoch: 8 Global Step: 82115 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:48:56,121-Speed 130.29 samples/sec Loss 10.8399 Epoch: 8 Global Step: 82120 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:49:16,112-Speed 128.07 samples/sec Loss 10.9406 Epoch: 8 Global Step: 82125 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:49:35,792-Speed 130.08 samples/sec Loss 10.5867 Epoch: 8 Global Step: 82130 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:49:55,579-Speed 129.39 samples/sec Loss 10.7994 Epoch: 8 Global Step: 82135 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:50:15,458-Speed 128.79 samples/sec Loss 10.6725 Epoch: 8 Global Step: 82140 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:50:34,935-Speed 131.44 samples/sec Loss 10.6833 Epoch: 8 Global Step: 82145 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:50:54,944-Speed 127.95 samples/sec Loss 10.6496 Epoch: 8 Global Step: 82150 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:51:14,612-Speed 130.17 samples/sec Loss 10.8029 Epoch: 8 Global Step: 82155 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:51:34,585-Speed 128.18 samples/sec Loss 10.7021 Epoch: 8 Global Step: 82160 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:51:54,041-Speed 131.59 samples/sec Loss 11.1111 Epoch: 8 Global Step: 82165 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:52:14,045-Speed 127.98 samples/sec Loss 10.8126 Epoch: 8 Global Step: 82170 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:52:33,766-Speed 129.81 samples/sec Loss 10.6353 Epoch: 8 Global Step: 82175 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:52:53,415-Speed 130.29 samples/sec Loss 10.8664 Epoch: 8 Global Step: 82180 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:53:13,046-Speed 130.41 samples/sec Loss 10.9203 Epoch: 8 Global Step: 82185 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:53:32,631-Speed 130.73 samples/sec Loss 10.9578 Epoch: 8 Global Step: 82190 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:53:52,512-Speed 128.77 samples/sec Loss 10.8744 Epoch: 8 Global Step: 82195 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:54:12,100-Speed 130.70 samples/sec Loss 10.7540 Epoch: 8 Global Step: 82200 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:54:32,090-Speed 128.07 samples/sec Loss 10.8007 Epoch: 8 Global Step: 82205 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:54:51,547-Speed 131.58 samples/sec Loss 10.6614 Epoch: 8 Global Step: 82210 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:55:11,653-Speed 127.33 samples/sec Loss 10.7339 Epoch: 8 Global Step: 82215 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:55:31,762-Speed 127.31 samples/sec Loss 10.5312 Epoch: 8 Global Step: 82220 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:55:51,351-Speed 130.69 samples/sec Loss 10.8769 Epoch: 8 Global Step: 82225 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:56:10,998-Speed 130.31 samples/sec Loss 10.6976 Epoch: 8 Global Step: 82230 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:56:30,565-Speed 130.84 samples/sec Loss 10.4957 Epoch: 8 Global Step: 82235 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:56:51,604-Speed 121.68 samples/sec Loss 10.9832 Epoch: 8 Global Step: 82240 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:57:11,245-Speed 130.35 samples/sec Loss 10.8235 Epoch: 8 Global Step: 82245 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:57:31,016-Speed 129.49 samples/sec Loss 10.8701 Epoch: 8 Global Step: 82250 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:57:50,580-Speed 130.86 samples/sec Loss 11.1178 Epoch: 8 Global Step: 82255 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:58:10,680-Speed 127.37 samples/sec Loss 10.7037 Epoch: 8 Global Step: 82260 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:58:30,741-Speed 127.61 samples/sec Loss 10.4033 Epoch: 8 Global Step: 82265 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:58:50,229-Speed 131.37 samples/sec Loss 10.8598 Epoch: 8 Global Step: 82270 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:59:10,016-Speed 129.38 samples/sec Loss 10.9494 Epoch: 8 Global Step: 82275 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:59:29,614-Speed 130.64 samples/sec Loss 10.9529 Epoch: 8 Global Step: 82280 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 20:59:49,366-Speed 129.61 samples/sec Loss 10.6904 Epoch: 8 Global Step: 82285 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 21:00:09,018-Speed 130.28 samples/sec Loss 10.6622 Epoch: 8 Global Step: 82290 Fp16 Grad Scale: 8192 Required: 199 hours Training: 2021-06-10 21:00:28,714-Speed 129.98 samples/sec Loss 10.8675 Epoch: 8 Global Step: 82295 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:00:48,208-Speed 131.33 samples/sec Loss 10.8576 Epoch: 8 Global Step: 82300 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:01:08,230-Speed 127.87 samples/sec Loss 10.9369 Epoch: 8 Global Step: 82305 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:01:28,250-Speed 127.88 samples/sec Loss 10.8634 Epoch: 8 Global Step: 82310 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:01:47,814-Speed 130.86 samples/sec Loss 10.7428 Epoch: 8 Global Step: 82315 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:02:07,486-Speed 130.14 samples/sec Loss 10.6622 Epoch: 8 Global Step: 82320 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:02:27,096-Speed 130.56 samples/sec Loss 10.6599 Epoch: 8 Global Step: 82325 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:02:46,900-Speed 129.28 samples/sec Loss 10.8121 Epoch: 8 Global Step: 82330 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:03:06,527-Speed 130.44 samples/sec Loss 10.5465 Epoch: 8 Global Step: 82335 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:03:26,236-Speed 129.89 samples/sec Loss 10.8938 Epoch: 8 Global Step: 82340 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:03:45,856-Speed 130.49 samples/sec Loss 10.8485 Epoch: 8 Global Step: 82345 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:04:05,813-Speed 128.28 samples/sec Loss 10.7894 Epoch: 8 Global Step: 82350 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:04:25,829-Speed 127.91 samples/sec Loss 11.1164 Epoch: 8 Global Step: 82355 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:04:45,316-Speed 131.37 samples/sec Loss 11.0572 Epoch: 8 Global Step: 82360 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:05:05,208-Speed 128.71 samples/sec Loss 10.7020 Epoch: 8 Global Step: 82365 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:05:24,755-Speed 130.97 samples/sec Loss 10.4914 Epoch: 8 Global Step: 82370 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:05:44,553-Speed 129.31 samples/sec Loss 10.8304 Epoch: 8 Global Step: 82375 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:06:04,163-Speed 130.55 samples/sec Loss 10.5659 Epoch: 8 Global Step: 82380 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:06:23,793-Speed 130.43 samples/sec Loss 10.7139 Epoch: 8 Global Step: 82385 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:06:43,424-Speed 130.41 samples/sec Loss 11.0719 Epoch: 8 Global Step: 82390 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:07:03,537-Speed 127.29 samples/sec Loss 10.4559 Epoch: 8 Global Step: 82395 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:07:23,420-Speed 128.76 samples/sec Loss 11.0079 Epoch: 8 Global Step: 82400 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:07:42,870-Speed 131.63 samples/sec Loss 10.5504 Epoch: 8 Global Step: 82405 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:08:02,802-Speed 128.44 samples/sec Loss 10.7920 Epoch: 8 Global Step: 82410 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:08:22,449-Speed 130.31 samples/sec Loss 11.1047 Epoch: 8 Global Step: 82415 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:08:42,291-Speed 129.03 samples/sec Loss 10.9630 Epoch: 8 Global Step: 82420 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:09:01,816-Speed 131.12 samples/sec Loss 10.9835 Epoch: 8 Global Step: 82425 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:09:21,528-Speed 129.87 samples/sec Loss 10.5497 Epoch: 8 Global Step: 82430 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:09:41,116-Speed 130.70 samples/sec Loss 10.8202 Epoch: 8 Global Step: 82435 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:10:01,262-Speed 127.08 samples/sec Loss 11.0448 Epoch: 8 Global Step: 82440 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:10:20,889-Speed 130.43 samples/sec Loss 10.8410 Epoch: 8 Global Step: 82445 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:10:40,429-Speed 131.02 samples/sec Loss 10.7507 Epoch: 8 Global Step: 82450 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:11:00,244-Speed 129.20 samples/sec Loss 10.5122 Epoch: 8 Global Step: 82455 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:11:19,872-Speed 130.44 samples/sec Loss 10.8850 Epoch: 8 Global Step: 82460 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:11:39,759-Speed 128.73 samples/sec Loss 10.9121 Epoch: 8 Global Step: 82465 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:11:59,379-Speed 130.48 samples/sec Loss 10.6974 Epoch: 8 Global Step: 82470 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:12:19,207-Speed 129.12 samples/sec Loss 10.6098 Epoch: 8 Global Step: 82475 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:12:38,786-Speed 130.76 samples/sec Loss 11.1557 Epoch: 8 Global Step: 82480 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:12:58,979-Speed 126.78 samples/sec Loss 10.7976 Epoch: 8 Global Step: 82485 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:13:18,664-Speed 130.06 samples/sec Loss 10.7325 Epoch: 8 Global Step: 82490 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:13:38,257-Speed 130.66 samples/sec Loss 10.9126 Epoch: 8 Global Step: 82495 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:13:58,012-Speed 129.59 samples/sec Loss 10.8334 Epoch: 8 Global Step: 82500 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:14:17,778-Speed 129.52 samples/sec Loss 10.8956 Epoch: 8 Global Step: 82505 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:14:37,404-Speed 130.44 samples/sec Loss 10.7036 Epoch: 8 Global Step: 82510 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:14:56,891-Speed 131.38 samples/sec Loss 10.6199 Epoch: 8 Global Step: 82515 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:15:16,784-Speed 128.70 samples/sec Loss 10.7489 Epoch: 8 Global Step: 82520 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:15:36,430-Speed 130.31 samples/sec Loss 11.0206 Epoch: 8 Global Step: 82525 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:15:56,441-Speed 127.94 samples/sec Loss 11.2643 Epoch: 8 Global Step: 82530 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:16:16,233-Speed 129.35 samples/sec Loss 10.6696 Epoch: 8 Global Step: 82535 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:16:35,976-Speed 129.67 samples/sec Loss 10.9985 Epoch: 8 Global Step: 82540 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:16:55,827-Speed 128.97 samples/sec Loss 10.8391 Epoch: 8 Global Step: 82545 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:17:15,409-Speed 130.75 samples/sec Loss 10.9304 Epoch: 8 Global Step: 82550 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:17:35,131-Speed 129.81 samples/sec Loss 10.5331 Epoch: 8 Global Step: 82555 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:17:54,755-Speed 130.46 samples/sec Loss 10.9554 Epoch: 8 Global Step: 82560 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:18:14,501-Speed 129.65 samples/sec Loss 10.8052 Epoch: 8 Global Step: 82565 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:18:34,142-Speed 130.35 samples/sec Loss 10.6954 Epoch: 8 Global Step: 82570 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:18:54,023-Speed 128.77 samples/sec Loss 10.8144 Epoch: 8 Global Step: 82575 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:19:13,907-Speed 128.75 samples/sec Loss 10.9510 Epoch: 8 Global Step: 82580 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:19:33,532-Speed 130.45 samples/sec Loss 10.7741 Epoch: 8 Global Step: 82585 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:19:53,689-Speed 127.01 samples/sec Loss 10.6514 Epoch: 8 Global Step: 82590 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:20:13,178-Speed 131.37 samples/sec Loss 10.5932 Epoch: 8 Global Step: 82595 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:20:32,921-Speed 129.67 samples/sec Loss 10.7179 Epoch: 8 Global Step: 82600 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:20:52,489-Speed 130.83 samples/sec Loss 10.7795 Epoch: 8 Global Step: 82605 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:21:12,286-Speed 129.32 samples/sec Loss 10.5537 Epoch: 8 Global Step: 82610 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:21:31,922-Speed 130.38 samples/sec Loss 10.7851 Epoch: 8 Global Step: 82615 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:21:51,811-Speed 128.72 samples/sec Loss 10.9683 Epoch: 8 Global Step: 82620 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:22:11,653-Speed 129.03 samples/sec Loss 10.8733 Epoch: 8 Global Step: 82625 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:22:31,258-Speed 130.58 samples/sec Loss 10.9523 Epoch: 8 Global Step: 82630 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:22:51,056-Speed 129.32 samples/sec Loss 10.6856 Epoch: 8 Global Step: 82635 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:23:10,500-Speed 131.67 samples/sec Loss 10.6741 Epoch: 8 Global Step: 82640 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:23:30,421-Speed 128.51 samples/sec Loss 11.0583 Epoch: 8 Global Step: 82645 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:23:50,001-Speed 130.76 samples/sec Loss 10.9127 Epoch: 8 Global Step: 82650 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:24:09,854-Speed 128.95 samples/sec Loss 10.7601 Epoch: 8 Global Step: 82655 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:24:29,451-Speed 130.64 samples/sec Loss 10.6696 Epoch: 8 Global Step: 82660 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:24:49,380-Speed 128.46 samples/sec Loss 11.1411 Epoch: 8 Global Step: 82665 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:25:09,185-Speed 129.27 samples/sec Loss 10.5434 Epoch: 8 Global Step: 82670 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:25:28,730-Speed 130.98 samples/sec Loss 10.8431 Epoch: 8 Global Step: 82675 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:25:48,540-Speed 129.24 samples/sec Loss 10.9917 Epoch: 8 Global Step: 82680 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:26:08,063-Speed 131.13 samples/sec Loss 10.9786 Epoch: 8 Global Step: 82685 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:26:27,937-Speed 128.82 samples/sec Loss 10.9693 Epoch: 8 Global Step: 82690 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:26:47,565-Speed 130.44 samples/sec Loss 10.6218 Epoch: 8 Global Step: 82695 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:27:07,303-Speed 129.70 samples/sec Loss 10.5999 Epoch: 8 Global Step: 82700 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:27:26,942-Speed 130.36 samples/sec Loss 10.6406 Epoch: 8 Global Step: 82705 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:27:46,954-Speed 127.93 samples/sec Loss 10.8894 Epoch: 8 Global Step: 82710 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:28:06,900-Speed 128.35 samples/sec Loss 10.7631 Epoch: 8 Global Step: 82715 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:28:26,633-Speed 129.74 samples/sec Loss 10.9760 Epoch: 8 Global Step: 82720 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:28:46,445-Speed 129.23 samples/sec Loss 10.7473 Epoch: 8 Global Step: 82725 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:29:05,990-Speed 130.98 samples/sec Loss 10.8624 Epoch: 8 Global Step: 82730 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:29:25,909-Speed 128.53 samples/sec Loss 10.6952 Epoch: 8 Global Step: 82735 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:29:45,539-Speed 130.42 samples/sec Loss 10.7052 Epoch: 8 Global Step: 82740 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:30:05,230-Speed 130.01 samples/sec Loss 11.2771 Epoch: 8 Global Step: 82745 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:30:24,826-Speed 130.65 samples/sec Loss 10.5827 Epoch: 8 Global Step: 82750 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:30:44,876-Speed 127.69 samples/sec Loss 10.6264 Epoch: 8 Global Step: 82755 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:31:04,735-Speed 128.92 samples/sec Loss 10.9588 Epoch: 8 Global Step: 82760 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:31:24,177-Speed 131.68 samples/sec Loss 10.8782 Epoch: 8 Global Step: 82765 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:31:43,999-Speed 129.16 samples/sec Loss 10.9979 Epoch: 8 Global Step: 82770 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:32:03,542-Speed 131.00 samples/sec Loss 10.7936 Epoch: 8 Global Step: 82775 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:32:23,516-Speed 128.17 samples/sec Loss 10.8950 Epoch: 8 Global Step: 82780 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:32:43,083-Speed 130.84 samples/sec Loss 10.8208 Epoch: 8 Global Step: 82785 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:33:03,013-Speed 128.46 samples/sec Loss 10.5931 Epoch: 8 Global Step: 82790 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:33:22,575-Speed 130.87 samples/sec Loss 10.4683 Epoch: 8 Global Step: 82795 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:33:42,694-Speed 127.25 samples/sec Loss 10.7519 Epoch: 8 Global Step: 82800 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:34:02,467-Speed 129.48 samples/sec Loss 10.7602 Epoch: 8 Global Step: 82805 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:34:22,090-Speed 130.46 samples/sec Loss 10.8677 Epoch: 8 Global Step: 82810 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:34:41,837-Speed 129.65 samples/sec Loss 10.9432 Epoch: 8 Global Step: 82815 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:35:01,387-Speed 130.96 samples/sec Loss 11.4201 Epoch: 8 Global Step: 82820 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:35:21,340-Speed 128.31 samples/sec Loss 11.1449 Epoch: 8 Global Step: 82825 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:35:40,889-Speed 130.96 samples/sec Loss 11.0203 Epoch: 8 Global Step: 82830 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:36:00,714-Speed 129.14 samples/sec Loss 10.7817 Epoch: 8 Global Step: 82835 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:36:20,250-Speed 131.05 samples/sec Loss 10.8706 Epoch: 8 Global Step: 82840 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:36:40,426-Speed 126.89 samples/sec Loss 11.2929 Epoch: 8 Global Step: 82845 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:37:00,285-Speed 128.92 samples/sec Loss 10.4173 Epoch: 8 Global Step: 82850 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:37:19,891-Speed 130.58 samples/sec Loss 10.5611 Epoch: 8 Global Step: 82855 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:37:39,508-Speed 130.50 samples/sec Loss 10.6495 Epoch: 8 Global Step: 82860 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:37:59,134-Speed 130.45 samples/sec Loss 10.8903 Epoch: 8 Global Step: 82865 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:38:18,860-Speed 129.79 samples/sec Loss 10.9257 Epoch: 8 Global Step: 82870 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:38:38,346-Speed 131.38 samples/sec Loss 10.7363 Epoch: 8 Global Step: 82875 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:38:58,498-Speed 127.04 samples/sec Loss 10.7675 Epoch: 8 Global Step: 82880 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:39:18,291-Speed 129.35 samples/sec Loss 10.8047 Epoch: 8 Global Step: 82885 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:39:38,484-Speed 126.78 samples/sec Loss 10.4127 Epoch: 8 Global Step: 82890 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:39:58,403-Speed 128.53 samples/sec Loss 10.9252 Epoch: 8 Global Step: 82895 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:40:18,094-Speed 130.02 samples/sec Loss 11.1594 Epoch: 8 Global Step: 82900 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:40:37,908-Speed 129.21 samples/sec Loss 10.7465 Epoch: 8 Global Step: 82905 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:40:57,529-Speed 130.48 samples/sec Loss 10.9756 Epoch: 8 Global Step: 82910 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:41:17,257-Speed 129.78 samples/sec Loss 11.0144 Epoch: 8 Global Step: 82915 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:41:36,867-Speed 130.55 samples/sec Loss 10.7701 Epoch: 8 Global Step: 82920 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:41:56,499-Speed 130.41 samples/sec Loss 10.8369 Epoch: 8 Global Step: 82925 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:42:16,139-Speed 130.35 samples/sec Loss 10.8063 Epoch: 8 Global Step: 82930 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:42:36,200-Speed 127.62 samples/sec Loss 10.9346 Epoch: 8 Global Step: 82935 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:42:56,121-Speed 128.51 samples/sec Loss 10.3748 Epoch: 8 Global Step: 82940 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:43:15,698-Speed 130.78 samples/sec Loss 11.2203 Epoch: 8 Global Step: 82945 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:43:35,668-Speed 128.20 samples/sec Loss 10.4871 Epoch: 8 Global Step: 82950 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:43:55,224-Speed 130.91 samples/sec Loss 10.8279 Epoch: 8 Global Step: 82955 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:44:15,014-Speed 129.36 samples/sec Loss 10.6171 Epoch: 8 Global Step: 82960 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:44:34,612-Speed 130.63 samples/sec Loss 10.8084 Epoch: 8 Global Step: 82965 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:44:54,329-Speed 129.85 samples/sec Loss 10.3751 Epoch: 8 Global Step: 82970 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:45:13,920-Speed 130.68 samples/sec Loss 10.8571 Epoch: 8 Global Step: 82975 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:45:33,979-Speed 127.63 samples/sec Loss 10.6151 Epoch: 8 Global Step: 82980 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:45:53,834-Speed 128.94 samples/sec Loss 11.0309 Epoch: 8 Global Step: 82985 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:46:13,473-Speed 130.36 samples/sec Loss 10.6405 Epoch: 8 Global Step: 82990 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:46:33,294-Speed 129.16 samples/sec Loss 10.5393 Epoch: 8 Global Step: 82995 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:46:52,863-Speed 130.83 samples/sec Loss 10.4306 Epoch: 8 Global Step: 83000 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:47:12,555-Speed 130.01 samples/sec Loss 10.8829 Epoch: 8 Global Step: 83005 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:47:32,160-Speed 130.59 samples/sec Loss 10.5487 Epoch: 8 Global Step: 83010 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:47:52,004-Speed 129.02 samples/sec Loss 10.4286 Epoch: 8 Global Step: 83015 Fp16 Grad Scale: 8192 Required: 198 hours Training: 2021-06-10 21:48:11,655-Speed 130.28 samples/sec Loss 11.0936 Epoch: 8 Global Step: 83020 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:48:31,616-Speed 128.25 samples/sec Loss 10.9971 Epoch: 8 Global Step: 83025 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:48:51,430-Speed 129.21 samples/sec Loss 10.8037 Epoch: 8 Global Step: 83030 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:49:11,046-Speed 130.51 samples/sec Loss 10.5827 Epoch: 8 Global Step: 83035 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:49:30,963-Speed 128.55 samples/sec Loss 11.1243 Epoch: 8 Global Step: 83040 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:49:50,515-Speed 130.93 samples/sec Loss 10.8800 Epoch: 8 Global Step: 83045 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:50:10,880-Speed 125.71 samples/sec Loss 10.8398 Epoch: 8 Global Step: 83050 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:50:30,416-Speed 131.05 samples/sec Loss 10.6528 Epoch: 8 Global Step: 83055 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:50:50,166-Speed 129.62 samples/sec Loss 10.9444 Epoch: 8 Global Step: 83060 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:51:09,826-Speed 130.22 samples/sec Loss 10.6778 Epoch: 8 Global Step: 83065 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:51:29,780-Speed 128.30 samples/sec Loss 10.9382 Epoch: 8 Global Step: 83070 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:51:49,681-Speed 128.65 samples/sec Loss 10.5921 Epoch: 8 Global Step: 83075 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:52:09,254-Speed 130.79 samples/sec Loss 11.3079 Epoch: 8 Global Step: 83080 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:52:29,181-Speed 128.48 samples/sec Loss 11.0778 Epoch: 8 Global Step: 83085 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:52:48,721-Speed 131.02 samples/sec Loss 10.8239 Epoch: 8 Global Step: 83090 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:53:08,517-Speed 129.33 samples/sec Loss 10.6758 Epoch: 8 Global Step: 83095 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:53:28,142-Speed 130.46 samples/sec Loss 10.7896 Epoch: 8 Global Step: 83100 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:53:47,842-Speed 129.96 samples/sec Loss 10.6543 Epoch: 8 Global Step: 83105 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:54:07,455-Speed 130.53 samples/sec Loss 10.6444 Epoch: 8 Global Step: 83110 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:54:27,544-Speed 127.44 samples/sec Loss 11.0405 Epoch: 8 Global Step: 83115 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:54:47,359-Speed 129.20 samples/sec Loss 10.9898 Epoch: 8 Global Step: 83120 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:55:06,867-Speed 131.24 samples/sec Loss 10.9160 Epoch: 8 Global Step: 83125 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:55:26,737-Speed 128.84 samples/sec Loss 10.3009 Epoch: 8 Global Step: 83130 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:55:46,222-Speed 131.39 samples/sec Loss 10.6683 Epoch: 8 Global Step: 83135 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:56:06,041-Speed 129.18 samples/sec Loss 10.5431 Epoch: 8 Global Step: 83140 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:56:25,782-Speed 129.68 samples/sec Loss 10.3529 Epoch: 8 Global Step: 83145 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:56:45,689-Speed 128.61 samples/sec Loss 10.6828 Epoch: 8 Global Step: 83150 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:57:05,189-Speed 131.29 samples/sec Loss 10.6824 Epoch: 8 Global Step: 83155 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:57:25,226-Speed 127.77 samples/sec Loss 10.7008 Epoch: 8 Global Step: 83160 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:57:45,035-Speed 129.24 samples/sec Loss 11.0069 Epoch: 8 Global Step: 83165 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:58:04,616-Speed 130.75 samples/sec Loss 10.8851 Epoch: 8 Global Step: 83170 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:58:24,457-Speed 129.03 samples/sec Loss 10.7805 Epoch: 8 Global Step: 83175 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:58:43,935-Speed 131.44 samples/sec Loss 10.4759 Epoch: 8 Global Step: 83180 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:59:03,811-Speed 128.81 samples/sec Loss 10.8930 Epoch: 8 Global Step: 83185 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:59:23,380-Speed 130.82 samples/sec Loss 10.9631 Epoch: 8 Global Step: 83190 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 21:59:43,351-Speed 128.19 samples/sec Loss 10.6511 Epoch: 8 Global Step: 83195 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:00:02,918-Speed 130.84 samples/sec Loss 10.8705 Epoch: 8 Global Step: 83200 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:00:22,929-Speed 127.94 samples/sec Loss 10.8395 Epoch: 8 Global Step: 83205 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:00:42,845-Speed 128.55 samples/sec Loss 10.5772 Epoch: 8 Global Step: 83210 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:01:02,451-Speed 130.58 samples/sec Loss 10.4580 Epoch: 8 Global Step: 83215 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:01:22,245-Speed 129.34 samples/sec Loss 10.8544 Epoch: 8 Global Step: 83220 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:01:41,872-Speed 130.44 samples/sec Loss 10.6538 Epoch: 8 Global Step: 83225 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:02:01,606-Speed 129.73 samples/sec Loss 10.6722 Epoch: 8 Global Step: 83230 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:02:21,169-Speed 130.86 samples/sec Loss 10.6286 Epoch: 8 Global Step: 83235 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:02:41,000-Speed 129.10 samples/sec Loss 10.8311 Epoch: 8 Global Step: 83240 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:03:00,517-Speed 131.18 samples/sec Loss 10.5486 Epoch: 8 Global Step: 83245 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:03:20,519-Speed 127.99 samples/sec Loss 11.0401 Epoch: 8 Global Step: 83250 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:03:40,627-Speed 127.32 samples/sec Loss 10.6126 Epoch: 8 Global Step: 83255 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:04:00,086-Speed 131.57 samples/sec Loss 10.9333 Epoch: 8 Global Step: 83260 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:04:19,785-Speed 129.96 samples/sec Loss 10.7763 Epoch: 8 Global Step: 83265 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:04:39,389-Speed 130.59 samples/sec Loss 10.8818 Epoch: 8 Global Step: 83270 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:04:59,197-Speed 129.25 samples/sec Loss 10.2893 Epoch: 8 Global Step: 83275 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:05:18,834-Speed 130.37 samples/sec Loss 10.9038 Epoch: 8 Global Step: 83280 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:05:38,536-Speed 129.94 samples/sec Loss 10.9977 Epoch: 8 Global Step: 83285 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:05:58,151-Speed 130.52 samples/sec Loss 10.9349 Epoch: 8 Global Step: 83290 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:06:18,280-Speed 127.19 samples/sec Loss 11.2063 Epoch: 8 Global Step: 83295 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:06:38,371-Speed 127.43 samples/sec Loss 10.5161 Epoch: 8 Global Step: 83300 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:06:57,911-Speed 131.02 samples/sec Loss 10.6215 Epoch: 8 Global Step: 83305 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:07:17,595-Speed 130.06 samples/sec Loss 11.1966 Epoch: 8 Global Step: 83310 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:07:37,201-Speed 130.58 samples/sec Loss 10.6936 Epoch: 8 Global Step: 83315 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:07:56,941-Speed 129.70 samples/sec Loss 10.6545 Epoch: 8 Global Step: 83320 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:08:16,583-Speed 130.34 samples/sec Loss 10.8126 Epoch: 8 Global Step: 83325 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:08:36,368-Speed 129.40 samples/sec Loss 10.8322 Epoch: 8 Global Step: 83330 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:08:55,865-Speed 131.31 samples/sec Loss 10.8405 Epoch: 8 Global Step: 83335 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:09:16,537-Speed 123.85 samples/sec Loss 10.5880 Epoch: 8 Global Step: 83340 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:09:36,624-Speed 127.45 samples/sec Loss 10.5402 Epoch: 8 Global Step: 83345 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:09:56,174-Speed 130.95 samples/sec Loss 10.5822 Epoch: 8 Global Step: 83350 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:10:16,100-Speed 128.48 samples/sec Loss 10.6804 Epoch: 8 Global Step: 83355 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:10:35,673-Speed 130.80 samples/sec Loss 10.4991 Epoch: 8 Global Step: 83360 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:10:55,438-Speed 129.53 samples/sec Loss 10.5721 Epoch: 8 Global Step: 83365 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:11:15,052-Speed 130.52 samples/sec Loss 10.8457 Epoch: 8 Global Step: 83370 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:11:34,802-Speed 129.63 samples/sec Loss 10.9624 Epoch: 8 Global Step: 83375 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:11:54,565-Speed 129.54 samples/sec Loss 10.7346 Epoch: 8 Global Step: 83380 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:12:14,658-Speed 127.41 samples/sec Loss 10.8108 Epoch: 8 Global Step: 83385 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:12:34,533-Speed 128.81 samples/sec Loss 10.9758 Epoch: 8 Global Step: 83390 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:12:54,092-Speed 130.89 samples/sec Loss 10.7256 Epoch: 8 Global Step: 83395 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:13:14,030-Speed 128.41 samples/sec Loss 11.0500 Epoch: 8 Global Step: 83400 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:13:33,658-Speed 130.43 samples/sec Loss 11.2680 Epoch: 8 Global Step: 83405 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:13:53,585-Speed 128.48 samples/sec Loss 10.6603 Epoch: 8 Global Step: 83410 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:14:13,156-Speed 130.82 samples/sec Loss 10.8979 Epoch: 8 Global Step: 83415 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:14:32,922-Speed 129.52 samples/sec Loss 10.7574 Epoch: 8 Global Step: 83420 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:14:52,539-Speed 130.51 samples/sec Loss 10.9598 Epoch: 8 Global Step: 83425 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:15:12,704-Speed 126.95 samples/sec Loss 10.5635 Epoch: 8 Global Step: 83430 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:15:32,424-Speed 129.82 samples/sec Loss 10.4785 Epoch: 8 Global Step: 83435 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:15:51,991-Speed 130.84 samples/sec Loss 10.8563 Epoch: 8 Global Step: 83440 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:16:11,921-Speed 128.46 samples/sec Loss 10.7867 Epoch: 8 Global Step: 83445 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:16:32,059-Speed 127.13 samples/sec Loss 10.4164 Epoch: 8 Global Step: 83450 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:16:51,954-Speed 128.68 samples/sec Loss 10.8161 Epoch: 8 Global Step: 83455 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:17:11,508-Speed 130.93 samples/sec Loss 10.7407 Epoch: 8 Global Step: 83460 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:17:31,350-Speed 129.02 samples/sec Loss 10.8185 Epoch: 8 Global Step: 83465 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:17:50,943-Speed 130.67 samples/sec Loss 10.9408 Epoch: 8 Global Step: 83470 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:18:11,103-Speed 126.98 samples/sec Loss 10.6573 Epoch: 8 Global Step: 83475 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:18:30,834-Speed 129.75 samples/sec Loss 10.8257 Epoch: 8 Global Step: 83480 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:18:50,468-Speed 130.39 samples/sec Loss 10.9777 Epoch: 8 Global Step: 83485 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:19:10,227-Speed 129.57 samples/sec Loss 10.7324 Epoch: 8 Global Step: 83490 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:19:30,025-Speed 129.31 samples/sec Loss 11.1190 Epoch: 8 Global Step: 83495 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:19:49,749-Speed 129.80 samples/sec Loss 10.6939 Epoch: 8 Global Step: 83500 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:20:09,271-Speed 131.14 samples/sec Loss 10.5056 Epoch: 8 Global Step: 83505 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:20:29,151-Speed 128.78 samples/sec Loss 10.5447 Epoch: 8 Global Step: 83510 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:20:48,743-Speed 130.67 samples/sec Loss 10.5423 Epoch: 8 Global Step: 83515 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:21:08,842-Speed 127.37 samples/sec Loss 10.6954 Epoch: 8 Global Step: 83520 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:21:28,831-Speed 128.08 samples/sec Loss 10.6499 Epoch: 8 Global Step: 83525 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:21:48,429-Speed 130.63 samples/sec Loss 10.5668 Epoch: 8 Global Step: 83530 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:22:08,202-Speed 129.47 samples/sec Loss 10.8053 Epoch: 8 Global Step: 83535 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:22:27,855-Speed 130.27 samples/sec Loss 10.4483 Epoch: 8 Global Step: 83540 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:22:47,601-Speed 129.66 samples/sec Loss 10.7025 Epoch: 8 Global Step: 83545 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:23:07,170-Speed 130.83 samples/sec Loss 10.9700 Epoch: 8 Global Step: 83550 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:23:27,044-Speed 128.82 samples/sec Loss 10.5680 Epoch: 8 Global Step: 83555 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:23:46,682-Speed 130.37 samples/sec Loss 11.0509 Epoch: 8 Global Step: 83560 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:24:06,792-Speed 127.31 samples/sec Loss 10.6682 Epoch: 8 Global Step: 83565 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:24:26,831-Speed 127.76 samples/sec Loss 10.7677 Epoch: 8 Global Step: 83570 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:24:46,400-Speed 130.82 samples/sec Loss 10.8799 Epoch: 8 Global Step: 83575 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:25:06,163-Speed 129.54 samples/sec Loss 10.7158 Epoch: 8 Global Step: 83580 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:25:25,745-Speed 130.74 samples/sec Loss 11.1202 Epoch: 8 Global Step: 83585 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:25:45,446-Speed 129.95 samples/sec Loss 10.8587 Epoch: 8 Global Step: 83590 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:26:05,099-Speed 130.26 samples/sec Loss 10.9766 Epoch: 8 Global Step: 83595 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:26:24,853-Speed 129.60 samples/sec Loss 10.8656 Epoch: 8 Global Step: 83600 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:26:44,402-Speed 130.96 samples/sec Loss 10.5108 Epoch: 8 Global Step: 83605 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:27:04,460-Speed 127.64 samples/sec Loss 10.9590 Epoch: 8 Global Step: 83610 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:27:24,460-Speed 128.00 samples/sec Loss 11.1578 Epoch: 8 Global Step: 83615 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:27:44,115-Speed 130.25 samples/sec Loss 10.7078 Epoch: 8 Global Step: 83620 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:28:04,034-Speed 128.53 samples/sec Loss 10.7337 Epoch: 8 Global Step: 83625 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:28:23,593-Speed 130.90 samples/sec Loss 10.8373 Epoch: 8 Global Step: 83630 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:28:43,378-Speed 129.40 samples/sec Loss 10.8695 Epoch: 8 Global Step: 83635 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:29:02,969-Speed 130.68 samples/sec Loss 10.8290 Epoch: 8 Global Step: 83640 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:29:22,769-Speed 129.30 samples/sec Loss 10.7362 Epoch: 8 Global Step: 83645 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:29:42,386-Speed 130.50 samples/sec Loss 10.6815 Epoch: 8 Global Step: 83650 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:30:02,399-Speed 127.93 samples/sec Loss 10.9195 Epoch: 8 Global Step: 83655 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:30:22,171-Speed 129.48 samples/sec Loss 10.9270 Epoch: 8 Global Step: 83660 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:30:41,782-Speed 130.54 samples/sec Loss 10.5035 Epoch: 8 Global Step: 83665 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:31:01,741-Speed 128.27 samples/sec Loss 10.7759 Epoch: 8 Global Step: 83670 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:31:21,310-Speed 130.83 samples/sec Loss 10.8886 Epoch: 8 Global Step: 83675 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:31:41,195-Speed 128.74 samples/sec Loss 10.8128 Epoch: 8 Global Step: 83680 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:32:00,809-Speed 130.53 samples/sec Loss 10.6930 Epoch: 8 Global Step: 83685 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:32:20,570-Speed 129.56 samples/sec Loss 10.7808 Epoch: 8 Global Step: 83690 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:32:40,216-Speed 130.31 samples/sec Loss 10.8331 Epoch: 8 Global Step: 83695 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:33:00,166-Speed 128.33 samples/sec Loss 10.7045 Epoch: 8 Global Step: 83700 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:33:20,117-Speed 128.32 samples/sec Loss 10.7223 Epoch: 8 Global Step: 83705 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:33:39,791-Speed 130.13 samples/sec Loss 10.8057 Epoch: 8 Global Step: 83710 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:33:59,694-Speed 128.63 samples/sec Loss 10.7673 Epoch: 8 Global Step: 83715 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:34:19,226-Speed 131.07 samples/sec Loss 10.7713 Epoch: 8 Global Step: 83720 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:34:39,057-Speed 129.10 samples/sec Loss 10.6218 Epoch: 8 Global Step: 83725 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:34:58,632-Speed 130.79 samples/sec Loss 10.8068 Epoch: 8 Global Step: 83730 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:35:18,410-Speed 129.44 samples/sec Loss 10.9893 Epoch: 8 Global Step: 83735 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:35:38,000-Speed 130.68 samples/sec Loss 10.6991 Epoch: 8 Global Step: 83740 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:35:57,990-Speed 128.07 samples/sec Loss 10.8621 Epoch: 8 Global Step: 83745 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:36:17,723-Speed 129.74 samples/sec Loss 10.9291 Epoch: 8 Global Step: 83750 Fp16 Grad Scale: 8192 Required: 197 hours Training: 2021-06-10 22:36:37,271-Speed 130.96 samples/sec Loss 10.6467 Epoch: 8 Global Step: 83755 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:36:57,157-Speed 128.74 samples/sec Loss 10.8729 Epoch: 8 Global Step: 83760 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:37:16,680-Speed 131.13 samples/sec Loss 10.5597 Epoch: 8 Global Step: 83765 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:37:36,639-Speed 128.27 samples/sec Loss 10.8717 Epoch: 8 Global Step: 83770 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:37:56,160-Speed 131.15 samples/sec Loss 10.7352 Epoch: 8 Global Step: 83775 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:38:15,988-Speed 129.12 samples/sec Loss 10.8016 Epoch: 8 Global Step: 83780 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:38:35,569-Speed 130.74 samples/sec Loss 10.6813 Epoch: 8 Global Step: 83785 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:38:55,593-Speed 127.85 samples/sec Loss 10.7788 Epoch: 8 Global Step: 83790 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:39:15,334-Speed 129.69 samples/sec Loss 10.7403 Epoch: 8 Global Step: 83795 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:39:34,896-Speed 130.87 samples/sec Loss 10.4652 Epoch: 8 Global Step: 83800 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:39:54,773-Speed 128.80 samples/sec Loss 10.7615 Epoch: 8 Global Step: 83805 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:40:14,322-Speed 130.96 samples/sec Loss 10.6671 Epoch: 8 Global Step: 83810 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:40:34,315-Speed 128.05 samples/sec Loss 10.7842 Epoch: 8 Global Step: 83815 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:40:53,884-Speed 130.83 samples/sec Loss 11.0168 Epoch: 8 Global Step: 83820 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:41:13,792-Speed 128.60 samples/sec Loss 10.4465 Epoch: 8 Global Step: 83825 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:41:33,345-Speed 130.93 samples/sec Loss 10.8295 Epoch: 8 Global Step: 83830 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:41:53,388-Speed 127.73 samples/sec Loss 11.0329 Epoch: 8 Global Step: 83835 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:42:13,120-Speed 129.74 samples/sec Loss 10.3558 Epoch: 8 Global Step: 83840 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:42:32,755-Speed 130.39 samples/sec Loss 11.1780 Epoch: 8 Global Step: 83845 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:42:52,377-Speed 130.47 samples/sec Loss 10.7363 Epoch: 8 Global Step: 83850 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:43:12,291-Speed 128.56 samples/sec Loss 10.4501 Epoch: 8 Global Step: 83855 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:43:32,175-Speed 128.76 samples/sec Loss 10.7423 Epoch: 8 Global Step: 83860 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:43:51,733-Speed 130.90 samples/sec Loss 10.2840 Epoch: 8 Global Step: 83865 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:44:11,673-Speed 128.40 samples/sec Loss 10.5489 Epoch: 8 Global Step: 83870 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:44:31,151-Speed 131.43 samples/sec Loss 10.9071 Epoch: 8 Global Step: 83875 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:44:51,287-Speed 127.15 samples/sec Loss 11.0192 Epoch: 8 Global Step: 83880 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:45:11,113-Speed 129.13 samples/sec Loss 10.7457 Epoch: 8 Global Step: 83885 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:45:30,876-Speed 129.53 samples/sec Loss 10.8495 Epoch: 8 Global Step: 83890 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:45:50,489-Speed 130.54 samples/sec Loss 10.8829 Epoch: 8 Global Step: 83895 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:46:10,132-Speed 130.33 samples/sec Loss 11.0044 Epoch: 8 Global Step: 83900 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:46:29,781-Speed 130.29 samples/sec Loss 10.8207 Epoch: 8 Global Step: 83905 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:46:49,318-Speed 131.04 samples/sec Loss 10.7224 Epoch: 8 Global Step: 83910 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:47:09,199-Speed 128.77 samples/sec Loss 10.7189 Epoch: 8 Global Step: 83915 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:47:28,670-Speed 131.48 samples/sec Loss 10.7965 Epoch: 8 Global Step: 83920 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:47:48,965-Speed 126.15 samples/sec Loss 10.6821 Epoch: 8 Global Step: 83925 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:48:08,992-Speed 127.83 samples/sec Loss 10.6417 Epoch: 8 Global Step: 83930 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:48:28,552-Speed 130.88 samples/sec Loss 10.9826 Epoch: 8 Global Step: 83935 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:48:48,459-Speed 128.61 samples/sec Loss 10.6508 Epoch: 8 Global Step: 83940 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:49:08,051-Speed 130.67 samples/sec Loss 10.4916 Epoch: 8 Global Step: 83945 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:49:27,795-Speed 129.66 samples/sec Loss 10.4431 Epoch: 8 Global Step: 83950 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:49:47,421-Speed 130.45 samples/sec Loss 10.6554 Epoch: 8 Global Step: 83955 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:50:07,180-Speed 129.57 samples/sec Loss 10.8401 Epoch: 8 Global Step: 83960 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:50:26,836-Speed 130.25 samples/sec Loss 11.0065 Epoch: 8 Global Step: 83965 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:50:46,710-Speed 128.82 samples/sec Loss 11.2648 Epoch: 8 Global Step: 83970 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:51:06,600-Speed 128.71 samples/sec Loss 10.7035 Epoch: 8 Global Step: 83975 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:51:26,286-Speed 130.05 samples/sec Loss 10.8560 Epoch: 8 Global Step: 83980 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:51:46,310-Speed 127.85 samples/sec Loss 10.6115 Epoch: 8 Global Step: 83985 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:52:05,757-Speed 131.65 samples/sec Loss 10.9279 Epoch: 8 Global Step: 83990 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:52:25,790-Speed 127.80 samples/sec Loss 10.7532 Epoch: 8 Global Step: 83995 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:52:45,345-Speed 130.91 samples/sec Loss 10.4996 Epoch: 8 Global Step: 84000 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:53:05,019-Speed 130.13 samples/sec Loss 10.3889 Epoch: 8 Global Step: 84005 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:53:24,650-Speed 130.42 samples/sec Loss 10.8454 Epoch: 8 Global Step: 84010 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:53:44,741-Speed 127.42 samples/sec Loss 10.7172 Epoch: 8 Global Step: 84015 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:54:04,474-Speed 129.74 samples/sec Loss 10.9321 Epoch: 8 Global Step: 84020 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:54:24,045-Speed 130.81 samples/sec Loss 10.7384 Epoch: 8 Global Step: 84025 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:54:43,925-Speed 128.78 samples/sec Loss 10.7195 Epoch: 8 Global Step: 84030 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:55:03,491-Speed 130.85 samples/sec Loss 10.6207 Epoch: 8 Global Step: 84035 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:55:23,411-Speed 128.52 samples/sec Loss 10.8999 Epoch: 8 Global Step: 84040 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:55:42,955-Speed 130.99 samples/sec Loss 10.9010 Epoch: 8 Global Step: 84045 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:56:02,878-Speed 128.50 samples/sec Loss 10.7290 Epoch: 8 Global Step: 84050 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:56:22,515-Speed 130.38 samples/sec Loss 10.8588 Epoch: 8 Global Step: 84055 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:56:42,689-Speed 126.90 samples/sec Loss 10.8496 Epoch: 8 Global Step: 84060 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:57:02,410-Speed 129.82 samples/sec Loss 10.6212 Epoch: 8 Global Step: 84065 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:57:21,997-Speed 130.70 samples/sec Loss 10.7803 Epoch: 8 Global Step: 84070 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:57:41,802-Speed 129.27 samples/sec Loss 11.0208 Epoch: 8 Global Step: 84075 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:58:01,347-Speed 130.99 samples/sec Loss 10.6740 Epoch: 8 Global Step: 84080 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:58:21,215-Speed 128.86 samples/sec Loss 11.0040 Epoch: 8 Global Step: 84085 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:58:40,764-Speed 130.96 samples/sec Loss 10.3740 Epoch: 8 Global Step: 84090 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:59:00,665-Speed 128.64 samples/sec Loss 10.4753 Epoch: 8 Global Step: 84095 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:59:20,224-Speed 130.90 samples/sec Loss 10.8284 Epoch: 8 Global Step: 84100 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 22:59:40,238-Speed 127.92 samples/sec Loss 10.6030 Epoch: 8 Global Step: 84105 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:00:00,106-Speed 128.86 samples/sec Loss 11.1339 Epoch: 8 Global Step: 84110 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:00:19,909-Speed 129.28 samples/sec Loss 10.8007 Epoch: 8 Global Step: 84115 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:00:39,616-Speed 129.91 samples/sec Loss 10.7139 Epoch: 8 Global Step: 84120 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:00:59,255-Speed 130.36 samples/sec Loss 10.7908 Epoch: 8 Global Step: 84125 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:01:19,096-Speed 129.03 samples/sec Loss 10.7028 Epoch: 8 Global Step: 84130 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:01:38,653-Speed 130.91 samples/sec Loss 10.6529 Epoch: 8 Global Step: 84135 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:01:58,557-Speed 128.63 samples/sec Loss 11.0170 Epoch: 8 Global Step: 84140 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:02:18,021-Speed 131.53 samples/sec Loss 10.5482 Epoch: 8 Global Step: 84145 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:02:38,096-Speed 127.53 samples/sec Loss 10.7307 Epoch: 8 Global Step: 84150 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:02:58,004-Speed 128.60 samples/sec Loss 10.9394 Epoch: 8 Global Step: 84155 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:03:17,593-Speed 130.69 samples/sec Loss 10.8186 Epoch: 8 Global Step: 84160 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:03:37,241-Speed 130.30 samples/sec Loss 10.7564 Epoch: 8 Global Step: 84165 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:03:56,983-Speed 129.68 samples/sec Loss 10.7099 Epoch: 8 Global Step: 84170 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:04:16,705-Speed 129.81 samples/sec Loss 10.7698 Epoch: 8 Global Step: 84175 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:04:36,277-Speed 130.80 samples/sec Loss 10.6899 Epoch: 8 Global Step: 84180 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:04:56,134-Speed 128.93 samples/sec Loss 10.6517 Epoch: 8 Global Step: 84185 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:05:15,777-Speed 130.34 samples/sec Loss 11.1950 Epoch: 8 Global Step: 84190 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:05:35,873-Speed 127.39 samples/sec Loss 10.9176 Epoch: 8 Global Step: 84195 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:05:55,968-Speed 127.40 samples/sec Loss 10.3846 Epoch: 8 Global Step: 84200 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:06:15,619-Speed 130.28 samples/sec Loss 11.0628 Epoch: 8 Global Step: 84205 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:06:35,267-Speed 130.30 samples/sec Loss 10.8078 Epoch: 8 Global Step: 84210 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:06:54,923-Speed 130.25 samples/sec Loss 10.9446 Epoch: 8 Global Step: 84215 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:07:14,702-Speed 129.44 samples/sec Loss 10.9091 Epoch: 8 Global Step: 84220 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:07:34,453-Speed 129.62 samples/sec Loss 10.6645 Epoch: 8 Global Step: 84225 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:07:54,160-Speed 129.91 samples/sec Loss 10.6696 Epoch: 8 Global Step: 84230 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:08:13,731-Speed 130.82 samples/sec Loss 10.6314 Epoch: 8 Global Step: 84235 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:08:33,842-Speed 127.30 samples/sec Loss 10.5712 Epoch: 8 Global Step: 84240 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:08:53,848-Speed 127.97 samples/sec Loss 10.8347 Epoch: 8 Global Step: 84245 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:09:13,347-Speed 131.29 samples/sec Loss 10.9642 Epoch: 8 Global Step: 84250 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:09:33,156-Speed 129.24 samples/sec Loss 11.0874 Epoch: 8 Global Step: 84255 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:09:52,752-Speed 130.64 samples/sec Loss 10.9047 Epoch: 8 Global Step: 84260 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:10:12,591-Speed 129.05 samples/sec Loss 10.7631 Epoch: 8 Global Step: 84265 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:10:32,274-Speed 130.07 samples/sec Loss 11.0563 Epoch: 8 Global Step: 84270 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:10:51,958-Speed 130.06 samples/sec Loss 10.5166 Epoch: 8 Global Step: 84275 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:11:11,570-Speed 130.54 samples/sec Loss 10.6590 Epoch: 8 Global Step: 84280 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:11:31,644-Speed 127.53 samples/sec Loss 10.4697 Epoch: 8 Global Step: 84285 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:11:51,647-Speed 127.99 samples/sec Loss 10.8079 Epoch: 8 Global Step: 84290 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:12:11,202-Speed 130.92 samples/sec Loss 10.6252 Epoch: 8 Global Step: 84295 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:12:31,037-Speed 129.07 samples/sec Loss 10.7731 Epoch: 8 Global Step: 84300 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:12:50,604-Speed 130.84 samples/sec Loss 10.8362 Epoch: 8 Global Step: 84305 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:13:10,445-Speed 129.03 samples/sec Loss 10.7597 Epoch: 8 Global Step: 84310 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:13:30,040-Speed 130.65 samples/sec Loss 10.8581 Epoch: 8 Global Step: 84315 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:13:49,796-Speed 129.59 samples/sec Loss 11.1210 Epoch: 8 Global Step: 84320 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:14:09,415-Speed 130.49 samples/sec Loss 10.9997 Epoch: 8 Global Step: 84325 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:14:29,341-Speed 128.49 samples/sec Loss 10.5659 Epoch: 8 Global Step: 84330 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:14:49,333-Speed 128.06 samples/sec Loss 10.4293 Epoch: 8 Global Step: 84335 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:15:08,902-Speed 130.82 samples/sec Loss 10.7012 Epoch: 8 Global Step: 84340 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:15:28,826-Speed 128.50 samples/sec Loss 10.8823 Epoch: 8 Global Step: 84345 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:15:48,380-Speed 130.92 samples/sec Loss 10.8366 Epoch: 8 Global Step: 84350 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:16:08,124-Speed 129.67 samples/sec Loss 10.9268 Epoch: 8 Global Step: 84355 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:16:27,717-Speed 130.67 samples/sec Loss 10.6082 Epoch: 8 Global Step: 84360 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:16:47,462-Speed 129.66 samples/sec Loss 10.9730 Epoch: 8 Global Step: 84365 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:17:07,096-Speed 130.39 samples/sec Loss 10.9690 Epoch: 8 Global Step: 84370 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:17:27,202-Speed 127.33 samples/sec Loss 10.6465 Epoch: 8 Global Step: 84375 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:17:47,047-Speed 129.01 samples/sec Loss 10.6688 Epoch: 8 Global Step: 84380 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:18:06,620-Speed 130.79 samples/sec Loss 10.9050 Epoch: 8 Global Step: 84385 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:18:26,535-Speed 128.56 samples/sec Loss 10.4869 Epoch: 8 Global Step: 84390 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:18:46,218-Speed 130.07 samples/sec Loss 10.9455 Epoch: 8 Global Step: 84395 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:19:06,122-Speed 128.63 samples/sec Loss 10.8161 Epoch: 8 Global Step: 84400 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:19:25,696-Speed 130.79 samples/sec Loss 10.9590 Epoch: 8 Global Step: 84405 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:19:45,397-Speed 129.94 samples/sec Loss 10.8802 Epoch: 8 Global Step: 84410 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:20:04,987-Speed 130.69 samples/sec Loss 10.5582 Epoch: 8 Global Step: 84415 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:20:25,503-Speed 124.79 samples/sec Loss 10.6529 Epoch: 8 Global Step: 84420 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:20:45,238-Speed 129.72 samples/sec Loss 10.6467 Epoch: 8 Global Step: 84425 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:21:05,087-Speed 128.98 samples/sec Loss 10.8239 Epoch: 8 Global Step: 84430 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:21:24,793-Speed 129.92 samples/sec Loss 10.6224 Epoch: 8 Global Step: 84435 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:21:44,405-Speed 130.54 samples/sec Loss 10.3469 Epoch: 8 Global Step: 84440 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:22:04,286-Speed 128.77 samples/sec Loss 10.7737 Epoch: 8 Global Step: 84445 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:22:23,861-Speed 130.78 samples/sec Loss 10.8804 Epoch: 8 Global Step: 84450 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:22:43,740-Speed 128.79 samples/sec Loss 11.2402 Epoch: 8 Global Step: 84455 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:23:03,272-Speed 131.07 samples/sec Loss 11.0304 Epoch: 8 Global Step: 84460 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:23:23,423-Speed 127.05 samples/sec Loss 10.6541 Epoch: 8 Global Step: 84465 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:23:43,238-Speed 129.20 samples/sec Loss 10.7414 Epoch: 8 Global Step: 84470 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:24:02,867-Speed 130.42 samples/sec Loss 10.8439 Epoch: 8 Global Step: 84475 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:24:22,590-Speed 129.80 samples/sec Loss 10.6621 Epoch: 8 Global Step: 84480 Fp16 Grad Scale: 8192 Required: 196 hours Training: 2021-06-10 23:24:42,217-Speed 130.44 samples/sec Loss 10.7023 Epoch: 8 Global Step: 84485 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:25:01,898-Speed 130.08 samples/sec Loss 10.4382 Epoch: 8 Global Step: 84490 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:25:21,445-Speed 130.97 samples/sec Loss 10.5147 Epoch: 8 Global Step: 84495 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:25:41,357-Speed 128.58 samples/sec Loss 11.0133 Epoch: 8 Global Step: 84500 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:26:00,912-Speed 130.91 samples/sec Loss 10.6591 Epoch: 8 Global Step: 84505 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:26:21,223-Speed 126.05 samples/sec Loss 10.7243 Epoch: 8 Global Step: 84510 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:26:40,971-Speed 129.65 samples/sec Loss 11.0558 Epoch: 8 Global Step: 84515 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:27:00,555-Speed 130.73 samples/sec Loss 10.6439 Epoch: 8 Global Step: 84520 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:27:20,290-Speed 129.73 samples/sec Loss 10.8483 Epoch: 8 Global Step: 84525 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:27:39,911-Speed 130.48 samples/sec Loss 10.7222 Epoch: 8 Global Step: 84530 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:27:59,638-Speed 129.78 samples/sec Loss 10.8304 Epoch: 8 Global Step: 84535 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:28:19,200-Speed 130.87 samples/sec Loss 11.0353 Epoch: 8 Global Step: 84540 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:28:38,974-Speed 129.47 samples/sec Loss 10.7666 Epoch: 8 Global Step: 84545 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:28:58,592-Speed 130.50 samples/sec Loss 10.9871 Epoch: 8 Global Step: 84550 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:29:18,809-Speed 126.63 samples/sec Loss 10.6233 Epoch: 8 Global Step: 84555 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:29:38,614-Speed 129.26 samples/sec Loss 10.9298 Epoch: 8 Global Step: 84560 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:29:58,214-Speed 130.62 samples/sec Loss 11.3644 Epoch: 8 Global Step: 84565 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:30:18,066-Speed 128.96 samples/sec Loss 10.4968 Epoch: 8 Global Step: 84570 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:30:37,657-Speed 130.68 samples/sec Loss 10.8423 Epoch: 8 Global Step: 84575 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:30:57,435-Speed 129.44 samples/sec Loss 10.6075 Epoch: 8 Global Step: 84580 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:31:17,054-Speed 130.49 samples/sec Loss 10.6866 Epoch: 8 Global Step: 84585 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:31:36,873-Speed 129.18 samples/sec Loss 10.6630 Epoch: 8 Global Step: 84590 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:31:56,491-Speed 130.50 samples/sec Loss 11.2901 Epoch: 8 Global Step: 84595 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:32:16,488-Speed 128.03 samples/sec Loss 10.6547 Epoch: 8 Global Step: 84600 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:32:36,399-Speed 128.57 samples/sec Loss 10.5598 Epoch: 8 Global Step: 84605 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:32:56,138-Speed 129.70 samples/sec Loss 10.8039 Epoch: 8 Global Step: 84610 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:33:15,949-Speed 129.23 samples/sec Loss 10.6902 Epoch: 8 Global Step: 84615 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:33:35,548-Speed 130.62 samples/sec Loss 10.7941 Epoch: 8 Global Step: 84620 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:33:55,197-Speed 130.29 samples/sec Loss 10.7145 Epoch: 8 Global Step: 84625 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:34:14,857-Speed 130.22 samples/sec Loss 10.8027 Epoch: 8 Global Step: 84630 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:34:34,596-Speed 129.70 samples/sec Loss 10.6793 Epoch: 8 Global Step: 84635 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:34:54,218-Speed 130.47 samples/sec Loss 10.7776 Epoch: 8 Global Step: 84640 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:35:14,290-Speed 127.55 samples/sec Loss 11.1554 Epoch: 8 Global Step: 84645 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:35:34,131-Speed 129.03 samples/sec Loss 10.9666 Epoch: 8 Global Step: 84650 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:35:53,741-Speed 130.55 samples/sec Loss 10.9961 Epoch: 8 Global Step: 84655 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:36:13,585-Speed 129.01 samples/sec Loss 10.8536 Epoch: 8 Global Step: 84660 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:36:33,210-Speed 130.45 samples/sec Loss 11.0669 Epoch: 8 Global Step: 84665 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:36:52,860-Speed 130.28 samples/sec Loss 10.7177 Epoch: 8 Global Step: 84670 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:37:12,500-Speed 130.35 samples/sec Loss 10.7457 Epoch: 8 Global Step: 84675 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:37:32,303-Speed 129.28 samples/sec Loss 10.4855 Epoch: 8 Global Step: 84680 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:37:51,940-Speed 130.38 samples/sec Loss 10.6974 Epoch: 8 Global Step: 84685 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:38:11,959-Speed 127.88 samples/sec Loss 11.0206 Epoch: 8 Global Step: 84690 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:38:31,971-Speed 127.93 samples/sec Loss 11.0201 Epoch: 8 Global Step: 84695 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:38:51,595-Speed 130.46 samples/sec Loss 10.6749 Epoch: 8 Global Step: 84700 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:39:11,592-Speed 128.03 samples/sec Loss 10.6729 Epoch: 8 Global Step: 84705 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:39:31,152-Speed 130.88 samples/sec Loss 10.7662 Epoch: 8 Global Step: 84710 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:39:50,967-Speed 129.21 samples/sec Loss 10.9195 Epoch: 8 Global Step: 84715 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:40:10,538-Speed 130.81 samples/sec Loss 10.8331 Epoch: 8 Global Step: 84720 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:40:30,389-Speed 128.97 samples/sec Loss 11.1112 Epoch: 8 Global Step: 84725 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:40:50,135-Speed 129.65 samples/sec Loss 10.9994 Epoch: 8 Global Step: 84730 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:41:10,137-Speed 127.99 samples/sec Loss 10.7870 Epoch: 8 Global Step: 84735 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:41:29,864-Speed 129.78 samples/sec Loss 11.1028 Epoch: 8 Global Step: 84740 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:41:49,493-Speed 130.42 samples/sec Loss 10.6593 Epoch: 8 Global Step: 84745 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:42:09,239-Speed 129.66 samples/sec Loss 10.9211 Epoch: 8 Global Step: 84750 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:42:28,724-Speed 131.39 samples/sec Loss 10.7841 Epoch: 8 Global Step: 84755 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:42:48,524-Speed 129.30 samples/sec Loss 11.0694 Epoch: 8 Global Step: 84760 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:43:08,096-Speed 130.81 samples/sec Loss 10.5986 Epoch: 8 Global Step: 84765 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:43:28,087-Speed 128.06 samples/sec Loss 10.8063 Epoch: 8 Global Step: 84770 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:43:47,693-Speed 130.58 samples/sec Loss 10.4864 Epoch: 8 Global Step: 84775 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:44:07,723-Speed 127.82 samples/sec Loss 10.9328 Epoch: 8 Global Step: 84780 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:44:27,715-Speed 128.06 samples/sec Loss 10.5575 Epoch: 8 Global Step: 84785 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:44:47,419-Speed 129.93 samples/sec Loss 10.8143 Epoch: 8 Global Step: 84790 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:45:07,111-Speed 130.01 samples/sec Loss 10.5374 Epoch: 8 Global Step: 84795 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:45:26,758-Speed 130.30 samples/sec Loss 10.7358 Epoch: 8 Global Step: 84800 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:45:46,502-Speed 129.67 samples/sec Loss 10.9140 Epoch: 8 Global Step: 84805 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:46:06,131-Speed 130.43 samples/sec Loss 10.8912 Epoch: 8 Global Step: 84810 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:46:26,066-Speed 128.43 samples/sec Loss 10.4162 Epoch: 8 Global Step: 84815 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:46:45,641-Speed 130.78 samples/sec Loss 10.5903 Epoch: 8 Global Step: 84820 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:47:05,662-Speed 127.87 samples/sec Loss 10.6045 Epoch: 8 Global Step: 84825 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:47:25,682-Speed 127.88 samples/sec Loss 10.3679 Epoch: 8 Global Step: 84830 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:47:45,273-Speed 130.68 samples/sec Loss 10.8421 Epoch: 8 Global Step: 84835 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:48:04,927-Speed 130.26 samples/sec Loss 10.9203 Epoch: 8 Global Step: 84840 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:48:24,656-Speed 129.77 samples/sec Loss 10.7722 Epoch: 8 Global Step: 84845 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:48:44,430-Speed 129.47 samples/sec Loss 10.6107 Epoch: 8 Global Step: 84850 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:49:04,037-Speed 130.57 samples/sec Loss 10.8084 Epoch: 8 Global Step: 84855 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:49:23,757-Speed 129.83 samples/sec Loss 10.5528 Epoch: 8 Global Step: 84860 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:49:43,343-Speed 130.71 samples/sec Loss 10.8644 Epoch: 8 Global Step: 84865 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:50:03,428-Speed 127.47 samples/sec Loss 10.6913 Epoch: 8 Global Step: 84870 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:50:23,468-Speed 127.75 samples/sec Loss 11.0411 Epoch: 8 Global Step: 84875 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:50:43,061-Speed 130.66 samples/sec Loss 10.9346 Epoch: 8 Global Step: 84880 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:51:02,873-Speed 129.22 samples/sec Loss 10.6436 Epoch: 8 Global Step: 84885 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:51:22,411-Speed 131.04 samples/sec Loss 11.0647 Epoch: 8 Global Step: 84890 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:51:42,248-Speed 129.06 samples/sec Loss 10.8973 Epoch: 8 Global Step: 84895 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:52:01,866-Speed 130.50 samples/sec Loss 10.6576 Epoch: 8 Global Step: 84900 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:52:21,607-Speed 129.69 samples/sec Loss 10.6185 Epoch: 8 Global Step: 84905 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:52:41,226-Speed 130.49 samples/sec Loss 10.8244 Epoch: 8 Global Step: 84910 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:53:01,334-Speed 127.32 samples/sec Loss 10.6418 Epoch: 8 Global Step: 84915 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:53:21,060-Speed 129.79 samples/sec Loss 10.5788 Epoch: 8 Global Step: 84920 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:53:40,554-Speed 131.33 samples/sec Loss 10.6576 Epoch: 8 Global Step: 84925 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:54:00,430-Speed 128.81 samples/sec Loss 10.6843 Epoch: 8 Global Step: 84930 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:54:19,979-Speed 130.96 samples/sec Loss 11.1224 Epoch: 8 Global Step: 84935 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:54:39,999-Speed 127.88 samples/sec Loss 10.8564 Epoch: 8 Global Step: 84940 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:54:59,582-Speed 130.73 samples/sec Loss 10.5425 Epoch: 8 Global Step: 84945 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:55:19,315-Speed 129.74 samples/sec Loss 10.6500 Epoch: 8 Global Step: 84950 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:55:39,124-Speed 129.24 samples/sec Loss 10.5683 Epoch: 8 Global Step: 84955 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:55:59,387-Speed 126.35 samples/sec Loss 10.6780 Epoch: 8 Global Step: 84960 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:56:19,144-Speed 129.58 samples/sec Loss 10.5974 Epoch: 8 Global Step: 84965 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:56:38,718-Speed 130.79 samples/sec Loss 10.7412 Epoch: 8 Global Step: 84970 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:56:58,598-Speed 128.78 samples/sec Loss 10.6600 Epoch: 8 Global Step: 84975 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:57:18,140-Speed 131.01 samples/sec Loss 10.4039 Epoch: 8 Global Step: 84980 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:57:38,125-Speed 128.10 samples/sec Loss 10.7519 Epoch: 8 Global Step: 84985 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:57:57,733-Speed 130.57 samples/sec Loss 10.5365 Epoch: 8 Global Step: 84990 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:58:17,466-Speed 129.74 samples/sec Loss 10.4746 Epoch: 8 Global Step: 84995 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:58:37,036-Speed 130.82 samples/sec Loss 10.6187 Epoch: 8 Global Step: 85000 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:58:57,163-Speed 127.20 samples/sec Loss 10.6150 Epoch: 8 Global Step: 85005 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:59:16,944-Speed 129.42 samples/sec Loss 11.0004 Epoch: 8 Global Step: 85010 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:59:36,559-Speed 130.52 samples/sec Loss 10.7519 Epoch: 8 Global Step: 85015 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-10 23:59:56,288-Speed 129.76 samples/sec Loss 10.5115 Epoch: 8 Global Step: 85020 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:00:15,920-Speed 130.41 samples/sec Loss 10.6771 Epoch: 8 Global Step: 85025 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:00:35,919-Speed 128.02 samples/sec Loss 10.9422 Epoch: 8 Global Step: 85030 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:00:55,536-Speed 130.51 samples/sec Loss 10.9150 Epoch: 8 Global Step: 85035 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:01:15,455-Speed 128.53 samples/sec Loss 10.7517 Epoch: 8 Global Step: 85040 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:01:35,011-Speed 130.91 samples/sec Loss 10.6376 Epoch: 8 Global Step: 85045 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:01:55,218-Speed 126.70 samples/sec Loss 10.8628 Epoch: 8 Global Step: 85050 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:02:14,885-Speed 130.17 samples/sec Loss 10.6218 Epoch: 8 Global Step: 85055 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:02:34,476-Speed 130.68 samples/sec Loss 10.7979 Epoch: 8 Global Step: 85060 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:02:54,121-Speed 130.32 samples/sec Loss 10.6646 Epoch: 8 Global Step: 85065 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:03:13,863-Speed 129.68 samples/sec Loss 10.5125 Epoch: 8 Global Step: 85070 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:03:33,666-Speed 129.28 samples/sec Loss 10.9833 Epoch: 8 Global Step: 85075 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:03:53,306-Speed 130.35 samples/sec Loss 10.6165 Epoch: 8 Global Step: 85080 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:04:13,225-Speed 128.53 samples/sec Loss 11.0651 Epoch: 8 Global Step: 85085 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:04:32,796-Speed 130.81 samples/sec Loss 10.8347 Epoch: 8 Global Step: 85090 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:04:52,919-Speed 127.22 samples/sec Loss 10.7112 Epoch: 8 Global Step: 85095 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:05:12,758-Speed 129.05 samples/sec Loss 10.6854 Epoch: 8 Global Step: 85100 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:05:32,375-Speed 130.51 samples/sec Loss 10.7443 Epoch: 8 Global Step: 85105 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:05:52,101-Speed 129.78 samples/sec Loss 10.7234 Epoch: 8 Global Step: 85110 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:06:11,885-Speed 129.40 samples/sec Loss 10.7741 Epoch: 8 Global Step: 85115 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:06:31,646-Speed 129.55 samples/sec Loss 10.7539 Epoch: 8 Global Step: 85120 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:06:51,135-Speed 131.37 samples/sec Loss 10.5641 Epoch: 8 Global Step: 85125 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:07:10,995-Speed 128.91 samples/sec Loss 11.1115 Epoch: 8 Global Step: 85130 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:07:30,631-Speed 130.38 samples/sec Loss 11.0360 Epoch: 8 Global Step: 85135 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:07:50,746-Speed 127.27 samples/sec Loss 10.5201 Epoch: 8 Global Step: 85140 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:08:10,683-Speed 128.41 samples/sec Loss 10.5562 Epoch: 8 Global Step: 85145 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:08:30,212-Speed 131.09 samples/sec Loss 10.9284 Epoch: 8 Global Step: 85150 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:08:50,072-Speed 128.91 samples/sec Loss 10.6837 Epoch: 8 Global Step: 85155 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:09:09,674-Speed 130.61 samples/sec Loss 10.5233 Epoch: 8 Global Step: 85160 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:09:29,581-Speed 128.61 samples/sec Loss 10.5970 Epoch: 8 Global Step: 85165 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:09:49,255-Speed 130.13 samples/sec Loss 10.7342 Epoch: 8 Global Step: 85170 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:10:08,968-Speed 129.87 samples/sec Loss 10.5248 Epoch: 8 Global Step: 85175 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:10:28,545-Speed 130.77 samples/sec Loss 10.7839 Epoch: 8 Global Step: 85180 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:10:48,731-Speed 126.83 samples/sec Loss 10.4936 Epoch: 8 Global Step: 85185 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:11:08,667-Speed 128.42 samples/sec Loss 10.8565 Epoch: 8 Global Step: 85190 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:11:28,291-Speed 130.46 samples/sec Loss 10.7716 Epoch: 8 Global Step: 85195 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:11:48,191-Speed 128.65 samples/sec Loss 10.7007 Epoch: 8 Global Step: 85200 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:12:07,860-Speed 130.17 samples/sec Loss 10.6227 Epoch: 8 Global Step: 85205 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:12:27,529-Speed 130.16 samples/sec Loss 10.8077 Epoch: 8 Global Step: 85210 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:12:47,145-Speed 130.52 samples/sec Loss 10.6209 Epoch: 8 Global Step: 85215 Fp16 Grad Scale: 8192 Required: 195 hours Training: 2021-06-11 00:13:06,954-Speed 129.24 samples/sec Loss 10.6239 Epoch: 8 Global Step: 85220 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:13:26,604-Speed 130.28 samples/sec Loss 10.9326 Epoch: 8 Global Step: 85225 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:13:46,459-Speed 128.94 samples/sec Loss 10.6986 Epoch: 8 Global Step: 85230 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:14:06,265-Speed 129.26 samples/sec Loss 10.7924 Epoch: 8 Global Step: 85235 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:14:25,845-Speed 130.75 samples/sec Loss 10.6599 Epoch: 8 Global Step: 85240 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:14:45,768-Speed 128.50 samples/sec Loss 10.7194 Epoch: 8 Global Step: 85245 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:15:05,220-Speed 131.61 samples/sec Loss 10.7248 Epoch: 8 Global Step: 85250 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:15:24,911-Speed 130.02 samples/sec Loss 10.6867 Epoch: 8 Global Step: 85255 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:15:44,611-Speed 129.96 samples/sec Loss 10.4606 Epoch: 8 Global Step: 85260 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:16:04,350-Speed 129.70 samples/sec Loss 10.8886 Epoch: 8 Global Step: 85265 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:16:23,970-Speed 130.49 samples/sec Loss 10.6396 Epoch: 8 Global Step: 85270 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:16:43,820-Speed 128.97 samples/sec Loss 10.8610 Epoch: 8 Global Step: 85275 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:17:03,843-Speed 127.86 samples/sec Loss 10.7429 Epoch: 8 Global Step: 85280 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:17:23,419-Speed 130.78 samples/sec Loss 10.5710 Epoch: 8 Global Step: 85285 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:17:43,353-Speed 128.43 samples/sec Loss 10.5091 Epoch: 8 Global Step: 85290 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:18:02,802-Speed 131.63 samples/sec Loss 10.8697 Epoch: 8 Global Step: 85295 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:18:22,873-Speed 127.56 samples/sec Loss 10.7767 Epoch: 8 Global Step: 85300 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:18:42,511-Speed 130.36 samples/sec Loss 10.8991 Epoch: 8 Global Step: 85305 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:19:02,269-Speed 129.58 samples/sec Loss 11.0718 Epoch: 8 Global Step: 85310 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:19:21,853-Speed 130.73 samples/sec Loss 10.6930 Epoch: 8 Global Step: 85315 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:19:41,861-Speed 127.96 samples/sec Loss 10.8027 Epoch: 8 Global Step: 85320 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:20:01,601-Speed 129.69 samples/sec Loss 10.7478 Epoch: 8 Global Step: 85325 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:20:21,140-Speed 131.03 samples/sec Loss 10.5454 Epoch: 8 Global Step: 85330 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:20:40,949-Speed 129.24 samples/sec Loss 10.8495 Epoch: 8 Global Step: 85335 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:21:00,410-Speed 131.55 samples/sec Loss 10.7055 Epoch: 8 Global Step: 85340 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:21:20,396-Speed 128.10 samples/sec Loss 10.5280 Epoch: 8 Global Step: 85345 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:21:39,957-Speed 130.88 samples/sec Loss 10.6243 Epoch: 8 Global Step: 85350 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:21:59,906-Speed 128.33 samples/sec Loss 10.4849 Epoch: 8 Global Step: 85355 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:22:19,457-Speed 130.95 samples/sec Loss 10.7730 Epoch: 8 Global Step: 85360 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:22:39,503-Speed 127.71 samples/sec Loss 10.6011 Epoch: 8 Global Step: 85365 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:22:59,220-Speed 129.85 samples/sec Loss 10.6446 Epoch: 8 Global Step: 85370 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:23:18,774-Speed 130.92 samples/sec Loss 10.4693 Epoch: 8 Global Step: 85375 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:23:38,823-Speed 127.69 samples/sec Loss 10.7642 Epoch: 8 Global Step: 85380 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:23:58,503-Speed 130.09 samples/sec Loss 10.8880 Epoch: 8 Global Step: 85385 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:24:18,406-Speed 128.63 samples/sec Loss 10.6436 Epoch: 8 Global Step: 85390 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:24:37,964-Speed 130.90 samples/sec Loss 10.8831 Epoch: 8 Global Step: 85395 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:24:57,877-Speed 128.57 samples/sec Loss 10.9301 Epoch: 8 Global Step: 85400 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:25:17,326-Speed 131.63 samples/sec Loss 11.0287 Epoch: 8 Global Step: 85405 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:25:37,476-Speed 127.06 samples/sec Loss 11.2032 Epoch: 8 Global Step: 85410 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:25:57,153-Speed 130.11 samples/sec Loss 10.4417 Epoch: 8 Global Step: 85415 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:26:16,764-Speed 130.54 samples/sec Loss 10.9621 Epoch: 8 Global Step: 85420 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:26:36,388-Speed 130.46 samples/sec Loss 11.0696 Epoch: 8 Global Step: 85425 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:26:55,998-Speed 130.55 samples/sec Loss 10.6619 Epoch: 8 Global Step: 85430 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:27:15,892-Speed 128.69 samples/sec Loss 10.8763 Epoch: 8 Global Step: 85435 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:27:35,363-Speed 131.49 samples/sec Loss 10.8914 Epoch: 8 Global Step: 85440 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:27:55,350-Speed 128.09 samples/sec Loss 10.4387 Epoch: 8 Global Step: 85445 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:28:14,904-Speed 130.93 samples/sec Loss 10.6723 Epoch: 8 Global Step: 85450 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:28:35,014-Speed 127.31 samples/sec Loss 11.0743 Epoch: 8 Global Step: 85455 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:28:54,832-Speed 129.18 samples/sec Loss 10.9286 Epoch: 8 Global Step: 85460 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:29:14,452-Speed 130.49 samples/sec Loss 10.9857 Epoch: 8 Global Step: 85465 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:29:34,177-Speed 129.80 samples/sec Loss 10.8072 Epoch: 8 Global Step: 85470 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:29:53,747-Speed 130.82 samples/sec Loss 10.4892 Epoch: 8 Global Step: 85475 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:30:13,598-Speed 128.97 samples/sec Loss 10.8183 Epoch: 8 Global Step: 85480 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:30:33,456-Speed 128.92 samples/sec Loss 10.9530 Epoch: 8 Global Step: 85485 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:30:53,275-Speed 129.18 samples/sec Loss 11.1400 Epoch: 8 Global Step: 85490 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:31:12,909-Speed 130.39 samples/sec Loss 11.0465 Epoch: 8 Global Step: 85495 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:31:33,006-Speed 127.39 samples/sec Loss 10.9192 Epoch: 8 Global Step: 85500 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:31:52,832-Speed 129.13 samples/sec Loss 10.2436 Epoch: 8 Global Step: 85505 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:32:12,413-Speed 130.74 samples/sec Loss 11.0160 Epoch: 8 Global Step: 85510 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:32:32,228-Speed 129.21 samples/sec Loss 10.6762 Epoch: 8 Global Step: 85515 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:32:51,880-Speed 130.27 samples/sec Loss 10.6745 Epoch: 8 Global Step: 85520 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:33:11,672-Speed 129.35 samples/sec Loss 10.2962 Epoch: 8 Global Step: 85525 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:33:31,259-Speed 130.70 samples/sec Loss 10.3957 Epoch: 8 Global Step: 85530 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:33:51,123-Speed 128.88 samples/sec Loss 10.4625 Epoch: 8 Global Step: 85535 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:34:10,744-Speed 130.48 samples/sec Loss 10.8794 Epoch: 8 Global Step: 85540 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:34:30,845-Speed 127.36 samples/sec Loss 10.7303 Epoch: 8 Global Step: 85545 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:34:50,640-Speed 129.33 samples/sec Loss 10.6634 Epoch: 8 Global Step: 85550 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:35:10,294-Speed 130.26 samples/sec Loss 10.7843 Epoch: 8 Global Step: 85555 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:35:30,018-Speed 129.80 samples/sec Loss 10.7792 Epoch: 8 Global Step: 85560 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:35:49,759-Speed 129.69 samples/sec Loss 10.6703 Epoch: 8 Global Step: 85565 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:36:09,479-Speed 129.83 samples/sec Loss 10.6246 Epoch: 8 Global Step: 85570 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:36:29,081-Speed 130.60 samples/sec Loss 10.7346 Epoch: 8 Global Step: 85575 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:36:48,893-Speed 129.22 samples/sec Loss 10.8202 Epoch: 8 Global Step: 85580 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:37:08,464-Speed 130.81 samples/sec Loss 11.1363 Epoch: 8 Global Step: 85585 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:37:28,602-Speed 127.13 samples/sec Loss 10.7710 Epoch: 8 Global Step: 85590 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:37:48,506-Speed 128.63 samples/sec Loss 10.9344 Epoch: 8 Global Step: 85595 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:38:08,141-Speed 130.38 samples/sec Loss 10.9583 Epoch: 8 Global Step: 85600 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:38:27,898-Speed 129.59 samples/sec Loss 10.6145 Epoch: 8 Global Step: 85605 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:38:47,849-Speed 128.32 samples/sec Loss 10.6024 Epoch: 8 Global Step: 85610 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:39:07,595-Speed 129.65 samples/sec Loss 10.6335 Epoch: 8 Global Step: 85615 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:39:27,219-Speed 130.46 samples/sec Loss 10.8478 Epoch: 8 Global Step: 85620 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:39:47,188-Speed 128.20 samples/sec Loss 10.6657 Epoch: 8 Global Step: 85625 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:40:06,767-Speed 130.76 samples/sec Loss 10.6718 Epoch: 8 Global Step: 85630 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:40:26,725-Speed 128.28 samples/sec Loss 10.5668 Epoch: 8 Global Step: 85635 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:40:46,780-Speed 127.65 samples/sec Loss 11.1800 Epoch: 8 Global Step: 85640 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:41:06,317-Speed 131.04 samples/sec Loss 10.7418 Epoch: 8 Global Step: 85645 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:41:26,135-Speed 129.18 samples/sec Loss 10.6167 Epoch: 8 Global Step: 85650 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:41:45,720-Speed 130.72 samples/sec Loss 10.6456 Epoch: 8 Global Step: 85655 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:42:05,632-Speed 128.57 samples/sec Loss 10.9561 Epoch: 8 Global Step: 85660 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:42:25,249-Speed 130.51 samples/sec Loss 10.7216 Epoch: 8 Global Step: 85665 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:42:44,978-Speed 129.76 samples/sec Loss 10.7806 Epoch: 8 Global Step: 85670 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:43:04,595-Speed 130.51 samples/sec Loss 10.8736 Epoch: 8 Global Step: 85675 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:43:24,621-Speed 127.84 samples/sec Loss 11.0076 Epoch: 8 Global Step: 85680 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:43:44,431-Speed 129.23 samples/sec Loss 10.8615 Epoch: 8 Global Step: 85685 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:44:03,987-Speed 130.91 samples/sec Loss 10.7551 Epoch: 8 Global Step: 85690 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:44:23,759-Speed 129.48 samples/sec Loss 10.6543 Epoch: 8 Global Step: 85695 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:44:43,244-Speed 131.39 samples/sec Loss 10.7974 Epoch: 8 Global Step: 85700 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:45:03,103-Speed 128.91 samples/sec Loss 10.5756 Epoch: 8 Global Step: 85705 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:45:22,750-Speed 130.30 samples/sec Loss 10.5166 Epoch: 8 Global Step: 85710 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:45:42,601-Speed 128.97 samples/sec Loss 11.0069 Epoch: 8 Global Step: 85715 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:46:02,127-Speed 131.11 samples/sec Loss 11.0767 Epoch: 8 Global Step: 85720 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:46:22,157-Speed 127.82 samples/sec Loss 10.6087 Epoch: 8 Global Step: 85725 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:46:41,969-Speed 129.22 samples/sec Loss 11.0234 Epoch: 8 Global Step: 85730 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:47:01,548-Speed 130.76 samples/sec Loss 11.0296 Epoch: 8 Global Step: 85735 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:47:21,430-Speed 128.77 samples/sec Loss 10.6429 Epoch: 8 Global Step: 85740 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:47:40,980-Speed 130.96 samples/sec Loss 10.5457 Epoch: 8 Global Step: 85745 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:48:00,886-Speed 128.61 samples/sec Loss 10.9932 Epoch: 8 Global Step: 85750 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:48:20,474-Speed 130.70 samples/sec Loss 10.8265 Epoch: 8 Global Step: 85755 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:48:40,281-Speed 129.25 samples/sec Loss 10.9036 Epoch: 8 Global Step: 85760 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:48:59,730-Speed 131.64 samples/sec Loss 11.0046 Epoch: 8 Global Step: 85765 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:49:19,818-Speed 127.45 samples/sec Loss 10.6508 Epoch: 8 Global Step: 85770 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:49:39,546-Speed 129.77 samples/sec Loss 10.9201 Epoch: 8 Global Step: 85775 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:49:59,201-Speed 130.25 samples/sec Loss 10.9812 Epoch: 8 Global Step: 85780 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:50:18,820-Speed 130.49 samples/sec Loss 10.9001 Epoch: 8 Global Step: 85785 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:50:38,391-Speed 130.81 samples/sec Loss 11.1715 Epoch: 8 Global Step: 85790 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:50:58,190-Speed 129.31 samples/sec Loss 10.8526 Epoch: 8 Global Step: 85795 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:51:17,749-Speed 130.90 samples/sec Loss 10.9725 Epoch: 8 Global Step: 85800 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:51:37,604-Speed 128.94 samples/sec Loss 10.7078 Epoch: 8 Global Step: 85805 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:51:57,076-Speed 131.48 samples/sec Loss 10.8813 Epoch: 8 Global Step: 85810 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:52:17,204-Speed 127.20 samples/sec Loss 10.8143 Epoch: 8 Global Step: 85815 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:52:37,085-Speed 128.77 samples/sec Loss 10.8748 Epoch: 8 Global Step: 85820 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:52:56,665-Speed 130.75 samples/sec Loss 10.9442 Epoch: 8 Global Step: 85825 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:53:16,300-Speed 130.38 samples/sec Loss 10.7618 Epoch: 8 Global Step: 85830 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:53:35,858-Speed 130.90 samples/sec Loss 10.8374 Epoch: 8 Global Step: 85835 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:53:55,811-Speed 128.31 samples/sec Loss 10.7143 Epoch: 8 Global Step: 85840 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:54:15,333-Speed 131.15 samples/sec Loss 10.3457 Epoch: 8 Global Step: 85845 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:54:35,160-Speed 129.12 samples/sec Loss 11.0402 Epoch: 8 Global Step: 85850 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:54:54,732-Speed 130.80 samples/sec Loss 10.3509 Epoch: 8 Global Step: 85855 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:55:14,850-Speed 127.25 samples/sec Loss 10.4884 Epoch: 8 Global Step: 85860 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:55:34,704-Speed 128.95 samples/sec Loss 10.8852 Epoch: 8 Global Step: 85865 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:55:54,416-Speed 129.87 samples/sec Loss 10.5974 Epoch: 8 Global Step: 85870 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:56:14,195-Speed 129.44 samples/sec Loss 10.6021 Epoch: 8 Global Step: 85875 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:56:33,840-Speed 130.32 samples/sec Loss 10.4958 Epoch: 8 Global Step: 85880 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:56:53,654-Speed 129.21 samples/sec Loss 10.6180 Epoch: 8 Global Step: 85885 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:57:13,247-Speed 130.67 samples/sec Loss 10.5845 Epoch: 8 Global Step: 85890 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:57:33,124-Speed 128.80 samples/sec Loss 10.4239 Epoch: 8 Global Step: 85895 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:57:52,694-Speed 130.82 samples/sec Loss 10.5725 Epoch: 8 Global Step: 85900 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:58:12,762-Speed 127.58 samples/sec Loss 10.3882 Epoch: 8 Global Step: 85905 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:58:32,702-Speed 128.39 samples/sec Loss 10.4818 Epoch: 8 Global Step: 85910 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:58:52,299-Speed 130.64 samples/sec Loss 10.7234 Epoch: 8 Global Step: 85915 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:59:12,086-Speed 129.39 samples/sec Loss 10.7656 Epoch: 8 Global Step: 85920 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:59:31,706-Speed 130.48 samples/sec Loss 10.8935 Epoch: 8 Global Step: 85925 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 00:59:51,446-Speed 129.70 samples/sec Loss 10.5612 Epoch: 8 Global Step: 85930 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 01:00:11,073-Speed 130.44 samples/sec Loss 11.0393 Epoch: 8 Global Step: 85935 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 01:00:30,829-Speed 129.59 samples/sec Loss 10.8322 Epoch: 8 Global Step: 85940 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 01:00:50,394-Speed 130.86 samples/sec Loss 10.7390 Epoch: 8 Global Step: 85945 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 01:01:10,525-Speed 127.17 samples/sec Loss 10.5375 Epoch: 8 Global Step: 85950 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 01:01:30,488-Speed 128.24 samples/sec Loss 10.5381 Epoch: 8 Global Step: 85955 Fp16 Grad Scale: 8192 Required: 194 hours Training: 2021-06-11 01:01:49,958-Speed 131.48 samples/sec Loss 10.4261 Epoch: 8 Global Step: 85960 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:02:09,736-Speed 129.44 samples/sec Loss 10.4445 Epoch: 8 Global Step: 85965 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:02:29,356-Speed 130.49 samples/sec Loss 10.5689 Epoch: 8 Global Step: 85970 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:02:48,984-Speed 130.43 samples/sec Loss 11.0129 Epoch: 8 Global Step: 85975 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:03:08,605-Speed 130.48 samples/sec Loss 10.6636 Epoch: 8 Global Step: 85980 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:03:28,311-Speed 129.92 samples/sec Loss 11.0076 Epoch: 8 Global Step: 85985 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:03:47,889-Speed 130.76 samples/sec Loss 10.5094 Epoch: 8 Global Step: 85990 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:04:08,245-Speed 125.77 samples/sec Loss 10.7619 Epoch: 8 Global Step: 85995 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:04:28,186-Speed 128.39 samples/sec Loss 10.3855 Epoch: 8 Global Step: 86000 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:04:47,732-Speed 130.98 samples/sec Loss 10.7287 Epoch: 8 Global Step: 86005 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:05:07,667-Speed 128.43 samples/sec Loss 10.9295 Epoch: 8 Global Step: 86010 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:05:27,235-Speed 130.83 samples/sec Loss 10.9149 Epoch: 8 Global Step: 86015 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:05:46,922-Speed 130.04 samples/sec Loss 10.7530 Epoch: 8 Global Step: 86020 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:06:06,495-Speed 130.80 samples/sec Loss 10.6528 Epoch: 8 Global Step: 86025 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:06:26,289-Speed 129.34 samples/sec Loss 10.7878 Epoch: 8 Global Step: 86030 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:06:45,919-Speed 130.42 samples/sec Loss 10.3707 Epoch: 8 Global Step: 86035 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:07:05,943-Speed 127.85 samples/sec Loss 11.2187 Epoch: 8 Global Step: 86040 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:07:25,682-Speed 129.70 samples/sec Loss 10.7213 Epoch: 8 Global Step: 86045 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:07:45,299-Speed 130.51 samples/sec Loss 10.7941 Epoch: 8 Global Step: 86050 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:08:05,120-Speed 129.16 samples/sec Loss 10.5332 Epoch: 8 Global Step: 86055 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:08:24,596-Speed 131.45 samples/sec Loss 10.8514 Epoch: 8 Global Step: 86060 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:08:44,472-Speed 128.81 samples/sec Loss 11.0162 Epoch: 8 Global Step: 86065 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:09:04,042-Speed 130.82 samples/sec Loss 10.6473 Epoch: 8 Global Step: 86070 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:09:23,986-Speed 128.37 samples/sec Loss 10.6164 Epoch: 8 Global Step: 86075 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:09:43,631-Speed 130.32 samples/sec Loss 11.1446 Epoch: 8 Global Step: 86080 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:10:03,660-Speed 127.82 samples/sec Loss 10.8382 Epoch: 8 Global Step: 86085 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:10:23,410-Speed 129.63 samples/sec Loss 10.9693 Epoch: 8 Global Step: 86090 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:10:43,047-Speed 130.37 samples/sec Loss 10.8100 Epoch: 8 Global Step: 86095 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:11:02,988-Speed 128.39 samples/sec Loss 10.6007 Epoch: 8 Global Step: 86100 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:11:22,536-Speed 130.97 samples/sec Loss 10.6302 Epoch: 8 Global Step: 86105 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:11:42,320-Speed 129.41 samples/sec Loss 10.8851 Epoch: 8 Global Step: 86110 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:12:01,881-Speed 130.88 samples/sec Loss 10.5943 Epoch: 8 Global Step: 86115 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:12:21,834-Speed 128.31 samples/sec Loss 10.9103 Epoch: 8 Global Step: 86120 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:12:41,688-Speed 128.95 samples/sec Loss 10.6080 Epoch: 8 Global Step: 86125 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:13:01,785-Speed 127.39 samples/sec Loss 10.5409 Epoch: 8 Global Step: 86130 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:13:21,684-Speed 128.66 samples/sec Loss 10.6500 Epoch: 8 Global Step: 86135 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:13:41,296-Speed 130.53 samples/sec Loss 10.6547 Epoch: 8 Global Step: 86140 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:14:00,927-Speed 130.41 samples/sec Loss 10.7055 Epoch: 8 Global Step: 86145 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:14:20,728-Speed 129.29 samples/sec Loss 10.5406 Epoch: 8 Global Step: 86150 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:14:40,491-Speed 129.54 samples/sec Loss 10.3999 Epoch: 8 Global Step: 86155 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:15:00,058-Speed 130.84 samples/sec Loss 10.7049 Epoch: 8 Global Step: 86160 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:15:20,061-Speed 127.99 samples/sec Loss 10.6395 Epoch: 8 Global Step: 86165 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:15:39,575-Speed 131.20 samples/sec Loss 10.8547 Epoch: 8 Global Step: 86170 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:15:59,666-Speed 127.43 samples/sec Loss 10.5684 Epoch: 8 Global Step: 86175 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:16:19,605-Speed 128.40 samples/sec Loss 11.0325 Epoch: 8 Global Step: 86180 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:16:39,227-Speed 130.47 samples/sec Loss 11.0930 Epoch: 8 Global Step: 86185 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:16:59,222-Speed 128.04 samples/sec Loss 10.7883 Epoch: 8 Global Step: 86190 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:17:18,763-Speed 131.01 samples/sec Loss 10.7250 Epoch: 8 Global Step: 86195 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:17:38,542-Speed 129.44 samples/sec Loss 10.5210 Epoch: 8 Global Step: 86200 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:17:58,136-Speed 130.66 samples/sec Loss 10.4472 Epoch: 8 Global Step: 86205 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:18:17,938-Speed 129.28 samples/sec Loss 10.9110 Epoch: 8 Global Step: 86210 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:18:37,586-Speed 130.31 samples/sec Loss 10.5489 Epoch: 8 Global Step: 86215 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:18:57,636-Speed 127.68 samples/sec Loss 10.6208 Epoch: 8 Global Step: 86220 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:19:17,557-Speed 128.52 samples/sec Loss 10.7987 Epoch: 8 Global Step: 86225 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:19:37,183-Speed 130.44 samples/sec Loss 10.7673 Epoch: 8 Global Step: 86230 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:19:57,087-Speed 128.62 samples/sec Loss 10.5045 Epoch: 8 Global Step: 86235 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:20:16,796-Speed 129.90 samples/sec Loss 10.7570 Epoch: 8 Global Step: 86240 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:20:36,603-Speed 129.25 samples/sec Loss 10.4236 Epoch: 8 Global Step: 86245 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:20:56,145-Speed 131.01 samples/sec Loss 10.4466 Epoch: 8 Global Step: 86250 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:21:15,996-Speed 128.97 samples/sec Loss 10.7516 Epoch: 8 Global Step: 86255 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:21:35,650-Speed 130.26 samples/sec Loss 10.9728 Epoch: 8 Global Step: 86260 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:21:55,696-Speed 127.71 samples/sec Loss 11.2240 Epoch: 8 Global Step: 86265 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:22:15,553-Speed 128.92 samples/sec Loss 10.7582 Epoch: 8 Global Step: 86270 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:22:35,146-Speed 130.66 samples/sec Loss 10.7494 Epoch: 8 Global Step: 86275 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:22:55,122-Speed 128.16 samples/sec Loss 10.7641 Epoch: 8 Global Step: 86280 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:23:14,687-Speed 130.85 samples/sec Loss 10.8570 Epoch: 8 Global Step: 86285 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:23:34,521-Speed 129.08 samples/sec Loss 10.6031 Epoch: 8 Global Step: 86290 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:23:54,141-Speed 130.49 samples/sec Loss 10.8851 Epoch: 8 Global Step: 86295 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:24:13,896-Speed 129.60 samples/sec Loss 10.4307 Epoch: 8 Global Step: 86300 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:24:33,474-Speed 130.76 samples/sec Loss 10.8984 Epoch: 8 Global Step: 86305 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:24:53,498-Speed 127.85 samples/sec Loss 10.4196 Epoch: 8 Global Step: 86310 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:25:13,268-Speed 129.50 samples/sec Loss 10.4411 Epoch: 8 Global Step: 86315 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:25:32,874-Speed 130.58 samples/sec Loss 10.6817 Epoch: 8 Global Step: 86320 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:25:52,619-Speed 129.66 samples/sec Loss 10.5471 Epoch: 8 Global Step: 86325 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:26:12,125-Speed 131.25 samples/sec Loss 10.9508 Epoch: 8 Global Step: 86330 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:26:32,063-Speed 128.41 samples/sec Loss 10.8164 Epoch: 8 Global Step: 86335 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:26:51,665-Speed 130.61 samples/sec Loss 10.4388 Epoch: 8 Global Step: 86340 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:27:11,518-Speed 128.95 samples/sec Loss 10.6952 Epoch: 8 Global Step: 86345 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:27:31,088-Speed 130.82 samples/sec Loss 10.6303 Epoch: 8 Global Step: 86350 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:27:51,093-Speed 127.97 samples/sec Loss 10.9196 Epoch: 8 Global Step: 86355 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:28:11,050-Speed 128.29 samples/sec Loss 10.9764 Epoch: 8 Global Step: 86360 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:28:30,641-Speed 130.68 samples/sec Loss 10.5423 Epoch: 8 Global Step: 86365 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:28:50,374-Speed 129.74 samples/sec Loss 10.9878 Epoch: 8 Global Step: 86370 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:29:09,976-Speed 130.60 samples/sec Loss 10.5737 Epoch: 8 Global Step: 86375 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:29:29,804-Speed 129.12 samples/sec Loss 10.4739 Epoch: 8 Global Step: 86380 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:29:49,367-Speed 130.87 samples/sec Loss 10.6505 Epoch: 8 Global Step: 86385 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:30:09,123-Speed 129.59 samples/sec Loss 10.6663 Epoch: 8 Global Step: 86390 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:30:28,693-Speed 130.82 samples/sec Loss 10.6510 Epoch: 8 Global Step: 86395 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:30:48,791-Speed 127.39 samples/sec Loss 10.7652 Epoch: 8 Global Step: 86400 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:31:08,695-Speed 128.62 samples/sec Loss 10.6523 Epoch: 8 Global Step: 86405 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:31:28,192-Speed 131.31 samples/sec Loss 10.7749 Epoch: 8 Global Step: 86410 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:31:47,917-Speed 129.79 samples/sec Loss 10.7714 Epoch: 8 Global Step: 86415 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:32:07,677-Speed 129.56 samples/sec Loss 10.8144 Epoch: 8 Global Step: 86420 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:32:27,519-Speed 129.03 samples/sec Loss 10.6401 Epoch: 8 Global Step: 86425 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:32:47,093-Speed 130.79 samples/sec Loss 10.2375 Epoch: 8 Global Step: 86430 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:33:06,858-Speed 129.53 samples/sec Loss 10.5734 Epoch: 8 Global Step: 86435 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:33:26,352-Speed 131.33 samples/sec Loss 10.9509 Epoch: 8 Global Step: 86440 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:33:46,578-Speed 126.58 samples/sec Loss 10.7111 Epoch: 8 Global Step: 86445 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:34:06,366-Speed 129.38 samples/sec Loss 10.7898 Epoch: 8 Global Step: 86450 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:34:25,921-Speed 130.92 samples/sec Loss 10.9198 Epoch: 8 Global Step: 86455 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:34:45,633-Speed 129.88 samples/sec Loss 10.6066 Epoch: 8 Global Step: 86460 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:35:05,951-Speed 126.00 samples/sec Loss 10.7600 Epoch: 8 Global Step: 86465 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:35:26,121-Speed 126.93 samples/sec Loss 10.4623 Epoch: 8 Global Step: 86470 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:35:45,882-Speed 129.55 samples/sec Loss 10.7544 Epoch: 8 Global Step: 86475 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:36:05,641-Speed 129.57 samples/sec Loss 10.8815 Epoch: 8 Global Step: 86480 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:36:25,220-Speed 130.76 samples/sec Loss 10.9449 Epoch: 8 Global Step: 86485 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:36:45,269-Speed 127.69 samples/sec Loss 10.3879 Epoch: 8 Global Step: 86490 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:37:05,083-Speed 129.21 samples/sec Loss 10.8583 Epoch: 8 Global Step: 86495 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:37:24,643-Speed 130.88 samples/sec Loss 10.8737 Epoch: 8 Global Step: 86500 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:37:44,827-Speed 126.84 samples/sec Loss 10.5243 Epoch: 8 Global Step: 86505 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:38:04,534-Speed 129.91 samples/sec Loss 10.9109 Epoch: 8 Global Step: 86510 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:38:24,232-Speed 129.97 samples/sec Loss 10.7673 Epoch: 8 Global Step: 86515 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:38:43,940-Speed 129.90 samples/sec Loss 10.9576 Epoch: 8 Global Step: 86520 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:39:03,720-Speed 129.43 samples/sec Loss 10.7865 Epoch: 8 Global Step: 86525 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:39:23,385-Speed 130.18 samples/sec Loss 10.7859 Epoch: 8 Global Step: 86530 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:39:43,269-Speed 128.75 samples/sec Loss 10.6339 Epoch: 8 Global Step: 86535 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:40:03,171-Speed 128.64 samples/sec Loss 10.7295 Epoch: 8 Global Step: 86540 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:40:22,753-Speed 130.73 samples/sec Loss 10.9985 Epoch: 8 Global Step: 86545 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:40:42,721-Speed 128.21 samples/sec Loss 10.6876 Epoch: 8 Global Step: 86550 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:41:02,196-Speed 131.46 samples/sec Loss 10.6462 Epoch: 8 Global Step: 86555 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:41:22,000-Speed 129.28 samples/sec Loss 10.8429 Epoch: 8 Global Step: 86560 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:41:41,562-Speed 130.87 samples/sec Loss 10.6928 Epoch: 8 Global Step: 86565 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:42:01,347-Speed 129.40 samples/sec Loss 10.8243 Epoch: 8 Global Step: 86570 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:42:21,380-Speed 127.80 samples/sec Loss 10.4957 Epoch: 8 Global Step: 86575 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:42:41,361-Speed 128.13 samples/sec Loss 10.5245 Epoch: 8 Global Step: 86580 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:43:01,110-Speed 129.63 samples/sec Loss 10.8416 Epoch: 8 Global Step: 86585 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:43:20,757-Speed 130.31 samples/sec Loss 10.4956 Epoch: 8 Global Step: 86590 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:43:40,628-Speed 128.84 samples/sec Loss 10.9266 Epoch: 8 Global Step: 86595 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:44:00,084-Speed 131.59 samples/sec Loss 10.4725 Epoch: 8 Global Step: 86600 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:44:19,899-Speed 129.21 samples/sec Loss 10.5709 Epoch: 8 Global Step: 86605 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:44:39,488-Speed 130.69 samples/sec Loss 10.7247 Epoch: 8 Global Step: 86610 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:44:59,533-Speed 127.72 samples/sec Loss 10.9378 Epoch: 8 Global Step: 86615 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:45:19,085-Speed 130.94 samples/sec Loss 10.3980 Epoch: 8 Global Step: 86620 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:45:39,102-Speed 127.90 samples/sec Loss 11.3256 Epoch: 8 Global Step: 86625 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:45:58,845-Speed 129.67 samples/sec Loss 10.6612 Epoch: 8 Global Step: 86630 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:46:18,401-Speed 130.91 samples/sec Loss 10.5285 Epoch: 8 Global Step: 86635 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:46:38,157-Speed 129.59 samples/sec Loss 10.6236 Epoch: 8 Global Step: 86640 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:46:57,606-Speed 131.64 samples/sec Loss 10.6384 Epoch: 8 Global Step: 86645 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:47:17,438-Speed 129.09 samples/sec Loss 10.5519 Epoch: 8 Global Step: 86650 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:47:37,056-Speed 130.50 samples/sec Loss 10.5515 Epoch: 8 Global Step: 86655 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:47:56,961-Speed 128.62 samples/sec Loss 10.9563 Epoch: 8 Global Step: 86660 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:48:16,413-Speed 131.62 samples/sec Loss 10.6714 Epoch: 8 Global Step: 86665 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:48:36,406-Speed 128.05 samples/sec Loss 11.0042 Epoch: 8 Global Step: 86670 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:48:56,205-Speed 129.31 samples/sec Loss 10.6565 Epoch: 8 Global Step: 86675 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:49:15,851-Speed 130.31 samples/sec Loss 10.7315 Epoch: 8 Global Step: 86680 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:49:35,526-Speed 130.12 samples/sec Loss 10.5437 Epoch: 8 Global Step: 86685 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:49:55,010-Speed 131.39 samples/sec Loss 10.6660 Epoch: 8 Global Step: 86690 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:50:14,973-Speed 128.24 samples/sec Loss 10.6141 Epoch: 8 Global Step: 86695 Fp16 Grad Scale: 8192 Required: 193 hours Training: 2021-06-11 01:50:34,844-Speed 128.84 samples/sec Loss 10.8178 Epoch: 8 Global Step: 86700 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:50:54,848-Speed 127.98 samples/sec Loss 10.7465 Epoch: 8 Global Step: 86705 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:51:14,284-Speed 131.72 samples/sec Loss 10.6065 Epoch: 8 Global Step: 86710 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:51:34,283-Speed 128.02 samples/sec Loss 11.0446 Epoch: 8 Global Step: 86715 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:51:54,142-Speed 128.91 samples/sec Loss 10.8708 Epoch: 8 Global Step: 86720 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:52:13,739-Speed 130.64 samples/sec Loss 10.8916 Epoch: 8 Global Step: 86725 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:52:33,363-Speed 130.46 samples/sec Loss 10.6989 Epoch: 8 Global Step: 86730 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:52:52,973-Speed 130.55 samples/sec Loss 10.6252 Epoch: 8 Global Step: 86735 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:53:12,716-Speed 129.67 samples/sec Loss 10.6971 Epoch: 8 Global Step: 86740 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:53:32,274-Speed 130.90 samples/sec Loss 10.5343 Epoch: 8 Global Step: 86745 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:53:51,981-Speed 129.91 samples/sec Loss 10.6359 Epoch: 8 Global Step: 86750 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:54:11,444-Speed 131.54 samples/sec Loss 10.4456 Epoch: 8 Global Step: 86755 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:54:31,562-Speed 127.25 samples/sec Loss 10.6085 Epoch: 8 Global Step: 86760 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:54:51,571-Speed 127.95 samples/sec Loss 10.9802 Epoch: 8 Global Step: 86765 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:55:11,038-Speed 131.51 samples/sec Loss 10.6988 Epoch: 8 Global Step: 86770 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:55:30,705-Speed 130.17 samples/sec Loss 10.5764 Epoch: 8 Global Step: 86775 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:55:50,382-Speed 130.11 samples/sec Loss 10.5603 Epoch: 8 Global Step: 86780 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:56:10,238-Speed 128.94 samples/sec Loss 10.7062 Epoch: 8 Global Step: 86785 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:56:29,829-Speed 130.68 samples/sec Loss 10.6146 Epoch: 8 Global Step: 86790 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:56:49,475-Speed 130.31 samples/sec Loss 10.8315 Epoch: 8 Global Step: 86795 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:57:09,154-Speed 130.10 samples/sec Loss 10.9409 Epoch: 8 Global Step: 86800 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:57:29,215-Speed 127.62 samples/sec Loss 10.6170 Epoch: 8 Global Step: 86805 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:57:49,348-Speed 127.16 samples/sec Loss 10.7429 Epoch: 8 Global Step: 86810 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:58:08,796-Speed 131.64 samples/sec Loss 10.9131 Epoch: 8 Global Step: 86815 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:58:28,715-Speed 128.53 samples/sec Loss 10.5744 Epoch: 8 Global Step: 86820 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:58:48,269-Speed 130.93 samples/sec Loss 10.7275 Epoch: 8 Global Step: 86825 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:59:08,054-Speed 129.40 samples/sec Loss 10.7683 Epoch: 8 Global Step: 86830 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:59:27,626-Speed 130.80 samples/sec Loss 10.7541 Epoch: 8 Global Step: 86835 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 01:59:47,376-Speed 129.63 samples/sec Loss 11.0129 Epoch: 8 Global Step: 86840 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:00:06,991-Speed 130.52 samples/sec Loss 11.0103 Epoch: 8 Global Step: 86845 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:00:26,969-Speed 128.14 samples/sec Loss 10.7165 Epoch: 8 Global Step: 86850 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:00:46,872-Speed 128.64 samples/sec Loss 10.7567 Epoch: 8 Global Step: 86855 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:01:06,421-Speed 130.95 samples/sec Loss 10.7170 Epoch: 8 Global Step: 86860 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:01:26,341-Speed 128.52 samples/sec Loss 10.7389 Epoch: 8 Global Step: 86865 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:01:46,016-Speed 130.12 samples/sec Loss 10.7213 Epoch: 8 Global Step: 86870 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:02:05,975-Speed 128.27 samples/sec Loss 11.0854 Epoch: 8 Global Step: 86875 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:02:25,492-Speed 131.17 samples/sec Loss 10.8501 Epoch: 8 Global Step: 86880 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:02:45,268-Speed 129.46 samples/sec Loss 10.7547 Epoch: 8 Global Step: 86885 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:03:04,868-Speed 130.62 samples/sec Loss 10.8159 Epoch: 8 Global Step: 86890 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:03:24,943-Speed 127.53 samples/sec Loss 10.6327 Epoch: 8 Global Step: 86895 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:03:44,556-Speed 130.53 samples/sec Loss 10.8114 Epoch: 8 Global Step: 86900 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:04:04,092-Speed 131.04 samples/sec Loss 10.7996 Epoch: 8 Global Step: 86905 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:04:23,942-Speed 128.98 samples/sec Loss 10.6958 Epoch: 8 Global Step: 86910 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:04:43,497-Speed 130.92 samples/sec Loss 10.9527 Epoch: 8 Global Step: 86915 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:05:03,423-Speed 128.48 samples/sec Loss 10.6078 Epoch: 8 Global Step: 86920 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:05:22,999-Speed 130.78 samples/sec Loss 10.6845 Epoch: 8 Global Step: 86925 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:05:42,865-Speed 128.87 samples/sec Loss 10.7532 Epoch: 8 Global Step: 86930 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:06:02,467-Speed 130.61 samples/sec Loss 10.7009 Epoch: 8 Global Step: 86935 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:06:22,601-Speed 127.15 samples/sec Loss 10.5155 Epoch: 8 Global Step: 86940 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:06:42,350-Speed 129.64 samples/sec Loss 10.5419 Epoch: 8 Global Step: 86945 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:07:01,986-Speed 130.38 samples/sec Loss 10.5823 Epoch: 8 Global Step: 86950 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:07:21,733-Speed 129.65 samples/sec Loss 10.6922 Epoch: 8 Global Step: 86955 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:07:41,379-Speed 130.31 samples/sec Loss 10.6649 Epoch: 8 Global Step: 86960 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:08:01,084-Speed 129.93 samples/sec Loss 10.6422 Epoch: 8 Global Step: 86965 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:08:20,554-Speed 131.49 samples/sec Loss 10.9543 Epoch: 8 Global Step: 86970 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:08:40,352-Speed 129.32 samples/sec Loss 10.8578 Epoch: 8 Global Step: 86975 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:09:00,040-Speed 130.03 samples/sec Loss 10.7333 Epoch: 8 Global Step: 86980 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:09:20,200-Speed 126.99 samples/sec Loss 10.6338 Epoch: 8 Global Step: 86985 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:09:39,963-Speed 129.55 samples/sec Loss 10.3164 Epoch: 8 Global Step: 86990 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:09:59,554-Speed 130.68 samples/sec Loss 10.2100 Epoch: 8 Global Step: 86995 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:10:19,329-Speed 129.46 samples/sec Loss 10.6678 Epoch: 8 Global Step: 87000 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:10:38,963-Speed 130.40 samples/sec Loss 10.9194 Epoch: 8 Global Step: 87005 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:10:58,682-Speed 129.83 samples/sec Loss 10.7020 Epoch: 8 Global Step: 87010 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:11:18,305-Speed 130.47 samples/sec Loss 10.7315 Epoch: 8 Global Step: 87015 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:11:38,166-Speed 128.90 samples/sec Loss 10.8082 Epoch: 8 Global Step: 87020 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:11:57,859-Speed 130.00 samples/sec Loss 10.8008 Epoch: 8 Global Step: 87025 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:12:18,045-Speed 126.83 samples/sec Loss 10.7953 Epoch: 8 Global Step: 87030 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:12:37,780-Speed 129.73 samples/sec Loss 10.3945 Epoch: 8 Global Step: 87035 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:12:57,357-Speed 130.77 samples/sec Loss 10.5801 Epoch: 8 Global Step: 87040 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:13:17,165-Speed 129.24 samples/sec Loss 10.7820 Epoch: 8 Global Step: 87045 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:13:36,797-Speed 130.41 samples/sec Loss 10.7969 Epoch: 8 Global Step: 87050 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:13:56,551-Speed 129.60 samples/sec Loss 10.8353 Epoch: 8 Global Step: 87055 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:14:16,043-Speed 131.34 samples/sec Loss 10.7435 Epoch: 8 Global Step: 87060 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:14:35,861-Speed 129.18 samples/sec Loss 10.7838 Epoch: 8 Global Step: 87065 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:14:55,486-Speed 130.46 samples/sec Loss 10.9561 Epoch: 8 Global Step: 87070 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:15:15,559-Speed 127.55 samples/sec Loss 10.5605 Epoch: 8 Global Step: 87075 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:15:35,396-Speed 129.06 samples/sec Loss 10.9100 Epoch: 8 Global Step: 87080 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:15:54,970-Speed 130.79 samples/sec Loss 10.3186 Epoch: 8 Global Step: 87085 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:16:14,850-Speed 128.78 samples/sec Loss 10.6256 Epoch: 8 Global Step: 87090 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:16:34,454-Speed 130.60 samples/sec Loss 10.7819 Epoch: 8 Global Step: 87095 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:16:54,099-Speed 130.32 samples/sec Loss 10.4641 Epoch: 8 Global Step: 87100 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:17:13,689-Speed 130.69 samples/sec Loss 10.7404 Epoch: 8 Global Step: 87105 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:17:33,441-Speed 129.61 samples/sec Loss 10.8102 Epoch: 8 Global Step: 87110 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:17:53,243-Speed 129.28 samples/sec Loss 10.7875 Epoch: 8 Global Step: 87115 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:18:13,152-Speed 128.59 samples/sec Loss 10.8633 Epoch: 8 Global Step: 87120 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:18:33,071-Speed 128.53 samples/sec Loss 10.7350 Epoch: 8 Global Step: 87125 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:18:52,631-Speed 130.89 samples/sec Loss 10.6615 Epoch: 8 Global Step: 87130 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:19:12,598-Speed 128.22 samples/sec Loss 10.6154 Epoch: 8 Global Step: 87135 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:19:32,049-Speed 131.62 samples/sec Loss 10.4414 Epoch: 8 Global Step: 87140 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:19:51,731-Speed 130.08 samples/sec Loss 10.7709 Epoch: 8 Global Step: 87145 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:20:11,348-Speed 130.50 samples/sec Loss 10.5159 Epoch: 8 Global Step: 87150 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:20:31,088-Speed 129.70 samples/sec Loss 10.7199 Epoch: 8 Global Step: 87155 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:20:50,726-Speed 130.37 samples/sec Loss 10.4494 Epoch: 8 Global Step: 87160 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:21:10,584-Speed 128.92 samples/sec Loss 10.7634 Epoch: 8 Global Step: 87165 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:21:30,378-Speed 129.34 samples/sec Loss 10.4965 Epoch: 8 Global Step: 87170 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:21:49,952-Speed 130.79 samples/sec Loss 10.8164 Epoch: 8 Global Step: 87175 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:22:09,948-Speed 128.04 samples/sec Loss 11.0538 Epoch: 8 Global Step: 87180 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:22:29,391-Speed 131.67 samples/sec Loss 11.1114 Epoch: 8 Global Step: 87185 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:22:49,583-Speed 126.79 samples/sec Loss 10.6091 Epoch: 8 Global Step: 87190 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:23:09,144-Speed 130.88 samples/sec Loss 10.5646 Epoch: 8 Global Step: 87195 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:23:28,920-Speed 129.45 samples/sec Loss 10.8099 Epoch: 8 Global Step: 87200 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:23:48,542-Speed 130.47 samples/sec Loss 10.6320 Epoch: 8 Global Step: 87205 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:24:08,525-Speed 128.12 samples/sec Loss 10.4763 Epoch: 8 Global Step: 87210 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:24:28,313-Speed 129.37 samples/sec Loss 10.6868 Epoch: 8 Global Step: 87215 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:24:48,039-Speed 129.78 samples/sec Loss 10.8784 Epoch: 8 Global Step: 87220 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:25:07,914-Speed 128.82 samples/sec Loss 10.7896 Epoch: 8 Global Step: 87225 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:25:27,479-Speed 130.85 samples/sec Loss 10.6955 Epoch: 8 Global Step: 87230 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:25:47,307-Speed 129.12 samples/sec Loss 10.6470 Epoch: 8 Global Step: 87235 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:26:06,870-Speed 130.87 samples/sec Loss 10.7923 Epoch: 8 Global Step: 87240 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:26:26,853-Speed 128.11 samples/sec Loss 10.7951 Epoch: 8 Global Step: 87245 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:26:46,309-Speed 131.58 samples/sec Loss 11.0083 Epoch: 8 Global Step: 87250 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:27:06,402-Speed 127.42 samples/sec Loss 10.5117 Epoch: 8 Global Step: 87255 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:27:26,164-Speed 129.54 samples/sec Loss 10.8621 Epoch: 8 Global Step: 87260 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:27:45,784-Speed 130.49 samples/sec Loss 10.3761 Epoch: 8 Global Step: 87265 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:28:05,397-Speed 130.53 samples/sec Loss 11.2179 Epoch: 8 Global Step: 87270 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:28:24,945-Speed 130.97 samples/sec Loss 10.7245 Epoch: 8 Global Step: 87275 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:28:44,820-Speed 128.81 samples/sec Loss 10.8662 Epoch: 8 Global Step: 87280 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:29:04,366-Speed 130.98 samples/sec Loss 10.9163 Epoch: 8 Global Step: 87285 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:29:24,334-Speed 128.22 samples/sec Loss 11.1230 Epoch: 8 Global Step: 87290 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:29:43,824-Speed 131.35 samples/sec Loss 10.6416 Epoch: 8 Global Step: 87295 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:30:03,926-Speed 127.35 samples/sec Loss 10.3468 Epoch: 8 Global Step: 87300 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:30:23,825-Speed 128.66 samples/sec Loss 10.6974 Epoch: 8 Global Step: 87305 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:30:43,413-Speed 130.69 samples/sec Loss 10.9723 Epoch: 8 Global Step: 87310 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:31:03,034-Speed 130.48 samples/sec Loss 10.9096 Epoch: 8 Global Step: 87315 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:31:22,688-Speed 130.26 samples/sec Loss 10.4057 Epoch: 8 Global Step: 87320 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:31:42,436-Speed 129.64 samples/sec Loss 10.6384 Epoch: 8 Global Step: 87325 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:32:02,102-Speed 130.18 samples/sec Loss 10.5170 Epoch: 8 Global Step: 87330 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:32:21,916-Speed 129.21 samples/sec Loss 10.9025 Epoch: 8 Global Step: 87335 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:32:41,391-Speed 131.46 samples/sec Loss 10.8868 Epoch: 8 Global Step: 87340 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:33:01,494-Speed 127.35 samples/sec Loss 10.6751 Epoch: 8 Global Step: 87345 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:33:21,626-Speed 127.17 samples/sec Loss 10.7642 Epoch: 8 Global Step: 87350 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:33:41,070-Speed 131.66 samples/sec Loss 10.6152 Epoch: 8 Global Step: 87355 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:34:00,845-Speed 129.47 samples/sec Loss 10.8918 Epoch: 8 Global Step: 87360 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:34:20,431-Speed 130.71 samples/sec Loss 10.8821 Epoch: 8 Global Step: 87365 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:34:40,173-Speed 129.68 samples/sec Loss 10.9507 Epoch: 8 Global Step: 87370 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:34:59,786-Speed 130.53 samples/sec Loss 10.6129 Epoch: 8 Global Step: 87375 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:35:19,408-Speed 130.47 samples/sec Loss 10.7001 Epoch: 8 Global Step: 87380 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:35:38,966-Speed 130.90 samples/sec Loss 10.9064 Epoch: 8 Global Step: 87385 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:35:58,992-Speed 127.84 samples/sec Loss 10.7166 Epoch: 8 Global Step: 87390 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:36:19,066-Speed 127.53 samples/sec Loss 10.8827 Epoch: 8 Global Step: 87395 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:36:40,046-Speed 122.03 samples/sec Loss 10.8287 Epoch: 8 Global Step: 87400 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:36:59,969-Speed 128.50 samples/sec Loss 10.8009 Epoch: 8 Global Step: 87405 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:37:19,504-Speed 131.05 samples/sec Loss 10.8299 Epoch: 8 Global Step: 87410 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:37:39,405-Speed 128.64 samples/sec Loss 10.8126 Epoch: 8 Global Step: 87415 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:37:58,947-Speed 131.01 samples/sec Loss 10.9884 Epoch: 8 Global Step: 87420 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:38:18,609-Speed 130.21 samples/sec Loss 10.6323 Epoch: 8 Global Step: 87425 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:38:38,259-Speed 130.29 samples/sec Loss 10.8334 Epoch: 8 Global Step: 87430 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:38:58,245-Speed 128.09 samples/sec Loss 10.6912 Epoch: 8 Global Step: 87435 Fp16 Grad Scale: 8192 Required: 192 hours Training: 2021-06-11 02:39:18,227-Speed 128.12 samples/sec Loss 10.8457 Epoch: 8 Global Step: 87440 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:39:37,764-Speed 131.04 samples/sec Loss 10.8269 Epoch: 8 Global Step: 87445 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:39:57,701-Speed 128.41 samples/sec Loss 10.6230 Epoch: 8 Global Step: 87450 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:40:17,274-Speed 130.80 samples/sec Loss 10.9486 Epoch: 8 Global Step: 87455 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:40:37,274-Speed 128.01 samples/sec Loss 10.3745 Epoch: 8 Global Step: 87460 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:40:56,701-Speed 131.79 samples/sec Loss 10.8163 Epoch: 8 Global Step: 87465 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:41:16,462-Speed 129.55 samples/sec Loss 10.4589 Epoch: 8 Global Step: 87470 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:41:36,380-Speed 128.54 samples/sec Loss 10.9511 Epoch: 8 Global Step: 87475 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:41:56,313-Speed 128.43 samples/sec Loss 10.9921 Epoch: 8 Global Step: 87480 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:42:15,926-Speed 130.53 samples/sec Loss 10.7344 Epoch: 8 Global Step: 87485 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:42:35,559-Speed 130.40 samples/sec Loss 10.8986 Epoch: 8 Global Step: 87490 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:42:55,228-Speed 130.16 samples/sec Loss 10.8333 Epoch: 8 Global Step: 87495 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:43:14,774-Speed 130.98 samples/sec Loss 10.5824 Epoch: 8 Global Step: 87500 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:43:34,643-Speed 128.85 samples/sec Loss 10.5189 Epoch: 8 Global Step: 87505 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:43:54,192-Speed 130.96 samples/sec Loss 11.1908 Epoch: 8 Global Step: 87510 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:44:14,289-Speed 127.38 samples/sec Loss 11.0439 Epoch: 8 Global Step: 87515 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:44:33,849-Speed 130.89 samples/sec Loss 10.7364 Epoch: 8 Global Step: 87520 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:44:54,028-Speed 126.87 samples/sec Loss 10.8246 Epoch: 8 Global Step: 87525 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:45:13,779-Speed 129.62 samples/sec Loss 10.6829 Epoch: 8 Global Step: 87530 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:45:33,314-Speed 131.06 samples/sec Loss 10.9923 Epoch: 8 Global Step: 87535 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:45:53,189-Speed 128.81 samples/sec Loss 10.8167 Epoch: 8 Global Step: 87540 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:46:12,817-Speed 130.44 samples/sec Loss 10.7616 Epoch: 8 Global Step: 87545 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:46:32,691-Speed 128.82 samples/sec Loss 10.4866 Epoch: 8 Global Step: 87550 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:46:52,256-Speed 130.85 samples/sec Loss 10.8474 Epoch: 8 Global Step: 87555 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:47:12,051-Speed 129.33 samples/sec Loss 10.6826 Epoch: 8 Global Step: 87560 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:47:31,615-Speed 130.86 samples/sec Loss 10.7260 Epoch: 8 Global Step: 87565 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:47:51,720-Speed 127.33 samples/sec Loss 10.9347 Epoch: 8 Global Step: 87570 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:48:11,482-Speed 129.55 samples/sec Loss 10.9486 Epoch: 8 Global Step: 87575 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:48:31,087-Speed 130.58 samples/sec Loss 10.8269 Epoch: 8 Global Step: 87580 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:48:50,835-Speed 129.64 samples/sec Loss 10.6822 Epoch: 8 Global Step: 87585 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:49:10,476-Speed 130.35 samples/sec Loss 10.8115 Epoch: 8 Global Step: 87590 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:49:30,435-Speed 128.27 samples/sec Loss 10.4720 Epoch: 8 Global Step: 87595 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:49:49,978-Speed 131.00 samples/sec Loss 10.4906 Epoch: 8 Global Step: 87600 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:50:09,752-Speed 129.47 samples/sec Loss 10.7563 Epoch: 8 Global Step: 87605 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:50:29,388-Speed 130.38 samples/sec Loss 10.6277 Epoch: 8 Global Step: 87610 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:50:49,487-Speed 127.38 samples/sec Loss 10.3422 Epoch: 8 Global Step: 87615 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:51:09,467-Speed 128.14 samples/sec Loss 10.5943 Epoch: 8 Global Step: 87620 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:51:29,005-Speed 131.03 samples/sec Loss 10.5838 Epoch: 8 Global Step: 87625 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:51:48,880-Speed 128.81 samples/sec Loss 10.3195 Epoch: 8 Global Step: 87630 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:52:08,550-Speed 130.16 samples/sec Loss 10.6861 Epoch: 8 Global Step: 87635 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:52:28,352-Speed 129.29 samples/sec Loss 10.7428 Epoch: 8 Global Step: 87640 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:52:48,006-Speed 130.26 samples/sec Loss 10.4575 Epoch: 8 Global Step: 87645 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:53:07,734-Speed 129.78 samples/sec Loss 10.7191 Epoch: 8 Global Step: 87650 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:53:27,346-Speed 130.54 samples/sec Loss 10.6983 Epoch: 8 Global Step: 87655 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:53:47,408-Speed 127.61 samples/sec Loss 10.9831 Epoch: 8 Global Step: 87660 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:54:07,329-Speed 128.52 samples/sec Loss 10.6905 Epoch: 8 Global Step: 87665 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:54:26,882-Speed 130.93 samples/sec Loss 10.5050 Epoch: 8 Global Step: 87670 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:54:46,795-Speed 128.56 samples/sec Loss 10.6078 Epoch: 8 Global Step: 87675 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:55:06,357-Speed 130.88 samples/sec Loss 11.0667 Epoch: 8 Global Step: 87680 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:55:26,097-Speed 129.69 samples/sec Loss 10.7061 Epoch: 8 Global Step: 87685 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:55:45,734-Speed 130.37 samples/sec Loss 10.5700 Epoch: 8 Global Step: 87690 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:56:05,472-Speed 129.71 samples/sec Loss 10.6543 Epoch: 8 Global Step: 87695 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:56:25,132-Speed 130.22 samples/sec Loss 10.5173 Epoch: 8 Global Step: 87700 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:56:45,047-Speed 128.55 samples/sec Loss 10.4257 Epoch: 8 Global Step: 87705 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:57:05,096-Speed 127.69 samples/sec Loss 11.0581 Epoch: 8 Global Step: 87710 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:57:24,707-Speed 130.54 samples/sec Loss 10.5925 Epoch: 8 Global Step: 87715 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:57:44,533-Speed 129.13 samples/sec Loss 10.6226 Epoch: 8 Global Step: 87720 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:58:04,075-Speed 131.01 samples/sec Loss 10.6553 Epoch: 8 Global Step: 87725 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:58:23,800-Speed 129.79 samples/sec Loss 10.8567 Epoch: 8 Global Step: 87730 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:58:43,365-Speed 130.86 samples/sec Loss 11.1087 Epoch: 8 Global Step: 87735 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:59:03,162-Speed 129.32 samples/sec Loss 10.3954 Epoch: 8 Global Step: 87740 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:59:22,791-Speed 130.42 samples/sec Loss 10.6120 Epoch: 8 Global Step: 87745 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 02:59:42,896-Speed 127.34 samples/sec Loss 10.8939 Epoch: 8 Global Step: 87750 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:00:02,635-Speed 129.70 samples/sec Loss 10.7812 Epoch: 8 Global Step: 87755 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:00:22,258-Speed 130.46 samples/sec Loss 10.6951 Epoch: 8 Global Step: 87760 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:00:42,137-Speed 128.79 samples/sec Loss 10.8635 Epoch: 8 Global Step: 87765 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:01:01,700-Speed 130.86 samples/sec Loss 10.7704 Epoch: 8 Global Step: 87770 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:01:21,711-Speed 127.94 samples/sec Loss 10.9266 Epoch: 8 Global Step: 87775 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:01:41,281-Speed 130.82 samples/sec Loss 10.6665 Epoch: 8 Global Step: 87780 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:02:01,147-Speed 128.87 samples/sec Loss 10.7918 Epoch: 8 Global Step: 87785 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:02:20,715-Speed 130.83 samples/sec Loss 10.8771 Epoch: 8 Global Step: 87790 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:02:40,729-Speed 127.91 samples/sec Loss 10.7887 Epoch: 8 Global Step: 87795 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:03:00,411-Speed 130.08 samples/sec Loss 10.7135 Epoch: 8 Global Step: 87800 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:03:20,030-Speed 130.49 samples/sec Loss 11.0336 Epoch: 8 Global Step: 87805 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:03:39,914-Speed 128.75 samples/sec Loss 10.7270 Epoch: 8 Global Step: 87810 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:03:59,388-Speed 131.47 samples/sec Loss 10.4930 Epoch: 8 Global Step: 87815 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:04:19,316-Speed 128.47 samples/sec Loss 10.3795 Epoch: 8 Global Step: 87820 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:04:38,895-Speed 130.76 samples/sec Loss 10.8867 Epoch: 8 Global Step: 87825 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:04:58,780-Speed 128.75 samples/sec Loss 10.7728 Epoch: 8 Global Step: 87830 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:05:18,404-Speed 130.45 samples/sec Loss 10.4939 Epoch: 8 Global Step: 87835 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:05:38,407-Speed 127.99 samples/sec Loss 10.8432 Epoch: 8 Global Step: 87840 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:05:58,188-Speed 129.42 samples/sec Loss 10.4884 Epoch: 8 Global Step: 87845 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:06:17,806-Speed 130.50 samples/sec Loss 10.7333 Epoch: 8 Global Step: 87850 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:06:37,576-Speed 129.49 samples/sec Loss 10.8441 Epoch: 8 Global Step: 87855 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:06:57,075-Speed 131.30 samples/sec Loss 10.7984 Epoch: 8 Global Step: 87860 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:07:16,987-Speed 128.58 samples/sec Loss 10.5657 Epoch: 8 Global Step: 87865 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:07:36,620-Speed 130.40 samples/sec Loss 10.7917 Epoch: 8 Global Step: 87870 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:07:56,522-Speed 128.64 samples/sec Loss 11.1967 Epoch: 8 Global Step: 87875 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:08:16,147-Speed 130.45 samples/sec Loss 10.5826 Epoch: 8 Global Step: 87880 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:08:36,165-Speed 127.89 samples/sec Loss 10.8625 Epoch: 8 Global Step: 87885 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:08:56,113-Speed 128.34 samples/sec Loss 10.6869 Epoch: 8 Global Step: 87890 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:09:15,713-Speed 130.61 samples/sec Loss 10.7732 Epoch: 8 Global Step: 87895 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:09:35,350-Speed 130.37 samples/sec Loss 10.3989 Epoch: 8 Global Step: 87900 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:09:54,980-Speed 130.42 samples/sec Loss 10.6963 Epoch: 8 Global Step: 87905 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:10:14,665-Speed 130.05 samples/sec Loss 10.9556 Epoch: 8 Global Step: 87910 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:10:34,301-Speed 130.38 samples/sec Loss 10.9818 Epoch: 8 Global Step: 87915 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:10:54,031-Speed 129.76 samples/sec Loss 10.5519 Epoch: 8 Global Step: 87920 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:11:13,503-Speed 131.48 samples/sec Loss 10.4622 Epoch: 8 Global Step: 87925 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:11:33,582-Speed 127.50 samples/sec Loss 10.7174 Epoch: 8 Global Step: 87930 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:11:53,617-Speed 127.79 samples/sec Loss 10.9656 Epoch: 8 Global Step: 87935 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:12:13,167-Speed 130.95 samples/sec Loss 10.5287 Epoch: 8 Global Step: 87940 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:12:32,873-Speed 129.91 samples/sec Loss 10.6744 Epoch: 8 Global Step: 87945 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:12:52,702-Speed 129.11 samples/sec Loss 10.6150 Epoch: 8 Global Step: 87950 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:13:12,508-Speed 129.26 samples/sec Loss 10.8303 Epoch: 8 Global Step: 87955 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:13:32,193-Speed 130.05 samples/sec Loss 10.8412 Epoch: 8 Global Step: 87960 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:13:51,822-Speed 130.43 samples/sec Loss 10.6079 Epoch: 8 Global Step: 87965 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:14:11,375-Speed 130.94 samples/sec Loss 10.7440 Epoch: 8 Global Step: 87970 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:14:31,467-Speed 127.42 samples/sec Loss 10.8954 Epoch: 8 Global Step: 87975 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:14:51,563-Speed 127.39 samples/sec Loss 10.7786 Epoch: 8 Global Step: 87980 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:15:10,987-Speed 131.80 samples/sec Loss 10.8401 Epoch: 8 Global Step: 87985 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:15:31,069-Speed 127.48 samples/sec Loss 10.8449 Epoch: 8 Global Step: 87990 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:15:50,582-Speed 131.20 samples/sec Loss 10.7572 Epoch: 8 Global Step: 87995 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:16:10,435-Speed 128.95 samples/sec Loss 10.7350 Epoch: 8 Global Step: 88000 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:16:30,257-Speed 129.16 samples/sec Loss 11.0641 Epoch: 8 Global Step: 88005 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:16:49,898-Speed 130.35 samples/sec Loss 10.7775 Epoch: 8 Global Step: 88010 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:17:09,534-Speed 130.38 samples/sec Loss 10.9748 Epoch: 8 Global Step: 88015 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:17:29,639-Speed 127.33 samples/sec Loss 10.6796 Epoch: 8 Global Step: 88020 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:17:49,346-Speed 129.91 samples/sec Loss 11.0063 Epoch: 8 Global Step: 88025 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:18:08,959-Speed 130.53 samples/sec Loss 10.7739 Epoch: 8 Global Step: 88030 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:18:28,867-Speed 128.60 samples/sec Loss 10.5376 Epoch: 8 Global Step: 88035 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:18:48,480-Speed 130.53 samples/sec Loss 10.5451 Epoch: 8 Global Step: 88040 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:19:08,379-Speed 128.66 samples/sec Loss 10.8785 Epoch: 8 Global Step: 88045 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:19:27,826-Speed 131.64 samples/sec Loss 10.4548 Epoch: 8 Global Step: 88050 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:19:47,605-Speed 129.44 samples/sec Loss 10.4710 Epoch: 8 Global Step: 88055 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:20:07,204-Speed 130.63 samples/sec Loss 10.5571 Epoch: 8 Global Step: 88060 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:20:27,479-Speed 126.27 samples/sec Loss 10.9975 Epoch: 8 Global Step: 88065 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:20:47,120-Speed 130.35 samples/sec Loss 10.6064 Epoch: 8 Global Step: 88070 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:21:06,687-Speed 130.84 samples/sec Loss 11.0067 Epoch: 8 Global Step: 88075 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:21:26,474-Speed 129.39 samples/sec Loss 10.5433 Epoch: 8 Global Step: 88080 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:21:46,006-Speed 131.07 samples/sec Loss 10.4469 Epoch: 8 Global Step: 88085 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:22:05,888-Speed 128.77 samples/sec Loss 10.9799 Epoch: 8 Global Step: 88090 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:22:25,426-Speed 131.03 samples/sec Loss 10.8576 Epoch: 8 Global Step: 88095 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:22:45,341-Speed 128.56 samples/sec Loss 10.6218 Epoch: 8 Global Step: 88100 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:23:04,985-Speed 130.33 samples/sec Loss 10.7477 Epoch: 8 Global Step: 88105 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:23:24,973-Speed 128.08 samples/sec Loss 10.8430 Epoch: 8 Global Step: 88110 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:23:44,742-Speed 129.50 samples/sec Loss 10.9255 Epoch: 8 Global Step: 88115 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:24:04,363-Speed 130.48 samples/sec Loss 10.6384 Epoch: 8 Global Step: 88120 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:24:24,150-Speed 129.38 samples/sec Loss 10.5536 Epoch: 8 Global Step: 88125 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:24:43,830-Speed 130.09 samples/sec Loss 10.7356 Epoch: 8 Global Step: 88130 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:25:03,707-Speed 128.80 samples/sec Loss 10.8364 Epoch: 8 Global Step: 88135 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:25:23,279-Speed 130.81 samples/sec Loss 10.6308 Epoch: 8 Global Step: 88140 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:25:43,210-Speed 128.45 samples/sec Loss 10.5132 Epoch: 8 Global Step: 88145 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:26:02,829-Speed 130.49 samples/sec Loss 10.7921 Epoch: 8 Global Step: 88150 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:26:22,909-Speed 127.50 samples/sec Loss 10.7510 Epoch: 8 Global Step: 88155 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:26:42,667-Speed 129.57 samples/sec Loss 10.6527 Epoch: 8 Global Step: 88160 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:27:02,290-Speed 130.47 samples/sec Loss 10.7128 Epoch: 8 Global Step: 88165 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:27:21,999-Speed 129.89 samples/sec Loss 10.6446 Epoch: 8 Global Step: 88170 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:27:41,562-Speed 130.87 samples/sec Loss 11.0377 Epoch: 8 Global Step: 88175 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:28:01,416-Speed 128.95 samples/sec Loss 10.6869 Epoch: 8 Global Step: 88180 Fp16 Grad Scale: 8192 Required: 191 hours Training: 2021-06-11 03:28:20,980-Speed 130.86 samples/sec Loss 10.7226 Epoch: 8 Global Step: 88185 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:28:40,976-Speed 128.03 samples/sec Loss 10.9045 Epoch: 8 Global Step: 88190 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:29:00,570-Speed 130.66 samples/sec Loss 10.5959 Epoch: 8 Global Step: 88195 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:29:20,686-Speed 127.26 samples/sec Loss 10.7633 Epoch: 8 Global Step: 88200 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:29:40,845-Speed 127.00 samples/sec Loss 10.6426 Epoch: 8 Global Step: 88205 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:30:00,466-Speed 130.48 samples/sec Loss 10.7166 Epoch: 8 Global Step: 88210 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:30:20,089-Speed 130.46 samples/sec Loss 10.5769 Epoch: 8 Global Step: 88215 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:30:39,798-Speed 129.90 samples/sec Loss 10.8884 Epoch: 8 Global Step: 88220 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:30:59,649-Speed 128.97 samples/sec Loss 10.9310 Epoch: 8 Global Step: 88225 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:31:19,204-Speed 130.92 samples/sec Loss 11.2760 Epoch: 8 Global Step: 88230 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:31:38,997-Speed 129.35 samples/sec Loss 10.9397 Epoch: 8 Global Step: 88235 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:31:58,562-Speed 130.85 samples/sec Loss 10.8443 Epoch: 8 Global Step: 88240 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:32:18,666-Speed 127.34 samples/sec Loss 11.0110 Epoch: 8 Global Step: 88245 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:32:38,703-Speed 127.77 samples/sec Loss 10.4979 Epoch: 8 Global Step: 88250 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:32:58,233-Speed 131.09 samples/sec Loss 10.7043 Epoch: 8 Global Step: 88255 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:33:18,155-Speed 128.50 samples/sec Loss 10.5873 Epoch: 8 Global Step: 88260 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:33:37,631-Speed 131.45 samples/sec Loss 10.7186 Epoch: 8 Global Step: 88265 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:33:57,398-Speed 129.52 samples/sec Loss 11.0953 Epoch: 8 Global Step: 88270 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:34:17,003-Speed 130.59 samples/sec Loss 10.7281 Epoch: 8 Global Step: 88275 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:34:36,765-Speed 129.54 samples/sec Loss 10.4265 Epoch: 8 Global Step: 88280 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:34:56,332-Speed 130.84 samples/sec Loss 10.9330 Epoch: 8 Global Step: 88285 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:35:16,207-Speed 128.81 samples/sec Loss 10.9803 Epoch: 8 Global Step: 88290 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:35:36,317-Speed 127.31 samples/sec Loss 10.7333 Epoch: 8 Global Step: 88295 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:35:55,867-Speed 130.95 samples/sec Loss 10.6825 Epoch: 8 Global Step: 88300 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:36:15,801-Speed 128.43 samples/sec Loss 11.1543 Epoch: 8 Global Step: 88305 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:36:35,278-Speed 131.44 samples/sec Loss 10.6662 Epoch: 8 Global Step: 88310 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:36:54,980-Speed 129.94 samples/sec Loss 10.8326 Epoch: 8 Global Step: 88315 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:37:14,593-Speed 130.54 samples/sec Loss 10.8533 Epoch: 8 Global Step: 88320 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:37:34,335-Speed 129.68 samples/sec Loss 10.7769 Epoch: 8 Global Step: 88325 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:37:53,958-Speed 130.47 samples/sec Loss 10.8035 Epoch: 8 Global Step: 88330 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:38:14,057-Speed 127.38 samples/sec Loss 10.7107 Epoch: 8 Global Step: 88335 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:38:34,086-Speed 127.82 samples/sec Loss 10.5512 Epoch: 8 Global Step: 88340 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:38:53,621-Speed 131.05 samples/sec Loss 10.8296 Epoch: 8 Global Step: 88345 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:39:13,453-Speed 129.09 samples/sec Loss 10.5729 Epoch: 8 Global Step: 88350 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:39:32,993-Speed 131.02 samples/sec Loss 10.8310 Epoch: 8 Global Step: 88355 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:39:52,873-Speed 128.78 samples/sec Loss 10.8368 Epoch: 8 Global Step: 88360 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:40:12,462-Speed 130.70 samples/sec Loss 10.4976 Epoch: 8 Global Step: 88365 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:40:32,175-Speed 129.87 samples/sec Loss 10.9072 Epoch: 8 Global Step: 88370 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:40:51,783-Speed 130.56 samples/sec Loss 10.6074 Epoch: 8 Global Step: 88375 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:41:11,749-Speed 128.22 samples/sec Loss 10.4196 Epoch: 8 Global Step: 88380 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:41:31,758-Speed 127.95 samples/sec Loss 10.5699 Epoch: 8 Global Step: 88385 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:41:51,303-Speed 130.98 samples/sec Loss 10.8728 Epoch: 8 Global Step: 88390 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:42:11,123-Speed 129.17 samples/sec Loss 10.8864 Epoch: 8 Global Step: 88395 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:42:30,690-Speed 130.84 samples/sec Loss 10.5750 Epoch: 8 Global Step: 88400 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:42:50,658-Speed 128.21 samples/sec Loss 10.7266 Epoch: 8 Global Step: 88405 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:43:10,191-Speed 131.07 samples/sec Loss 10.5099 Epoch: 8 Global Step: 88410 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:43:30,032-Speed 129.03 samples/sec Loss 10.7256 Epoch: 8 Global Step: 88415 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:43:49,571-Speed 131.02 samples/sec Loss 10.4513 Epoch: 8 Global Step: 88420 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:44:09,676-Speed 127.34 samples/sec Loss 10.6520 Epoch: 8 Global Step: 88425 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:44:29,391-Speed 129.86 samples/sec Loss 10.7890 Epoch: 8 Global Step: 88430 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:44:48,947-Speed 130.91 samples/sec Loss 10.6996 Epoch: 8 Global Step: 88435 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:45:08,770-Speed 129.15 samples/sec Loss 10.6186 Epoch: 8 Global Step: 88440 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:45:28,395-Speed 130.45 samples/sec Loss 10.3217 Epoch: 8 Global Step: 88445 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:45:48,675-Speed 126.24 samples/sec Loss 10.7782 Epoch: 8 Global Step: 88450 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:46:08,211-Speed 131.05 samples/sec Loss 10.7162 Epoch: 8 Global Step: 88455 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:46:28,010-Speed 129.31 samples/sec Loss 10.6837 Epoch: 8 Global Step: 88460 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:46:47,568-Speed 130.90 samples/sec Loss 10.7258 Epoch: 8 Global Step: 88465 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:47:07,679-Speed 127.30 samples/sec Loss 10.8143 Epoch: 8 Global Step: 88470 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:47:27,395-Speed 129.85 samples/sec Loss 10.7383 Epoch: 8 Global Step: 88475 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:47:47,018-Speed 130.46 samples/sec Loss 10.9942 Epoch: 8 Global Step: 88480 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:48:06,770-Speed 129.61 samples/sec Loss 10.6894 Epoch: 8 Global Step: 88485 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:48:26,369-Speed 130.63 samples/sec Loss 10.7068 Epoch: 8 Global Step: 88490 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:48:46,316-Speed 128.35 samples/sec Loss 10.8832 Epoch: 8 Global Step: 88495 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:49:05,866-Speed 130.96 samples/sec Loss 10.9545 Epoch: 8 Global Step: 88500 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:49:25,671-Speed 129.27 samples/sec Loss 10.4595 Epoch: 8 Global Step: 88505 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:49:45,242-Speed 130.81 samples/sec Loss 10.8191 Epoch: 8 Global Step: 88510 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:50:05,298-Speed 127.65 samples/sec Loss 10.5318 Epoch: 8 Global Step: 88515 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:50:25,200-Speed 128.64 samples/sec Loss 10.7317 Epoch: 8 Global Step: 88520 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:50:44,769-Speed 130.82 samples/sec Loss 10.7409 Epoch: 8 Global Step: 88525 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:51:04,534-Speed 129.53 samples/sec Loss 10.8690 Epoch: 8 Global Step: 88530 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:51:24,195-Speed 130.21 samples/sec Loss 10.7306 Epoch: 8 Global Step: 88535 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:51:43,916-Speed 129.82 samples/sec Loss 10.8315 Epoch: 8 Global Step: 88540 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:52:03,532-Speed 130.51 samples/sec Loss 10.4285 Epoch: 8 Global Step: 88545 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:52:23,160-Speed 130.43 samples/sec Loss 10.5035 Epoch: 8 Global Step: 88550 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:52:42,765-Speed 130.59 samples/sec Loss 10.6824 Epoch: 8 Global Step: 88555 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:53:02,900-Speed 127.14 samples/sec Loss 10.3808 Epoch: 8 Global Step: 88560 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:53:22,814-Speed 128.56 samples/sec Loss 10.7814 Epoch: 8 Global Step: 88565 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:53:42,385-Speed 130.81 samples/sec Loss 10.5590 Epoch: 8 Global Step: 88570 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:54:02,218-Speed 129.08 samples/sec Loss 10.4199 Epoch: 8 Global Step: 88575 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:54:21,834-Speed 130.51 samples/sec Loss 11.0476 Epoch: 8 Global Step: 88580 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:54:41,591-Speed 129.59 samples/sec Loss 10.7788 Epoch: 8 Global Step: 88585 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:55:01,442-Speed 128.97 samples/sec Loss 11.0109 Epoch: 8 Global Step: 88590 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:55:21,230-Speed 129.38 samples/sec Loss 10.9083 Epoch: 8 Global Step: 88595 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:55:40,834-Speed 130.59 samples/sec Loss 10.7466 Epoch: 8 Global Step: 88600 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:56:00,755-Speed 128.52 samples/sec Loss 10.3382 Epoch: 8 Global Step: 88605 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:56:20,758-Speed 127.99 samples/sec Loss 10.5292 Epoch: 8 Global Step: 88610 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:56:40,285-Speed 131.10 samples/sec Loss 10.7004 Epoch: 8 Global Step: 88615 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:57:00,374-Speed 127.44 samples/sec Loss 10.6599 Epoch: 8 Global Step: 88620 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:57:19,911-Speed 131.03 samples/sec Loss 10.4709 Epoch: 8 Global Step: 88625 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:57:39,803-Speed 128.70 samples/sec Loss 10.5392 Epoch: 8 Global Step: 88630 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:57:59,363-Speed 130.89 samples/sec Loss 10.5897 Epoch: 8 Global Step: 88635 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:58:19,090-Speed 129.78 samples/sec Loss 10.4054 Epoch: 8 Global Step: 88640 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:58:38,754-Speed 130.19 samples/sec Loss 10.8245 Epoch: 8 Global Step: 88645 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:58:58,846-Speed 127.42 samples/sec Loss 10.6397 Epoch: 8 Global Step: 88650 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:59:18,601-Speed 129.59 samples/sec Loss 10.7932 Epoch: 8 Global Step: 88655 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:59:38,157-Speed 130.92 samples/sec Loss 10.6914 Epoch: 8 Global Step: 88660 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 03:59:57,998-Speed 129.03 samples/sec Loss 10.4463 Epoch: 8 Global Step: 88665 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:00:17,464-Speed 131.52 samples/sec Loss 10.7750 Epoch: 8 Global Step: 88670 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:00:37,379-Speed 128.55 samples/sec Loss 10.7420 Epoch: 8 Global Step: 88675 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:00:56,983-Speed 130.59 samples/sec Loss 10.7281 Epoch: 8 Global Step: 88680 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:01:16,956-Speed 128.18 samples/sec Loss 10.5724 Epoch: 8 Global Step: 88685 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:01:36,441-Speed 131.39 samples/sec Loss 10.7282 Epoch: 8 Global Step: 88690 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:01:56,456-Speed 127.91 samples/sec Loss 10.9677 Epoch: 8 Global Step: 88695 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:02:16,194-Speed 129.70 samples/sec Loss 10.7299 Epoch: 8 Global Step: 88700 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:02:35,826-Speed 130.40 samples/sec Loss 10.7916 Epoch: 8 Global Step: 88705 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:02:55,609-Speed 129.41 samples/sec Loss 10.3395 Epoch: 8 Global Step: 88710 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:03:15,139-Speed 131.09 samples/sec Loss 10.6430 Epoch: 8 Global Step: 88715 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:03:34,927-Speed 129.38 samples/sec Loss 11.0475 Epoch: 8 Global Step: 88720 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:03:54,494-Speed 130.84 samples/sec Loss 10.8000 Epoch: 8 Global Step: 88725 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:04:14,490-Speed 128.03 samples/sec Loss 11.1255 Epoch: 8 Global Step: 88730 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:04:34,060-Speed 130.82 samples/sec Loss 10.6871 Epoch: 8 Global Step: 88735 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:04:54,126-Speed 127.59 samples/sec Loss 10.7001 Epoch: 8 Global Step: 88740 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:05:13,936-Speed 129.23 samples/sec Loss 10.6916 Epoch: 8 Global Step: 88745 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:05:33,547-Speed 130.54 samples/sec Loss 10.7615 Epoch: 8 Global Step: 88750 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:05:53,237-Speed 130.03 samples/sec Loss 10.5800 Epoch: 8 Global Step: 88755 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:06:12,872-Speed 130.39 samples/sec Loss 10.9964 Epoch: 8 Global Step: 88760 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:06:32,601-Speed 129.76 samples/sec Loss 10.7914 Epoch: 8 Global Step: 88765 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:06:52,147-Speed 130.98 samples/sec Loss 10.5531 Epoch: 8 Global Step: 88770 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:07:11,987-Speed 129.04 samples/sec Loss 10.7562 Epoch: 8 Global Step: 88775 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:07:31,546-Speed 130.89 samples/sec Loss 10.9839 Epoch: 8 Global Step: 88780 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:07:51,660-Speed 127.28 samples/sec Loss 10.6858 Epoch: 8 Global Step: 88785 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:08:11,734-Speed 127.54 samples/sec Loss 10.6176 Epoch: 8 Global Step: 88790 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:08:31,275-Speed 131.02 samples/sec Loss 11.1932 Epoch: 8 Global Step: 88795 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:08:51,193-Speed 128.53 samples/sec Loss 10.5195 Epoch: 8 Global Step: 88800 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:09:10,802-Speed 130.56 samples/sec Loss 11.0433 Epoch: 8 Global Step: 88805 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:09:30,522-Speed 129.82 samples/sec Loss 10.9349 Epoch: 8 Global Step: 88810 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:09:50,154-Speed 130.41 samples/sec Loss 10.8404 Epoch: 8 Global Step: 88815 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:10:10,020-Speed 128.87 samples/sec Loss 10.5791 Epoch: 8 Global Step: 88820 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:10:29,559-Speed 131.03 samples/sec Loss 10.2145 Epoch: 8 Global Step: 88825 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:10:49,633-Speed 127.56 samples/sec Loss 10.7594 Epoch: 8 Global Step: 88830 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:11:09,658-Speed 127.84 samples/sec Loss 10.9489 Epoch: 8 Global Step: 88835 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:11:29,213-Speed 130.92 samples/sec Loss 10.7293 Epoch: 8 Global Step: 88840 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:11:48,929-Speed 129.85 samples/sec Loss 10.5759 Epoch: 8 Global Step: 88845 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:12:08,488-Speed 130.89 samples/sec Loss 10.7481 Epoch: 8 Global Step: 88850 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:12:28,190-Speed 129.94 samples/sec Loss 10.3562 Epoch: 8 Global Step: 88855 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:12:47,814-Speed 130.46 samples/sec Loss 10.3922 Epoch: 8 Global Step: 88860 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:13:07,538-Speed 129.80 samples/sec Loss 10.6436 Epoch: 8 Global Step: 88865 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:13:27,158-Speed 130.48 samples/sec Loss 10.8312 Epoch: 8 Global Step: 88870 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:13:47,227-Speed 127.57 samples/sec Loss 10.8453 Epoch: 8 Global Step: 88875 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:14:07,215-Speed 128.08 samples/sec Loss 10.8220 Epoch: 8 Global Step: 88880 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:14:26,729-Speed 131.19 samples/sec Loss 10.5751 Epoch: 8 Global Step: 88885 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:14:46,660-Speed 128.46 samples/sec Loss 10.8428 Epoch: 8 Global Step: 88890 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:15:06,211-Speed 130.94 samples/sec Loss 10.9192 Epoch: 8 Global Step: 88895 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:15:26,091-Speed 128.78 samples/sec Loss 10.8807 Epoch: 8 Global Step: 88900 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:15:45,655-Speed 130.86 samples/sec Loss 10.6394 Epoch: 8 Global Step: 88905 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:16:05,286-Speed 130.42 samples/sec Loss 10.9814 Epoch: 8 Global Step: 88910 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:16:24,922-Speed 130.38 samples/sec Loss 11.2013 Epoch: 8 Global Step: 88915 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:16:44,840-Speed 128.53 samples/sec Loss 10.8103 Epoch: 8 Global Step: 88920 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:17:04,839-Speed 128.01 samples/sec Loss 10.7250 Epoch: 8 Global Step: 88925 Fp16 Grad Scale: 8192 Required: 190 hours Training: 2021-06-11 04:17:24,353-Speed 131.20 samples/sec Loss 10.4636 Epoch: 8 Global Step: 88930 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:17:44,281-Speed 128.47 samples/sec Loss 10.7247 Epoch: 8 Global Step: 88935 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:18:04,004-Speed 129.81 samples/sec Loss 10.7554 Epoch: 8 Global Step: 88940 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:18:23,976-Speed 128.18 samples/sec Loss 10.6846 Epoch: 8 Global Step: 88945 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:18:43,531-Speed 130.92 samples/sec Loss 10.7717 Epoch: 8 Global Step: 88950 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:19:03,297-Speed 129.52 samples/sec Loss 10.5090 Epoch: 8 Global Step: 88955 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:19:22,869-Speed 130.81 samples/sec Loss 10.7815 Epoch: 8 Global Step: 88960 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:19:42,891-Speed 127.86 samples/sec Loss 10.8384 Epoch: 8 Global Step: 88965 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:20:02,522-Speed 130.41 samples/sec Loss 10.7414 Epoch: 8 Global Step: 88970 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:20:22,065-Speed 131.00 samples/sec Loss 10.6875 Epoch: 8 Global Step: 88975 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:20:41,872-Speed 129.25 samples/sec Loss 10.4549 Epoch: 8 Global Step: 88980 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:21:01,407-Speed 131.06 samples/sec Loss 10.3914 Epoch: 8 Global Step: 88985 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:21:21,347-Speed 128.39 samples/sec Loss 10.5688 Epoch: 8 Global Step: 88990 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:21:40,889-Speed 131.00 samples/sec Loss 10.8551 Epoch: 8 Global Step: 88995 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:22:00,807-Speed 128.54 samples/sec Loss 10.7901 Epoch: 8 Global Step: 89000 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:22:20,386-Speed 130.76 samples/sec Loss 11.0235 Epoch: 8 Global Step: 89005 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:22:40,494-Speed 127.32 samples/sec Loss 10.6325 Epoch: 8 Global Step: 89010 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:23:00,232-Speed 129.70 samples/sec Loss 10.6590 Epoch: 8 Global Step: 89015 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:23:19,857-Speed 130.45 samples/sec Loss 10.8309 Epoch: 8 Global Step: 89020 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:23:39,601-Speed 129.66 samples/sec Loss 10.9532 Epoch: 8 Global Step: 89025 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:23:59,230-Speed 130.43 samples/sec Loss 11.2030 Epoch: 8 Global Step: 89030 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:24:19,007-Speed 129.45 samples/sec Loss 10.9747 Epoch: 8 Global Step: 89035 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:24:38,575-Speed 130.83 samples/sec Loss 10.7561 Epoch: 8 Global Step: 89040 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:24:58,654-Speed 127.50 samples/sec Loss 10.5527 Epoch: 8 Global Step: 89045 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:25:18,213-Speed 130.89 samples/sec Loss 10.8905 Epoch: 8 Global Step: 89050 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:25:38,411-Speed 126.75 samples/sec Loss 10.8520 Epoch: 8 Global Step: 89055 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:25:58,033-Speed 130.47 samples/sec Loss 10.7576 Epoch: 8 Global Step: 89060 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:26:17,653-Speed 130.48 samples/sec Loss 10.3087 Epoch: 8 Global Step: 89065 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:26:37,269-Speed 130.52 samples/sec Loss 10.7228 Epoch: 8 Global Step: 89070 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:26:56,894-Speed 130.45 samples/sec Loss 10.5201 Epoch: 8 Global Step: 89075 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:27:16,765-Speed 128.84 samples/sec Loss 10.8951 Epoch: 8 Global Step: 89080 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:27:36,307-Speed 131.01 samples/sec Loss 10.4536 Epoch: 8 Global Step: 89085 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:27:56,217-Speed 128.58 samples/sec Loss 10.5520 Epoch: 8 Global Step: 89090 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:28:15,765-Speed 130.97 samples/sec Loss 10.8258 Epoch: 8 Global Step: 89095 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:28:35,880-Speed 127.28 samples/sec Loss 10.9230 Epoch: 8 Global Step: 89100 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:28:55,599-Speed 129.83 samples/sec Loss 10.9042 Epoch: 8 Global Step: 89105 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:29:15,236-Speed 130.37 samples/sec Loss 10.7016 Epoch: 8 Global Step: 89110 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:29:35,028-Speed 129.35 samples/sec Loss 10.4659 Epoch: 8 Global Step: 89115 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:29:54,646-Speed 130.50 samples/sec Loss 10.5287 Epoch: 8 Global Step: 89120 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:30:14,287-Speed 130.35 samples/sec Loss 10.9647 Epoch: 8 Global Step: 89125 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:30:33,832-Speed 130.98 samples/sec Loss 10.6715 Epoch: 8 Global Step: 89130 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:30:53,868-Speed 127.78 samples/sec Loss 10.5431 Epoch: 8 Global Step: 89135 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:31:13,499-Speed 130.42 samples/sec Loss 10.7338 Epoch: 8 Global Step: 89140 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:31:33,811-Speed 126.04 samples/sec Loss 10.7651 Epoch: 8 Global Step: 89145 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:31:53,745-Speed 128.43 samples/sec Loss 10.6153 Epoch: 8 Global Step: 89150 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:32:13,317-Speed 130.81 samples/sec Loss 10.4823 Epoch: 8 Global Step: 89155 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:32:33,047-Speed 129.76 samples/sec Loss 10.8039 Epoch: 8 Global Step: 89160 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:32:52,665-Speed 130.50 samples/sec Loss 10.8724 Epoch: 8 Global Step: 89165 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:33:12,283-Speed 130.50 samples/sec Loss 10.6292 Epoch: 8 Global Step: 89170 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:33:31,837-Speed 130.92 samples/sec Loss 10.4165 Epoch: 8 Global Step: 89175 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:33:51,713-Speed 128.81 samples/sec Loss 10.7935 Epoch: 8 Global Step: 89180 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:34:11,287-Speed 130.79 samples/sec Loss 10.6155 Epoch: 8 Global Step: 89185 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:34:31,295-Speed 127.95 samples/sec Loss 10.6835 Epoch: 8 Global Step: 89190 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:34:51,205-Speed 128.59 samples/sec Loss 10.7998 Epoch: 8 Global Step: 89195 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:35:10,833-Speed 130.43 samples/sec Loss 10.6029 Epoch: 8 Global Step: 89200 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:35:30,610-Speed 129.45 samples/sec Loss 10.9218 Epoch: 8 Global Step: 89205 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:35:50,182-Speed 130.80 samples/sec Loss 10.5059 Epoch: 8 Global Step: 89210 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:36:09,852-Speed 130.15 samples/sec Loss 10.4916 Epoch: 8 Global Step: 89215 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:36:29,458-Speed 130.58 samples/sec Loss 11.1141 Epoch: 8 Global Step: 89220 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:36:49,250-Speed 129.35 samples/sec Loss 11.0054 Epoch: 8 Global Step: 89225 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:37:08,800-Speed 130.95 samples/sec Loss 10.8076 Epoch: 8 Global Step: 89230 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:37:28,734-Speed 128.43 samples/sec Loss 10.7566 Epoch: 8 Global Step: 89235 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:37:48,786-Speed 127.67 samples/sec Loss 11.0957 Epoch: 8 Global Step: 89240 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:38:08,434-Speed 130.30 samples/sec Loss 11.0346 Epoch: 8 Global Step: 89245 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:38:28,239-Speed 129.26 samples/sec Loss 10.8011 Epoch: 8 Global Step: 89250 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:38:47,692-Speed 131.61 samples/sec Loss 10.6475 Epoch: 8 Global Step: 89255 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:39:07,430-Speed 129.71 samples/sec Loss 10.5817 Epoch: 8 Global Step: 89260 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:39:27,044-Speed 130.53 samples/sec Loss 10.7891 Epoch: 8 Global Step: 89265 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:39:46,848-Speed 129.27 samples/sec Loss 10.7015 Epoch: 8 Global Step: 89270 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:40:06,468-Speed 130.48 samples/sec Loss 10.6859 Epoch: 8 Global Step: 89275 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:40:26,372-Speed 128.62 samples/sec Loss 10.5736 Epoch: 8 Global Step: 89280 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:40:46,379-Speed 127.96 samples/sec Loss 10.9489 Epoch: 8 Global Step: 89285 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:41:05,897-Speed 131.17 samples/sec Loss 10.7443 Epoch: 8 Global Step: 89290 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:41:25,734-Speed 129.06 samples/sec Loss 10.9970 Epoch: 8 Global Step: 89295 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:41:45,271-Speed 131.04 samples/sec Loss 10.6691 Epoch: 8 Global Step: 89300 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:42:05,092-Speed 129.16 samples/sec Loss 10.4550 Epoch: 8 Global Step: 89305 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:42:24,715-Speed 130.47 samples/sec Loss 11.0148 Epoch: 8 Global Step: 89310 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:42:44,339-Speed 130.46 samples/sec Loss 10.2473 Epoch: 8 Global Step: 89315 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:43:04,067-Speed 129.77 samples/sec Loss 10.9614 Epoch: 8 Global Step: 89320 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:43:24,017-Speed 128.33 samples/sec Loss 10.5482 Epoch: 8 Global Step: 89325 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:43:43,988-Speed 128.20 samples/sec Loss 11.0516 Epoch: 8 Global Step: 89330 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:44:03,457-Speed 131.49 samples/sec Loss 10.7049 Epoch: 8 Global Step: 89335 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:44:23,312-Speed 128.95 samples/sec Loss 10.7050 Epoch: 8 Global Step: 89340 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:44:42,770-Speed 131.57 samples/sec Loss 10.8940 Epoch: 8 Global Step: 89345 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:45:02,741-Speed 128.19 samples/sec Loss 10.4519 Epoch: 8 Global Step: 89350 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:45:22,302-Speed 130.88 samples/sec Loss 10.9543 Epoch: 8 Global Step: 89355 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:45:41,978-Speed 130.12 samples/sec Loss 10.5234 Epoch: 8 Global Step: 89360 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:46:01,570-Speed 130.67 samples/sec Loss 11.0050 Epoch: 8 Global Step: 89365 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:46:21,605-Speed 127.78 samples/sec Loss 10.7764 Epoch: 8 Global Step: 89370 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:46:41,304-Speed 129.96 samples/sec Loss 10.6660 Epoch: 8 Global Step: 89375 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:47:01,189-Speed 128.75 samples/sec Loss 10.8449 Epoch: 8 Global Step: 89380 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:47:20,888-Speed 129.97 samples/sec Loss 10.6326 Epoch: 8 Global Step: 89385 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:47:40,438-Speed 130.95 samples/sec Loss 10.7981 Epoch: 8 Global Step: 89390 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:48:00,376-Speed 128.41 samples/sec Loss 10.9074 Epoch: 8 Global Step: 89395 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:48:19,943-Speed 130.84 samples/sec Loss 11.0037 Epoch: 8 Global Step: 89400 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:48:39,864-Speed 128.51 samples/sec Loss 10.8371 Epoch: 8 Global Step: 89405 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:48:59,314-Speed 131.63 samples/sec Loss 10.5359 Epoch: 8 Global Step: 89410 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:49:19,423-Speed 127.31 samples/sec Loss 10.8792 Epoch: 8 Global Step: 89415 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:49:39,253-Speed 129.10 samples/sec Loss 11.0165 Epoch: 8 Global Step: 89420 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:49:58,883-Speed 130.42 samples/sec Loss 10.6999 Epoch: 8 Global Step: 89425 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:50:18,558-Speed 130.12 samples/sec Loss 10.7307 Epoch: 8 Global Step: 89430 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:50:38,126-Speed 130.83 samples/sec Loss 10.5883 Epoch: 8 Global Step: 89435 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:50:58,055-Speed 128.46 samples/sec Loss 10.5782 Epoch: 8 Global Step: 89440 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:51:17,684-Speed 130.43 samples/sec Loss 10.8082 Epoch: 8 Global Step: 89445 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:51:37,585-Speed 128.65 samples/sec Loss 10.7383 Epoch: 8 Global Step: 89450 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:51:57,148-Speed 130.86 samples/sec Loss 10.5585 Epoch: 8 Global Step: 89455 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:52:17,239-Speed 127.43 samples/sec Loss 10.7859 Epoch: 8 Global Step: 89460 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:52:37,144-Speed 128.61 samples/sec Loss 10.7561 Epoch: 8 Global Step: 89465 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:52:56,715-Speed 130.81 samples/sec Loss 10.7505 Epoch: 8 Global Step: 89470 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:53:16,418-Speed 129.94 samples/sec Loss 10.4174 Epoch: 8 Global Step: 89475 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:53:36,043-Speed 130.45 samples/sec Loss 10.5402 Epoch: 8 Global Step: 89480 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:53:55,776-Speed 129.75 samples/sec Loss 10.7310 Epoch: 8 Global Step: 89485 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:54:15,328-Speed 130.94 samples/sec Loss 10.9425 Epoch: 8 Global Step: 89490 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:54:35,105-Speed 129.45 samples/sec Loss 10.7371 Epoch: 8 Global Step: 89495 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:54:54,689-Speed 130.73 samples/sec Loss 10.8795 Epoch: 8 Global Step: 89500 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:55:14,883-Speed 126.78 samples/sec Loss 10.6856 Epoch: 8 Global Step: 89505 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:55:34,783-Speed 128.65 samples/sec Loss 10.4386 Epoch: 8 Global Step: 89510 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:55:54,232-Speed 131.63 samples/sec Loss 10.6385 Epoch: 8 Global Step: 89515 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:56:14,006-Speed 129.47 samples/sec Loss 10.4393 Epoch: 8 Global Step: 89520 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:56:33,582-Speed 130.78 samples/sec Loss 10.6303 Epoch: 8 Global Step: 89525 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:56:53,295-Speed 129.87 samples/sec Loss 10.9623 Epoch: 8 Global Step: 89530 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:57:12,901-Speed 130.58 samples/sec Loss 10.8383 Epoch: 8 Global Step: 89535 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:57:32,610-Speed 129.90 samples/sec Loss 10.6968 Epoch: 8 Global Step: 89540 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:57:52,230-Speed 130.49 samples/sec Loss 10.8029 Epoch: 8 Global Step: 89545 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:58:12,489-Speed 126.37 samples/sec Loss 10.6430 Epoch: 8 Global Step: 89550 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:58:32,420-Speed 128.45 samples/sec Loss 10.6566 Epoch: 8 Global Step: 89555 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:58:51,981-Speed 130.88 samples/sec Loss 10.3425 Epoch: 8 Global Step: 89560 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:59:11,876-Speed 128.69 samples/sec Loss 10.3698 Epoch: 8 Global Step: 89565 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:59:31,427-Speed 130.95 samples/sec Loss 10.5035 Epoch: 8 Global Step: 89570 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 04:59:51,367-Speed 128.39 samples/sec Loss 10.7445 Epoch: 8 Global Step: 89575 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:00:10,969-Speed 130.61 samples/sec Loss 10.7335 Epoch: 8 Global Step: 89580 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:00:30,730-Speed 129.56 samples/sec Loss 10.8497 Epoch: 8 Global Step: 89585 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:00:50,348-Speed 130.50 samples/sec Loss 10.4947 Epoch: 8 Global Step: 89590 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:01:10,325-Speed 128.16 samples/sec Loss 10.9931 Epoch: 8 Global Step: 89595 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:01:30,263-Speed 128.40 samples/sec Loss 10.7783 Epoch: 8 Global Step: 89600 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:01:49,881-Speed 130.50 samples/sec Loss 11.0859 Epoch: 8 Global Step: 89605 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:02:09,797-Speed 128.55 samples/sec Loss 10.8318 Epoch: 8 Global Step: 89610 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:02:29,351-Speed 130.93 samples/sec Loss 10.7029 Epoch: 8 Global Step: 89615 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:02:49,076-Speed 129.79 samples/sec Loss 11.1846 Epoch: 8 Global Step: 89620 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:03:08,656-Speed 130.75 samples/sec Loss 10.7183 Epoch: 8 Global Step: 89625 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:03:28,459-Speed 129.28 samples/sec Loss 10.8113 Epoch: 8 Global Step: 89630 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:03:48,086-Speed 130.45 samples/sec Loss 10.7767 Epoch: 8 Global Step: 89635 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:04:08,019-Speed 128.43 samples/sec Loss 10.7531 Epoch: 8 Global Step: 89640 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:04:27,891-Speed 128.83 samples/sec Loss 10.3563 Epoch: 8 Global Step: 89645 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:04:47,500-Speed 130.56 samples/sec Loss 10.8836 Epoch: 8 Global Step: 89650 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:05:07,420-Speed 128.52 samples/sec Loss 10.4436 Epoch: 8 Global Step: 89655 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:05:26,962-Speed 131.01 samples/sec Loss 10.7820 Epoch: 8 Global Step: 89660 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:05:46,890-Speed 128.47 samples/sec Loss 10.8716 Epoch: 8 Global Step: 89665 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:06:06,371-Speed 131.42 samples/sec Loss 10.8778 Epoch: 8 Global Step: 89670 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:06:26,065-Speed 130.00 samples/sec Loss 10.8205 Epoch: 8 Global Step: 89675 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:07:06,225-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-11 05:07:28,889-Speed 40.75 samples/sec Loss 10.3156 Epoch: 9 Global Step: 89680 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:07:48,943-Speed 127.66 samples/sec Loss 9.8087 Epoch: 9 Global Step: 89685 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:08:08,984-Speed 127.74 samples/sec Loss 9.7630 Epoch: 9 Global Step: 89690 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 05:08:29,144-Speed 126.99 samples/sec Loss 10.0974 Epoch: 9 Global Step: 89695 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:08:48,966-Speed 129.16 samples/sec Loss 10.1458 Epoch: 9 Global Step: 89700 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:09:08,894-Speed 128.47 samples/sec Loss 9.7365 Epoch: 9 Global Step: 89705 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:09:28,659-Speed 129.53 samples/sec Loss 9.9784 Epoch: 9 Global Step: 89710 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:09:48,457-Speed 129.31 samples/sec Loss 10.1366 Epoch: 9 Global Step: 89715 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:10:08,373-Speed 128.55 samples/sec Loss 10.2221 Epoch: 9 Global Step: 89720 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:10:28,049-Speed 130.12 samples/sec Loss 10.0110 Epoch: 9 Global Step: 89725 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:10:47,991-Speed 128.38 samples/sec Loss 10.0149 Epoch: 9 Global Step: 89730 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:11:07,922-Speed 128.45 samples/sec Loss 9.8653 Epoch: 9 Global Step: 89735 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:11:28,064-Speed 127.10 samples/sec Loss 10.3068 Epoch: 9 Global Step: 89740 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:11:47,963-Speed 128.65 samples/sec Loss 9.8795 Epoch: 9 Global Step: 89745 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:12:07,856-Speed 128.70 samples/sec Loss 10.0015 Epoch: 9 Global Step: 89750 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:12:27,617-Speed 129.55 samples/sec Loss 10.3142 Epoch: 9 Global Step: 89755 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:12:47,569-Speed 128.32 samples/sec Loss 9.8914 Epoch: 9 Global Step: 89760 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:13:07,399-Speed 129.10 samples/sec Loss 10.3165 Epoch: 9 Global Step: 89765 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:13:27,319-Speed 128.53 samples/sec Loss 10.1738 Epoch: 9 Global Step: 89770 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:13:47,012-Speed 130.00 samples/sec Loss 10.0893 Epoch: 9 Global Step: 89775 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:14:07,369-Speed 125.76 samples/sec Loss 9.9810 Epoch: 9 Global Step: 89780 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:14:27,490-Speed 127.24 samples/sec Loss 9.9886 Epoch: 9 Global Step: 89785 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:14:47,502-Speed 127.93 samples/sec Loss 9.9676 Epoch: 9 Global Step: 89790 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:15:07,163-Speed 130.21 samples/sec Loss 10.1369 Epoch: 9 Global Step: 89795 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:15:27,119-Speed 128.29 samples/sec Loss 10.1273 Epoch: 9 Global Step: 89800 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:15:46,857-Speed 129.71 samples/sec Loss 10.0395 Epoch: 9 Global Step: 89805 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:16:06,674-Speed 129.19 samples/sec Loss 9.9477 Epoch: 9 Global Step: 89810 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:16:26,599-Speed 128.49 samples/sec Loss 10.0282 Epoch: 9 Global Step: 89815 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:16:46,258-Speed 130.23 samples/sec Loss 10.3660 Epoch: 9 Global Step: 89820 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:17:06,578-Speed 125.99 samples/sec Loss 9.7258 Epoch: 9 Global Step: 89825 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:17:26,694-Speed 127.27 samples/sec Loss 10.2879 Epoch: 9 Global Step: 89830 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:17:46,370-Speed 130.12 samples/sec Loss 10.1614 Epoch: 9 Global Step: 89835 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:18:06,168-Speed 129.31 samples/sec Loss 10.1814 Epoch: 9 Global Step: 89840 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:18:26,059-Speed 128.71 samples/sec Loss 9.8678 Epoch: 9 Global Step: 89845 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:18:45,901-Speed 129.02 samples/sec Loss 10.0821 Epoch: 9 Global Step: 89850 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:19:05,842-Speed 128.38 samples/sec Loss 10.3787 Epoch: 9 Global Step: 89855 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:19:25,525-Speed 130.07 samples/sec Loss 10.1209 Epoch: 9 Global Step: 89860 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:19:45,393-Speed 128.86 samples/sec Loss 10.1199 Epoch: 9 Global Step: 89865 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:20:05,588-Speed 126.77 samples/sec Loss 10.1827 Epoch: 9 Global Step: 89870 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:20:25,709-Speed 127.24 samples/sec Loss 10.0618 Epoch: 9 Global Step: 89875 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:20:45,999-Speed 126.17 samples/sec Loss 10.3825 Epoch: 9 Global Step: 89880 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:21:05,790-Speed 129.36 samples/sec Loss 10.1620 Epoch: 9 Global Step: 89885 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:21:25,787-Speed 128.02 samples/sec Loss 10.8310 Epoch: 9 Global Step: 89890 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:21:45,548-Speed 129.56 samples/sec Loss 10.2538 Epoch: 9 Global Step: 89895 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:22:05,406-Speed 128.92 samples/sec Loss 10.3660 Epoch: 9 Global Step: 89900 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:22:25,194-Speed 129.37 samples/sec Loss 10.1730 Epoch: 9 Global Step: 89905 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:22:45,138-Speed 128.37 samples/sec Loss 10.2331 Epoch: 9 Global Step: 89910 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:23:05,212-Speed 127.53 samples/sec Loss 10.1293 Epoch: 9 Global Step: 89915 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:23:25,275-Speed 127.60 samples/sec Loss 10.2323 Epoch: 9 Global Step: 89920 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:23:45,032-Speed 129.58 samples/sec Loss 10.1126 Epoch: 9 Global Step: 89925 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:24:04,972-Speed 128.39 samples/sec Loss 10.2837 Epoch: 9 Global Step: 89930 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:24:24,726-Speed 129.60 samples/sec Loss 10.4142 Epoch: 9 Global Step: 89935 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:24:44,778-Speed 127.67 samples/sec Loss 10.3071 Epoch: 9 Global Step: 89940 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:25:04,728-Speed 128.33 samples/sec Loss 10.2892 Epoch: 9 Global Step: 89945 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:25:24,505-Speed 129.45 samples/sec Loss 10.4882 Epoch: 9 Global Step: 89950 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:25:44,553-Speed 127.70 samples/sec Loss 10.0996 Epoch: 9 Global Step: 89955 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:26:04,447-Speed 128.69 samples/sec Loss 10.2647 Epoch: 9 Global Step: 89960 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:26:24,619-Speed 126.92 samples/sec Loss 10.3710 Epoch: 9 Global Step: 89965 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:26:44,465-Speed 129.00 samples/sec Loss 10.4473 Epoch: 9 Global Step: 89970 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:27:04,313-Speed 128.99 samples/sec Loss 10.2132 Epoch: 9 Global Step: 89975 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:27:24,066-Speed 129.61 samples/sec Loss 9.9341 Epoch: 9 Global Step: 89980 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:27:44,313-Speed 126.44 samples/sec Loss 10.2076 Epoch: 9 Global Step: 89985 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:28:04,072-Speed 129.57 samples/sec Loss 10.0735 Epoch: 9 Global Step: 89990 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:28:23,976-Speed 128.62 samples/sec Loss 10.4248 Epoch: 9 Global Step: 89995 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:28:43,810-Speed 129.08 samples/sec Loss 10.3945 Epoch: 9 Global Step: 90000 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 05:43:31,371-[lfw][90000]XNorm: 22.507632 Training: 2021-06-11 05:43:31,372-[lfw][90000]Accuracy-Flip: 0.99667+-0.00333 Training: 2021-06-11 05:43:31,372-[lfw][90000]Accuracy-Highest: 0.99750 Training: 2021-06-11 06:00:44,744-[cfp_fp][90000]XNorm: 20.686747 Training: 2021-06-11 06:00:44,744-[cfp_fp][90000]Accuracy-Flip: 0.97557+-0.00467 Training: 2021-06-11 06:00:44,744-[cfp_fp][90000]Accuracy-Highest: 0.97557 Training: 2021-06-11 06:15:31,170-[agedb_30][90000]XNorm: 22.140105 Training: 2021-06-11 06:15:31,171-[agedb_30][90000]Accuracy-Flip: 0.97267+-0.00616 Training: 2021-06-11 06:15:31,171-[agedb_30][90000]Accuracy-Highest: 0.97267 Training: 2021-06-11 06:15:51,262-Speed 0.91 samples/sec Loss 10.5091 Epoch: 9 Global Step: 90005 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:16:11,219-Speed 128.28 samples/sec Loss 10.3516 Epoch: 9 Global Step: 90010 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:16:31,323-Speed 127.35 samples/sec Loss 10.0340 Epoch: 9 Global Step: 90015 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:16:50,885-Speed 130.87 samples/sec Loss 10.5303 Epoch: 9 Global Step: 90020 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:17:10,950-Speed 127.59 samples/sec Loss 10.2161 Epoch: 9 Global Step: 90025 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:17:30,835-Speed 128.75 samples/sec Loss 10.2793 Epoch: 9 Global Step: 90030 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:17:50,563-Speed 129.77 samples/sec Loss 10.3722 Epoch: 9 Global Step: 90035 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:18:10,236-Speed 130.13 samples/sec Loss 10.5309 Epoch: 9 Global Step: 90040 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:18:30,185-Speed 128.33 samples/sec Loss 10.2934 Epoch: 9 Global Step: 90045 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:18:50,235-Speed 127.69 samples/sec Loss 10.1083 Epoch: 9 Global Step: 90050 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:19:10,206-Speed 128.19 samples/sec Loss 10.3102 Epoch: 9 Global Step: 90055 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:19:30,001-Speed 129.33 samples/sec Loss 10.2332 Epoch: 9 Global Step: 90060 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:19:49,771-Speed 129.50 samples/sec Loss 10.2280 Epoch: 9 Global Step: 90065 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:20:09,801-Speed 127.81 samples/sec Loss 10.6087 Epoch: 9 Global Step: 90070 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:20:29,665-Speed 128.88 samples/sec Loss 10.2853 Epoch: 9 Global Step: 90075 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:20:49,404-Speed 129.70 samples/sec Loss 10.5176 Epoch: 9 Global Step: 90080 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:21:09,282-Speed 128.79 samples/sec Loss 10.6995 Epoch: 9 Global Step: 90085 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:21:29,059-Speed 129.45 samples/sec Loss 10.5789 Epoch: 9 Global Step: 90090 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:21:49,029-Speed 128.20 samples/sec Loss 10.2352 Epoch: 9 Global Step: 90095 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:22:08,886-Speed 128.93 samples/sec Loss 10.7616 Epoch: 9 Global Step: 90100 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:22:28,597-Speed 129.89 samples/sec Loss 10.3671 Epoch: 9 Global Step: 90105 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:22:48,478-Speed 128.77 samples/sec Loss 10.4648 Epoch: 9 Global Step: 90110 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:23:08,788-Speed 126.05 samples/sec Loss 9.9766 Epoch: 9 Global Step: 90115 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:23:28,565-Speed 129.45 samples/sec Loss 10.6081 Epoch: 9 Global Step: 90120 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:23:48,360-Speed 129.33 samples/sec Loss 10.5088 Epoch: 9 Global Step: 90125 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:24:08,266-Speed 128.61 samples/sec Loss 10.4369 Epoch: 9 Global Step: 90130 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:24:28,060-Speed 129.34 samples/sec Loss 10.2900 Epoch: 9 Global Step: 90135 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:24:48,347-Speed 126.19 samples/sec Loss 10.5907 Epoch: 9 Global Step: 90140 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:25:08,092-Speed 129.66 samples/sec Loss 10.7774 Epoch: 9 Global Step: 90145 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:25:27,928-Speed 129.07 samples/sec Loss 10.3813 Epoch: 9 Global Step: 90150 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:25:47,696-Speed 129.51 samples/sec Loss 10.5151 Epoch: 9 Global Step: 90155 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:26:07,864-Speed 126.94 samples/sec Loss 10.2457 Epoch: 9 Global Step: 90160 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:26:27,750-Speed 128.74 samples/sec Loss 10.3413 Epoch: 9 Global Step: 90165 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:26:47,675-Speed 128.49 samples/sec Loss 10.3133 Epoch: 9 Global Step: 90170 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:27:08,507-Speed 122.89 samples/sec Loss 10.2996 Epoch: 9 Global Step: 90175 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:27:28,462-Speed 128.30 samples/sec Loss 10.5010 Epoch: 9 Global Step: 90180 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:27:48,537-Speed 127.53 samples/sec Loss 10.5048 Epoch: 9 Global Step: 90185 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:28:08,213-Speed 130.12 samples/sec Loss 10.5431 Epoch: 9 Global Step: 90190 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:28:28,139-Speed 128.48 samples/sec Loss 10.3537 Epoch: 9 Global Step: 90195 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:28:47,898-Speed 129.57 samples/sec Loss 10.2566 Epoch: 9 Global Step: 90200 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:29:07,915-Speed 127.90 samples/sec Loss 10.5534 Epoch: 9 Global Step: 90205 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:29:28,790-Speed 122.64 samples/sec Loss 10.2793 Epoch: 9 Global Step: 90210 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:29:48,543-Speed 129.61 samples/sec Loss 10.3007 Epoch: 9 Global Step: 90215 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:30:08,340-Speed 129.32 samples/sec Loss 9.9694 Epoch: 9 Global Step: 90220 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:30:28,241-Speed 128.64 samples/sec Loss 10.6640 Epoch: 9 Global Step: 90225 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:30:48,318-Speed 127.51 samples/sec Loss 10.3599 Epoch: 9 Global Step: 90230 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:31:08,229-Speed 128.58 samples/sec Loss 10.5892 Epoch: 9 Global Step: 90235 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:31:28,012-Speed 129.41 samples/sec Loss 10.6409 Epoch: 9 Global Step: 90240 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:31:47,832-Speed 129.18 samples/sec Loss 10.5190 Epoch: 9 Global Step: 90245 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:32:07,973-Speed 127.11 samples/sec Loss 10.4584 Epoch: 9 Global Step: 90250 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:32:27,937-Speed 128.23 samples/sec Loss 10.6073 Epoch: 9 Global Step: 90255 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:32:47,684-Speed 129.65 samples/sec Loss 10.8477 Epoch: 9 Global Step: 90260 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:33:07,658-Speed 128.17 samples/sec Loss 10.3524 Epoch: 9 Global Step: 90265 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:33:27,856-Speed 126.76 samples/sec Loss 10.3298 Epoch: 9 Global Step: 90270 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:33:47,936-Speed 127.50 samples/sec Loss 10.8663 Epoch: 9 Global Step: 90275 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:34:07,850-Speed 128.56 samples/sec Loss 10.6063 Epoch: 9 Global Step: 90280 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:34:27,628-Speed 129.44 samples/sec Loss 10.3455 Epoch: 9 Global Step: 90285 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:34:47,635-Speed 127.97 samples/sec Loss 10.5873 Epoch: 9 Global Step: 90290 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:35:07,511-Speed 128.80 samples/sec Loss 10.1522 Epoch: 9 Global Step: 90295 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:35:27,402-Speed 128.71 samples/sec Loss 10.6644 Epoch: 9 Global Step: 90300 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:35:47,171-Speed 129.50 samples/sec Loss 10.5513 Epoch: 9 Global Step: 90305 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:36:07,060-Speed 128.72 samples/sec Loss 10.5232 Epoch: 9 Global Step: 90310 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:36:26,848-Speed 129.37 samples/sec Loss 10.5723 Epoch: 9 Global Step: 90315 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:36:47,084-Speed 126.51 samples/sec Loss 10.4882 Epoch: 9 Global Step: 90320 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:37:06,840-Speed 129.59 samples/sec Loss 10.5707 Epoch: 9 Global Step: 90325 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:37:26,716-Speed 128.80 samples/sec Loss 10.3799 Epoch: 9 Global Step: 90330 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:37:46,544-Speed 129.11 samples/sec Loss 11.0429 Epoch: 9 Global Step: 90335 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:38:06,675-Speed 127.18 samples/sec Loss 10.7420 Epoch: 9 Global Step: 90340 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:38:26,439-Speed 129.53 samples/sec Loss 10.7129 Epoch: 9 Global Step: 90345 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:38:46,367-Speed 128.47 samples/sec Loss 10.6721 Epoch: 9 Global Step: 90350 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:39:06,169-Speed 129.29 samples/sec Loss 10.7934 Epoch: 9 Global Step: 90355 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:39:25,901-Speed 129.75 samples/sec Loss 10.7275 Epoch: 9 Global Step: 90360 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:39:46,093-Speed 126.79 samples/sec Loss 10.7662 Epoch: 9 Global Step: 90365 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:40:05,940-Speed 128.99 samples/sec Loss 10.5764 Epoch: 9 Global Step: 90370 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:40:25,795-Speed 128.95 samples/sec Loss 10.6411 Epoch: 9 Global Step: 90375 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:40:45,498-Speed 129.93 samples/sec Loss 10.3258 Epoch: 9 Global Step: 90380 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:41:05,622-Speed 127.22 samples/sec Loss 10.8671 Epoch: 9 Global Step: 90385 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:41:25,391-Speed 129.51 samples/sec Loss 10.5516 Epoch: 9 Global Step: 90390 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:41:45,337-Speed 128.35 samples/sec Loss 10.5242 Epoch: 9 Global Step: 90395 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:42:05,029-Speed 130.01 samples/sec Loss 10.5971 Epoch: 9 Global Step: 90400 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:42:24,899-Speed 128.84 samples/sec Loss 10.2013 Epoch: 9 Global Step: 90405 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:42:45,100-Speed 126.73 samples/sec Loss 10.2491 Epoch: 9 Global Step: 90410 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:43:04,819-Speed 129.83 samples/sec Loss 10.1523 Epoch: 9 Global Step: 90415 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:43:24,649-Speed 129.11 samples/sec Loss 10.5678 Epoch: 9 Global Step: 90420 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:43:44,410-Speed 129.55 samples/sec Loss 10.7301 Epoch: 9 Global Step: 90425 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:44:04,552-Speed 127.11 samples/sec Loss 10.7126 Epoch: 9 Global Step: 90430 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:44:24,404-Speed 128.96 samples/sec Loss 10.5490 Epoch: 9 Global Step: 90435 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:44:44,169-Speed 129.53 samples/sec Loss 10.5149 Epoch: 9 Global Step: 90440 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:45:03,979-Speed 129.24 samples/sec Loss 10.6349 Epoch: 9 Global Step: 90445 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:45:23,876-Speed 128.66 samples/sec Loss 10.4543 Epoch: 9 Global Step: 90450 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:45:43,923-Speed 127.71 samples/sec Loss 10.6683 Epoch: 9 Global Step: 90455 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:46:03,784-Speed 128.90 samples/sec Loss 10.8658 Epoch: 9 Global Step: 90460 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:46:23,551-Speed 129.51 samples/sec Loss 10.7838 Epoch: 9 Global Step: 90465 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:46:43,397-Speed 129.00 samples/sec Loss 10.8857 Epoch: 9 Global Step: 90470 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:47:03,522-Speed 127.21 samples/sec Loss 10.7046 Epoch: 9 Global Step: 90475 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:47:23,365-Speed 129.02 samples/sec Loss 10.6483 Epoch: 9 Global Step: 90480 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:47:43,033-Speed 130.17 samples/sec Loss 10.2014 Epoch: 9 Global Step: 90485 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:48:02,983-Speed 128.33 samples/sec Loss 10.5723 Epoch: 9 Global Step: 90490 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:48:22,769-Speed 129.39 samples/sec Loss 10.4492 Epoch: 9 Global Step: 90495 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:48:42,910-Speed 127.11 samples/sec Loss 10.6036 Epoch: 9 Global Step: 90500 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:49:02,825-Speed 128.55 samples/sec Loss 10.3365 Epoch: 9 Global Step: 90505 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:49:22,602-Speed 129.45 samples/sec Loss 10.5539 Epoch: 9 Global Step: 90510 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:49:42,636-Speed 127.79 samples/sec Loss 10.6127 Epoch: 9 Global Step: 90515 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:50:02,522-Speed 128.74 samples/sec Loss 10.7022 Epoch: 9 Global Step: 90520 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:50:22,384-Speed 128.90 samples/sec Loss 10.4344 Epoch: 9 Global Step: 90525 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:50:42,058-Speed 130.13 samples/sec Loss 10.3687 Epoch: 9 Global Step: 90530 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:51:02,056-Speed 128.02 samples/sec Loss 10.6284 Epoch: 9 Global Step: 90535 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:51:21,913-Speed 128.93 samples/sec Loss 10.5326 Epoch: 9 Global Step: 90540 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:51:42,025-Speed 127.29 samples/sec Loss 10.9454 Epoch: 9 Global Step: 90545 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:52:01,808-Speed 129.41 samples/sec Loss 10.6741 Epoch: 9 Global Step: 90550 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:52:21,605-Speed 129.32 samples/sec Loss 10.5632 Epoch: 9 Global Step: 90555 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:52:41,557-Speed 128.32 samples/sec Loss 10.8496 Epoch: 9 Global Step: 90560 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:53:01,650-Speed 127.42 samples/sec Loss 10.4727 Epoch: 9 Global Step: 90565 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:53:21,316-Speed 130.18 samples/sec Loss 10.6462 Epoch: 9 Global Step: 90570 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:53:41,092-Speed 129.45 samples/sec Loss 10.5101 Epoch: 9 Global Step: 90575 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:54:01,074-Speed 128.12 samples/sec Loss 10.4572 Epoch: 9 Global Step: 90580 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:54:20,832-Speed 129.58 samples/sec Loss 10.5794 Epoch: 9 Global Step: 90585 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:54:41,050-Speed 126.63 samples/sec Loss 10.9897 Epoch: 9 Global Step: 90590 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:55:00,813-Speed 129.54 samples/sec Loss 10.5361 Epoch: 9 Global Step: 90595 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:55:20,815-Speed 128.00 samples/sec Loss 10.1397 Epoch: 9 Global Step: 90600 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:55:40,571-Speed 129.59 samples/sec Loss 10.7165 Epoch: 9 Global Step: 90605 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:56:00,707-Speed 127.14 samples/sec Loss 10.5165 Epoch: 9 Global Step: 90610 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:56:20,464-Speed 129.58 samples/sec Loss 10.5638 Epoch: 9 Global Step: 90615 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:56:40,408-Speed 128.36 samples/sec Loss 10.5124 Epoch: 9 Global Step: 90620 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:57:00,261-Speed 128.95 samples/sec Loss 10.3364 Epoch: 9 Global Step: 90625 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:57:20,078-Speed 129.19 samples/sec Loss 10.7528 Epoch: 9 Global Step: 90630 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:57:40,195-Speed 127.27 samples/sec Loss 10.7051 Epoch: 9 Global Step: 90635 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:57:59,991-Speed 129.32 samples/sec Loss 10.7198 Epoch: 9 Global Step: 90640 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:58:19,946-Speed 128.30 samples/sec Loss 10.6669 Epoch: 9 Global Step: 90645 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:58:39,626-Speed 130.09 samples/sec Loss 10.5660 Epoch: 9 Global Step: 90650 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:58:59,739-Speed 127.29 samples/sec Loss 10.9938 Epoch: 9 Global Step: 90655 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:59:19,519-Speed 129.43 samples/sec Loss 10.7652 Epoch: 9 Global Step: 90660 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:59:39,486-Speed 128.22 samples/sec Loss 10.4607 Epoch: 9 Global Step: 90665 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 06:59:59,249-Speed 129.54 samples/sec Loss 10.6764 Epoch: 9 Global Step: 90670 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:00:19,093-Speed 129.01 samples/sec Loss 10.5526 Epoch: 9 Global Step: 90675 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:00:39,185-Speed 127.42 samples/sec Loss 10.5998 Epoch: 9 Global Step: 90680 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:00:59,149-Speed 128.24 samples/sec Loss 10.9298 Epoch: 9 Global Step: 90685 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:01:18,986-Speed 129.06 samples/sec Loss 10.5386 Epoch: 9 Global Step: 90690 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:01:38,648-Speed 130.21 samples/sec Loss 10.9102 Epoch: 9 Global Step: 90695 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:01:58,810-Speed 126.98 samples/sec Loss 10.5845 Epoch: 9 Global Step: 90700 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:02:18,770-Speed 128.26 samples/sec Loss 10.4895 Epoch: 9 Global Step: 90705 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:02:38,597-Speed 129.12 samples/sec Loss 10.4399 Epoch: 9 Global Step: 90710 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:02:58,289-Speed 130.01 samples/sec Loss 10.4811 Epoch: 9 Global Step: 90715 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:03:18,160-Speed 128.84 samples/sec Loss 10.7947 Epoch: 9 Global Step: 90720 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:03:38,319-Speed 126.99 samples/sec Loss 10.8832 Epoch: 9 Global Step: 90725 Fp16 Grad Scale: 8192 Required: 189 hours Training: 2021-06-11 07:03:58,257-Speed 128.40 samples/sec Loss 10.7670 Epoch: 9 Global Step: 90730 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:04:17,968-Speed 129.89 samples/sec Loss 11.0612 Epoch: 9 Global Step: 90735 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:04:37,718-Speed 129.63 samples/sec Loss 11.0154 Epoch: 9 Global Step: 90740 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:04:57,903-Speed 126.84 samples/sec Loss 10.3665 Epoch: 9 Global Step: 90745 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:05:17,940-Speed 127.76 samples/sec Loss 10.5444 Epoch: 9 Global Step: 90750 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:05:37,704-Speed 129.54 samples/sec Loss 10.7337 Epoch: 9 Global Step: 90755 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:05:57,490-Speed 129.39 samples/sec Loss 10.9363 Epoch: 9 Global Step: 90760 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:06:17,406-Speed 128.54 samples/sec Loss 10.5726 Epoch: 9 Global Step: 90765 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:06:37,463-Speed 127.65 samples/sec Loss 10.8321 Epoch: 9 Global Step: 90770 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:06:57,342-Speed 128.79 samples/sec Loss 10.1629 Epoch: 9 Global Step: 90775 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:07:17,016-Speed 130.12 samples/sec Loss 10.7608 Epoch: 9 Global Step: 90780 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:07:37,034-Speed 127.90 samples/sec Loss 10.8015 Epoch: 9 Global Step: 90785 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:07:57,084-Speed 127.68 samples/sec Loss 10.4372 Epoch: 9 Global Step: 90790 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:08:17,031-Speed 128.35 samples/sec Loss 10.9129 Epoch: 9 Global Step: 90795 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:08:36,673-Speed 130.34 samples/sec Loss 10.8257 Epoch: 9 Global Step: 90800 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:08:56,621-Speed 128.34 samples/sec Loss 10.4841 Epoch: 9 Global Step: 90805 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:09:16,395-Speed 129.47 samples/sec Loss 11.0583 Epoch: 9 Global Step: 90810 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:09:36,524-Speed 127.18 samples/sec Loss 10.3129 Epoch: 9 Global Step: 90815 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:09:56,343-Speed 129.18 samples/sec Loss 10.8909 Epoch: 9 Global Step: 90820 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:10:16,105-Speed 129.55 samples/sec Loss 10.5979 Epoch: 9 Global Step: 90825 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:10:36,116-Speed 127.94 samples/sec Loss 10.7802 Epoch: 9 Global Step: 90830 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:10:56,247-Speed 127.18 samples/sec Loss 10.5401 Epoch: 9 Global Step: 90835 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:11:15,984-Speed 129.71 samples/sec Loss 10.5281 Epoch: 9 Global Step: 90840 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:11:36,015-Speed 127.81 samples/sec Loss 10.7555 Epoch: 9 Global Step: 90845 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:11:55,946-Speed 128.45 samples/sec Loss 10.6942 Epoch: 9 Global Step: 90850 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:12:15,727-Speed 129.42 samples/sec Loss 10.4390 Epoch: 9 Global Step: 90855 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:12:35,837-Speed 127.31 samples/sec Loss 11.0502 Epoch: 9 Global Step: 90860 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:12:55,605-Speed 129.50 samples/sec Loss 10.9134 Epoch: 9 Global Step: 90865 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:13:15,519-Speed 128.56 samples/sec Loss 10.7543 Epoch: 9 Global Step: 90870 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:13:35,313-Speed 129.34 samples/sec Loss 10.6664 Epoch: 9 Global Step: 90875 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:13:55,356-Speed 127.73 samples/sec Loss 10.7686 Epoch: 9 Global Step: 90880 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:14:15,044-Speed 130.03 samples/sec Loss 10.6299 Epoch: 9 Global Step: 90885 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:14:34,871-Speed 129.12 samples/sec Loss 10.8084 Epoch: 9 Global Step: 90890 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:14:54,743-Speed 128.83 samples/sec Loss 10.4789 Epoch: 9 Global Step: 90895 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:15:14,572-Speed 129.11 samples/sec Loss 10.4305 Epoch: 9 Global Step: 90900 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:15:34,680-Speed 127.32 samples/sec Loss 10.7056 Epoch: 9 Global Step: 90905 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:15:54,425-Speed 129.66 samples/sec Loss 10.6802 Epoch: 9 Global Step: 90910 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:16:14,381-Speed 128.29 samples/sec Loss 10.6423 Epoch: 9 Global Step: 90915 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:16:34,137-Speed 129.59 samples/sec Loss 10.9536 Epoch: 9 Global Step: 90920 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:16:54,174-Speed 127.77 samples/sec Loss 10.5910 Epoch: 9 Global Step: 90925 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:17:13,954-Speed 129.43 samples/sec Loss 10.6582 Epoch: 9 Global Step: 90930 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:17:34,119-Speed 126.96 samples/sec Loss 10.7158 Epoch: 9 Global Step: 90935 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:17:53,932-Speed 129.21 samples/sec Loss 10.6161 Epoch: 9 Global Step: 90940 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:18:13,719-Speed 129.39 samples/sec Loss 10.5137 Epoch: 9 Global Step: 90945 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:18:33,802-Speed 127.48 samples/sec Loss 10.5202 Epoch: 9 Global Step: 90950 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:18:53,709-Speed 128.60 samples/sec Loss 10.5275 Epoch: 9 Global Step: 90955 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:19:13,630-Speed 128.51 samples/sec Loss 10.8491 Epoch: 9 Global Step: 90960 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:19:33,562-Speed 128.44 samples/sec Loss 10.7857 Epoch: 9 Global Step: 90965 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:19:53,795-Speed 126.53 samples/sec Loss 10.6968 Epoch: 9 Global Step: 90970 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:20:13,745-Speed 128.33 samples/sec Loss 10.6367 Epoch: 9 Global Step: 90975 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:20:33,568-Speed 129.15 samples/sec Loss 10.4090 Epoch: 9 Global Step: 90980 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:20:53,334-Speed 129.52 samples/sec Loss 10.5612 Epoch: 9 Global Step: 90985 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:21:13,171-Speed 129.06 samples/sec Loss 10.4530 Epoch: 9 Global Step: 90990 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:21:33,252-Speed 127.48 samples/sec Loss 10.3544 Epoch: 9 Global Step: 90995 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:21:53,089-Speed 129.06 samples/sec Loss 10.7276 Epoch: 9 Global Step: 91000 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:22:12,965-Speed 128.81 samples/sec Loss 10.4200 Epoch: 9 Global Step: 91005 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:22:32,779-Speed 129.21 samples/sec Loss 10.8073 Epoch: 9 Global Step: 91010 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:22:52,883-Speed 127.34 samples/sec Loss 10.5660 Epoch: 9 Global Step: 91015 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:23:12,911-Speed 127.82 samples/sec Loss 10.4971 Epoch: 9 Global Step: 91020 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:23:32,660-Speed 129.64 samples/sec Loss 10.8197 Epoch: 9 Global Step: 91025 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:23:52,433-Speed 129.47 samples/sec Loss 10.6321 Epoch: 9 Global Step: 91030 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:24:12,297-Speed 128.89 samples/sec Loss 10.5846 Epoch: 9 Global Step: 91035 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:24:32,524-Speed 126.56 samples/sec Loss 10.6109 Epoch: 9 Global Step: 91040 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:24:52,477-Speed 128.31 samples/sec Loss 10.5858 Epoch: 9 Global Step: 91045 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:25:12,151-Speed 130.13 samples/sec Loss 10.9342 Epoch: 9 Global Step: 91050 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:25:32,043-Speed 128.70 samples/sec Loss 10.7377 Epoch: 9 Global Step: 91055 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:25:52,086-Speed 127.73 samples/sec Loss 10.7638 Epoch: 9 Global Step: 91060 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:26:12,043-Speed 128.28 samples/sec Loss 10.6214 Epoch: 9 Global Step: 91065 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:26:31,773-Speed 129.76 samples/sec Loss 10.4849 Epoch: 9 Global Step: 91070 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:26:51,725-Speed 128.31 samples/sec Loss 10.9111 Epoch: 9 Global Step: 91075 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:27:11,503-Speed 129.45 samples/sec Loss 10.6776 Epoch: 9 Global Step: 91080 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:27:31,710-Speed 126.69 samples/sec Loss 10.4025 Epoch: 9 Global Step: 91085 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:27:51,540-Speed 129.11 samples/sec Loss 10.9599 Epoch: 9 Global Step: 91090 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:28:11,298-Speed 129.57 samples/sec Loss 10.5222 Epoch: 9 Global Step: 91095 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:28:31,194-Speed 128.68 samples/sec Loss 10.6484 Epoch: 9 Global Step: 91100 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:28:51,169-Speed 128.17 samples/sec Loss 10.8435 Epoch: 9 Global Step: 91105 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:29:11,069-Speed 128.65 samples/sec Loss 10.8392 Epoch: 9 Global Step: 91110 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:29:30,716-Speed 130.31 samples/sec Loss 10.4975 Epoch: 9 Global Step: 91115 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:29:50,666-Speed 128.33 samples/sec Loss 10.5962 Epoch: 9 Global Step: 91120 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:30:10,517-Speed 128.97 samples/sec Loss 10.5086 Epoch: 9 Global Step: 91125 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:30:30,660-Speed 127.10 samples/sec Loss 10.5360 Epoch: 9 Global Step: 91130 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:30:50,450-Speed 129.37 samples/sec Loss 10.6855 Epoch: 9 Global Step: 91135 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:31:10,193-Speed 129.67 samples/sec Loss 10.6148 Epoch: 9 Global Step: 91140 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:31:30,179-Speed 128.10 samples/sec Loss 10.6719 Epoch: 9 Global Step: 91145 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:31:50,601-Speed 125.36 samples/sec Loss 10.6226 Epoch: 9 Global Step: 91150 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:32:10,367-Speed 129.52 samples/sec Loss 10.6199 Epoch: 9 Global Step: 91155 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:32:30,157-Speed 129.36 samples/sec Loss 10.9734 Epoch: 9 Global Step: 91160 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:32:50,085-Speed 128.47 samples/sec Loss 10.2684 Epoch: 9 Global Step: 91165 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:33:09,928-Speed 129.02 samples/sec Loss 10.7147 Epoch: 9 Global Step: 91170 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:33:30,137-Speed 126.68 samples/sec Loss 10.7828 Epoch: 9 Global Step: 91175 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:33:49,831-Speed 130.00 samples/sec Loss 10.4658 Epoch: 9 Global Step: 91180 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:34:09,661-Speed 129.11 samples/sec Loss 10.7863 Epoch: 9 Global Step: 91185 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:34:30,102-Speed 125.24 samples/sec Loss 10.7926 Epoch: 9 Global Step: 91190 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:34:50,229-Speed 127.20 samples/sec Loss 10.9868 Epoch: 9 Global Step: 91195 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:35:10,004-Speed 129.46 samples/sec Loss 10.4903 Epoch: 9 Global Step: 91200 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:35:29,846-Speed 129.03 samples/sec Loss 10.4770 Epoch: 9 Global Step: 91205 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:35:49,638-Speed 129.35 samples/sec Loss 10.7265 Epoch: 9 Global Step: 91210 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:36:09,442-Speed 129.27 samples/sec Loss 11.0295 Epoch: 9 Global Step: 91215 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:36:29,615-Speed 126.91 samples/sec Loss 10.6280 Epoch: 9 Global Step: 91220 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:36:49,276-Speed 130.22 samples/sec Loss 10.6345 Epoch: 9 Global Step: 91225 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:37:09,296-Speed 127.88 samples/sec Loss 10.6666 Epoch: 9 Global Step: 91230 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:37:29,042-Speed 129.65 samples/sec Loss 10.6571 Epoch: 9 Global Step: 91235 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:37:49,156-Speed 127.28 samples/sec Loss 10.5830 Epoch: 9 Global Step: 91240 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:38:08,902-Speed 129.66 samples/sec Loss 10.3694 Epoch: 9 Global Step: 91245 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:38:28,863-Speed 128.25 samples/sec Loss 10.3563 Epoch: 9 Global Step: 91250 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:38:48,663-Speed 129.30 samples/sec Loss 10.7804 Epoch: 9 Global Step: 91255 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:39:08,546-Speed 128.76 samples/sec Loss 10.7363 Epoch: 9 Global Step: 91260 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:39:28,671-Speed 127.21 samples/sec Loss 10.5876 Epoch: 9 Global Step: 91265 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:39:48,513-Speed 129.03 samples/sec Loss 10.8695 Epoch: 9 Global Step: 91270 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:40:08,366-Speed 128.96 samples/sec Loss 10.6812 Epoch: 9 Global Step: 91275 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:40:28,128-Speed 129.55 samples/sec Loss 10.5097 Epoch: 9 Global Step: 91280 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:40:48,176-Speed 127.70 samples/sec Loss 10.4590 Epoch: 9 Global Step: 91285 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:41:08,102-Speed 128.48 samples/sec Loss 10.6243 Epoch: 9 Global Step: 91290 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:41:27,862-Speed 129.56 samples/sec Loss 10.8079 Epoch: 9 Global Step: 91295 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:41:47,666-Speed 129.28 samples/sec Loss 10.8779 Epoch: 9 Global Step: 91300 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:42:07,589-Speed 128.50 samples/sec Loss 10.6659 Epoch: 9 Global Step: 91305 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:42:27,518-Speed 128.46 samples/sec Loss 10.7970 Epoch: 9 Global Step: 91310 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:42:47,510-Speed 128.06 samples/sec Loss 10.5310 Epoch: 9 Global Step: 91315 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:43:07,260-Speed 129.63 samples/sec Loss 10.8982 Epoch: 9 Global Step: 91320 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:43:27,087-Speed 129.12 samples/sec Loss 10.4759 Epoch: 9 Global Step: 91325 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:43:47,166-Speed 127.51 samples/sec Loss 10.6204 Epoch: 9 Global Step: 91330 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:44:07,017-Speed 128.96 samples/sec Loss 10.9529 Epoch: 9 Global Step: 91335 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:44:26,844-Speed 129.13 samples/sec Loss 10.9674 Epoch: 9 Global Step: 91340 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:44:46,961-Speed 127.26 samples/sec Loss 10.6602 Epoch: 9 Global Step: 91345 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:45:06,724-Speed 129.54 samples/sec Loss 10.9956 Epoch: 9 Global Step: 91350 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:45:26,725-Speed 128.00 samples/sec Loss 10.4785 Epoch: 9 Global Step: 91355 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:45:46,711-Speed 128.10 samples/sec Loss 10.6395 Epoch: 9 Global Step: 91360 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:46:06,483-Speed 129.49 samples/sec Loss 10.7377 Epoch: 9 Global Step: 91365 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:46:26,418-Speed 128.42 samples/sec Loss 10.3477 Epoch: 9 Global Step: 91370 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:46:46,332-Speed 128.57 samples/sec Loss 10.8230 Epoch: 9 Global Step: 91375 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:47:06,273-Speed 128.38 samples/sec Loss 10.5781 Epoch: 9 Global Step: 91380 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:47:26,233-Speed 128.26 samples/sec Loss 10.7186 Epoch: 9 Global Step: 91385 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:47:46,229-Speed 128.03 samples/sec Loss 10.9073 Epoch: 9 Global Step: 91390 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:48:06,003-Speed 129.46 samples/sec Loss 10.4215 Epoch: 9 Global Step: 91395 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:48:26,083-Speed 127.50 samples/sec Loss 10.9823 Epoch: 9 Global Step: 91400 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:48:46,031-Speed 128.34 samples/sec Loss 11.0222 Epoch: 9 Global Step: 91405 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:49:05,797-Speed 129.52 samples/sec Loss 10.8603 Epoch: 9 Global Step: 91410 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:49:25,620-Speed 129.15 samples/sec Loss 10.4974 Epoch: 9 Global Step: 91415 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:49:45,833-Speed 126.66 samples/sec Loss 10.6414 Epoch: 9 Global Step: 91420 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:50:05,663-Speed 129.10 samples/sec Loss 10.5215 Epoch: 9 Global Step: 91425 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:50:25,489-Speed 129.13 samples/sec Loss 10.7135 Epoch: 9 Global Step: 91430 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:50:45,346-Speed 128.93 samples/sec Loss 10.5957 Epoch: 9 Global Step: 91435 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:51:05,070-Speed 129.80 samples/sec Loss 10.3548 Epoch: 9 Global Step: 91440 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:51:25,273-Speed 126.72 samples/sec Loss 11.1038 Epoch: 9 Global Step: 91445 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:51:45,025-Speed 129.61 samples/sec Loss 10.6059 Epoch: 9 Global Step: 91450 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:52:04,998-Speed 128.18 samples/sec Loss 10.6164 Epoch: 9 Global Step: 91455 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:52:24,732-Speed 129.73 samples/sec Loss 10.8866 Epoch: 9 Global Step: 91460 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:52:44,917-Speed 126.83 samples/sec Loss 10.3741 Epoch: 9 Global Step: 91465 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:53:04,683-Speed 129.52 samples/sec Loss 10.4044 Epoch: 9 Global Step: 91470 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:53:24,619-Speed 128.42 samples/sec Loss 10.6157 Epoch: 9 Global Step: 91475 Fp16 Grad Scale: 8192 Required: 188 hours Training: 2021-06-11 07:53:44,382-Speed 129.54 samples/sec Loss 10.8138 Epoch: 9 Global Step: 91480 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:54:04,573-Speed 126.79 samples/sec Loss 10.6523 Epoch: 9 Global Step: 91485 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:54:24,714-Speed 127.11 samples/sec Loss 10.4331 Epoch: 9 Global Step: 91490 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:54:44,477-Speed 129.54 samples/sec Loss 10.4364 Epoch: 9 Global Step: 91495 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:55:04,378-Speed 128.64 samples/sec Loss 10.9910 Epoch: 9 Global Step: 91500 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:55:24,078-Speed 129.96 samples/sec Loss 10.7142 Epoch: 9 Global Step: 91505 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:55:44,233-Speed 127.02 samples/sec Loss 10.6242 Epoch: 9 Global Step: 91510 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:56:04,044-Speed 129.23 samples/sec Loss 10.5899 Epoch: 9 Global Step: 91515 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:56:23,917-Speed 128.82 samples/sec Loss 10.7235 Epoch: 9 Global Step: 91520 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:56:43,674-Speed 129.58 samples/sec Loss 10.4828 Epoch: 9 Global Step: 91525 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:57:03,580-Speed 128.61 samples/sec Loss 11.0346 Epoch: 9 Global Step: 91530 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:57:23,677-Speed 127.39 samples/sec Loss 10.8963 Epoch: 9 Global Step: 91535 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:57:43,528-Speed 128.97 samples/sec Loss 11.0215 Epoch: 9 Global Step: 91540 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:58:03,331-Speed 129.28 samples/sec Loss 10.7758 Epoch: 9 Global Step: 91545 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:58:23,078-Speed 129.65 samples/sec Loss 10.4160 Epoch: 9 Global Step: 91550 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:58:43,210-Speed 127.17 samples/sec Loss 10.5517 Epoch: 9 Global Step: 91555 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:59:03,048-Speed 129.04 samples/sec Loss 10.3513 Epoch: 9 Global Step: 91560 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:59:22,877-Speed 129.11 samples/sec Loss 10.4997 Epoch: 9 Global Step: 91565 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 07:59:42,552-Speed 130.12 samples/sec Loss 10.7835 Epoch: 9 Global Step: 91570 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:00:02,712-Speed 126.99 samples/sec Loss 10.8530 Epoch: 9 Global Step: 91575 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:00:22,741-Speed 127.82 samples/sec Loss 10.6171 Epoch: 9 Global Step: 91580 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:00:42,579-Speed 129.05 samples/sec Loss 11.0530 Epoch: 9 Global Step: 91585 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:01:02,496-Speed 128.55 samples/sec Loss 10.7910 Epoch: 9 Global Step: 91590 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:01:22,376-Speed 128.78 samples/sec Loss 11.0323 Epoch: 9 Global Step: 91595 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:01:42,495-Speed 127.25 samples/sec Loss 10.7924 Epoch: 9 Global Step: 91600 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:02:02,464-Speed 128.20 samples/sec Loss 10.6206 Epoch: 9 Global Step: 91605 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:02:22,121-Speed 130.24 samples/sec Loss 10.5010 Epoch: 9 Global Step: 91610 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:02:41,936-Speed 129.20 samples/sec Loss 10.6525 Epoch: 9 Global Step: 91615 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:03:01,918-Speed 128.13 samples/sec Loss 10.9825 Epoch: 9 Global Step: 91620 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:03:21,962-Speed 127.72 samples/sec Loss 10.5743 Epoch: 9 Global Step: 91625 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:03:41,894-Speed 128.44 samples/sec Loss 10.8238 Epoch: 9 Global Step: 91630 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:04:01,639-Speed 129.66 samples/sec Loss 10.7907 Epoch: 9 Global Step: 91635 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:04:21,554-Speed 128.56 samples/sec Loss 11.0044 Epoch: 9 Global Step: 91640 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:04:41,805-Speed 126.42 samples/sec Loss 10.8884 Epoch: 9 Global Step: 91645 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:05:01,733-Speed 128.47 samples/sec Loss 10.7693 Epoch: 9 Global Step: 91650 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:05:21,392-Speed 130.23 samples/sec Loss 10.4110 Epoch: 9 Global Step: 91655 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:05:41,416-Speed 127.85 samples/sec Loss 10.5833 Epoch: 9 Global Step: 91660 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:06:01,243-Speed 129.12 samples/sec Loss 10.7359 Epoch: 9 Global Step: 91665 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:06:21,324-Speed 127.49 samples/sec Loss 10.6783 Epoch: 9 Global Step: 91670 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:06:41,132-Speed 129.25 samples/sec Loss 10.5012 Epoch: 9 Global Step: 91675 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:07:00,979-Speed 128.99 samples/sec Loss 10.7087 Epoch: 9 Global Step: 91680 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:07:21,044-Speed 127.59 samples/sec Loss 10.8299 Epoch: 9 Global Step: 91685 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:07:41,072-Speed 127.83 samples/sec Loss 11.0342 Epoch: 9 Global Step: 91690 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:08:00,951-Speed 128.79 samples/sec Loss 10.5151 Epoch: 9 Global Step: 91695 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:08:20,711-Speed 129.56 samples/sec Loss 10.5428 Epoch: 9 Global Step: 91700 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:08:40,794-Speed 127.47 samples/sec Loss 10.8712 Epoch: 9 Global Step: 91705 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:09:00,584-Speed 129.37 samples/sec Loss 10.4181 Epoch: 9 Global Step: 91710 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:09:20,999-Speed 125.40 samples/sec Loss 10.7019 Epoch: 9 Global Step: 91715 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:09:40,828-Speed 129.11 samples/sec Loss 10.5500 Epoch: 9 Global Step: 91720 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:10:00,620-Speed 129.35 samples/sec Loss 10.5720 Epoch: 9 Global Step: 91725 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:10:20,523-Speed 128.63 samples/sec Loss 10.4668 Epoch: 9 Global Step: 91730 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:10:40,661-Speed 127.13 samples/sec Loss 10.7837 Epoch: 9 Global Step: 91735 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:11:00,316-Speed 130.25 samples/sec Loss 11.0006 Epoch: 9 Global Step: 91740 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:11:20,294-Speed 128.14 samples/sec Loss 10.6004 Epoch: 9 Global Step: 91745 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:11:40,207-Speed 128.57 samples/sec Loss 10.7638 Epoch: 9 Global Step: 91750 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:11:59,985-Speed 129.44 samples/sec Loss 10.7026 Epoch: 9 Global Step: 91755 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:12:20,123-Speed 127.13 samples/sec Loss 10.5245 Epoch: 9 Global Step: 91760 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:12:39,980-Speed 128.93 samples/sec Loss 10.2207 Epoch: 9 Global Step: 91765 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:12:59,913-Speed 128.44 samples/sec Loss 10.6583 Epoch: 9 Global Step: 91770 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:13:19,674-Speed 129.55 samples/sec Loss 10.8653 Epoch: 9 Global Step: 91775 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:13:40,090-Speed 125.40 samples/sec Loss 10.7274 Epoch: 9 Global Step: 91780 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:13:59,841-Speed 129.62 samples/sec Loss 10.4410 Epoch: 9 Global Step: 91785 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:14:19,850-Speed 127.95 samples/sec Loss 10.7238 Epoch: 9 Global Step: 91790 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:14:39,604-Speed 129.60 samples/sec Loss 10.7197 Epoch: 9 Global Step: 91795 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:14:59,397-Speed 129.35 samples/sec Loss 10.4644 Epoch: 9 Global Step: 91800 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:15:19,375-Speed 128.14 samples/sec Loss 10.5743 Epoch: 9 Global Step: 91805 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:15:39,240-Speed 128.88 samples/sec Loss 10.8669 Epoch: 9 Global Step: 91810 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:15:59,200-Speed 128.26 samples/sec Loss 10.5957 Epoch: 9 Global Step: 91815 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:16:18,853-Speed 130.27 samples/sec Loss 10.5649 Epoch: 9 Global Step: 91820 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:16:38,914-Speed 127.62 samples/sec Loss 10.6067 Epoch: 9 Global Step: 91825 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:16:58,727-Speed 129.21 samples/sec Loss 10.5332 Epoch: 9 Global Step: 91830 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:17:18,745-Speed 127.89 samples/sec Loss 11.1585 Epoch: 9 Global Step: 91835 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:17:38,514-Speed 129.52 samples/sec Loss 10.8648 Epoch: 9 Global Step: 91840 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:17:58,268-Speed 129.60 samples/sec Loss 10.7440 Epoch: 9 Global Step: 91845 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:18:18,331-Speed 127.60 samples/sec Loss 10.4767 Epoch: 9 Global Step: 91850 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:18:38,171-Speed 129.04 samples/sec Loss 10.5903 Epoch: 9 Global Step: 91855 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:18:58,111-Speed 128.40 samples/sec Loss 10.5015 Epoch: 9 Global Step: 91860 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:19:17,790-Speed 130.09 samples/sec Loss 10.7045 Epoch: 9 Global Step: 91865 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:19:37,817-Speed 127.83 samples/sec Loss 10.3250 Epoch: 9 Global Step: 91870 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:19:57,755-Speed 128.41 samples/sec Loss 10.9514 Epoch: 9 Global Step: 91875 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:20:17,867-Speed 127.29 samples/sec Loss 10.5277 Epoch: 9 Global Step: 91880 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:20:37,650-Speed 129.41 samples/sec Loss 10.4675 Epoch: 9 Global Step: 91885 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:20:57,487-Speed 129.06 samples/sec Loss 10.4581 Epoch: 9 Global Step: 91890 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:21:17,516-Speed 127.82 samples/sec Loss 10.6078 Epoch: 9 Global Step: 91895 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:21:37,471-Speed 128.30 samples/sec Loss 10.6323 Epoch: 9 Global Step: 91900 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:21:57,140-Speed 130.16 samples/sec Loss 10.4840 Epoch: 9 Global Step: 91905 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:22:16,848-Speed 129.91 samples/sec Loss 10.6701 Epoch: 9 Global Step: 91910 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:22:36,900-Speed 127.67 samples/sec Loss 10.4100 Epoch: 9 Global Step: 91915 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:22:56,835-Speed 128.43 samples/sec Loss 10.7849 Epoch: 9 Global Step: 91920 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:23:16,589-Speed 129.60 samples/sec Loss 10.6647 Epoch: 9 Global Step: 91925 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:23:36,445-Speed 128.93 samples/sec Loss 10.6603 Epoch: 9 Global Step: 91930 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:23:56,119-Speed 130.13 samples/sec Loss 10.9377 Epoch: 9 Global Step: 91935 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:24:16,260-Speed 127.11 samples/sec Loss 10.5386 Epoch: 9 Global Step: 91940 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:24:36,230-Speed 128.20 samples/sec Loss 10.9790 Epoch: 9 Global Step: 91945 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:24:55,930-Speed 129.96 samples/sec Loss 10.6303 Epoch: 9 Global Step: 91950 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:25:15,764-Speed 129.08 samples/sec Loss 10.4688 Epoch: 9 Global Step: 91955 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:25:35,753-Speed 128.08 samples/sec Loss 10.5274 Epoch: 9 Global Step: 91960 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:25:55,672-Speed 128.52 samples/sec Loss 10.6644 Epoch: 9 Global Step: 91965 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:26:15,476-Speed 129.27 samples/sec Loss 10.8908 Epoch: 9 Global Step: 91970 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:26:35,335-Speed 128.92 samples/sec Loss 10.5093 Epoch: 9 Global Step: 91975 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:26:55,088-Speed 129.61 samples/sec Loss 10.5367 Epoch: 9 Global Step: 91980 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:27:15,397-Speed 126.06 samples/sec Loss 10.4315 Epoch: 9 Global Step: 91985 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:27:35,187-Speed 129.36 samples/sec Loss 10.7321 Epoch: 9 Global Step: 91990 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:27:54,968-Speed 129.42 samples/sec Loss 10.5705 Epoch: 9 Global Step: 91995 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:28:14,797-Speed 129.11 samples/sec Loss 10.8437 Epoch: 9 Global Step: 92000 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:28:34,940-Speed 127.10 samples/sec Loss 10.8551 Epoch: 9 Global Step: 92005 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:28:54,756-Speed 129.20 samples/sec Loss 10.8339 Epoch: 9 Global Step: 92010 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:29:14,592-Speed 129.06 samples/sec Loss 10.8930 Epoch: 9 Global Step: 92015 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:29:34,439-Speed 128.99 samples/sec Loss 10.7916 Epoch: 9 Global Step: 92020 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:29:54,215-Speed 129.46 samples/sec Loss 10.3405 Epoch: 9 Global Step: 92025 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:30:14,536-Speed 125.99 samples/sec Loss 10.6281 Epoch: 9 Global Step: 92030 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:30:34,299-Speed 129.54 samples/sec Loss 10.7611 Epoch: 9 Global Step: 92035 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:30:54,237-Speed 128.41 samples/sec Loss 10.7599 Epoch: 9 Global Step: 92040 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:31:13,987-Speed 129.62 samples/sec Loss 10.3949 Epoch: 9 Global Step: 92045 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:31:34,139-Speed 127.04 samples/sec Loss 10.3237 Epoch: 9 Global Step: 92050 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:31:53,889-Speed 129.62 samples/sec Loss 10.4249 Epoch: 9 Global Step: 92055 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:32:13,771-Speed 128.77 samples/sec Loss 10.7280 Epoch: 9 Global Step: 92060 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:32:33,530-Speed 129.57 samples/sec Loss 10.4333 Epoch: 9 Global Step: 92065 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:32:53,343-Speed 129.21 samples/sec Loss 10.6064 Epoch: 9 Global Step: 92070 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:33:13,513-Speed 126.93 samples/sec Loss 10.5493 Epoch: 9 Global Step: 92075 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:33:33,278-Speed 129.52 samples/sec Loss 10.5905 Epoch: 9 Global Step: 92080 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:33:53,401-Speed 127.23 samples/sec Loss 10.3807 Epoch: 9 Global Step: 92085 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:34:13,177-Speed 129.46 samples/sec Loss 10.5175 Epoch: 9 Global Step: 92090 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:34:33,383-Speed 126.70 samples/sec Loss 10.6883 Epoch: 9 Global Step: 92095 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:34:53,114-Speed 129.75 samples/sec Loss 10.5227 Epoch: 9 Global Step: 92100 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:35:13,018-Speed 128.62 samples/sec Loss 10.7498 Epoch: 9 Global Step: 92105 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:35:32,788-Speed 129.50 samples/sec Loss 10.7420 Epoch: 9 Global Step: 92110 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:35:52,624-Speed 129.07 samples/sec Loss 10.8312 Epoch: 9 Global Step: 92115 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:36:12,742-Speed 127.25 samples/sec Loss 10.5710 Epoch: 9 Global Step: 92120 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:36:32,503-Speed 129.56 samples/sec Loss 11.0660 Epoch: 9 Global Step: 92125 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:36:52,289-Speed 129.39 samples/sec Loss 10.6601 Epoch: 9 Global Step: 92130 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:37:12,068-Speed 129.44 samples/sec Loss 10.9091 Epoch: 9 Global Step: 92135 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:37:32,144-Speed 127.53 samples/sec Loss 10.6903 Epoch: 9 Global Step: 92140 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:37:52,090-Speed 128.35 samples/sec Loss 10.4476 Epoch: 9 Global Step: 92145 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:38:11,843-Speed 129.60 samples/sec Loss 10.7277 Epoch: 9 Global Step: 92150 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:38:31,640-Speed 129.32 samples/sec Loss 10.7252 Epoch: 9 Global Step: 92155 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:38:51,601-Speed 128.25 samples/sec Loss 10.9940 Epoch: 9 Global Step: 92160 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:39:11,647-Speed 127.71 samples/sec Loss 10.7674 Epoch: 9 Global Step: 92165 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:39:31,511-Speed 128.89 samples/sec Loss 10.6849 Epoch: 9 Global Step: 92170 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:39:51,273-Speed 129.55 samples/sec Loss 10.7298 Epoch: 9 Global Step: 92175 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:40:11,140-Speed 128.86 samples/sec Loss 10.6653 Epoch: 9 Global Step: 92180 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:40:31,238-Speed 127.39 samples/sec Loss 10.5018 Epoch: 9 Global Step: 92185 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:40:51,194-Speed 128.29 samples/sec Loss 10.5344 Epoch: 9 Global Step: 92190 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:41:10,873-Speed 130.09 samples/sec Loss 10.5470 Epoch: 9 Global Step: 92195 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:41:30,805-Speed 128.44 samples/sec Loss 10.5043 Epoch: 9 Global Step: 92200 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:41:50,620-Speed 129.20 samples/sec Loss 10.7422 Epoch: 9 Global Step: 92205 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:42:10,576-Speed 128.29 samples/sec Loss 10.4805 Epoch: 9 Global Step: 92210 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:42:30,514-Speed 128.41 samples/sec Loss 10.4009 Epoch: 9 Global Step: 92215 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:42:50,365-Speed 128.97 samples/sec Loss 10.3506 Epoch: 9 Global Step: 92220 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:43:10,661-Speed 126.14 samples/sec Loss 10.7856 Epoch: 9 Global Step: 92225 Fp16 Grad Scale: 8192 Required: 187 hours Training: 2021-06-11 08:43:30,712-Speed 127.68 samples/sec Loss 10.5305 Epoch: 9 Global Step: 92230 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:43:50,499-Speed 129.38 samples/sec Loss 10.6344 Epoch: 9 Global Step: 92235 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:44:10,225-Speed 129.78 samples/sec Loss 10.6345 Epoch: 9 Global Step: 92240 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:44:30,260-Speed 127.79 samples/sec Loss 10.5969 Epoch: 9 Global Step: 92245 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:44:50,006-Speed 129.65 samples/sec Loss 10.5358 Epoch: 9 Global Step: 92250 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:45:10,146-Speed 127.12 samples/sec Loss 10.6487 Epoch: 9 Global Step: 92255 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:45:29,969-Speed 129.14 samples/sec Loss 10.7200 Epoch: 9 Global Step: 92260 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:45:49,826-Speed 128.93 samples/sec Loss 10.7976 Epoch: 9 Global Step: 92265 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:46:09,664-Speed 129.05 samples/sec Loss 10.5303 Epoch: 9 Global Step: 92270 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:46:29,684-Speed 127.88 samples/sec Loss 10.5803 Epoch: 9 Global Step: 92275 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:46:49,448-Speed 129.53 samples/sec Loss 10.3924 Epoch: 9 Global Step: 92280 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:47:09,322-Speed 128.82 samples/sec Loss 10.9999 Epoch: 9 Global Step: 92285 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:47:29,240-Speed 128.53 samples/sec Loss 10.7212 Epoch: 9 Global Step: 92290 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:47:49,000-Speed 129.57 samples/sec Loss 10.6029 Epoch: 9 Global Step: 92295 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:48:09,077-Speed 127.51 samples/sec Loss 10.5644 Epoch: 9 Global Step: 92300 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:48:28,911-Speed 129.08 samples/sec Loss 10.8672 Epoch: 9 Global Step: 92305 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:48:48,766-Speed 128.95 samples/sec Loss 10.6720 Epoch: 9 Global Step: 92310 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:49:08,444-Speed 130.10 samples/sec Loss 10.8301 Epoch: 9 Global Step: 92315 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:49:28,638-Speed 126.78 samples/sec Loss 10.7953 Epoch: 9 Global Step: 92320 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:49:48,435-Speed 129.32 samples/sec Loss 10.4736 Epoch: 9 Global Step: 92325 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:50:08,364-Speed 128.46 samples/sec Loss 10.3344 Epoch: 9 Global Step: 92330 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:50:28,917-Speed 124.56 samples/sec Loss 10.8265 Epoch: 9 Global Step: 92335 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:50:48,650-Speed 129.74 samples/sec Loss 10.9359 Epoch: 9 Global Step: 92340 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:51:08,862-Speed 126.66 samples/sec Loss 10.4915 Epoch: 9 Global Step: 92345 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:51:28,627-Speed 129.53 samples/sec Loss 10.7479 Epoch: 9 Global Step: 92350 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:51:48,551-Speed 128.50 samples/sec Loss 10.4853 Epoch: 9 Global Step: 92355 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:52:08,313-Speed 129.55 samples/sec Loss 10.6224 Epoch: 9 Global Step: 92360 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:52:28,476-Speed 126.97 samples/sec Loss 10.6584 Epoch: 9 Global Step: 92365 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:52:48,266-Speed 129.36 samples/sec Loss 10.8485 Epoch: 9 Global Step: 92370 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:53:08,200-Speed 128.43 samples/sec Loss 10.7966 Epoch: 9 Global Step: 92375 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:53:27,945-Speed 129.66 samples/sec Loss 10.4069 Epoch: 9 Global Step: 92380 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:53:47,904-Speed 128.27 samples/sec Loss 10.8036 Epoch: 9 Global Step: 92385 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:54:08,032-Speed 127.20 samples/sec Loss 10.6282 Epoch: 9 Global Step: 92390 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:54:27,826-Speed 129.34 samples/sec Loss 10.7179 Epoch: 9 Global Step: 92395 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:54:47,773-Speed 128.34 samples/sec Loss 10.6975 Epoch: 9 Global Step: 92400 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:55:07,460-Speed 130.05 samples/sec Loss 10.4816 Epoch: 9 Global Step: 92405 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:55:27,626-Speed 126.95 samples/sec Loss 10.6686 Epoch: 9 Global Step: 92410 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:55:47,490-Speed 128.88 samples/sec Loss 10.6135 Epoch: 9 Global Step: 92415 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:56:07,330-Speed 129.04 samples/sec Loss 10.7681 Epoch: 9 Global Step: 92420 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:56:27,117-Speed 129.39 samples/sec Loss 10.5491 Epoch: 9 Global Step: 92425 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:56:46,978-Speed 128.90 samples/sec Loss 10.6027 Epoch: 9 Global Step: 92430 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:57:07,095-Speed 127.26 samples/sec Loss 10.7820 Epoch: 9 Global Step: 92435 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:57:27,005-Speed 128.59 samples/sec Loss 10.5921 Epoch: 9 Global Step: 92440 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:57:46,745-Speed 129.69 samples/sec Loss 10.9547 Epoch: 9 Global Step: 92445 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:58:06,559-Speed 129.21 samples/sec Loss 10.9633 Epoch: 9 Global Step: 92450 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:58:26,731-Speed 126.91 samples/sec Loss 10.5629 Epoch: 9 Global Step: 92455 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:58:46,749-Speed 127.89 samples/sec Loss 10.7493 Epoch: 9 Global Step: 92460 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:59:06,452-Speed 129.93 samples/sec Loss 10.7287 Epoch: 9 Global Step: 92465 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:59:26,324-Speed 128.83 samples/sec Loss 10.7967 Epoch: 9 Global Step: 92470 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 08:59:46,006-Speed 130.07 samples/sec Loss 10.8048 Epoch: 9 Global Step: 92475 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:00:06,165-Speed 126.99 samples/sec Loss 10.5810 Epoch: 9 Global Step: 92480 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:00:26,077-Speed 128.58 samples/sec Loss 10.7985 Epoch: 9 Global Step: 92485 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:00:45,831-Speed 129.60 samples/sec Loss 10.4138 Epoch: 9 Global Step: 92490 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:01:05,758-Speed 128.48 samples/sec Loss 10.5293 Epoch: 9 Global Step: 92495 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:01:25,826-Speed 127.57 samples/sec Loss 10.5625 Epoch: 9 Global Step: 92500 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:01:45,682-Speed 128.94 samples/sec Loss 10.7704 Epoch: 9 Global Step: 92505 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:02:05,394-Speed 129.87 samples/sec Loss 10.6379 Epoch: 9 Global Step: 92510 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:02:25,331-Speed 128.41 samples/sec Loss 10.8078 Epoch: 9 Global Step: 92515 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:02:45,110-Speed 129.43 samples/sec Loss 10.4494 Epoch: 9 Global Step: 92520 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:03:05,176-Speed 127.58 samples/sec Loss 10.7405 Epoch: 9 Global Step: 92525 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:03:25,060-Speed 128.75 samples/sec Loss 10.6382 Epoch: 9 Global Step: 92530 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:03:44,840-Speed 129.44 samples/sec Loss 10.9043 Epoch: 9 Global Step: 92535 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:04:04,757-Speed 128.54 samples/sec Loss 10.8862 Epoch: 9 Global Step: 92540 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:04:24,817-Speed 127.62 samples/sec Loss 10.5050 Epoch: 9 Global Step: 92545 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:04:44,627-Speed 129.23 samples/sec Loss 10.5739 Epoch: 9 Global Step: 92550 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:05:04,353-Speed 129.79 samples/sec Loss 10.7252 Epoch: 9 Global Step: 92555 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:05:24,300-Speed 128.34 samples/sec Loss 10.3946 Epoch: 9 Global Step: 92560 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:05:44,052-Speed 129.62 samples/sec Loss 11.3373 Epoch: 9 Global Step: 92565 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:06:04,347-Speed 126.14 samples/sec Loss 11.0236 Epoch: 9 Global Step: 92570 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:06:24,047-Speed 129.96 samples/sec Loss 10.7999 Epoch: 9 Global Step: 92575 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:06:43,877-Speed 129.10 samples/sec Loss 10.3218 Epoch: 9 Global Step: 92580 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:07:03,754-Speed 128.81 samples/sec Loss 10.3718 Epoch: 9 Global Step: 92585 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:07:23,910-Speed 127.02 samples/sec Loss 10.7785 Epoch: 9 Global Step: 92590 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:07:43,671-Speed 129.55 samples/sec Loss 10.6098 Epoch: 9 Global Step: 92595 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:08:03,753-Speed 127.48 samples/sec Loss 10.5962 Epoch: 9 Global Step: 92600 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:08:23,441-Speed 130.04 samples/sec Loss 10.4592 Epoch: 9 Global Step: 92605 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:08:43,238-Speed 129.32 samples/sec Loss 11.1524 Epoch: 9 Global Step: 92610 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:09:03,417-Speed 126.87 samples/sec Loss 10.8722 Epoch: 9 Global Step: 92615 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:09:23,181-Speed 129.54 samples/sec Loss 10.8020 Epoch: 9 Global Step: 92620 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:09:43,123-Speed 128.38 samples/sec Loss 10.9107 Epoch: 9 Global Step: 92625 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:10:02,826-Speed 129.94 samples/sec Loss 10.7114 Epoch: 9 Global Step: 92630 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:10:22,916-Speed 127.44 samples/sec Loss 10.5870 Epoch: 9 Global Step: 92635 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:10:42,752-Speed 129.06 samples/sec Loss 10.3452 Epoch: 9 Global Step: 92640 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:11:02,579-Speed 129.12 samples/sec Loss 10.9930 Epoch: 9 Global Step: 92645 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:11:22,327-Speed 129.64 samples/sec Loss 10.8353 Epoch: 9 Global Step: 92650 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:11:42,298-Speed 128.20 samples/sec Loss 10.7935 Epoch: 9 Global Step: 92655 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:12:02,327-Speed 127.83 samples/sec Loss 10.4908 Epoch: 9 Global Step: 92660 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:12:22,164-Speed 129.06 samples/sec Loss 10.4014 Epoch: 9 Global Step: 92665 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:12:42,287-Speed 127.22 samples/sec Loss 10.6080 Epoch: 9 Global Step: 92670 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:13:01,980-Speed 130.00 samples/sec Loss 10.6371 Epoch: 9 Global Step: 92675 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:13:22,078-Speed 127.38 samples/sec Loss 10.7572 Epoch: 9 Global Step: 92680 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:13:41,889-Speed 129.23 samples/sec Loss 10.6957 Epoch: 9 Global Step: 92685 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:14:01,691-Speed 129.28 samples/sec Loss 10.9462 Epoch: 9 Global Step: 92690 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:14:21,478-Speed 129.39 samples/sec Loss 10.3023 Epoch: 9 Global Step: 92695 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:14:41,327-Speed 128.98 samples/sec Loss 10.8070 Epoch: 9 Global Step: 92700 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:15:01,395-Speed 127.57 samples/sec Loss 11.2198 Epoch: 9 Global Step: 92705 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:15:21,212-Speed 129.19 samples/sec Loss 10.9672 Epoch: 9 Global Step: 92710 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:15:41,149-Speed 128.41 samples/sec Loss 10.4526 Epoch: 9 Global Step: 92715 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:16:00,799-Speed 130.29 samples/sec Loss 10.6514 Epoch: 9 Global Step: 92720 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:16:20,989-Speed 126.81 samples/sec Loss 10.6725 Epoch: 9 Global Step: 92725 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:16:40,937-Speed 128.34 samples/sec Loss 10.3454 Epoch: 9 Global Step: 92730 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:17:00,705-Speed 129.51 samples/sec Loss 10.7562 Epoch: 9 Global Step: 92735 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:17:20,565-Speed 128.91 samples/sec Loss 10.8744 Epoch: 9 Global Step: 92740 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:17:40,432-Speed 128.86 samples/sec Loss 10.8991 Epoch: 9 Global Step: 92745 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:18:00,541-Speed 127.32 samples/sec Loss 10.4622 Epoch: 9 Global Step: 92750 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:18:20,480-Speed 128.40 samples/sec Loss 10.7666 Epoch: 9 Global Step: 92755 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:18:40,156-Speed 130.12 samples/sec Loss 10.4679 Epoch: 9 Global Step: 92760 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:18:59,952-Speed 129.33 samples/sec Loss 10.5750 Epoch: 9 Global Step: 92765 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:19:20,119-Speed 126.94 samples/sec Loss 10.8068 Epoch: 9 Global Step: 92770 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:19:40,082-Speed 128.25 samples/sec Loss 10.9254 Epoch: 9 Global Step: 92775 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:19:59,792-Speed 129.89 samples/sec Loss 11.0050 Epoch: 9 Global Step: 92780 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:20:19,724-Speed 128.44 samples/sec Loss 10.7463 Epoch: 9 Global Step: 92785 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:20:39,499-Speed 129.46 samples/sec Loss 10.8360 Epoch: 9 Global Step: 92790 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:20:59,589-Speed 127.44 samples/sec Loss 10.8637 Epoch: 9 Global Step: 92795 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:21:19,399-Speed 129.23 samples/sec Loss 10.7066 Epoch: 9 Global Step: 92800 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:21:39,165-Speed 129.53 samples/sec Loss 10.8767 Epoch: 9 Global Step: 92805 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:21:59,194-Speed 127.82 samples/sec Loss 10.9788 Epoch: 9 Global Step: 92810 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:22:19,222-Speed 127.83 samples/sec Loss 10.4985 Epoch: 9 Global Step: 92815 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:22:39,073-Speed 128.96 samples/sec Loss 10.7117 Epoch: 9 Global Step: 92820 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:22:58,817-Speed 129.67 samples/sec Loss 10.5803 Epoch: 9 Global Step: 92825 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:23:18,805-Speed 128.08 samples/sec Loss 10.6012 Epoch: 9 Global Step: 92830 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:23:38,561-Speed 129.58 samples/sec Loss 10.7085 Epoch: 9 Global Step: 92835 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:23:58,796-Speed 126.52 samples/sec Loss 10.8538 Epoch: 9 Global Step: 92840 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:24:18,538-Speed 129.68 samples/sec Loss 10.5846 Epoch: 9 Global Step: 92845 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:24:38,392-Speed 128.95 samples/sec Loss 10.9189 Epoch: 9 Global Step: 92850 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:24:58,287-Speed 128.68 samples/sec Loss 10.8792 Epoch: 9 Global Step: 92855 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:25:18,370-Speed 127.48 samples/sec Loss 10.7932 Epoch: 9 Global Step: 92860 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:25:38,093-Speed 129.80 samples/sec Loss 10.9253 Epoch: 9 Global Step: 92865 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:25:57,944-Speed 128.96 samples/sec Loss 10.4568 Epoch: 9 Global Step: 92870 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:26:17,786-Speed 129.03 samples/sec Loss 10.3681 Epoch: 9 Global Step: 92875 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:26:37,595-Speed 129.24 samples/sec Loss 10.8729 Epoch: 9 Global Step: 92880 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:26:57,778-Speed 126.84 samples/sec Loss 10.3830 Epoch: 9 Global Step: 92885 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:27:17,597-Speed 129.18 samples/sec Loss 10.4928 Epoch: 9 Global Step: 92890 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:27:37,453-Speed 128.93 samples/sec Loss 10.5988 Epoch: 9 Global Step: 92895 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:27:57,239-Speed 129.40 samples/sec Loss 10.8897 Epoch: 9 Global Step: 92900 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:28:17,415-Speed 126.89 samples/sec Loss 10.4784 Epoch: 9 Global Step: 92905 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:28:37,227-Speed 129.22 samples/sec Loss 10.6066 Epoch: 9 Global Step: 92910 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:28:57,248-Speed 127.88 samples/sec Loss 10.5519 Epoch: 9 Global Step: 92915 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:29:17,030-Speed 129.42 samples/sec Loss 10.6611 Epoch: 9 Global Step: 92920 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:29:36,825-Speed 129.33 samples/sec Loss 10.5269 Epoch: 9 Global Step: 92925 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:29:57,000-Speed 126.89 samples/sec Loss 10.8193 Epoch: 9 Global Step: 92930 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:30:16,938-Speed 128.40 samples/sec Loss 11.0429 Epoch: 9 Global Step: 92935 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:30:36,890-Speed 128.32 samples/sec Loss 10.5282 Epoch: 9 Global Step: 92940 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:30:56,693-Speed 129.28 samples/sec Loss 10.9825 Epoch: 9 Global Step: 92945 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:31:16,697-Speed 127.98 samples/sec Loss 10.1064 Epoch: 9 Global Step: 92950 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:31:36,562-Speed 128.87 samples/sec Loss 10.7153 Epoch: 9 Global Step: 92955 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:31:56,453-Speed 128.71 samples/sec Loss 10.8944 Epoch: 9 Global Step: 92960 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:32:16,232-Speed 129.44 samples/sec Loss 10.3908 Epoch: 9 Global Step: 92965 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:32:36,215-Speed 128.11 samples/sec Loss 10.7786 Epoch: 9 Global Step: 92970 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:32:56,548-Speed 125.91 samples/sec Loss 11.0632 Epoch: 9 Global Step: 92975 Fp16 Grad Scale: 8192 Required: 186 hours Training: 2021-06-11 09:33:16,354-Speed 129.26 samples/sec Loss 10.8240 Epoch: 9 Global Step: 92980 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:33:36,319-Speed 128.23 samples/sec Loss 10.8210 Epoch: 9 Global Step: 92985 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:33:57,020-Speed 123.68 samples/sec Loss 10.7278 Epoch: 9 Global Step: 92990 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:34:17,170-Speed 127.06 samples/sec Loss 10.5690 Epoch: 9 Global Step: 92995 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:34:37,103-Speed 128.43 samples/sec Loss 10.6667 Epoch: 9 Global Step: 93000 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:34:56,876-Speed 129.47 samples/sec Loss 10.8066 Epoch: 9 Global Step: 93005 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:35:16,723-Speed 128.99 samples/sec Loss 10.4930 Epoch: 9 Global Step: 93010 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:35:36,520-Speed 129.32 samples/sec Loss 10.8860 Epoch: 9 Global Step: 93015 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:35:57,158-Speed 124.05 samples/sec Loss 10.5998 Epoch: 9 Global Step: 93020 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:36:17,128-Speed 128.20 samples/sec Loss 10.6236 Epoch: 9 Global Step: 93025 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:36:36,856-Speed 129.77 samples/sec Loss 10.6838 Epoch: 9 Global Step: 93030 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:36:56,720-Speed 128.88 samples/sec Loss 10.5240 Epoch: 9 Global Step: 93035 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:37:16,690-Speed 128.20 samples/sec Loss 10.3109 Epoch: 9 Global Step: 93040 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:37:36,597-Speed 128.60 samples/sec Loss 10.4791 Epoch: 9 Global Step: 93045 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:37:56,369-Speed 129.48 samples/sec Loss 10.5733 Epoch: 9 Global Step: 93050 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:38:16,284-Speed 128.55 samples/sec Loss 10.8821 Epoch: 9 Global Step: 93055 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:38:35,946-Speed 130.21 samples/sec Loss 11.0791 Epoch: 9 Global Step: 93060 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:38:56,361-Speed 125.40 samples/sec Loss 10.6878 Epoch: 9 Global Step: 93065 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:39:16,138-Speed 129.45 samples/sec Loss 10.8217 Epoch: 9 Global Step: 93070 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:39:35,867-Speed 129.77 samples/sec Loss 11.0530 Epoch: 9 Global Step: 93075 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:39:55,785-Speed 128.53 samples/sec Loss 10.6632 Epoch: 9 Global Step: 93080 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:40:15,880-Speed 127.40 samples/sec Loss 10.6346 Epoch: 9 Global Step: 93085 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:40:35,659-Speed 129.43 samples/sec Loss 10.8227 Epoch: 9 Global Step: 93090 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:40:55,438-Speed 129.44 samples/sec Loss 10.3868 Epoch: 9 Global Step: 93095 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:41:15,200-Speed 129.55 samples/sec Loss 10.5338 Epoch: 9 Global Step: 93100 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:41:34,966-Speed 129.52 samples/sec Loss 10.4828 Epoch: 9 Global Step: 93105 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:41:55,155-Speed 126.80 samples/sec Loss 10.8433 Epoch: 9 Global Step: 93110 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:42:14,836-Speed 130.08 samples/sec Loss 10.7082 Epoch: 9 Global Step: 93115 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:42:34,763-Speed 128.48 samples/sec Loss 10.8293 Epoch: 9 Global Step: 93120 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:42:54,427-Speed 130.20 samples/sec Loss 10.8113 Epoch: 9 Global Step: 93125 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:43:14,907-Speed 125.00 samples/sec Loss 10.5490 Epoch: 9 Global Step: 93130 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:43:34,666-Speed 129.57 samples/sec Loss 10.4205 Epoch: 9 Global Step: 93135 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:43:54,624-Speed 128.27 samples/sec Loss 10.6531 Epoch: 9 Global Step: 93140 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:44:14,294-Speed 130.16 samples/sec Loss 10.8691 Epoch: 9 Global Step: 93145 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:44:34,010-Speed 129.85 samples/sec Loss 10.5828 Epoch: 9 Global Step: 93150 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:44:54,169-Speed 126.99 samples/sec Loss 10.9466 Epoch: 9 Global Step: 93155 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:45:13,951-Speed 129.42 samples/sec Loss 10.6208 Epoch: 9 Global Step: 93160 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:45:33,790-Speed 129.05 samples/sec Loss 11.0076 Epoch: 9 Global Step: 93165 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:45:53,597-Speed 129.25 samples/sec Loss 10.8275 Epoch: 9 Global Step: 93170 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:46:13,740-Speed 127.10 samples/sec Loss 11.0667 Epoch: 9 Global Step: 93175 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:46:33,521-Speed 129.42 samples/sec Loss 10.6557 Epoch: 9 Global Step: 93180 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:46:53,363-Speed 129.03 samples/sec Loss 10.3902 Epoch: 9 Global Step: 93185 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:47:13,047-Speed 130.06 samples/sec Loss 10.5447 Epoch: 9 Global Step: 93190 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:47:33,112-Speed 127.59 samples/sec Loss 10.6846 Epoch: 9 Global Step: 93195 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:47:53,155-Speed 127.73 samples/sec Loss 10.5779 Epoch: 9 Global Step: 93200 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:48:12,888-Speed 129.74 samples/sec Loss 10.6494 Epoch: 9 Global Step: 93205 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:48:32,696-Speed 129.25 samples/sec Loss 10.5715 Epoch: 9 Global Step: 93210 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:48:52,524-Speed 129.11 samples/sec Loss 10.8492 Epoch: 9 Global Step: 93215 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:49:12,627-Speed 127.35 samples/sec Loss 10.5630 Epoch: 9 Global Step: 93220 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:49:32,560-Speed 128.44 samples/sec Loss 10.7997 Epoch: 9 Global Step: 93225 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:49:52,228-Speed 130.17 samples/sec Loss 10.5510 Epoch: 9 Global Step: 93230 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:50:12,026-Speed 129.31 samples/sec Loss 10.8811 Epoch: 9 Global Step: 93235 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:50:31,963-Speed 128.42 samples/sec Loss 10.5909 Epoch: 9 Global Step: 93240 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:50:52,794-Speed 122.90 samples/sec Loss 10.5968 Epoch: 9 Global Step: 93245 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:51:12,649-Speed 128.94 samples/sec Loss 10.5367 Epoch: 9 Global Step: 93250 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:51:32,400-Speed 129.62 samples/sec Loss 10.6134 Epoch: 9 Global Step: 93255 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:51:52,775-Speed 125.65 samples/sec Loss 10.7513 Epoch: 9 Global Step: 93260 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:52:12,731-Speed 128.29 samples/sec Loss 10.8244 Epoch: 9 Global Step: 93265 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:52:32,588-Speed 128.93 samples/sec Loss 10.7719 Epoch: 9 Global Step: 93270 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:52:52,421-Speed 129.08 samples/sec Loss 10.8423 Epoch: 9 Global Step: 93275 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:53:12,443-Speed 127.87 samples/sec Loss 11.1108 Epoch: 9 Global Step: 93280 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:53:32,159-Speed 129.85 samples/sec Loss 10.9207 Epoch: 9 Global Step: 93285 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:53:52,146-Speed 128.09 samples/sec Loss 10.9391 Epoch: 9 Global Step: 93290 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:54:12,041-Speed 128.68 samples/sec Loss 10.9566 Epoch: 9 Global Step: 93295 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:54:31,829-Speed 129.38 samples/sec Loss 10.3918 Epoch: 9 Global Step: 93300 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:54:51,755-Speed 128.48 samples/sec Loss 10.6866 Epoch: 9 Global Step: 93305 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:55:11,629-Speed 128.82 samples/sec Loss 11.0282 Epoch: 9 Global Step: 93310 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:55:31,477-Speed 128.99 samples/sec Loss 10.4283 Epoch: 9 Global Step: 93315 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:55:51,305-Speed 129.12 samples/sec Loss 10.7458 Epoch: 9 Global Step: 93320 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:56:11,400-Speed 127.40 samples/sec Loss 10.7490 Epoch: 9 Global Step: 93325 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:56:31,242-Speed 129.03 samples/sec Loss 10.4903 Epoch: 9 Global Step: 93330 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:56:51,321-Speed 127.50 samples/sec Loss 10.7056 Epoch: 9 Global Step: 93335 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:57:11,112-Speed 129.36 samples/sec Loss 10.3907 Epoch: 9 Global Step: 93340 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:57:30,877-Speed 129.53 samples/sec Loss 10.5684 Epoch: 9 Global Step: 93345 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:57:50,711-Speed 129.08 samples/sec Loss 10.7025 Epoch: 9 Global Step: 93350 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:58:10,731-Speed 127.88 samples/sec Loss 10.2754 Epoch: 9 Global Step: 93355 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:58:30,572-Speed 129.03 samples/sec Loss 10.5371 Epoch: 9 Global Step: 93360 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:58:50,368-Speed 129.33 samples/sec Loss 10.7228 Epoch: 9 Global Step: 93365 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:59:10,285-Speed 128.54 samples/sec Loss 10.8721 Epoch: 9 Global Step: 93370 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:59:29,950-Speed 130.18 samples/sec Loss 10.6592 Epoch: 9 Global Step: 93375 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 09:59:50,163-Speed 126.65 samples/sec Loss 10.6200 Epoch: 9 Global Step: 93380 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:00:09,916-Speed 129.61 samples/sec Loss 10.7291 Epoch: 9 Global Step: 93385 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:00:29,859-Speed 128.37 samples/sec Loss 10.5960 Epoch: 9 Global Step: 93390 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:00:49,533-Speed 130.13 samples/sec Loss 10.7446 Epoch: 9 Global Step: 93395 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:01:09,754-Speed 126.61 samples/sec Loss 10.6959 Epoch: 9 Global Step: 93400 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:01:29,536-Speed 129.42 samples/sec Loss 10.6772 Epoch: 9 Global Step: 93405 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:01:49,476-Speed 128.39 samples/sec Loss 10.4419 Epoch: 9 Global Step: 93410 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:02:09,147-Speed 130.16 samples/sec Loss 10.7505 Epoch: 9 Global Step: 93415 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:02:28,946-Speed 129.30 samples/sec Loss 10.8198 Epoch: 9 Global Step: 93420 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:02:49,216-Speed 126.31 samples/sec Loss 10.5734 Epoch: 9 Global Step: 93425 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:03:08,980-Speed 129.53 samples/sec Loss 10.7689 Epoch: 9 Global Step: 93430 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:03:28,821-Speed 129.03 samples/sec Loss 10.6251 Epoch: 9 Global Step: 93435 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:03:48,560-Speed 129.70 samples/sec Loss 10.8717 Epoch: 9 Global Step: 93440 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:04:08,698-Speed 127.13 samples/sec Loss 11.0642 Epoch: 9 Global Step: 93445 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:04:28,554-Speed 128.94 samples/sec Loss 10.8915 Epoch: 9 Global Step: 93450 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:04:48,465-Speed 128.58 samples/sec Loss 10.7646 Epoch: 9 Global Step: 93455 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:05:08,134-Speed 130.16 samples/sec Loss 10.5296 Epoch: 9 Global Step: 93460 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:05:28,094-Speed 128.26 samples/sec Loss 10.9500 Epoch: 9 Global Step: 93465 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:05:48,116-Speed 127.87 samples/sec Loss 10.5466 Epoch: 9 Global Step: 93470 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:06:08,060-Speed 128.36 samples/sec Loss 10.9588 Epoch: 9 Global Step: 93475 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:06:27,802-Speed 129.68 samples/sec Loss 11.0515 Epoch: 9 Global Step: 93480 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:06:47,623-Speed 129.16 samples/sec Loss 10.9036 Epoch: 9 Global Step: 93485 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:07:07,888-Speed 126.33 samples/sec Loss 10.7085 Epoch: 9 Global Step: 93490 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:07:27,809-Speed 128.51 samples/sec Loss 10.5232 Epoch: 9 Global Step: 93495 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:07:47,518-Speed 129.89 samples/sec Loss 10.8186 Epoch: 9 Global Step: 93500 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:08:07,455-Speed 128.42 samples/sec Loss 10.7292 Epoch: 9 Global Step: 93505 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:08:27,340-Speed 128.75 samples/sec Loss 10.5710 Epoch: 9 Global Step: 93510 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:08:47,498-Speed 127.01 samples/sec Loss 10.6283 Epoch: 9 Global Step: 93515 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:09:07,332-Speed 129.08 samples/sec Loss 10.4620 Epoch: 9 Global Step: 93520 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:09:26,982-Speed 130.29 samples/sec Loss 10.8468 Epoch: 9 Global Step: 93525 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:09:47,002-Speed 127.88 samples/sec Loss 10.6656 Epoch: 9 Global Step: 93530 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:10:07,093-Speed 127.43 samples/sec Loss 10.7243 Epoch: 9 Global Step: 93535 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:10:26,866-Speed 129.47 samples/sec Loss 10.8537 Epoch: 9 Global Step: 93540 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:10:46,617-Speed 129.62 samples/sec Loss 10.7676 Epoch: 9 Global Step: 93545 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:11:06,588-Speed 128.20 samples/sec Loss 10.7151 Epoch: 9 Global Step: 93550 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:11:26,377-Speed 129.37 samples/sec Loss 10.8476 Epoch: 9 Global Step: 93555 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:11:46,561-Speed 126.84 samples/sec Loss 10.4117 Epoch: 9 Global Step: 93560 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:12:06,322-Speed 129.55 samples/sec Loss 10.8230 Epoch: 9 Global Step: 93565 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:12:26,212-Speed 128.72 samples/sec Loss 10.7241 Epoch: 9 Global Step: 93570 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:12:46,070-Speed 128.92 samples/sec Loss 11.2312 Epoch: 9 Global Step: 93575 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:13:06,198-Speed 127.19 samples/sec Loss 10.6468 Epoch: 9 Global Step: 93580 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:13:25,945-Speed 129.65 samples/sec Loss 10.6179 Epoch: 9 Global Step: 93585 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:13:45,882-Speed 128.41 samples/sec Loss 10.6307 Epoch: 9 Global Step: 93590 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:14:05,682-Speed 129.30 samples/sec Loss 10.6227 Epoch: 9 Global Step: 93595 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:14:25,410-Speed 129.77 samples/sec Loss 10.5717 Epoch: 9 Global Step: 93600 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:14:45,583-Speed 126.91 samples/sec Loss 10.5505 Epoch: 9 Global Step: 93605 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:15:05,431-Speed 128.99 samples/sec Loss 10.7233 Epoch: 9 Global Step: 93610 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:15:25,282-Speed 128.97 samples/sec Loss 10.8375 Epoch: 9 Global Step: 93615 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:15:45,058-Speed 129.46 samples/sec Loss 10.7743 Epoch: 9 Global Step: 93620 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:16:05,169-Speed 127.30 samples/sec Loss 10.8357 Epoch: 9 Global Step: 93625 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:16:24,996-Speed 129.12 samples/sec Loss 10.7348 Epoch: 9 Global Step: 93630 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:16:44,927-Speed 128.45 samples/sec Loss 10.6034 Epoch: 9 Global Step: 93635 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:17:04,716-Speed 129.37 samples/sec Loss 11.0923 Epoch: 9 Global Step: 93640 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:17:24,467-Speed 129.62 samples/sec Loss 10.2710 Epoch: 9 Global Step: 93645 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:17:44,743-Speed 126.26 samples/sec Loss 10.7584 Epoch: 9 Global Step: 93650 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:18:04,422-Speed 130.10 samples/sec Loss 10.6011 Epoch: 9 Global Step: 93655 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:18:24,369-Speed 128.35 samples/sec Loss 10.6774 Epoch: 9 Global Step: 93660 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:18:44,033-Speed 130.20 samples/sec Loss 10.7712 Epoch: 9 Global Step: 93665 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:19:04,160-Speed 127.20 samples/sec Loss 10.7118 Epoch: 9 Global Step: 93670 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:19:23,982-Speed 129.15 samples/sec Loss 10.7207 Epoch: 9 Global Step: 93675 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:19:43,872-Speed 128.72 samples/sec Loss 10.7947 Epoch: 9 Global Step: 93680 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:20:03,636-Speed 129.53 samples/sec Loss 10.5999 Epoch: 9 Global Step: 93685 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:20:23,578-Speed 128.37 samples/sec Loss 10.7986 Epoch: 9 Global Step: 93690 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:20:43,611-Speed 127.80 samples/sec Loss 10.9245 Epoch: 9 Global Step: 93695 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:21:03,507-Speed 128.68 samples/sec Loss 10.6903 Epoch: 9 Global Step: 93700 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:21:23,395-Speed 128.73 samples/sec Loss 10.3693 Epoch: 9 Global Step: 93705 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:21:43,192-Speed 129.32 samples/sec Loss 10.5939 Epoch: 9 Global Step: 93710 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:22:03,321-Speed 127.19 samples/sec Loss 10.7088 Epoch: 9 Global Step: 93715 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:22:23,158-Speed 129.06 samples/sec Loss 10.5052 Epoch: 9 Global Step: 93720 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:22:43,083-Speed 128.49 samples/sec Loss 10.9582 Epoch: 9 Global Step: 93725 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:23:02,888-Speed 129.27 samples/sec Loss 10.5571 Epoch: 9 Global Step: 93730 Fp16 Grad Scale: 8192 Required: 185 hours Training: 2021-06-11 10:23:22,893-Speed 127.98 samples/sec Loss 10.5611 Epoch: 9 Global Step: 93735 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:23:42,770-Speed 128.80 samples/sec Loss 10.6508 Epoch: 9 Global Step: 93740 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:24:02,801-Speed 127.81 samples/sec Loss 10.7376 Epoch: 9 Global Step: 93745 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:24:22,558-Speed 129.58 samples/sec Loss 10.6332 Epoch: 9 Global Step: 93750 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:24:42,432-Speed 128.81 samples/sec Loss 10.7894 Epoch: 9 Global Step: 93755 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:25:02,531-Speed 127.38 samples/sec Loss 10.6210 Epoch: 9 Global Step: 93760 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:25:22,430-Speed 128.65 samples/sec Loss 10.5259 Epoch: 9 Global Step: 93765 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:25:42,190-Speed 129.56 samples/sec Loss 10.5538 Epoch: 9 Global Step: 93770 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:26:02,104-Speed 128.56 samples/sec Loss 10.8554 Epoch: 9 Global Step: 93775 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:26:21,922-Speed 129.18 samples/sec Loss 10.7485 Epoch: 9 Global Step: 93780 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:26:41,920-Speed 128.02 samples/sec Loss 10.5686 Epoch: 9 Global Step: 93785 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:27:01,892-Speed 128.19 samples/sec Loss 10.7220 Epoch: 9 Global Step: 93790 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:27:21,677-Speed 129.39 samples/sec Loss 10.6028 Epoch: 9 Global Step: 93795 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:27:41,687-Speed 127.95 samples/sec Loss 10.4051 Epoch: 9 Global Step: 93800 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:28:01,808-Speed 127.24 samples/sec Loss 10.4364 Epoch: 9 Global Step: 93805 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:28:21,556-Speed 129.64 samples/sec Loss 10.5448 Epoch: 9 Global Step: 93810 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:28:41,326-Speed 129.50 samples/sec Loss 10.5734 Epoch: 9 Global Step: 93815 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:29:01,240-Speed 128.56 samples/sec Loss 10.6484 Epoch: 9 Global Step: 93820 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:29:20,993-Speed 129.60 samples/sec Loss 10.5583 Epoch: 9 Global Step: 93825 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:29:41,256-Speed 126.35 samples/sec Loss 10.8502 Epoch: 9 Global Step: 93830 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:30:01,116-Speed 128.91 samples/sec Loss 10.7072 Epoch: 9 Global Step: 93835 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:30:21,137-Speed 127.87 samples/sec Loss 10.7366 Epoch: 9 Global Step: 93840 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:30:40,924-Speed 129.39 samples/sec Loss 10.8085 Epoch: 9 Global Step: 93845 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:31:01,117-Speed 126.78 samples/sec Loss 10.6948 Epoch: 9 Global Step: 93850 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:31:20,876-Speed 129.57 samples/sec Loss 10.5069 Epoch: 9 Global Step: 93855 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:31:40,796-Speed 128.52 samples/sec Loss 10.5773 Epoch: 9 Global Step: 93860 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:32:00,784-Speed 128.08 samples/sec Loss 10.7211 Epoch: 9 Global Step: 93865 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:32:20,596-Speed 129.22 samples/sec Loss 10.8732 Epoch: 9 Global Step: 93870 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:32:40,831-Speed 126.52 samples/sec Loss 10.8208 Epoch: 9 Global Step: 93875 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:33:00,651-Speed 129.17 samples/sec Loss 10.7922 Epoch: 9 Global Step: 93880 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:33:20,484-Speed 129.09 samples/sec Loss 10.7453 Epoch: 9 Global Step: 93885 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:33:40,256-Speed 129.49 samples/sec Loss 10.8949 Epoch: 9 Global Step: 93890 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:34:00,413-Speed 127.01 samples/sec Loss 10.4330 Epoch: 9 Global Step: 93895 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:34:20,178-Speed 129.52 samples/sec Loss 10.5249 Epoch: 9 Global Step: 93900 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:34:40,078-Speed 128.65 samples/sec Loss 10.8564 Epoch: 9 Global Step: 93905 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:34:59,779-Speed 129.95 samples/sec Loss 10.3950 Epoch: 9 Global Step: 93910 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:35:19,652-Speed 128.82 samples/sec Loss 10.7403 Epoch: 9 Global Step: 93915 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:35:39,867-Speed 126.65 samples/sec Loss 10.1904 Epoch: 9 Global Step: 93920 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:35:59,588-Speed 129.81 samples/sec Loss 10.7536 Epoch: 9 Global Step: 93925 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:36:19,431-Speed 129.02 samples/sec Loss 10.9387 Epoch: 9 Global Step: 93930 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:36:39,189-Speed 129.58 samples/sec Loss 10.7340 Epoch: 9 Global Step: 93935 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:36:59,299-Speed 127.30 samples/sec Loss 10.4690 Epoch: 9 Global Step: 93940 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:37:19,318-Speed 127.89 samples/sec Loss 10.8586 Epoch: 9 Global Step: 93945 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:37:38,956-Speed 130.36 samples/sec Loss 10.7995 Epoch: 9 Global Step: 93950 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:37:58,820-Speed 128.88 samples/sec Loss 10.8236 Epoch: 9 Global Step: 93955 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:38:18,687-Speed 128.87 samples/sec Loss 10.5260 Epoch: 9 Global Step: 93960 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:38:38,743-Speed 127.65 samples/sec Loss 10.4401 Epoch: 9 Global Step: 93965 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:38:58,576-Speed 129.08 samples/sec Loss 10.7493 Epoch: 9 Global Step: 93970 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:39:18,350-Speed 129.47 samples/sec Loss 10.4883 Epoch: 9 Global Step: 93975 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:39:38,288-Speed 128.41 samples/sec Loss 11.0501 Epoch: 9 Global Step: 93980 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:39:58,332-Speed 127.73 samples/sec Loss 10.6238 Epoch: 9 Global Step: 93985 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:40:18,145-Speed 129.21 samples/sec Loss 10.2998 Epoch: 9 Global Step: 93990 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:40:37,814-Speed 130.15 samples/sec Loss 10.7957 Epoch: 9 Global Step: 93995 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:40:57,772-Speed 128.28 samples/sec Loss 10.5012 Epoch: 9 Global Step: 94000 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:41:17,563-Speed 129.36 samples/sec Loss 10.5839 Epoch: 9 Global Step: 94005 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:41:37,771-Speed 126.69 samples/sec Loss 10.5901 Epoch: 9 Global Step: 94010 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:41:57,537-Speed 129.53 samples/sec Loss 10.7190 Epoch: 9 Global Step: 94015 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:42:17,332-Speed 129.33 samples/sec Loss 10.8167 Epoch: 9 Global Step: 94020 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:42:37,252-Speed 128.52 samples/sec Loss 10.5326 Epoch: 9 Global Step: 94025 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:42:57,387-Speed 127.15 samples/sec Loss 11.0751 Epoch: 9 Global Step: 94030 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:43:17,069-Speed 130.07 samples/sec Loss 10.8990 Epoch: 9 Global Step: 94035 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:43:37,001-Speed 128.45 samples/sec Loss 10.7622 Epoch: 9 Global Step: 94040 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:43:56,829-Speed 129.12 samples/sec Loss 10.4804 Epoch: 9 Global Step: 94045 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:44:16,759-Speed 128.45 samples/sec Loss 10.7312 Epoch: 9 Global Step: 94050 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:44:36,842-Speed 127.48 samples/sec Loss 10.7052 Epoch: 9 Global Step: 94055 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:44:56,861-Speed 127.89 samples/sec Loss 10.8896 Epoch: 9 Global Step: 94060 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:45:16,800-Speed 128.40 samples/sec Loss 10.6084 Epoch: 9 Global Step: 94065 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:45:36,607-Speed 129.26 samples/sec Loss 11.0025 Epoch: 9 Global Step: 94070 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:45:56,623-Speed 127.90 samples/sec Loss 10.8153 Epoch: 9 Global Step: 94075 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:46:16,301-Speed 130.10 samples/sec Loss 10.3954 Epoch: 9 Global Step: 94080 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:46:36,457-Speed 127.02 samples/sec Loss 10.9533 Epoch: 9 Global Step: 94085 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:46:56,277-Speed 129.17 samples/sec Loss 10.9944 Epoch: 9 Global Step: 94090 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:47:16,134-Speed 128.93 samples/sec Loss 10.6330 Epoch: 9 Global Step: 94095 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:47:36,084-Speed 128.33 samples/sec Loss 11.0170 Epoch: 9 Global Step: 94100 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:47:55,856-Speed 129.48 samples/sec Loss 10.8062 Epoch: 9 Global Step: 94105 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:48:15,858-Speed 127.99 samples/sec Loss 10.5767 Epoch: 9 Global Step: 94110 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:48:35,660-Speed 129.29 samples/sec Loss 10.7120 Epoch: 9 Global Step: 94115 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:48:55,677-Speed 127.90 samples/sec Loss 10.9057 Epoch: 9 Global Step: 94120 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:49:15,607-Speed 128.46 samples/sec Loss 10.9970 Epoch: 9 Global Step: 94125 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:49:35,359-Speed 129.61 samples/sec Loss 10.8476 Epoch: 9 Global Step: 94130 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:49:55,240-Speed 128.77 samples/sec Loss 10.7611 Epoch: 9 Global Step: 94135 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:50:15,061-Speed 129.16 samples/sec Loss 10.4553 Epoch: 9 Global Step: 94140 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:50:35,014-Speed 128.30 samples/sec Loss 10.4362 Epoch: 9 Global Step: 94145 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:50:54,942-Speed 128.47 samples/sec Loss 10.7700 Epoch: 9 Global Step: 94150 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:51:14,797-Speed 128.94 samples/sec Loss 10.8317 Epoch: 9 Global Step: 94155 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:51:34,507-Speed 129.90 samples/sec Loss 10.8528 Epoch: 9 Global Step: 94160 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:51:54,684-Speed 126.88 samples/sec Loss 11.0748 Epoch: 9 Global Step: 94165 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:52:14,626-Speed 128.38 samples/sec Loss 10.8276 Epoch: 9 Global Step: 94170 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:52:34,359-Speed 129.74 samples/sec Loss 10.7189 Epoch: 9 Global Step: 94175 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:52:54,308-Speed 128.33 samples/sec Loss 10.6473 Epoch: 9 Global Step: 94180 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:53:14,017-Speed 129.89 samples/sec Loss 10.4543 Epoch: 9 Global Step: 94185 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:53:34,099-Speed 127.48 samples/sec Loss 10.4036 Epoch: 9 Global Step: 94190 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:53:54,084-Speed 128.10 samples/sec Loss 11.0762 Epoch: 9 Global Step: 94195 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:54:13,768-Speed 130.06 samples/sec Loss 10.6882 Epoch: 9 Global Step: 94200 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:54:33,778-Speed 127.94 samples/sec Loss 10.7598 Epoch: 9 Global Step: 94205 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:54:53,811-Speed 127.80 samples/sec Loss 10.7817 Epoch: 9 Global Step: 94210 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:55:13,657-Speed 129.00 samples/sec Loss 10.7473 Epoch: 9 Global Step: 94215 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:55:33,418-Speed 129.55 samples/sec Loss 10.3520 Epoch: 9 Global Step: 94220 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:55:53,305-Speed 128.73 samples/sec Loss 10.9858 Epoch: 9 Global Step: 94225 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:56:13,170-Speed 128.88 samples/sec Loss 10.7408 Epoch: 9 Global Step: 94230 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:56:33,377-Speed 126.70 samples/sec Loss 10.7351 Epoch: 9 Global Step: 94235 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:56:53,118-Speed 129.68 samples/sec Loss 10.5212 Epoch: 9 Global Step: 94240 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:57:12,967-Speed 128.98 samples/sec Loss 10.6480 Epoch: 9 Global Step: 94245 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:57:32,852-Speed 128.75 samples/sec Loss 10.5599 Epoch: 9 Global Step: 94250 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:57:52,973-Speed 127.23 samples/sec Loss 10.6253 Epoch: 9 Global Step: 94255 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:58:12,806-Speed 129.08 samples/sec Loss 10.9131 Epoch: 9 Global Step: 94260 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:58:32,601-Speed 129.34 samples/sec Loss 10.4446 Epoch: 9 Global Step: 94265 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:58:52,633-Speed 127.80 samples/sec Loss 10.6365 Epoch: 9 Global Step: 94270 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:59:12,328-Speed 129.99 samples/sec Loss 10.7141 Epoch: 9 Global Step: 94275 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:59:32,642-Speed 126.03 samples/sec Loss 11.0278 Epoch: 9 Global Step: 94280 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 10:59:52,387-Speed 129.66 samples/sec Loss 10.7664 Epoch: 9 Global Step: 94285 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:00:12,324-Speed 128.41 samples/sec Loss 10.5586 Epoch: 9 Global Step: 94290 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:00:32,079-Speed 129.60 samples/sec Loss 10.3747 Epoch: 9 Global Step: 94295 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:00:52,260-Speed 126.86 samples/sec Loss 10.5218 Epoch: 9 Global Step: 94300 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:01:12,092-Speed 129.09 samples/sec Loss 10.8370 Epoch: 9 Global Step: 94305 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:01:32,011-Speed 128.53 samples/sec Loss 10.6924 Epoch: 9 Global Step: 94310 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:01:51,733-Speed 129.81 samples/sec Loss 11.0333 Epoch: 9 Global Step: 94315 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:02:11,668-Speed 128.43 samples/sec Loss 10.6143 Epoch: 9 Global Step: 94320 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:02:31,765-Speed 127.39 samples/sec Loss 10.9601 Epoch: 9 Global Step: 94325 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:02:51,547-Speed 129.42 samples/sec Loss 10.7175 Epoch: 9 Global Step: 94330 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:03:11,403-Speed 128.93 samples/sec Loss 10.6573 Epoch: 9 Global Step: 94335 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:03:31,117-Speed 129.86 samples/sec Loss 10.4630 Epoch: 9 Global Step: 94340 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:03:51,303-Speed 126.83 samples/sec Loss 10.3405 Epoch: 9 Global Step: 94345 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:04:11,116-Speed 129.21 samples/sec Loss 10.4946 Epoch: 9 Global Step: 94350 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:04:30,905-Speed 129.37 samples/sec Loss 10.7106 Epoch: 9 Global Step: 94355 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:04:50,688-Speed 129.41 samples/sec Loss 10.4861 Epoch: 9 Global Step: 94360 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:05:10,645-Speed 128.28 samples/sec Loss 10.5459 Epoch: 9 Global Step: 94365 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:05:30,765-Speed 127.25 samples/sec Loss 10.4906 Epoch: 9 Global Step: 94370 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:05:50,600-Speed 129.07 samples/sec Loss 10.7195 Epoch: 9 Global Step: 94375 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:06:10,391-Speed 129.35 samples/sec Loss 10.5483 Epoch: 9 Global Step: 94380 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:06:30,276-Speed 128.75 samples/sec Loss 10.8902 Epoch: 9 Global Step: 94385 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:06:50,351-Speed 127.53 samples/sec Loss 10.4364 Epoch: 9 Global Step: 94390 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:07:10,313-Speed 128.25 samples/sec Loss 10.6913 Epoch: 9 Global Step: 94395 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:07:29,952-Speed 130.36 samples/sec Loss 10.7462 Epoch: 9 Global Step: 94400 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:07:49,741-Speed 129.37 samples/sec Loss 10.4788 Epoch: 9 Global Step: 94405 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:08:09,731-Speed 128.07 samples/sec Loss 10.4614 Epoch: 9 Global Step: 94410 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:08:29,885-Speed 127.02 samples/sec Loss 10.4357 Epoch: 9 Global Step: 94415 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:08:49,820-Speed 128.42 samples/sec Loss 10.6651 Epoch: 9 Global Step: 94420 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:09:09,629-Speed 129.24 samples/sec Loss 10.6653 Epoch: 9 Global Step: 94425 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:09:29,577-Speed 128.34 samples/sec Loss 10.7074 Epoch: 9 Global Step: 94430 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:09:49,631-Speed 127.66 samples/sec Loss 10.6864 Epoch: 9 Global Step: 94435 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:10:09,514-Speed 128.76 samples/sec Loss 10.9363 Epoch: 9 Global Step: 94440 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:10:29,249-Speed 129.72 samples/sec Loss 10.8354 Epoch: 9 Global Step: 94445 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:10:49,197-Speed 128.34 samples/sec Loss 10.5316 Epoch: 9 Global Step: 94450 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:11:08,945-Speed 129.64 samples/sec Loss 10.6657 Epoch: 9 Global Step: 94455 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:11:29,158-Speed 126.66 samples/sec Loss 10.3114 Epoch: 9 Global Step: 94460 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:11:48,890-Speed 129.74 samples/sec Loss 10.6865 Epoch: 9 Global Step: 94465 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:12:08,723-Speed 129.08 samples/sec Loss 10.5857 Epoch: 9 Global Step: 94470 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:12:28,766-Speed 127.73 samples/sec Loss 10.8432 Epoch: 9 Global Step: 94475 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:12:48,813-Speed 127.71 samples/sec Loss 10.6266 Epoch: 9 Global Step: 94480 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:13:08,777-Speed 128.24 samples/sec Loss 10.8409 Epoch: 9 Global Step: 94485 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:13:28,590-Speed 129.21 samples/sec Loss 10.5830 Epoch: 9 Global Step: 94490 Fp16 Grad Scale: 8192 Required: 184 hours Training: 2021-06-11 11:13:48,564-Speed 128.18 samples/sec Loss 10.3952 Epoch: 9 Global Step: 94495 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:14:08,302-Speed 129.70 samples/sec Loss 10.6008 Epoch: 9 Global Step: 94500 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:14:28,509-Speed 126.70 samples/sec Loss 10.4356 Epoch: 9 Global Step: 94505 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:14:48,268-Speed 129.57 samples/sec Loss 10.5758 Epoch: 9 Global Step: 94510 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:15:08,168-Speed 128.65 samples/sec Loss 10.7385 Epoch: 9 Global Step: 94515 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:15:28,083-Speed 128.56 samples/sec Loss 10.7191 Epoch: 9 Global Step: 94520 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:15:48,097-Speed 127.92 samples/sec Loss 10.7102 Epoch: 9 Global Step: 94525 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:16:07,841-Speed 129.66 samples/sec Loss 10.7292 Epoch: 9 Global Step: 94530 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:16:28,084-Speed 126.47 samples/sec Loss 10.7811 Epoch: 9 Global Step: 94535 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:16:47,753-Speed 130.16 samples/sec Loss 10.7989 Epoch: 9 Global Step: 94540 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:17:07,698-Speed 128.36 samples/sec Loss 10.6152 Epoch: 9 Global Step: 94545 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:17:27,738-Speed 127.75 samples/sec Loss 10.4120 Epoch: 9 Global Step: 94550 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:17:47,636-Speed 128.66 samples/sec Loss 10.7298 Epoch: 9 Global Step: 94555 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:18:07,546-Speed 128.59 samples/sec Loss 10.9169 Epoch: 9 Global Step: 94560 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:18:27,322-Speed 129.46 samples/sec Loss 10.6912 Epoch: 9 Global Step: 94565 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:18:47,627-Speed 126.08 samples/sec Loss 10.4993 Epoch: 9 Global Step: 94570 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:19:07,760-Speed 127.16 samples/sec Loss 10.7943 Epoch: 9 Global Step: 94575 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:19:27,654-Speed 128.69 samples/sec Loss 10.6914 Epoch: 9 Global Step: 94580 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:19:47,471-Speed 129.19 samples/sec Loss 11.0164 Epoch: 9 Global Step: 94585 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:20:07,388-Speed 128.54 samples/sec Loss 10.7654 Epoch: 9 Global Step: 94590 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:20:27,398-Speed 127.94 samples/sec Loss 10.6260 Epoch: 9 Global Step: 94595 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:20:47,900-Speed 124.88 samples/sec Loss 10.8051 Epoch: 9 Global Step: 94600 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:21:07,656-Speed 129.59 samples/sec Loss 10.7721 Epoch: 9 Global Step: 94605 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:21:27,468-Speed 129.22 samples/sec Loss 10.5114 Epoch: 9 Global Step: 94610 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:21:47,539-Speed 127.56 samples/sec Loss 10.5594 Epoch: 9 Global Step: 94615 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:22:07,466-Speed 128.47 samples/sec Loss 10.5653 Epoch: 9 Global Step: 94620 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:22:27,418-Speed 128.31 samples/sec Loss 10.6538 Epoch: 9 Global Step: 94625 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:22:47,351-Speed 128.44 samples/sec Loss 10.4381 Epoch: 9 Global Step: 94630 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:23:07,120-Speed 129.50 samples/sec Loss 10.7892 Epoch: 9 Global Step: 94635 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:23:27,143-Speed 127.86 samples/sec Loss 10.5216 Epoch: 9 Global Step: 94640 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:23:47,343-Speed 126.74 samples/sec Loss 10.6615 Epoch: 9 Global Step: 94645 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:24:07,118-Speed 129.47 samples/sec Loss 10.4963 Epoch: 9 Global Step: 94650 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:24:27,066-Speed 128.34 samples/sec Loss 10.6623 Epoch: 9 Global Step: 94655 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:24:46,974-Speed 128.60 samples/sec Loss 10.5472 Epoch: 9 Global Step: 94660 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:25:06,801-Speed 129.12 samples/sec Loss 10.7513 Epoch: 9 Global Step: 94665 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:25:26,652-Speed 128.97 samples/sec Loss 10.4330 Epoch: 9 Global Step: 94670 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:25:46,582-Speed 128.46 samples/sec Loss 10.4625 Epoch: 9 Global Step: 94675 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:26:06,367-Speed 129.39 samples/sec Loss 10.6457 Epoch: 9 Global Step: 94680 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:26:26,673-Speed 126.08 samples/sec Loss 10.8865 Epoch: 9 Global Step: 94685 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:26:46,400-Speed 129.79 samples/sec Loss 10.0011 Epoch: 9 Global Step: 94690 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:27:06,418-Speed 127.89 samples/sec Loss 10.3187 Epoch: 9 Global Step: 94695 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:27:26,278-Speed 128.91 samples/sec Loss 10.9252 Epoch: 9 Global Step: 94700 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:27:46,308-Speed 127.81 samples/sec Loss 10.5944 Epoch: 9 Global Step: 94705 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:28:06,086-Speed 129.44 samples/sec Loss 10.9845 Epoch: 9 Global Step: 94710 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:28:26,032-Speed 128.35 samples/sec Loss 10.4855 Epoch: 9 Global Step: 94715 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:28:45,788-Speed 129.59 samples/sec Loss 10.6040 Epoch: 9 Global Step: 94720 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:29:05,587-Speed 129.31 samples/sec Loss 10.9976 Epoch: 9 Global Step: 94725 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:29:25,672-Speed 127.47 samples/sec Loss 10.4858 Epoch: 9 Global Step: 94730 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:29:45,521-Speed 128.98 samples/sec Loss 10.6227 Epoch: 9 Global Step: 94735 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:30:05,450-Speed 128.46 samples/sec Loss 10.7426 Epoch: 9 Global Step: 94740 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:30:25,200-Speed 129.63 samples/sec Loss 10.7764 Epoch: 9 Global Step: 94745 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:30:45,227-Speed 127.84 samples/sec Loss 10.9004 Epoch: 9 Global Step: 94750 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:31:04,963-Speed 129.72 samples/sec Loss 10.5493 Epoch: 9 Global Step: 94755 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:31:24,908-Speed 128.36 samples/sec Loss 10.4074 Epoch: 9 Global Step: 94760 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:31:44,680-Speed 129.48 samples/sec Loss 10.8085 Epoch: 9 Global Step: 94765 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:32:04,600-Speed 128.53 samples/sec Loss 10.2986 Epoch: 9 Global Step: 94770 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:32:24,644-Speed 127.72 samples/sec Loss 10.4534 Epoch: 9 Global Step: 94775 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:32:44,462-Speed 129.19 samples/sec Loss 10.7909 Epoch: 9 Global Step: 94780 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:33:04,367-Speed 128.61 samples/sec Loss 10.7842 Epoch: 9 Global Step: 94785 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:33:24,032-Speed 130.19 samples/sec Loss 10.7888 Epoch: 9 Global Step: 94790 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:33:44,138-Speed 127.34 samples/sec Loss 10.7396 Epoch: 9 Global Step: 94795 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:34:04,146-Speed 127.95 samples/sec Loss 10.8829 Epoch: 9 Global Step: 94800 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:34:23,900-Speed 129.60 samples/sec Loss 10.6178 Epoch: 9 Global Step: 94805 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:34:43,717-Speed 129.19 samples/sec Loss 10.6055 Epoch: 9 Global Step: 94810 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:35:03,634-Speed 128.54 samples/sec Loss 10.5289 Epoch: 9 Global Step: 94815 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:35:23,685-Speed 127.68 samples/sec Loss 10.8234 Epoch: 9 Global Step: 94820 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:35:43,781-Speed 127.39 samples/sec Loss 10.4708 Epoch: 9 Global Step: 94825 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:36:03,481-Speed 129.96 samples/sec Loss 10.4719 Epoch: 9 Global Step: 94830 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:36:23,305-Speed 129.14 samples/sec Loss 11.0191 Epoch: 9 Global Step: 94835 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:36:43,436-Speed 127.18 samples/sec Loss 10.9019 Epoch: 9 Global Step: 94840 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:37:03,412-Speed 128.16 samples/sec Loss 10.1717 Epoch: 9 Global Step: 94845 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:37:23,168-Speed 129.58 samples/sec Loss 10.3791 Epoch: 9 Global Step: 94850 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:37:43,124-Speed 128.29 samples/sec Loss 10.6313 Epoch: 9 Global Step: 94855 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:38:02,921-Speed 129.31 samples/sec Loss 10.7090 Epoch: 9 Global Step: 94860 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:38:23,096-Speed 126.90 samples/sec Loss 10.5177 Epoch: 9 Global Step: 94865 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:38:43,014-Speed 128.53 samples/sec Loss 10.8152 Epoch: 9 Global Step: 94870 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:39:02,715-Speed 129.95 samples/sec Loss 10.4706 Epoch: 9 Global Step: 94875 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:39:22,619-Speed 128.63 samples/sec Loss 10.6523 Epoch: 9 Global Step: 94880 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:39:42,655-Speed 127.78 samples/sec Loss 10.5973 Epoch: 9 Global Step: 94885 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:40:02,578-Speed 128.50 samples/sec Loss 10.5979 Epoch: 9 Global Step: 94890 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:40:22,331-Speed 129.60 samples/sec Loss 10.2877 Epoch: 9 Global Step: 94895 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:40:42,170-Speed 129.05 samples/sec Loss 10.4614 Epoch: 9 Global Step: 94900 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:41:01,953-Speed 129.41 samples/sec Loss 10.6871 Epoch: 9 Global Step: 94905 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:41:22,239-Speed 126.20 samples/sec Loss 10.5643 Epoch: 9 Global Step: 94910 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:41:42,005-Speed 129.52 samples/sec Loss 10.6230 Epoch: 9 Global Step: 94915 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:42:01,794-Speed 129.38 samples/sec Loss 11.0526 Epoch: 9 Global Step: 94920 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:42:21,611-Speed 129.18 samples/sec Loss 10.7484 Epoch: 9 Global Step: 94925 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:42:41,804-Speed 126.79 samples/sec Loss 10.7992 Epoch: 9 Global Step: 94930 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:43:01,811-Speed 127.96 samples/sec Loss 10.8535 Epoch: 9 Global Step: 94935 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:43:21,637-Speed 129.12 samples/sec Loss 10.5026 Epoch: 9 Global Step: 94940 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:43:41,408-Speed 129.49 samples/sec Loss 10.5634 Epoch: 9 Global Step: 94945 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:44:01,208-Speed 129.30 samples/sec Loss 10.7869 Epoch: 9 Global Step: 94950 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:44:21,634-Speed 125.34 samples/sec Loss 10.5492 Epoch: 9 Global Step: 94955 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:44:41,293-Speed 130.23 samples/sec Loss 10.5925 Epoch: 9 Global Step: 94960 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:45:01,227-Speed 128.43 samples/sec Loss 10.7056 Epoch: 9 Global Step: 94965 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:45:21,086-Speed 128.92 samples/sec Loss 10.3920 Epoch: 9 Global Step: 94970 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:45:41,221-Speed 127.15 samples/sec Loss 10.8725 Epoch: 9 Global Step: 94975 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:46:00,873-Speed 130.27 samples/sec Loss 10.7091 Epoch: 9 Global Step: 94980 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:46:20,694-Speed 129.17 samples/sec Loss 10.8200 Epoch: 9 Global Step: 94985 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:46:40,450-Speed 129.58 samples/sec Loss 10.7531 Epoch: 9 Global Step: 94990 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:47:00,460-Speed 127.95 samples/sec Loss 10.5602 Epoch: 9 Global Step: 94995 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:47:20,434-Speed 128.17 samples/sec Loss 10.7120 Epoch: 9 Global Step: 95000 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:47:40,309-Speed 128.81 samples/sec Loss 10.9703 Epoch: 9 Global Step: 95005 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:48:00,289-Speed 128.13 samples/sec Loss 10.6700 Epoch: 9 Global Step: 95010 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:48:20,209-Speed 128.52 samples/sec Loss 10.6542 Epoch: 9 Global Step: 95015 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:48:40,300-Speed 127.42 samples/sec Loss 10.5022 Epoch: 9 Global Step: 95020 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:49:00,140-Speed 129.04 samples/sec Loss 10.5345 Epoch: 9 Global Step: 95025 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:49:19,891-Speed 129.62 samples/sec Loss 10.6896 Epoch: 9 Global Step: 95030 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:49:39,686-Speed 129.34 samples/sec Loss 10.6671 Epoch: 9 Global Step: 95035 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:49:59,602-Speed 128.55 samples/sec Loss 10.4988 Epoch: 9 Global Step: 95040 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:50:19,553-Speed 128.32 samples/sec Loss 10.3582 Epoch: 9 Global Step: 95045 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:50:39,494-Speed 128.39 samples/sec Loss 11.1407 Epoch: 9 Global Step: 95050 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:50:59,383-Speed 128.72 samples/sec Loss 10.8254 Epoch: 9 Global Step: 95055 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:51:19,222-Speed 129.05 samples/sec Loss 10.9171 Epoch: 9 Global Step: 95060 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:51:39,175-Speed 128.31 samples/sec Loss 10.8731 Epoch: 9 Global Step: 95065 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:51:59,131-Speed 128.29 samples/sec Loss 10.8490 Epoch: 9 Global Step: 95070 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:52:18,866-Speed 129.73 samples/sec Loss 10.7791 Epoch: 9 Global Step: 95075 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:52:38,876-Speed 127.94 samples/sec Loss 10.4348 Epoch: 9 Global Step: 95080 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:52:58,642-Speed 129.52 samples/sec Loss 10.7895 Epoch: 9 Global Step: 95085 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:53:18,639-Speed 128.03 samples/sec Loss 10.7744 Epoch: 9 Global Step: 95090 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:53:38,543-Speed 128.62 samples/sec Loss 10.7363 Epoch: 9 Global Step: 95095 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:53:58,311-Speed 129.52 samples/sec Loss 10.6878 Epoch: 9 Global Step: 95100 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:54:18,260-Speed 128.33 samples/sec Loss 10.7224 Epoch: 9 Global Step: 95105 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:54:38,229-Speed 128.21 samples/sec Loss 10.6987 Epoch: 9 Global Step: 95110 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:54:58,380-Speed 127.05 samples/sec Loss 10.6365 Epoch: 9 Global Step: 95115 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:55:18,048-Speed 130.16 samples/sec Loss 10.4884 Epoch: 9 Global Step: 95120 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:55:37,993-Speed 128.36 samples/sec Loss 10.8960 Epoch: 9 Global Step: 95125 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:55:57,759-Speed 129.52 samples/sec Loss 10.3662 Epoch: 9 Global Step: 95130 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:56:17,972-Speed 126.66 samples/sec Loss 10.7698 Epoch: 9 Global Step: 95135 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:56:37,721-Speed 129.64 samples/sec Loss 10.7878 Epoch: 9 Global Step: 95140 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:56:57,605-Speed 128.75 samples/sec Loss 10.6762 Epoch: 9 Global Step: 95145 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:57:17,509-Speed 128.63 samples/sec Loss 10.5613 Epoch: 9 Global Step: 95150 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:57:37,525-Speed 127.90 samples/sec Loss 10.6383 Epoch: 9 Global Step: 95155 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:57:57,306-Speed 129.43 samples/sec Loss 10.6087 Epoch: 9 Global Step: 95160 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:58:17,292-Speed 128.09 samples/sec Loss 10.7147 Epoch: 9 Global Step: 95165 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:58:37,160-Speed 128.86 samples/sec Loss 10.9682 Epoch: 9 Global Step: 95170 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:58:56,929-Speed 129.50 samples/sec Loss 10.8849 Epoch: 9 Global Step: 95175 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:59:17,095-Speed 126.96 samples/sec Loss 10.5000 Epoch: 9 Global Step: 95180 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:59:36,929-Speed 129.07 samples/sec Loss 10.5493 Epoch: 9 Global Step: 95185 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 11:59:56,872-Speed 128.38 samples/sec Loss 10.6390 Epoch: 9 Global Step: 95190 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:00:17,218-Speed 125.83 samples/sec Loss 10.8573 Epoch: 9 Global Step: 95195 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:00:37,363-Speed 127.08 samples/sec Loss 10.6861 Epoch: 9 Global Step: 95200 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:00:57,098-Speed 129.72 samples/sec Loss 10.5272 Epoch: 9 Global Step: 95205 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:01:17,064-Speed 128.22 samples/sec Loss 10.5931 Epoch: 9 Global Step: 95210 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:01:36,830-Speed 129.52 samples/sec Loss 10.7457 Epoch: 9 Global Step: 95215 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:01:56,548-Speed 129.84 samples/sec Loss 10.6628 Epoch: 9 Global Step: 95220 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:02:16,790-Speed 126.48 samples/sec Loss 10.8123 Epoch: 9 Global Step: 95225 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:02:36,546-Speed 129.59 samples/sec Loss 10.7099 Epoch: 9 Global Step: 95230 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:02:56,399-Speed 128.95 samples/sec Loss 10.8538 Epoch: 9 Global Step: 95235 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:03:16,282-Speed 128.76 samples/sec Loss 10.4604 Epoch: 9 Global Step: 95240 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:03:36,357-Speed 127.53 samples/sec Loss 10.9155 Epoch: 9 Global Step: 95245 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:03:56,215-Speed 128.92 samples/sec Loss 10.4717 Epoch: 9 Global Step: 95250 Fp16 Grad Scale: 8192 Required: 183 hours Training: 2021-06-11 12:04:16,114-Speed 128.65 samples/sec Loss 10.6408 Epoch: 9 Global Step: 95255 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:04:35,786-Speed 130.15 samples/sec Loss 10.6771 Epoch: 9 Global Step: 95260 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:04:55,851-Speed 127.59 samples/sec Loss 10.6288 Epoch: 9 Global Step: 95265 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:05:15,787-Speed 128.42 samples/sec Loss 10.2759 Epoch: 9 Global Step: 95270 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:05:35,613-Speed 129.13 samples/sec Loss 10.3997 Epoch: 9 Global Step: 95275 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:05:55,520-Speed 128.60 samples/sec Loss 10.5122 Epoch: 9 Global Step: 95280 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:06:15,169-Speed 130.29 samples/sec Loss 10.6144 Epoch: 9 Global Step: 95285 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:06:35,290-Speed 127.24 samples/sec Loss 10.7868 Epoch: 9 Global Step: 95290 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:06:55,228-Speed 128.40 samples/sec Loss 10.8870 Epoch: 9 Global Step: 95295 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:07:14,980-Speed 129.61 samples/sec Loss 10.8332 Epoch: 9 Global Step: 95300 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:07:34,751-Speed 129.49 samples/sec Loss 10.6190 Epoch: 9 Global Step: 95305 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:07:54,660-Speed 128.59 samples/sec Loss 10.5247 Epoch: 9 Global Step: 95310 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:08:14,693-Speed 127.79 samples/sec Loss 10.6464 Epoch: 9 Global Step: 95315 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:08:34,717-Speed 127.85 samples/sec Loss 10.6913 Epoch: 9 Global Step: 95320 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:08:54,491-Speed 129.47 samples/sec Loss 11.0813 Epoch: 9 Global Step: 95325 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:09:14,263-Speed 129.49 samples/sec Loss 10.8259 Epoch: 9 Global Step: 95330 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:09:34,345-Speed 127.48 samples/sec Loss 10.8447 Epoch: 9 Global Step: 95335 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:09:54,284-Speed 128.39 samples/sec Loss 10.4976 Epoch: 9 Global Step: 95340 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:10:14,026-Speed 129.68 samples/sec Loss 10.7331 Epoch: 9 Global Step: 95345 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:10:33,877-Speed 128.97 samples/sec Loss 10.9507 Epoch: 9 Global Step: 95350 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:10:53,619-Speed 129.68 samples/sec Loss 10.7962 Epoch: 9 Global Step: 95355 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:11:13,888-Speed 126.31 samples/sec Loss 11.0889 Epoch: 9 Global Step: 95360 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:11:33,585-Speed 129.98 samples/sec Loss 10.9160 Epoch: 9 Global Step: 95365 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:11:53,276-Speed 130.01 samples/sec Loss 10.8203 Epoch: 9 Global Step: 95370 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:12:13,224-Speed 128.34 samples/sec Loss 10.7647 Epoch: 9 Global Step: 95375 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:12:33,124-Speed 128.65 samples/sec Loss 10.7441 Epoch: 9 Global Step: 95380 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:12:53,109-Speed 128.10 samples/sec Loss 10.4224 Epoch: 9 Global Step: 95385 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:13:12,762-Speed 130.27 samples/sec Loss 10.6357 Epoch: 9 Global Step: 95390 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:13:32,693-Speed 128.45 samples/sec Loss 10.5393 Epoch: 9 Global Step: 95395 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:13:52,451-Speed 129.57 samples/sec Loss 10.9918 Epoch: 9 Global Step: 95400 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:14:12,738-Speed 126.19 samples/sec Loss 10.6786 Epoch: 9 Global Step: 95405 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:14:32,396-Speed 130.23 samples/sec Loss 11.0161 Epoch: 9 Global Step: 95410 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:14:52,325-Speed 128.46 samples/sec Loss 10.9101 Epoch: 9 Global Step: 95415 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:15:12,047-Speed 129.81 samples/sec Loss 10.7978 Epoch: 9 Global Step: 95420 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:15:32,173-Speed 127.21 samples/sec Loss 10.5494 Epoch: 9 Global Step: 95425 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:15:52,009-Speed 129.06 samples/sec Loss 10.7202 Epoch: 9 Global Step: 95430 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:16:11,856-Speed 128.99 samples/sec Loss 10.8464 Epoch: 9 Global Step: 95435 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:16:31,643-Speed 129.39 samples/sec Loss 10.2899 Epoch: 9 Global Step: 95440 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:16:51,367-Speed 129.80 samples/sec Loss 10.7436 Epoch: 9 Global Step: 95445 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:17:11,680-Speed 126.03 samples/sec Loss 10.6934 Epoch: 9 Global Step: 95450 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:17:31,454-Speed 129.47 samples/sec Loss 11.1084 Epoch: 9 Global Step: 95455 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:17:51,371-Speed 128.54 samples/sec Loss 10.6521 Epoch: 9 Global Step: 95460 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:18:11,125-Speed 129.60 samples/sec Loss 10.8403 Epoch: 9 Global Step: 95465 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:18:31,381-Speed 126.39 samples/sec Loss 10.6311 Epoch: 9 Global Step: 95470 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:18:51,066-Speed 130.06 samples/sec Loss 10.7465 Epoch: 9 Global Step: 95475 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:19:11,003-Speed 128.41 samples/sec Loss 10.4270 Epoch: 9 Global Step: 95480 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:19:30,794-Speed 129.36 samples/sec Loss 10.8150 Epoch: 9 Global Step: 95485 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:19:50,740-Speed 128.36 samples/sec Loss 10.4803 Epoch: 9 Global Step: 95490 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:20:10,882-Speed 127.11 samples/sec Loss 10.6059 Epoch: 9 Global Step: 95495 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:20:30,682-Speed 129.30 samples/sec Loss 10.5284 Epoch: 9 Global Step: 95500 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:20:50,576-Speed 128.69 samples/sec Loss 10.4446 Epoch: 9 Global Step: 95505 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:21:10,336-Speed 129.57 samples/sec Loss 10.2858 Epoch: 9 Global Step: 95510 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:21:30,443-Speed 127.32 samples/sec Loss 10.3484 Epoch: 9 Global Step: 95515 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:21:50,380-Speed 128.41 samples/sec Loss 10.5154 Epoch: 9 Global Step: 95520 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:22:10,061-Speed 130.08 samples/sec Loss 10.8422 Epoch: 9 Global Step: 95525 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:22:29,855-Speed 129.34 samples/sec Loss 10.8662 Epoch: 9 Global Step: 95530 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:22:49,964-Speed 127.31 samples/sec Loss 10.5358 Epoch: 9 Global Step: 95535 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:23:09,978-Speed 127.92 samples/sec Loss 10.5602 Epoch: 9 Global Step: 95540 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:23:30,230-Speed 126.41 samples/sec Loss 10.4583 Epoch: 9 Global Step: 95545 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:23:49,970-Speed 129.69 samples/sec Loss 10.3173 Epoch: 9 Global Step: 95550 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:24:09,812-Speed 129.03 samples/sec Loss 10.6441 Epoch: 9 Global Step: 95555 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:24:29,858-Speed 127.72 samples/sec Loss 10.6170 Epoch: 9 Global Step: 95560 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:24:49,779-Speed 128.52 samples/sec Loss 10.6581 Epoch: 9 Global Step: 95565 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:25:09,504-Speed 129.79 samples/sec Loss 10.5184 Epoch: 9 Global Step: 95570 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:25:29,468-Speed 128.24 samples/sec Loss 10.4566 Epoch: 9 Global Step: 95575 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:25:49,357-Speed 128.72 samples/sec Loss 10.5900 Epoch: 9 Global Step: 95580 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:26:09,427-Speed 127.56 samples/sec Loss 10.8787 Epoch: 9 Global Step: 95585 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:26:29,330-Speed 128.63 samples/sec Loss 10.4658 Epoch: 9 Global Step: 95590 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:26:49,171-Speed 129.04 samples/sec Loss 10.3695 Epoch: 9 Global Step: 95595 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:27:09,081-Speed 128.59 samples/sec Loss 10.5653 Epoch: 9 Global Step: 95600 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:27:29,193-Speed 127.29 samples/sec Loss 10.7481 Epoch: 9 Global Step: 95605 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:27:48,945-Speed 129.61 samples/sec Loss 10.7660 Epoch: 9 Global Step: 95610 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:28:08,718-Speed 129.48 samples/sec Loss 10.6038 Epoch: 9 Global Step: 95615 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:28:28,741-Speed 127.86 samples/sec Loss 10.5846 Epoch: 9 Global Step: 95620 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:28:48,489-Speed 129.64 samples/sec Loss 10.5962 Epoch: 9 Global Step: 95625 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:29:08,630-Speed 127.11 samples/sec Loss 10.8616 Epoch: 9 Global Step: 95630 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:29:28,410-Speed 129.43 samples/sec Loss 10.7927 Epoch: 9 Global Step: 95635 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:29:48,353-Speed 128.38 samples/sec Loss 10.6713 Epoch: 9 Global Step: 95640 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:30:08,146-Speed 129.34 samples/sec Loss 10.9581 Epoch: 9 Global Step: 95645 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:30:28,267-Speed 127.24 samples/sec Loss 10.6563 Epoch: 9 Global Step: 95650 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:30:48,048-Speed 129.43 samples/sec Loss 10.8137 Epoch: 9 Global Step: 95655 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:31:07,993-Speed 128.35 samples/sec Loss 10.5300 Epoch: 9 Global Step: 95660 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:31:27,795-Speed 129.29 samples/sec Loss 10.5567 Epoch: 9 Global Step: 95665 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:31:47,638-Speed 129.02 samples/sec Loss 10.4903 Epoch: 9 Global Step: 95670 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:32:07,666-Speed 127.82 samples/sec Loss 10.7957 Epoch: 9 Global Step: 95675 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:32:27,466-Speed 129.30 samples/sec Loss 10.4919 Epoch: 9 Global Step: 95680 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:32:47,437-Speed 128.19 samples/sec Loss 10.9321 Epoch: 9 Global Step: 95685 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:33:07,243-Speed 129.27 samples/sec Loss 10.5802 Epoch: 9 Global Step: 95690 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:33:27,341-Speed 127.38 samples/sec Loss 10.5737 Epoch: 9 Global Step: 95695 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:33:47,217-Speed 128.80 samples/sec Loss 10.8453 Epoch: 9 Global Step: 95700 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:34:07,109-Speed 128.70 samples/sec Loss 10.4667 Epoch: 9 Global Step: 95705 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:34:26,910-Speed 129.29 samples/sec Loss 10.5430 Epoch: 9 Global Step: 95710 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:34:46,761-Speed 128.97 samples/sec Loss 10.8129 Epoch: 9 Global Step: 95715 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:35:06,802-Speed 127.74 samples/sec Loss 10.8280 Epoch: 9 Global Step: 95720 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:35:26,818-Speed 127.91 samples/sec Loss 10.6563 Epoch: 9 Global Step: 95725 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:35:46,478-Speed 130.22 samples/sec Loss 10.6064 Epoch: 9 Global Step: 95730 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:36:06,337-Speed 128.91 samples/sec Loss 10.5734 Epoch: 9 Global Step: 95735 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:36:26,355-Speed 127.90 samples/sec Loss 11.0024 Epoch: 9 Global Step: 95740 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:36:46,288-Speed 128.44 samples/sec Loss 10.7168 Epoch: 9 Global Step: 95745 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:37:06,026-Speed 129.70 samples/sec Loss 10.6737 Epoch: 9 Global Step: 95750 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:37:25,914-Speed 128.73 samples/sec Loss 10.6208 Epoch: 9 Global Step: 95755 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:37:45,682-Speed 129.50 samples/sec Loss 10.6286 Epoch: 9 Global Step: 95760 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:38:05,926-Speed 126.47 samples/sec Loss 10.5511 Epoch: 9 Global Step: 95765 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:38:25,806-Speed 128.78 samples/sec Loss 10.8025 Epoch: 9 Global Step: 95770 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:38:45,486-Speed 130.09 samples/sec Loss 10.7954 Epoch: 9 Global Step: 95775 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:39:05,397-Speed 128.59 samples/sec Loss 10.5154 Epoch: 9 Global Step: 95780 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:39:25,496-Speed 127.37 samples/sec Loss 10.6241 Epoch: 9 Global Step: 95785 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:39:45,397-Speed 128.65 samples/sec Loss 10.6141 Epoch: 9 Global Step: 95790 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:40:05,063-Speed 130.18 samples/sec Loss 10.4040 Epoch: 9 Global Step: 95795 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:40:24,956-Speed 128.70 samples/sec Loss 10.4026 Epoch: 9 Global Step: 95800 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:40:44,669-Speed 129.87 samples/sec Loss 10.7598 Epoch: 9 Global Step: 95805 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:41:04,950-Speed 126.23 samples/sec Loss 10.4670 Epoch: 9 Global Step: 95810 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:41:24,628-Speed 130.10 samples/sec Loss 10.5850 Epoch: 9 Global Step: 95815 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:41:44,433-Speed 129.26 samples/sec Loss 10.5226 Epoch: 9 Global Step: 95820 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:42:04,244-Speed 129.23 samples/sec Loss 10.4307 Epoch: 9 Global Step: 95825 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:42:24,457-Speed 126.66 samples/sec Loss 10.6671 Epoch: 9 Global Step: 95830 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:42:44,212-Speed 129.59 samples/sec Loss 10.8631 Epoch: 9 Global Step: 95835 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:43:03,997-Speed 129.40 samples/sec Loss 10.6081 Epoch: 9 Global Step: 95840 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:43:23,886-Speed 128.72 samples/sec Loss 10.7856 Epoch: 9 Global Step: 95845 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:43:43,649-Speed 129.54 samples/sec Loss 10.7725 Epoch: 9 Global Step: 95850 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:44:03,859-Speed 126.67 samples/sec Loss 10.7239 Epoch: 9 Global Step: 95855 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:44:23,681-Speed 129.16 samples/sec Loss 10.7177 Epoch: 9 Global Step: 95860 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:44:43,613-Speed 128.44 samples/sec Loss 10.5310 Epoch: 9 Global Step: 95865 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:45:03,430-Speed 129.19 samples/sec Loss 10.4305 Epoch: 9 Global Step: 95870 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:45:23,535-Speed 127.34 samples/sec Loss 10.7794 Epoch: 9 Global Step: 95875 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:45:43,287-Speed 129.61 samples/sec Loss 10.6235 Epoch: 9 Global Step: 95880 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:46:03,192-Speed 128.62 samples/sec Loss 10.7563 Epoch: 9 Global Step: 95885 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:46:23,131-Speed 128.40 samples/sec Loss 10.4847 Epoch: 9 Global Step: 95890 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:46:43,041-Speed 128.59 samples/sec Loss 10.4234 Epoch: 9 Global Step: 95895 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:47:03,031-Speed 128.07 samples/sec Loss 10.1307 Epoch: 9 Global Step: 95900 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:47:22,724-Speed 130.01 samples/sec Loss 10.5636 Epoch: 9 Global Step: 95905 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:47:42,641-Speed 128.54 samples/sec Loss 11.0238 Epoch: 9 Global Step: 95910 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:48:02,432-Speed 129.36 samples/sec Loss 10.8267 Epoch: 9 Global Step: 95915 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:48:22,462-Speed 127.82 samples/sec Loss 11.0649 Epoch: 9 Global Step: 95920 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:48:42,169-Speed 129.91 samples/sec Loss 10.7755 Epoch: 9 Global Step: 95925 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:49:02,000-Speed 129.09 samples/sec Loss 10.5117 Epoch: 9 Global Step: 95930 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:49:21,843-Speed 129.02 samples/sec Loss 10.6479 Epoch: 9 Global Step: 95935 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:49:41,709-Speed 128.87 samples/sec Loss 10.4975 Epoch: 9 Global Step: 95940 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:50:01,647-Speed 128.40 samples/sec Loss 10.8082 Epoch: 9 Global Step: 95945 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:50:21,468-Speed 129.16 samples/sec Loss 10.6327 Epoch: 9 Global Step: 95950 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:50:41,465-Speed 128.02 samples/sec Loss 10.4697 Epoch: 9 Global Step: 95955 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:51:01,193-Speed 129.77 samples/sec Loss 10.7635 Epoch: 9 Global Step: 95960 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:51:21,262-Speed 127.57 samples/sec Loss 10.2111 Epoch: 9 Global Step: 95965 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:51:41,086-Speed 129.14 samples/sec Loss 10.9286 Epoch: 9 Global Step: 95970 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:52:00,816-Speed 129.76 samples/sec Loss 11.1304 Epoch: 9 Global Step: 95975 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:52:20,691-Speed 128.81 samples/sec Loss 11.1855 Epoch: 9 Global Step: 95980 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:52:40,609-Speed 128.54 samples/sec Loss 10.4590 Epoch: 9 Global Step: 95985 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:53:00,487-Speed 128.79 samples/sec Loss 10.7635 Epoch: 9 Global Step: 95990 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:53:20,454-Speed 128.22 samples/sec Loss 10.5836 Epoch: 9 Global Step: 95995 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:53:40,317-Speed 128.89 samples/sec Loss 10.9813 Epoch: 9 Global Step: 96000 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:54:00,182-Speed 128.88 samples/sec Loss 10.7310 Epoch: 9 Global Step: 96005 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:54:20,203-Speed 127.87 samples/sec Loss 10.4061 Epoch: 9 Global Step: 96010 Fp16 Grad Scale: 8192 Required: 182 hours Training: 2021-06-11 12:54:40,319-Speed 127.27 samples/sec Loss 10.4696 Epoch: 9 Global Step: 96015 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:55:00,056-Speed 129.71 samples/sec Loss 10.6360 Epoch: 9 Global Step: 96020 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:55:20,058-Speed 127.99 samples/sec Loss 10.5930 Epoch: 9 Global Step: 96025 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:55:39,849-Speed 129.36 samples/sec Loss 10.5194 Epoch: 9 Global Step: 96030 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:55:59,900-Speed 127.68 samples/sec Loss 10.7704 Epoch: 9 Global Step: 96035 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:56:19,874-Speed 128.17 samples/sec Loss 10.6223 Epoch: 9 Global Step: 96040 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:56:39,538-Speed 130.19 samples/sec Loss 10.7362 Epoch: 9 Global Step: 96045 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:56:59,573-Speed 127.79 samples/sec Loss 10.4804 Epoch: 9 Global Step: 96050 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:57:19,526-Speed 128.31 samples/sec Loss 10.4302 Epoch: 9 Global Step: 96055 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:57:39,381-Speed 128.94 samples/sec Loss 10.5917 Epoch: 9 Global Step: 96060 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:57:59,259-Speed 128.79 samples/sec Loss 10.3207 Epoch: 9 Global Step: 96065 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:58:19,207-Speed 128.34 samples/sec Loss 10.8806 Epoch: 9 Global Step: 96070 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:58:38,905-Speed 129.97 samples/sec Loss 10.5624 Epoch: 9 Global Step: 96075 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:58:59,190-Speed 126.21 samples/sec Loss 10.6088 Epoch: 9 Global Step: 96080 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:59:18,947-Speed 129.58 samples/sec Loss 10.8722 Epoch: 9 Global Step: 96085 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:59:38,650-Speed 129.93 samples/sec Loss 10.5479 Epoch: 9 Global Step: 96090 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 12:59:58,538-Speed 128.73 samples/sec Loss 10.6037 Epoch: 9 Global Step: 96095 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:00:18,638-Speed 127.37 samples/sec Loss 10.7233 Epoch: 9 Global Step: 96100 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:00:38,469-Speed 129.09 samples/sec Loss 10.7768 Epoch: 9 Global Step: 96105 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:00:58,204-Speed 129.73 samples/sec Loss 10.5298 Epoch: 9 Global Step: 96110 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:01:17,987-Speed 129.41 samples/sec Loss 10.8623 Epoch: 9 Global Step: 96115 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:01:37,819-Speed 129.08 samples/sec Loss 10.6627 Epoch: 9 Global Step: 96120 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:01:58,043-Speed 126.59 samples/sec Loss 10.8754 Epoch: 9 Global Step: 96125 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:02:17,720-Speed 130.10 samples/sec Loss 10.8508 Epoch: 9 Global Step: 96130 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:02:37,649-Speed 128.47 samples/sec Loss 10.8174 Epoch: 9 Global Step: 96135 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:02:57,412-Speed 129.54 samples/sec Loss 10.9290 Epoch: 9 Global Step: 96140 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:03:17,626-Speed 126.65 samples/sec Loss 10.6608 Epoch: 9 Global Step: 96145 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:03:37,345-Speed 129.83 samples/sec Loss 10.7211 Epoch: 9 Global Step: 96150 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:03:57,144-Speed 129.31 samples/sec Loss 10.6791 Epoch: 9 Global Step: 96155 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:04:16,893-Speed 129.63 samples/sec Loss 10.7779 Epoch: 9 Global Step: 96160 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:04:36,849-Speed 128.29 samples/sec Loss 10.8946 Epoch: 9 Global Step: 96165 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:04:56,862-Speed 127.92 samples/sec Loss 10.4489 Epoch: 9 Global Step: 96170 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:05:16,569-Speed 129.91 samples/sec Loss 10.7944 Epoch: 9 Global Step: 96175 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:05:36,492-Speed 128.50 samples/sec Loss 10.8318 Epoch: 9 Global Step: 96180 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:05:56,423-Speed 128.46 samples/sec Loss 10.8551 Epoch: 9 Global Step: 96185 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:06:16,539-Speed 127.27 samples/sec Loss 11.2059 Epoch: 9 Global Step: 96190 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:06:36,402-Speed 128.89 samples/sec Loss 10.7081 Epoch: 9 Global Step: 96195 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:06:56,126-Speed 129.79 samples/sec Loss 10.4694 Epoch: 9 Global Step: 96200 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:07:15,932-Speed 129.26 samples/sec Loss 10.7815 Epoch: 9 Global Step: 96205 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:07:35,911-Speed 128.14 samples/sec Loss 10.8206 Epoch: 9 Global Step: 96210 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:07:55,813-Speed 128.64 samples/sec Loss 10.5665 Epoch: 9 Global Step: 96215 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:08:15,738-Speed 128.49 samples/sec Loss 10.5260 Epoch: 9 Global Step: 96220 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:08:35,500-Speed 129.55 samples/sec Loss 10.6569 Epoch: 9 Global Step: 96225 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:08:55,282-Speed 129.42 samples/sec Loss 10.4756 Epoch: 9 Global Step: 96230 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:09:15,338-Speed 127.65 samples/sec Loss 10.1569 Epoch: 9 Global Step: 96235 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:09:35,179-Speed 129.03 samples/sec Loss 10.6732 Epoch: 9 Global Step: 96240 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:09:54,905-Speed 129.78 samples/sec Loss 10.7029 Epoch: 9 Global Step: 96245 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:10:14,695-Speed 129.37 samples/sec Loss 10.6185 Epoch: 9 Global Step: 96250 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:10:34,645-Speed 128.33 samples/sec Loss 10.8179 Epoch: 9 Global Step: 96255 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:10:54,589-Speed 128.36 samples/sec Loss 10.5365 Epoch: 9 Global Step: 96260 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:11:14,526-Speed 128.41 samples/sec Loss 10.5651 Epoch: 9 Global Step: 96265 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:11:34,301-Speed 129.46 samples/sec Loss 10.7703 Epoch: 9 Global Step: 96270 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:11:54,179-Speed 128.80 samples/sec Loss 10.4079 Epoch: 9 Global Step: 96275 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:12:14,150-Speed 128.19 samples/sec Loss 10.5591 Epoch: 9 Global Step: 96280 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:12:34,316-Speed 126.95 samples/sec Loss 10.7961 Epoch: 9 Global Step: 96285 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:12:54,100-Speed 129.40 samples/sec Loss 10.4891 Epoch: 9 Global Step: 96290 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:13:14,053-Speed 128.31 samples/sec Loss 10.5513 Epoch: 9 Global Step: 96295 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:13:33,857-Speed 129.27 samples/sec Loss 10.5153 Epoch: 9 Global Step: 96300 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:13:53,997-Speed 127.11 samples/sec Loss 10.6091 Epoch: 9 Global Step: 96305 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:14:13,819-Speed 129.16 samples/sec Loss 10.9242 Epoch: 9 Global Step: 96310 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:14:33,653-Speed 129.08 samples/sec Loss 10.5498 Epoch: 9 Global Step: 96315 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:14:53,534-Speed 128.77 samples/sec Loss 10.6019 Epoch: 9 Global Step: 96320 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:15:13,652-Speed 127.26 samples/sec Loss 10.6529 Epoch: 9 Global Step: 96325 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:15:33,306-Speed 130.26 samples/sec Loss 10.4484 Epoch: 9 Global Step: 96330 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:15:53,213-Speed 128.60 samples/sec Loss 10.6443 Epoch: 9 Global Step: 96335 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:16:13,066-Speed 128.96 samples/sec Loss 10.2964 Epoch: 9 Global Step: 96340 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:16:32,802-Speed 129.72 samples/sec Loss 10.5433 Epoch: 9 Global Step: 96345 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:16:52,878-Speed 127.52 samples/sec Loss 10.6664 Epoch: 9 Global Step: 96350 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:17:12,696-Speed 129.18 samples/sec Loss 11.0529 Epoch: 9 Global Step: 96355 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:17:32,555-Speed 128.91 samples/sec Loss 11.0158 Epoch: 9 Global Step: 96360 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:17:52,292-Speed 129.72 samples/sec Loss 10.4427 Epoch: 9 Global Step: 96365 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:18:12,315-Speed 127.86 samples/sec Loss 10.9762 Epoch: 9 Global Step: 96370 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:18:32,087-Speed 129.48 samples/sec Loss 10.7238 Epoch: 9 Global Step: 96375 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:18:52,054-Speed 128.22 samples/sec Loss 10.6704 Epoch: 9 Global Step: 96380 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:19:11,746-Speed 130.01 samples/sec Loss 10.1610 Epoch: 9 Global Step: 96385 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:19:31,674-Speed 128.47 samples/sec Loss 10.6399 Epoch: 9 Global Step: 96390 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:19:51,626-Speed 128.31 samples/sec Loss 10.8052 Epoch: 9 Global Step: 96395 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:20:11,546-Speed 128.52 samples/sec Loss 11.1082 Epoch: 9 Global Step: 96400 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:20:31,387-Speed 129.03 samples/sec Loss 10.4911 Epoch: 9 Global Step: 96405 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:20:51,177-Speed 129.37 samples/sec Loss 10.6813 Epoch: 9 Global Step: 96410 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:21:11,179-Speed 127.99 samples/sec Loss 10.6497 Epoch: 9 Global Step: 96415 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:21:31,113-Speed 128.43 samples/sec Loss 10.5503 Epoch: 9 Global Step: 96420 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:21:50,942-Speed 129.11 samples/sec Loss 10.2989 Epoch: 9 Global Step: 96425 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:22:10,875-Speed 128.44 samples/sec Loss 10.7415 Epoch: 9 Global Step: 96430 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:22:30,559-Speed 130.06 samples/sec Loss 10.8159 Epoch: 9 Global Step: 96435 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:22:50,496-Speed 128.41 samples/sec Loss 10.8035 Epoch: 9 Global Step: 96440 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:23:10,527-Speed 127.81 samples/sec Loss 10.3506 Epoch: 9 Global Step: 96445 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:23:30,280-Speed 129.61 samples/sec Loss 10.4270 Epoch: 9 Global Step: 96450 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:23:50,072-Speed 129.35 samples/sec Loss 10.7956 Epoch: 9 Global Step: 96455 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:24:10,145-Speed 127.54 samples/sec Loss 10.6089 Epoch: 9 Global Step: 96460 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:24:30,068-Speed 128.50 samples/sec Loss 10.9060 Epoch: 9 Global Step: 96465 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:24:49,835-Speed 129.51 samples/sec Loss 10.7913 Epoch: 9 Global Step: 96470 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:25:09,786-Speed 128.33 samples/sec Loss 10.3636 Epoch: 9 Global Step: 96475 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:25:29,551-Speed 129.53 samples/sec Loss 10.7444 Epoch: 9 Global Step: 96480 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:25:49,629-Speed 127.51 samples/sec Loss 10.7890 Epoch: 9 Global Step: 96485 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:26:09,546-Speed 128.54 samples/sec Loss 10.8878 Epoch: 9 Global Step: 96490 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:26:29,320-Speed 129.47 samples/sec Loss 10.8359 Epoch: 9 Global Step: 96495 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:26:49,259-Speed 128.40 samples/sec Loss 10.6114 Epoch: 9 Global Step: 96500 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:27:09,329-Speed 127.56 samples/sec Loss 11.0616 Epoch: 9 Global Step: 96505 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:27:29,304-Speed 128.17 samples/sec Loss 10.3209 Epoch: 9 Global Step: 96510 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:27:49,057-Speed 129.60 samples/sec Loss 10.9808 Epoch: 9 Global Step: 96515 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:28:09,002-Speed 128.36 samples/sec Loss 10.8444 Epoch: 9 Global Step: 96520 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:28:28,873-Speed 128.83 samples/sec Loss 10.8464 Epoch: 9 Global Step: 96525 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:28:49,083-Speed 126.68 samples/sec Loss 10.8887 Epoch: 9 Global Step: 96530 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:29:08,832-Speed 129.63 samples/sec Loss 10.8537 Epoch: 9 Global Step: 96535 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:29:28,485-Speed 130.27 samples/sec Loss 10.9885 Epoch: 9 Global Step: 96540 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:29:48,395-Speed 128.58 samples/sec Loss 10.9023 Epoch: 9 Global Step: 96545 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:30:08,608-Speed 126.66 samples/sec Loss 11.2046 Epoch: 9 Global Step: 96550 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:30:28,378-Speed 129.50 samples/sec Loss 10.8536 Epoch: 9 Global Step: 96555 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:30:48,165-Speed 129.39 samples/sec Loss 10.6811 Epoch: 9 Global Step: 96560 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:31:08,274-Speed 127.31 samples/sec Loss 10.4885 Epoch: 9 Global Step: 96565 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:31:28,040-Speed 129.52 samples/sec Loss 10.4983 Epoch: 9 Global Step: 96570 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:31:48,307-Speed 126.32 samples/sec Loss 10.4308 Epoch: 9 Global Step: 96575 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:32:08,056-Speed 129.63 samples/sec Loss 10.5425 Epoch: 9 Global Step: 96580 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:32:28,008-Speed 128.32 samples/sec Loss 10.9769 Epoch: 9 Global Step: 96585 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:32:47,839-Speed 129.10 samples/sec Loss 11.0491 Epoch: 9 Global Step: 96590 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:33:07,964-Speed 127.21 samples/sec Loss 10.6788 Epoch: 9 Global Step: 96595 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:33:27,735-Speed 129.49 samples/sec Loss 10.4961 Epoch: 9 Global Step: 96600 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:33:47,592-Speed 128.93 samples/sec Loss 10.5694 Epoch: 9 Global Step: 96605 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:34:07,339-Speed 129.64 samples/sec Loss 10.6211 Epoch: 9 Global Step: 96610 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:34:27,494-Speed 127.02 samples/sec Loss 10.6311 Epoch: 9 Global Step: 96615 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:34:47,548-Speed 127.66 samples/sec Loss 10.8774 Epoch: 9 Global Step: 96620 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:35:07,567-Speed 127.89 samples/sec Loss 10.3785 Epoch: 9 Global Step: 96625 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:35:27,407-Speed 129.04 samples/sec Loss 10.5571 Epoch: 9 Global Step: 96630 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:35:47,206-Speed 129.31 samples/sec Loss 10.4535 Epoch: 9 Global Step: 96635 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:36:07,244-Speed 127.76 samples/sec Loss 10.9849 Epoch: 9 Global Step: 96640 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:36:27,124-Speed 128.78 samples/sec Loss 10.6181 Epoch: 9 Global Step: 96645 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:36:46,941-Speed 129.19 samples/sec Loss 10.5913 Epoch: 9 Global Step: 96650 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:37:06,785-Speed 129.01 samples/sec Loss 10.7470 Epoch: 9 Global Step: 96655 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:37:26,719-Speed 128.43 samples/sec Loss 10.4631 Epoch: 9 Global Step: 96660 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:37:46,665-Speed 128.35 samples/sec Loss 10.5275 Epoch: 9 Global Step: 96665 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:38:06,555-Speed 128.71 samples/sec Loss 10.5764 Epoch: 9 Global Step: 96670 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:38:26,298-Speed 129.67 samples/sec Loss 10.4357 Epoch: 9 Global Step: 96675 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:38:46,092-Speed 129.34 samples/sec Loss 10.5065 Epoch: 9 Global Step: 96680 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:39:06,236-Speed 127.10 samples/sec Loss 10.7551 Epoch: 9 Global Step: 96685 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:39:26,174-Speed 128.40 samples/sec Loss 10.2748 Epoch: 9 Global Step: 96690 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:39:45,913-Speed 129.70 samples/sec Loss 10.6494 Epoch: 9 Global Step: 96695 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:40:05,727-Speed 129.21 samples/sec Loss 10.8064 Epoch: 9 Global Step: 96700 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:40:25,643-Speed 128.55 samples/sec Loss 10.7610 Epoch: 9 Global Step: 96705 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:40:45,706-Speed 127.60 samples/sec Loss 10.7111 Epoch: 9 Global Step: 96710 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:41:05,654-Speed 128.34 samples/sec Loss 10.8619 Epoch: 9 Global Step: 96715 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:41:25,522-Speed 128.85 samples/sec Loss 10.2636 Epoch: 9 Global Step: 96720 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:41:45,337-Speed 129.20 samples/sec Loss 10.5484 Epoch: 9 Global Step: 96725 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:42:05,404-Speed 127.58 samples/sec Loss 10.4902 Epoch: 9 Global Step: 96730 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:42:25,237-Speed 129.08 samples/sec Loss 10.8536 Epoch: 9 Global Step: 96735 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:42:44,980-Speed 129.67 samples/sec Loss 10.4194 Epoch: 9 Global Step: 96740 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:43:04,910-Speed 128.46 samples/sec Loss 10.5598 Epoch: 9 Global Step: 96745 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:43:24,681-Speed 129.49 samples/sec Loss 10.2787 Epoch: 9 Global Step: 96750 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:43:44,703-Speed 127.87 samples/sec Loss 10.5522 Epoch: 9 Global Step: 96755 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:44:04,822-Speed 127.25 samples/sec Loss 10.8301 Epoch: 9 Global Step: 96760 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:44:24,581-Speed 129.57 samples/sec Loss 10.4352 Epoch: 9 Global Step: 96765 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:44:44,375-Speed 129.34 samples/sec Loss 10.8357 Epoch: 9 Global Step: 96770 Fp16 Grad Scale: 8192 Required: 181 hours Training: 2021-06-11 13:45:04,352-Speed 128.15 samples/sec Loss 10.4522 Epoch: 9 Global Step: 96775 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:45:24,280-Speed 128.47 samples/sec Loss 10.6704 Epoch: 9 Global Step: 96780 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:45:44,099-Speed 129.18 samples/sec Loss 10.7863 Epoch: 9 Global Step: 96785 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:46:04,049-Speed 128.33 samples/sec Loss 10.8784 Epoch: 9 Global Step: 96790 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:46:23,822-Speed 129.48 samples/sec Loss 10.6363 Epoch: 9 Global Step: 96795 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:46:43,883-Speed 127.62 samples/sec Loss 10.5744 Epoch: 9 Global Step: 96800 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:47:03,872-Speed 128.08 samples/sec Loss 10.8673 Epoch: 9 Global Step: 96805 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:47:23,618-Speed 129.65 samples/sec Loss 10.5218 Epoch: 9 Global Step: 96810 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:47:43,552-Speed 128.43 samples/sec Loss 10.8877 Epoch: 9 Global Step: 96815 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:48:03,656-Speed 127.34 samples/sec Loss 10.7481 Epoch: 9 Global Step: 96820 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:48:23,502-Speed 129.00 samples/sec Loss 10.8386 Epoch: 9 Global Step: 96825 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:48:43,261-Speed 129.57 samples/sec Loss 10.5136 Epoch: 9 Global Step: 96830 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:49:03,176-Speed 128.55 samples/sec Loss 10.8719 Epoch: 9 Global Step: 96835 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:49:22,926-Speed 129.62 samples/sec Loss 10.8698 Epoch: 9 Global Step: 96840 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:49:43,158-Speed 126.54 samples/sec Loss 10.7195 Epoch: 9 Global Step: 96845 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:50:02,926-Speed 129.51 samples/sec Loss 10.6181 Epoch: 9 Global Step: 96850 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:50:22,856-Speed 128.46 samples/sec Loss 10.8653 Epoch: 9 Global Step: 96855 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:50:42,623-Speed 129.51 samples/sec Loss 10.7802 Epoch: 9 Global Step: 96860 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:51:02,717-Speed 127.41 samples/sec Loss 10.6875 Epoch: 9 Global Step: 96865 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:51:22,701-Speed 128.11 samples/sec Loss 10.7195 Epoch: 9 Global Step: 96870 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:51:42,886-Speed 126.83 samples/sec Loss 10.6936 Epoch: 9 Global Step: 96875 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:52:02,645-Speed 129.57 samples/sec Loss 10.7487 Epoch: 9 Global Step: 96880 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:52:22,364-Speed 129.83 samples/sec Loss 10.6653 Epoch: 9 Global Step: 96885 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:52:42,857-Speed 124.93 samples/sec Loss 10.9669 Epoch: 9 Global Step: 96890 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:53:02,623-Speed 129.52 samples/sec Loss 10.8091 Epoch: 9 Global Step: 96895 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:53:22,463-Speed 129.04 samples/sec Loss 10.6238 Epoch: 9 Global Step: 96900 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:53:42,132-Speed 130.16 samples/sec Loss 10.9592 Epoch: 9 Global Step: 96905 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:54:02,330-Speed 126.76 samples/sec Loss 10.4550 Epoch: 9 Global Step: 96910 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:54:22,136-Speed 129.26 samples/sec Loss 10.9360 Epoch: 9 Global Step: 96915 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:54:42,033-Speed 128.66 samples/sec Loss 10.8374 Epoch: 9 Global Step: 96920 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:55:01,697-Speed 130.20 samples/sec Loss 10.6065 Epoch: 9 Global Step: 96925 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:55:21,630-Speed 128.44 samples/sec Loss 10.9819 Epoch: 9 Global Step: 96930 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:55:41,677-Speed 127.70 samples/sec Loss 10.4457 Epoch: 9 Global Step: 96935 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:56:01,430-Speed 129.61 samples/sec Loss 10.4182 Epoch: 9 Global Step: 96940 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:56:21,263-Speed 129.09 samples/sec Loss 10.5887 Epoch: 9 Global Step: 96945 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:56:41,096-Speed 129.09 samples/sec Loss 10.7943 Epoch: 9 Global Step: 96950 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:57:01,223-Speed 127.20 samples/sec Loss 10.6113 Epoch: 9 Global Step: 96955 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:57:21,023-Speed 129.30 samples/sec Loss 10.7387 Epoch: 9 Global Step: 96960 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:57:40,835-Speed 129.22 samples/sec Loss 10.8232 Epoch: 9 Global Step: 96965 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:58:00,511-Speed 130.12 samples/sec Loss 10.5930 Epoch: 9 Global Step: 96970 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:58:20,567-Speed 127.65 samples/sec Loss 10.7233 Epoch: 9 Global Step: 96975 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:58:40,594-Speed 127.83 samples/sec Loss 10.7644 Epoch: 9 Global Step: 96980 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:59:00,394-Speed 129.30 samples/sec Loss 10.3725 Epoch: 9 Global Step: 96985 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:59:20,354-Speed 128.26 samples/sec Loss 10.7744 Epoch: 9 Global Step: 96990 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 13:59:40,248-Speed 128.69 samples/sec Loss 10.7517 Epoch: 9 Global Step: 96995 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:00:00,361-Speed 127.29 samples/sec Loss 10.6470 Epoch: 9 Global Step: 97000 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:00:20,194-Speed 129.08 samples/sec Loss 10.8395 Epoch: 9 Global Step: 97005 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:00:39,888-Speed 130.00 samples/sec Loss 10.7406 Epoch: 9 Global Step: 97010 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:00:59,805-Speed 128.54 samples/sec Loss 10.8411 Epoch: 9 Global Step: 97015 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:01:19,733-Speed 128.46 samples/sec Loss 10.8363 Epoch: 9 Global Step: 97020 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:01:39,675-Speed 128.38 samples/sec Loss 10.5864 Epoch: 9 Global Step: 97025 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:01:59,548-Speed 128.82 samples/sec Loss 10.5558 Epoch: 9 Global Step: 97030 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:02:19,382-Speed 129.08 samples/sec Loss 10.7315 Epoch: 9 Global Step: 97035 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:02:39,196-Speed 129.21 samples/sec Loss 10.9503 Epoch: 9 Global Step: 97040 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:02:59,189-Speed 128.05 samples/sec Loss 10.6115 Epoch: 9 Global Step: 97045 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:03:19,030-Speed 129.03 samples/sec Loss 10.6073 Epoch: 9 Global Step: 97050 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:03:38,731-Speed 129.95 samples/sec Loss 10.7121 Epoch: 9 Global Step: 97055 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:03:59,300-Speed 124.47 samples/sec Loss 10.6931 Epoch: 9 Global Step: 97060 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:04:19,091-Speed 129.36 samples/sec Loss 10.7247 Epoch: 9 Global Step: 97065 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:04:39,107-Speed 127.91 samples/sec Loss 10.6349 Epoch: 9 Global Step: 97070 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:04:59,097-Speed 128.07 samples/sec Loss 10.8464 Epoch: 9 Global Step: 97075 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:05:18,843-Speed 129.65 samples/sec Loss 10.6264 Epoch: 9 Global Step: 97080 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:05:38,794-Speed 128.32 samples/sec Loss 10.7881 Epoch: 9 Global Step: 97085 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:05:58,781-Speed 128.09 samples/sec Loss 10.6749 Epoch: 9 Global Step: 97090 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:06:18,552-Speed 129.49 samples/sec Loss 10.5287 Epoch: 9 Global Step: 97095 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:06:38,433-Speed 128.77 samples/sec Loss 10.7237 Epoch: 9 Global Step: 97100 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:06:58,343-Speed 128.59 samples/sec Loss 10.7741 Epoch: 9 Global Step: 97105 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:07:18,013-Speed 130.15 samples/sec Loss 10.7332 Epoch: 9 Global Step: 97110 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:07:38,249-Speed 126.51 samples/sec Loss 10.5048 Epoch: 9 Global Step: 97115 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:07:57,993-Speed 129.67 samples/sec Loss 10.6028 Epoch: 9 Global Step: 97120 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:08:17,819-Speed 129.13 samples/sec Loss 10.8603 Epoch: 9 Global Step: 97125 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:08:37,629-Speed 129.23 samples/sec Loss 10.2214 Epoch: 9 Global Step: 97130 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:08:57,792-Speed 126.97 samples/sec Loss 10.8681 Epoch: 9 Global Step: 97135 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:09:18,363-Speed 124.46 samples/sec Loss 10.6024 Epoch: 9 Global Step: 97140 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:09:38,188-Speed 129.13 samples/sec Loss 10.6200 Epoch: 9 Global Step: 97145 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:09:58,047-Speed 128.92 samples/sec Loss 10.5454 Epoch: 9 Global Step: 97150 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:10:17,750-Speed 129.93 samples/sec Loss 10.5131 Epoch: 9 Global Step: 97155 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:10:37,998-Speed 126.44 samples/sec Loss 10.3977 Epoch: 9 Global Step: 97160 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:10:57,752-Speed 129.60 samples/sec Loss 10.4450 Epoch: 9 Global Step: 97165 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:11:17,608-Speed 128.94 samples/sec Loss 10.3612 Epoch: 9 Global Step: 97170 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:11:37,368-Speed 129.56 samples/sec Loss 10.9947 Epoch: 9 Global Step: 97175 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:11:57,484-Speed 127.27 samples/sec Loss 10.9118 Epoch: 9 Global Step: 97180 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:12:17,282-Speed 129.31 samples/sec Loss 10.6602 Epoch: 9 Global Step: 97185 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:12:37,206-Speed 128.50 samples/sec Loss 10.8311 Epoch: 9 Global Step: 97190 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:12:56,955-Speed 129.63 samples/sec Loss 10.7395 Epoch: 9 Global Step: 97195 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:13:16,682-Speed 129.78 samples/sec Loss 10.9352 Epoch: 9 Global Step: 97200 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:13:37,044-Speed 125.73 samples/sec Loss 10.3875 Epoch: 9 Global Step: 97205 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:13:56,806-Speed 129.55 samples/sec Loss 10.7770 Epoch: 9 Global Step: 97210 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:14:16,756-Speed 128.33 samples/sec Loss 10.9354 Epoch: 9 Global Step: 97215 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:14:36,410-Speed 130.26 samples/sec Loss 10.6834 Epoch: 9 Global Step: 97220 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:14:56,610-Speed 126.74 samples/sec Loss 10.6399 Epoch: 9 Global Step: 97225 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:15:16,612-Speed 127.99 samples/sec Loss 10.8450 Epoch: 9 Global Step: 97230 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:15:36,685-Speed 127.53 samples/sec Loss 10.8986 Epoch: 9 Global Step: 97235 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:15:56,548-Speed 128.89 samples/sec Loss 11.0518 Epoch: 9 Global Step: 97240 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:16:16,364-Speed 129.19 samples/sec Loss 10.6032 Epoch: 9 Global Step: 97245 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:16:36,508-Speed 127.09 samples/sec Loss 10.6207 Epoch: 9 Global Step: 97250 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:16:56,413-Speed 128.62 samples/sec Loss 10.4235 Epoch: 9 Global Step: 97255 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:17:16,123-Speed 129.89 samples/sec Loss 10.5360 Epoch: 9 Global Step: 97260 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:17:36,137-Speed 127.92 samples/sec Loss 10.6358 Epoch: 9 Global Step: 97265 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:17:56,218-Speed 127.49 samples/sec Loss 10.5968 Epoch: 9 Global Step: 97270 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:18:16,183-Speed 128.23 samples/sec Loss 10.9166 Epoch: 9 Global Step: 97275 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:18:35,871-Speed 130.03 samples/sec Loss 10.7434 Epoch: 9 Global Step: 97280 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:18:55,757-Speed 128.74 samples/sec Loss 10.5918 Epoch: 9 Global Step: 97285 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:19:15,611-Speed 128.95 samples/sec Loss 10.5965 Epoch: 9 Global Step: 97290 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:19:35,574-Speed 128.24 samples/sec Loss 10.3833 Epoch: 9 Global Step: 97295 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:19:55,517-Speed 128.37 samples/sec Loss 10.4706 Epoch: 9 Global Step: 97300 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:20:15,283-Speed 129.52 samples/sec Loss 10.8908 Epoch: 9 Global Step: 97305 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:20:35,185-Speed 128.64 samples/sec Loss 10.8033 Epoch: 9 Global Step: 97310 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:20:55,192-Speed 127.96 samples/sec Loss 10.5039 Epoch: 9 Global Step: 97315 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:21:15,116-Speed 128.49 samples/sec Loss 10.9967 Epoch: 9 Global Step: 97320 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:21:34,749-Speed 130.39 samples/sec Loss 10.6932 Epoch: 9 Global Step: 97325 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:21:54,768-Speed 127.89 samples/sec Loss 10.5749 Epoch: 9 Global Step: 97330 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:22:14,622-Speed 128.95 samples/sec Loss 10.7352 Epoch: 9 Global Step: 97335 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:22:34,709-Speed 127.45 samples/sec Loss 10.3517 Epoch: 9 Global Step: 97340 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:22:54,534-Speed 129.14 samples/sec Loss 10.5976 Epoch: 9 Global Step: 97345 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:23:14,382-Speed 128.99 samples/sec Loss 10.6824 Epoch: 9 Global Step: 97350 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:23:34,308-Speed 128.48 samples/sec Loss 11.1096 Epoch: 9 Global Step: 97355 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:23:54,227-Speed 128.53 samples/sec Loss 10.8739 Epoch: 9 Global Step: 97360 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:24:14,132-Speed 128.61 samples/sec Loss 10.6297 Epoch: 9 Global Step: 97365 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:24:33,885-Speed 129.61 samples/sec Loss 10.6213 Epoch: 9 Global Step: 97370 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:24:53,824-Speed 128.40 samples/sec Loss 10.6436 Epoch: 9 Global Step: 97375 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:25:13,481-Speed 130.24 samples/sec Loss 10.5382 Epoch: 9 Global Step: 97380 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:25:33,629-Speed 127.07 samples/sec Loss 10.3686 Epoch: 9 Global Step: 97385 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:25:53,469-Speed 129.04 samples/sec Loss 10.6072 Epoch: 9 Global Step: 97390 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:26:13,325-Speed 128.93 samples/sec Loss 10.6953 Epoch: 9 Global Step: 97395 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:26:33,221-Speed 128.68 samples/sec Loss 10.7770 Epoch: 9 Global Step: 97400 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:26:53,233-Speed 127.93 samples/sec Loss 10.6167 Epoch: 9 Global Step: 97405 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:27:13,006-Speed 129.47 samples/sec Loss 10.9250 Epoch: 9 Global Step: 97410 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:27:32,838-Speed 129.09 samples/sec Loss 10.7407 Epoch: 9 Global Step: 97415 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:27:52,739-Speed 128.64 samples/sec Loss 10.8396 Epoch: 9 Global Step: 97420 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:28:12,431-Speed 130.01 samples/sec Loss 10.5009 Epoch: 9 Global Step: 97425 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:28:32,703-Speed 126.29 samples/sec Loss 10.9773 Epoch: 9 Global Step: 97430 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:28:52,537-Speed 129.08 samples/sec Loss 10.7859 Epoch: 9 Global Step: 97435 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:29:12,383-Speed 129.00 samples/sec Loss 10.7071 Epoch: 9 Global Step: 97440 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:29:32,083-Speed 129.96 samples/sec Loss 10.6042 Epoch: 9 Global Step: 97445 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:29:52,229-Speed 127.08 samples/sec Loss 10.6973 Epoch: 9 Global Step: 97450 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:30:12,055-Speed 129.13 samples/sec Loss 10.8000 Epoch: 9 Global Step: 97455 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:30:31,981-Speed 128.48 samples/sec Loss 10.8267 Epoch: 9 Global Step: 97460 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:30:51,706-Speed 129.79 samples/sec Loss 10.4972 Epoch: 9 Global Step: 97465 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:31:11,676-Speed 128.20 samples/sec Loss 11.0956 Epoch: 9 Global Step: 97470 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:31:31,717-Speed 127.74 samples/sec Loss 11.1014 Epoch: 9 Global Step: 97475 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:31:51,505-Speed 129.38 samples/sec Loss 10.6330 Epoch: 9 Global Step: 97480 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:32:11,396-Speed 128.71 samples/sec Loss 11.0174 Epoch: 9 Global Step: 97485 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:32:31,164-Speed 129.51 samples/sec Loss 10.9987 Epoch: 9 Global Step: 97490 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:32:51,363-Speed 126.75 samples/sec Loss 10.7935 Epoch: 9 Global Step: 97495 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:33:11,285-Speed 128.50 samples/sec Loss 10.8157 Epoch: 9 Global Step: 97500 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:33:31,032-Speed 129.65 samples/sec Loss 10.7346 Epoch: 9 Global Step: 97505 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:33:50,754-Speed 129.81 samples/sec Loss 10.7113 Epoch: 9 Global Step: 97510 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:34:10,729-Speed 128.17 samples/sec Loss 10.1584 Epoch: 9 Global Step: 97515 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:34:30,825-Speed 127.39 samples/sec Loss 10.7812 Epoch: 9 Global Step: 97520 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:34:50,753-Speed 128.47 samples/sec Loss 10.7910 Epoch: 9 Global Step: 97525 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:35:10,409-Speed 130.25 samples/sec Loss 10.9642 Epoch: 9 Global Step: 97530 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:35:30,206-Speed 129.32 samples/sec Loss 10.6218 Epoch: 9 Global Step: 97535 Fp16 Grad Scale: 8192 Required: 180 hours Training: 2021-06-11 14:35:50,329-Speed 127.23 samples/sec Loss 10.9246 Epoch: 9 Global Step: 97540 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:36:10,241-Speed 128.57 samples/sec Loss 10.6373 Epoch: 9 Global Step: 97545 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:36:29,940-Speed 129.97 samples/sec Loss 10.5024 Epoch: 9 Global Step: 97550 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:36:49,826-Speed 128.74 samples/sec Loss 10.6405 Epoch: 9 Global Step: 97555 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:37:09,651-Speed 129.13 samples/sec Loss 10.2944 Epoch: 9 Global Step: 97560 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:37:29,710-Speed 127.63 samples/sec Loss 10.2825 Epoch: 9 Global Step: 97565 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:37:49,599-Speed 128.72 samples/sec Loss 10.7064 Epoch: 9 Global Step: 97570 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:38:09,544-Speed 128.36 samples/sec Loss 10.5782 Epoch: 9 Global Step: 97575 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:38:29,527-Speed 128.12 samples/sec Loss 10.6138 Epoch: 9 Global Step: 97580 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:38:49,413-Speed 128.74 samples/sec Loss 10.6287 Epoch: 9 Global Step: 97585 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:39:09,294-Speed 128.77 samples/sec Loss 10.7266 Epoch: 9 Global Step: 97590 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:39:29,032-Speed 129.71 samples/sec Loss 10.6143 Epoch: 9 Global Step: 97595 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:39:48,982-Speed 128.33 samples/sec Loss 10.5587 Epoch: 9 Global Step: 97600 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:40:08,801-Speed 129.17 samples/sec Loss 10.6333 Epoch: 9 Global Step: 97605 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:40:29,014-Speed 126.65 samples/sec Loss 10.7013 Epoch: 9 Global Step: 97610 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:40:48,753-Speed 129.70 samples/sec Loss 10.6867 Epoch: 9 Global Step: 97615 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:41:08,559-Speed 129.26 samples/sec Loss 10.6443 Epoch: 9 Global Step: 97620 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:41:28,433-Speed 128.82 samples/sec Loss 11.0248 Epoch: 9 Global Step: 97625 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:41:48,783-Speed 125.81 samples/sec Loss 10.4231 Epoch: 9 Global Step: 97630 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:42:08,571-Speed 129.38 samples/sec Loss 10.6823 Epoch: 9 Global Step: 97635 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:42:28,397-Speed 129.13 samples/sec Loss 10.4976 Epoch: 9 Global Step: 97640 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:42:48,168-Speed 129.49 samples/sec Loss 10.8038 Epoch: 9 Global Step: 97645 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:43:08,052-Speed 128.75 samples/sec Loss 10.4865 Epoch: 9 Global Step: 97650 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:43:28,194-Speed 127.10 samples/sec Loss 10.7777 Epoch: 9 Global Step: 97655 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:43:47,978-Speed 129.41 samples/sec Loss 10.5493 Epoch: 9 Global Step: 97660 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:44:07,805-Speed 129.12 samples/sec Loss 10.5372 Epoch: 9 Global Step: 97665 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:44:27,629-Speed 129.14 samples/sec Loss 10.8100 Epoch: 9 Global Step: 97670 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:44:47,744-Speed 127.28 samples/sec Loss 10.9383 Epoch: 9 Global Step: 97675 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:45:07,599-Speed 128.94 samples/sec Loss 10.5365 Epoch: 9 Global Step: 97680 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:45:27,500-Speed 128.69 samples/sec Loss 10.3968 Epoch: 9 Global Step: 97685 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:45:47,345-Speed 129.01 samples/sec Loss 10.5630 Epoch: 9 Global Step: 97690 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:46:08,121-Speed 123.23 samples/sec Loss 10.5574 Epoch: 9 Global Step: 97695 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:46:28,142-Speed 127.87 samples/sec Loss 10.6558 Epoch: 9 Global Step: 97700 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:46:47,830-Speed 130.04 samples/sec Loss 10.5956 Epoch: 9 Global Step: 97705 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:47:08,837-Speed 121.87 samples/sec Loss 10.7574 Epoch: 9 Global Step: 97710 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:47:28,572-Speed 129.73 samples/sec Loss 10.7189 Epoch: 9 Global Step: 97715 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:47:48,695-Speed 127.22 samples/sec Loss 10.5863 Epoch: 9 Global Step: 97720 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:48:08,653-Speed 128.28 samples/sec Loss 10.9048 Epoch: 9 Global Step: 97725 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:48:28,435-Speed 129.42 samples/sec Loss 10.7371 Epoch: 9 Global Step: 97730 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:48:48,446-Speed 127.94 samples/sec Loss 10.8382 Epoch: 9 Global Step: 97735 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:49:08,100-Speed 130.26 samples/sec Loss 10.6538 Epoch: 9 Global Step: 97740 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:49:28,138-Speed 127.76 samples/sec Loss 10.8267 Epoch: 9 Global Step: 97745 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:49:48,093-Speed 128.30 samples/sec Loss 10.6154 Epoch: 9 Global Step: 97750 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:50:07,873-Speed 129.43 samples/sec Loss 10.4758 Epoch: 9 Global Step: 97755 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:50:27,806-Speed 128.44 samples/sec Loss 10.8634 Epoch: 9 Global Step: 97760 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:50:47,815-Speed 127.95 samples/sec Loss 10.4156 Epoch: 9 Global Step: 97765 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:51:07,629-Speed 129.20 samples/sec Loss 10.9080 Epoch: 9 Global Step: 97770 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:51:27,442-Speed 129.22 samples/sec Loss 10.5392 Epoch: 9 Global Step: 97775 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:51:47,340-Speed 128.66 samples/sec Loss 10.4490 Epoch: 9 Global Step: 97780 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:52:07,108-Speed 129.51 samples/sec Loss 10.9546 Epoch: 9 Global Step: 97785 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:52:27,284-Speed 126.89 samples/sec Loss 10.5712 Epoch: 9 Global Step: 97790 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:52:47,113-Speed 129.11 samples/sec Loss 10.4331 Epoch: 9 Global Step: 97795 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:53:06,872-Speed 129.57 samples/sec Loss 10.6972 Epoch: 9 Global Step: 97800 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:53:26,814-Speed 128.38 samples/sec Loss 10.6286 Epoch: 9 Global Step: 97805 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:53:46,840-Speed 127.84 samples/sec Loss 10.1620 Epoch: 9 Global Step: 97810 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:54:06,696-Speed 128.93 samples/sec Loss 10.7491 Epoch: 9 Global Step: 97815 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:54:26,884-Speed 126.81 samples/sec Loss 10.7507 Epoch: 9 Global Step: 97820 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:54:46,815-Speed 128.45 samples/sec Loss 10.5498 Epoch: 9 Global Step: 97825 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:55:06,676-Speed 128.91 samples/sec Loss 10.8857 Epoch: 9 Global Step: 97830 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:55:26,887-Speed 126.66 samples/sec Loss 10.6108 Epoch: 9 Global Step: 97835 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:55:46,646-Speed 129.57 samples/sec Loss 10.5488 Epoch: 9 Global Step: 97840 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:56:06,665-Speed 127.89 samples/sec Loss 10.3854 Epoch: 9 Global Step: 97845 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:56:26,434-Speed 129.50 samples/sec Loss 10.8388 Epoch: 9 Global Step: 97850 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:56:46,457-Speed 127.86 samples/sec Loss 10.5191 Epoch: 9 Global Step: 97855 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:57:06,112-Speed 130.25 samples/sec Loss 10.4101 Epoch: 9 Global Step: 97860 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:57:26,116-Speed 127.98 samples/sec Loss 10.2188 Epoch: 9 Global Step: 97865 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:57:45,848-Speed 129.75 samples/sec Loss 10.5011 Epoch: 9 Global Step: 97870 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:58:05,685-Speed 129.06 samples/sec Loss 10.5110 Epoch: 9 Global Step: 97875 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:58:25,839-Speed 127.03 samples/sec Loss 10.7169 Epoch: 9 Global Step: 97880 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:58:45,606-Speed 129.51 samples/sec Loss 10.4771 Epoch: 9 Global Step: 97885 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:59:05,431-Speed 129.14 samples/sec Loss 10.3877 Epoch: 9 Global Step: 97890 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:59:25,199-Speed 129.51 samples/sec Loss 10.3936 Epoch: 9 Global Step: 97895 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 14:59:45,235-Speed 127.78 samples/sec Loss 10.5777 Epoch: 9 Global Step: 97900 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:00:05,064-Speed 129.11 samples/sec Loss 10.8723 Epoch: 9 Global Step: 97905 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:00:24,976-Speed 128.57 samples/sec Loss 10.7965 Epoch: 9 Global Step: 97910 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:00:44,657-Speed 130.07 samples/sec Loss 10.4579 Epoch: 9 Global Step: 97915 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:01:04,594-Speed 128.41 samples/sec Loss 10.7196 Epoch: 9 Global Step: 97920 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:01:24,731-Speed 127.14 samples/sec Loss 10.6250 Epoch: 9 Global Step: 97925 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:01:44,516-Speed 129.40 samples/sec Loss 11.0025 Epoch: 9 Global Step: 97930 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:02:04,329-Speed 129.21 samples/sec Loss 10.5981 Epoch: 9 Global Step: 97935 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:02:24,119-Speed 129.37 samples/sec Loss 10.9209 Epoch: 9 Global Step: 97940 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:02:44,246-Speed 127.20 samples/sec Loss 10.5888 Epoch: 9 Global Step: 97945 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:03:04,252-Speed 127.97 samples/sec Loss 10.6552 Epoch: 9 Global Step: 97950 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:03:23,982-Speed 129.76 samples/sec Loss 10.9021 Epoch: 9 Global Step: 97955 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:03:43,799-Speed 129.18 samples/sec Loss 10.3292 Epoch: 9 Global Step: 97960 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:04:03,628-Speed 129.11 samples/sec Loss 10.5840 Epoch: 9 Global Step: 97965 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:04:23,723-Speed 127.40 samples/sec Loss 10.4821 Epoch: 9 Global Step: 97970 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:04:43,620-Speed 128.67 samples/sec Loss 10.6657 Epoch: 9 Global Step: 97975 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:05:03,376-Speed 129.59 samples/sec Loss 10.3267 Epoch: 9 Global Step: 97980 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:05:23,154-Speed 129.44 samples/sec Loss 10.6674 Epoch: 9 Global Step: 97985 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:05:43,157-Speed 127.99 samples/sec Loss 10.4808 Epoch: 9 Global Step: 97990 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:06:03,175-Speed 127.88 samples/sec Loss 10.8630 Epoch: 9 Global Step: 97995 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:06:23,060-Speed 128.75 samples/sec Loss 10.6329 Epoch: 9 Global Step: 98000 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:06:42,984-Speed 128.50 samples/sec Loss 10.4973 Epoch: 9 Global Step: 98005 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:07:02,752-Speed 129.51 samples/sec Loss 10.6835 Epoch: 9 Global Step: 98010 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:07:22,940-Speed 126.81 samples/sec Loss 10.8810 Epoch: 9 Global Step: 98015 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:07:42,831-Speed 128.71 samples/sec Loss 10.8439 Epoch: 9 Global Step: 98020 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:08:02,577-Speed 129.66 samples/sec Loss 10.9036 Epoch: 9 Global Step: 98025 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:08:22,426-Speed 128.98 samples/sec Loss 10.5442 Epoch: 9 Global Step: 98030 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:08:42,530-Speed 127.34 samples/sec Loss 10.4777 Epoch: 9 Global Step: 98035 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:09:02,353-Speed 129.15 samples/sec Loss 10.4472 Epoch: 9 Global Step: 98040 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:09:22,128-Speed 129.46 samples/sec Loss 10.5012 Epoch: 9 Global Step: 98045 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:09:42,038-Speed 128.59 samples/sec Loss 10.7031 Epoch: 9 Global Step: 98050 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:10:01,696-Speed 130.23 samples/sec Loss 10.5443 Epoch: 9 Global Step: 98055 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:10:21,961-Speed 126.34 samples/sec Loss 10.4591 Epoch: 9 Global Step: 98060 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:10:41,704-Speed 129.67 samples/sec Loss 10.4094 Epoch: 9 Global Step: 98065 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:11:01,553-Speed 128.98 samples/sec Loss 10.8048 Epoch: 9 Global Step: 98070 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:11:21,422-Speed 128.85 samples/sec Loss 10.8225 Epoch: 9 Global Step: 98075 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:11:41,530-Speed 127.32 samples/sec Loss 10.9362 Epoch: 9 Global Step: 98080 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:12:01,278-Speed 129.64 samples/sec Loss 10.9470 Epoch: 9 Global Step: 98085 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:12:21,242-Speed 128.23 samples/sec Loss 10.6121 Epoch: 9 Global Step: 98090 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:12:40,957-Speed 129.86 samples/sec Loss 10.8281 Epoch: 9 Global Step: 98095 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:13:00,956-Speed 128.01 samples/sec Loss 10.5798 Epoch: 9 Global Step: 98100 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:13:21,099-Speed 127.09 samples/sec Loss 10.7759 Epoch: 9 Global Step: 98105 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:13:40,786-Speed 130.04 samples/sec Loss 10.2298 Epoch: 9 Global Step: 98110 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:14:00,599-Speed 129.21 samples/sec Loss 10.8694 Epoch: 9 Global Step: 98115 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:14:20,444-Speed 129.01 samples/sec Loss 10.7960 Epoch: 9 Global Step: 98120 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:14:40,556-Speed 127.30 samples/sec Loss 10.4460 Epoch: 9 Global Step: 98125 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:15:00,417-Speed 128.90 samples/sec Loss 10.4992 Epoch: 9 Global Step: 98130 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:15:20,192-Speed 129.46 samples/sec Loss 10.4343 Epoch: 9 Global Step: 98135 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:15:39,960-Speed 129.51 samples/sec Loss 10.6439 Epoch: 9 Global Step: 98140 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:16:00,150-Speed 126.80 samples/sec Loss 10.5320 Epoch: 9 Global Step: 98145 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:16:20,142-Speed 128.06 samples/sec Loss 10.6580 Epoch: 9 Global Step: 98150 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:16:39,993-Speed 128.97 samples/sec Loss 10.7347 Epoch: 9 Global Step: 98155 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:16:59,649-Speed 130.25 samples/sec Loss 10.5518 Epoch: 9 Global Step: 98160 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:17:19,534-Speed 128.74 samples/sec Loss 10.7185 Epoch: 9 Global Step: 98165 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:17:39,633-Speed 127.38 samples/sec Loss 10.3891 Epoch: 9 Global Step: 98170 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:17:59,528-Speed 128.68 samples/sec Loss 10.7731 Epoch: 9 Global Step: 98175 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:18:19,260-Speed 129.74 samples/sec Loss 10.4588 Epoch: 9 Global Step: 98180 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:18:39,204-Speed 128.37 samples/sec Loss 10.5980 Epoch: 9 Global Step: 98185 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:18:59,041-Speed 129.06 samples/sec Loss 10.2456 Epoch: 9 Global Step: 98190 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:19:19,037-Speed 128.03 samples/sec Loss 10.7834 Epoch: 9 Global Step: 98195 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:19:38,920-Speed 128.76 samples/sec Loss 10.5175 Epoch: 9 Global Step: 98200 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:19:58,769-Speed 128.98 samples/sec Loss 10.5535 Epoch: 9 Global Step: 98205 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:20:18,709-Speed 128.39 samples/sec Loss 10.5880 Epoch: 9 Global Step: 98210 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:20:38,715-Speed 127.97 samples/sec Loss 10.6655 Epoch: 9 Global Step: 98215 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:20:58,518-Speed 129.28 samples/sec Loss 10.5951 Epoch: 9 Global Step: 98220 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:21:18,266-Speed 129.64 samples/sec Loss 10.9850 Epoch: 9 Global Step: 98225 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:21:38,203-Speed 128.41 samples/sec Loss 10.5969 Epoch: 9 Global Step: 98230 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:21:57,956-Speed 129.61 samples/sec Loss 10.9277 Epoch: 9 Global Step: 98235 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:22:18,001-Speed 127.71 samples/sec Loss 10.5477 Epoch: 9 Global Step: 98240 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:22:37,848-Speed 129.00 samples/sec Loss 10.4247 Epoch: 9 Global Step: 98245 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:22:57,640-Speed 129.35 samples/sec Loss 10.8450 Epoch: 9 Global Step: 98250 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:23:17,553-Speed 128.57 samples/sec Loss 10.7312 Epoch: 9 Global Step: 98255 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:23:37,579-Speed 127.84 samples/sec Loss 11.0450 Epoch: 9 Global Step: 98260 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:23:57,398-Speed 129.18 samples/sec Loss 10.7737 Epoch: 9 Global Step: 98265 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:24:17,121-Speed 129.81 samples/sec Loss 10.4914 Epoch: 9 Global Step: 98270 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:24:37,415-Speed 126.15 samples/sec Loss 11.1905 Epoch: 9 Global Step: 98275 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:24:57,134-Speed 129.83 samples/sec Loss 10.6506 Epoch: 9 Global Step: 98280 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:25:17,324-Speed 126.80 samples/sec Loss 10.6032 Epoch: 9 Global Step: 98285 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:25:37,091-Speed 129.52 samples/sec Loss 10.6917 Epoch: 9 Global Step: 98290 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:25:57,038-Speed 128.35 samples/sec Loss 10.4610 Epoch: 9 Global Step: 98295 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:26:16,697-Speed 130.23 samples/sec Loss 10.7873 Epoch: 9 Global Step: 98300 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:26:36,710-Speed 127.93 samples/sec Loss 11.1141 Epoch: 9 Global Step: 98305 Fp16 Grad Scale: 8192 Required: 179 hours Training: 2021-06-11 15:26:56,529-Speed 129.17 samples/sec Loss 10.5368 Epoch: 9 Global Step: 98310 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:27:16,599-Speed 127.56 samples/sec Loss 10.5292 Epoch: 9 Global Step: 98315 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:27:36,371-Speed 129.49 samples/sec Loss 10.7893 Epoch: 9 Global Step: 98320 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:27:56,064-Speed 130.00 samples/sec Loss 10.3281 Epoch: 9 Global Step: 98325 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:28:16,272-Speed 126.69 samples/sec Loss 10.6402 Epoch: 9 Global Step: 98330 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:28:36,046-Speed 129.48 samples/sec Loss 10.7639 Epoch: 9 Global Step: 98335 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:28:55,966-Speed 128.52 samples/sec Loss 10.7915 Epoch: 9 Global Step: 98340 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:29:15,630-Speed 130.19 samples/sec Loss 10.7908 Epoch: 9 Global Step: 98345 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:29:35,686-Speed 127.65 samples/sec Loss 10.9993 Epoch: 9 Global Step: 98350 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:29:55,449-Speed 129.54 samples/sec Loss 10.6739 Epoch: 9 Global Step: 98355 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:30:15,356-Speed 128.60 samples/sec Loss 10.5040 Epoch: 9 Global Step: 98360 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:30:35,119-Speed 129.55 samples/sec Loss 10.5539 Epoch: 9 Global Step: 98365 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:30:54,963-Speed 129.01 samples/sec Loss 10.3975 Epoch: 9 Global Step: 98370 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:31:15,017-Speed 127.66 samples/sec Loss 10.7298 Epoch: 9 Global Step: 98375 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:31:34,896-Speed 128.78 samples/sec Loss 10.3025 Epoch: 9 Global Step: 98380 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:31:54,805-Speed 128.59 samples/sec Loss 10.4754 Epoch: 9 Global Step: 98385 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:32:14,460-Speed 130.25 samples/sec Loss 10.9726 Epoch: 9 Global Step: 98390 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:32:34,575-Speed 127.28 samples/sec Loss 10.6085 Epoch: 9 Global Step: 98395 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:32:54,502-Speed 128.47 samples/sec Loss 10.3310 Epoch: 9 Global Step: 98400 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:33:14,317-Speed 129.20 samples/sec Loss 10.4937 Epoch: 9 Global Step: 98405 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:33:34,002-Speed 130.06 samples/sec Loss 10.2236 Epoch: 9 Global Step: 98410 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:33:53,849-Speed 128.99 samples/sec Loss 10.7434 Epoch: 9 Global Step: 98415 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:34:13,931-Speed 127.48 samples/sec Loss 10.6693 Epoch: 9 Global Step: 98420 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:34:33,898-Speed 128.22 samples/sec Loss 10.0510 Epoch: 9 Global Step: 98425 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:34:53,559-Speed 130.22 samples/sec Loss 10.5916 Epoch: 9 Global Step: 98430 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:35:13,360-Speed 129.29 samples/sec Loss 10.4656 Epoch: 9 Global Step: 98435 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:35:33,366-Speed 127.97 samples/sec Loss 10.4263 Epoch: 9 Global Step: 98440 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:35:53,302-Speed 128.42 samples/sec Loss 10.7286 Epoch: 9 Global Step: 98445 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:36:13,189-Speed 128.73 samples/sec Loss 10.8116 Epoch: 9 Global Step: 98450 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:36:33,045-Speed 128.93 samples/sec Loss 10.6912 Epoch: 9 Global Step: 98455 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:36:52,802-Speed 129.58 samples/sec Loss 10.4911 Epoch: 9 Global Step: 98460 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:37:12,781-Speed 128.14 samples/sec Loss 10.5214 Epoch: 9 Global Step: 98465 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:37:32,769-Speed 128.09 samples/sec Loss 10.6710 Epoch: 9 Global Step: 98470 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:37:52,527-Speed 129.57 samples/sec Loss 10.4576 Epoch: 9 Global Step: 98475 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:38:12,444-Speed 128.55 samples/sec Loss 10.3341 Epoch: 9 Global Step: 98480 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:38:32,448-Speed 127.98 samples/sec Loss 10.9775 Epoch: 9 Global Step: 98485 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:38:52,384-Speed 128.42 samples/sec Loss 10.4621 Epoch: 9 Global Step: 98490 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:39:12,131-Speed 129.64 samples/sec Loss 10.8090 Epoch: 9 Global Step: 98495 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:39:32,056-Speed 128.49 samples/sec Loss 10.4493 Epoch: 9 Global Step: 98500 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:39:51,839-Speed 129.41 samples/sec Loss 10.8944 Epoch: 9 Global Step: 98505 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:40:12,026-Speed 126.82 samples/sec Loss 10.6558 Epoch: 9 Global Step: 98510 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:40:31,830-Speed 129.27 samples/sec Loss 10.2892 Epoch: 9 Global Step: 98515 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:40:51,664-Speed 129.08 samples/sec Loss 10.5506 Epoch: 9 Global Step: 98520 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:41:11,535-Speed 128.84 samples/sec Loss 10.8052 Epoch: 9 Global Step: 98525 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:41:31,551-Speed 127.90 samples/sec Loss 10.7215 Epoch: 9 Global Step: 98530 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:41:51,308-Speed 129.58 samples/sec Loss 10.4665 Epoch: 9 Global Step: 98535 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:42:11,281-Speed 128.18 samples/sec Loss 10.1938 Epoch: 9 Global Step: 98540 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:42:31,166-Speed 128.75 samples/sec Loss 10.3113 Epoch: 9 Global Step: 98545 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:42:50,989-Speed 129.15 samples/sec Loss 10.5415 Epoch: 9 Global Step: 98550 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:43:11,100-Speed 127.30 samples/sec Loss 10.8908 Epoch: 9 Global Step: 98555 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:43:30,864-Speed 129.54 samples/sec Loss 10.4158 Epoch: 9 Global Step: 98560 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:43:50,814-Speed 128.33 samples/sec Loss 10.7341 Epoch: 9 Global Step: 98565 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:44:10,573-Speed 129.57 samples/sec Loss 10.7607 Epoch: 9 Global Step: 98570 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:44:30,611-Speed 127.76 samples/sec Loss 10.7148 Epoch: 9 Global Step: 98575 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:44:50,593-Speed 128.12 samples/sec Loss 10.5794 Epoch: 9 Global Step: 98580 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:45:10,462-Speed 128.85 samples/sec Loss 10.7189 Epoch: 9 Global Step: 98585 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:45:30,326-Speed 128.88 samples/sec Loss 10.7780 Epoch: 9 Global Step: 98590 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:45:50,180-Speed 128.94 samples/sec Loss 10.8302 Epoch: 9 Global Step: 98595 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:46:10,283-Speed 127.35 samples/sec Loss 10.7667 Epoch: 9 Global Step: 98600 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:46:30,204-Speed 128.52 samples/sec Loss 10.5038 Epoch: 9 Global Step: 98605 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:46:49,984-Speed 129.43 samples/sec Loss 10.5812 Epoch: 9 Global Step: 98610 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:47:09,789-Speed 129.26 samples/sec Loss 10.4102 Epoch: 9 Global Step: 98615 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:47:29,934-Speed 127.09 samples/sec Loss 10.3266 Epoch: 9 Global Step: 98620 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:47:49,881-Speed 128.35 samples/sec Loss 10.8719 Epoch: 9 Global Step: 98625 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:48:09,557-Speed 130.11 samples/sec Loss 10.6078 Epoch: 9 Global Step: 98630 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:48:29,415-Speed 128.92 samples/sec Loss 10.7300 Epoch: 9 Global Step: 98635 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:48:49,188-Speed 129.48 samples/sec Loss 10.7598 Epoch: 9 Global Step: 98640 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:49:09,408-Speed 126.61 samples/sec Loss 10.8104 Epoch: 9 Global Step: 98645 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:49:29,372-Speed 128.24 samples/sec Loss 10.8408 Epoch: 9 Global Step: 98650 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:49:49,078-Speed 129.91 samples/sec Loss 10.4841 Epoch: 9 Global Step: 98655 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:50:09,075-Speed 128.03 samples/sec Loss 10.7785 Epoch: 9 Global Step: 98660 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:50:29,097-Speed 127.87 samples/sec Loss 10.4450 Epoch: 9 Global Step: 98665 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:50:48,985-Speed 128.72 samples/sec Loss 10.7291 Epoch: 9 Global Step: 98670 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:51:08,727-Speed 129.68 samples/sec Loss 10.4908 Epoch: 9 Global Step: 98675 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:51:28,699-Speed 128.18 samples/sec Loss 10.7511 Epoch: 9 Global Step: 98680 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:51:48,539-Speed 129.04 samples/sec Loss 10.6147 Epoch: 9 Global Step: 98685 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:52:08,503-Speed 128.23 samples/sec Loss 10.6502 Epoch: 9 Global Step: 98690 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:52:28,395-Speed 128.70 samples/sec Loss 10.9805 Epoch: 9 Global Step: 98695 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:52:48,162-Speed 129.52 samples/sec Loss 10.6329 Epoch: 9 Global Step: 98700 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:53:08,085-Speed 128.51 samples/sec Loss 10.4678 Epoch: 9 Global Step: 98705 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:53:28,210-Speed 127.21 samples/sec Loss 10.6844 Epoch: 9 Global Step: 98710 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:53:47,947-Speed 129.71 samples/sec Loss 10.7693 Epoch: 9 Global Step: 98715 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:54:07,748-Speed 129.29 samples/sec Loss 10.7291 Epoch: 9 Global Step: 98720 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:54:27,742-Speed 128.04 samples/sec Loss 10.6050 Epoch: 9 Global Step: 98725 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:54:47,441-Speed 129.96 samples/sec Loss 10.6186 Epoch: 9 Global Step: 98730 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:55:07,587-Speed 127.08 samples/sec Loss 10.7453 Epoch: 9 Global Step: 98735 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:55:27,542-Speed 128.30 samples/sec Loss 10.7894 Epoch: 9 Global Step: 98740 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:55:47,382-Speed 129.04 samples/sec Loss 10.7295 Epoch: 9 Global Step: 98745 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:56:07,156-Speed 129.48 samples/sec Loss 10.3654 Epoch: 9 Global Step: 98750 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:56:27,169-Speed 127.92 samples/sec Loss 10.6400 Epoch: 9 Global Step: 98755 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:56:46,927-Speed 129.58 samples/sec Loss 10.6881 Epoch: 9 Global Step: 98760 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:57:06,922-Speed 128.04 samples/sec Loss 10.8772 Epoch: 9 Global Step: 98765 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:57:26,606-Speed 130.06 samples/sec Loss 10.4989 Epoch: 9 Global Step: 98770 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:57:46,345-Speed 129.70 samples/sec Loss 10.3892 Epoch: 9 Global Step: 98775 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:58:06,440-Speed 127.40 samples/sec Loss 10.6206 Epoch: 9 Global Step: 98780 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:58:26,296-Speed 128.94 samples/sec Loss 10.6305 Epoch: 9 Global Step: 98785 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:58:46,227-Speed 128.45 samples/sec Loss 10.7312 Epoch: 9 Global Step: 98790 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:59:05,910-Speed 130.07 samples/sec Loss 10.4822 Epoch: 9 Global Step: 98795 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:59:26,038-Speed 127.19 samples/sec Loss 10.5317 Epoch: 9 Global Step: 98800 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 15:59:45,799-Speed 129.56 samples/sec Loss 10.6490 Epoch: 9 Global Step: 98805 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:00:05,766-Speed 128.22 samples/sec Loss 10.7782 Epoch: 9 Global Step: 98810 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:00:25,614-Speed 128.99 samples/sec Loss 10.6881 Epoch: 9 Global Step: 98815 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:00:45,508-Speed 128.69 samples/sec Loss 10.3139 Epoch: 9 Global Step: 98820 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:01:05,810-Speed 126.10 samples/sec Loss 10.6947 Epoch: 9 Global Step: 98825 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:01:25,563-Speed 129.61 samples/sec Loss 10.4812 Epoch: 9 Global Step: 98830 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:01:45,606-Speed 127.73 samples/sec Loss 10.5926 Epoch: 9 Global Step: 98835 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:02:05,373-Speed 129.52 samples/sec Loss 10.5280 Epoch: 9 Global Step: 98840 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:02:25,529-Speed 127.01 samples/sec Loss 10.4312 Epoch: 9 Global Step: 98845 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:02:45,374-Speed 129.01 samples/sec Loss 10.7543 Epoch: 9 Global Step: 98850 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:03:05,328-Speed 128.30 samples/sec Loss 10.4382 Epoch: 9 Global Step: 98855 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:03:25,083-Speed 129.59 samples/sec Loss 10.5275 Epoch: 9 Global Step: 98860 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:03:44,785-Speed 129.95 samples/sec Loss 10.4955 Epoch: 9 Global Step: 98865 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:04:05,049-Speed 126.34 samples/sec Loss 10.4665 Epoch: 9 Global Step: 98870 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:04:24,798-Speed 129.64 samples/sec Loss 10.3152 Epoch: 9 Global Step: 98875 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:04:44,646-Speed 128.99 samples/sec Loss 10.4189 Epoch: 9 Global Step: 98880 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:05:04,429-Speed 129.41 samples/sec Loss 10.7650 Epoch: 9 Global Step: 98885 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:05:24,561-Speed 127.17 samples/sec Loss 10.5084 Epoch: 9 Global Step: 98890 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:05:44,473-Speed 128.57 samples/sec Loss 10.7775 Epoch: 9 Global Step: 98895 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:06:04,350-Speed 128.80 samples/sec Loss 10.6180 Epoch: 9 Global Step: 98900 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:06:24,041-Speed 130.01 samples/sec Loss 10.6171 Epoch: 9 Global Step: 98905 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:06:43,897-Speed 128.93 samples/sec Loss 10.5205 Epoch: 9 Global Step: 98910 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:07:04,009-Speed 127.29 samples/sec Loss 10.4506 Epoch: 9 Global Step: 98915 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:07:23,939-Speed 128.46 samples/sec Loss 10.7529 Epoch: 9 Global Step: 98920 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:07:43,761-Speed 129.15 samples/sec Loss 10.7797 Epoch: 9 Global Step: 98925 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:08:03,692-Speed 128.45 samples/sec Loss 10.7354 Epoch: 9 Global Step: 98930 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:08:23,794-Speed 127.36 samples/sec Loss 10.4738 Epoch: 9 Global Step: 98935 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:08:43,833-Speed 127.76 samples/sec Loss 10.8230 Epoch: 9 Global Step: 98940 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:09:03,491-Speed 130.23 samples/sec Loss 10.9537 Epoch: 9 Global Step: 98945 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:09:23,271-Speed 129.43 samples/sec Loss 11.0387 Epoch: 9 Global Step: 98950 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:09:43,105-Speed 129.08 samples/sec Loss 10.5563 Epoch: 9 Global Step: 98955 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:10:03,213-Speed 127.32 samples/sec Loss 10.6425 Epoch: 9 Global Step: 98960 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:10:23,062-Speed 128.98 samples/sec Loss 11.1167 Epoch: 9 Global Step: 98965 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:10:42,749-Speed 130.04 samples/sec Loss 10.3971 Epoch: 9 Global Step: 98970 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:11:02,540-Speed 129.36 samples/sec Loss 10.8601 Epoch: 9 Global Step: 98975 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:11:22,670-Speed 127.18 samples/sec Loss 10.4865 Epoch: 9 Global Step: 98980 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:11:42,513-Speed 129.02 samples/sec Loss 10.4088 Epoch: 9 Global Step: 98985 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:12:02,150-Speed 130.37 samples/sec Loss 10.9110 Epoch: 9 Global Step: 98990 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:12:22,021-Speed 128.84 samples/sec Loss 10.4220 Epoch: 9 Global Step: 98995 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:12:41,844-Speed 129.15 samples/sec Loss 10.3741 Epoch: 9 Global Step: 99000 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:13:01,847-Speed 127.99 samples/sec Loss 10.7036 Epoch: 9 Global Step: 99005 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:13:21,665-Speed 129.18 samples/sec Loss 10.5782 Epoch: 9 Global Step: 99010 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:13:41,430-Speed 129.53 samples/sec Loss 11.1014 Epoch: 9 Global Step: 99015 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:14:01,463-Speed 127.80 samples/sec Loss 10.5948 Epoch: 9 Global Step: 99020 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:14:21,421-Speed 128.28 samples/sec Loss 10.5747 Epoch: 9 Global Step: 99025 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:14:41,218-Speed 129.31 samples/sec Loss 10.6981 Epoch: 9 Global Step: 99030 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:15:00,967-Speed 129.63 samples/sec Loss 11.0082 Epoch: 9 Global Step: 99035 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:15:21,338-Speed 125.68 samples/sec Loss 10.6843 Epoch: 9 Global Step: 99040 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:15:41,074-Speed 129.72 samples/sec Loss 10.7442 Epoch: 9 Global Step: 99045 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:16:01,206-Speed 127.17 samples/sec Loss 10.6239 Epoch: 9 Global Step: 99050 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:16:20,963-Speed 129.58 samples/sec Loss 10.7574 Epoch: 9 Global Step: 99055 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:16:40,966-Speed 127.99 samples/sec Loss 10.8123 Epoch: 9 Global Step: 99060 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:17:00,635-Speed 130.17 samples/sec Loss 11.0207 Epoch: 9 Global Step: 99065 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:17:20,739-Speed 127.34 samples/sec Loss 10.6538 Epoch: 9 Global Step: 99070 Fp16 Grad Scale: 8192 Required: 178 hours Training: 2021-06-11 16:17:40,388-Speed 130.29 samples/sec Loss 10.8078 Epoch: 9 Global Step: 99075 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:18:00,316-Speed 128.47 samples/sec Loss 10.7083 Epoch: 9 Global Step: 99080 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:18:20,080-Speed 129.54 samples/sec Loss 10.4882 Epoch: 9 Global Step: 99085 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:18:40,246-Speed 126.95 samples/sec Loss 10.8689 Epoch: 9 Global Step: 99090 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:19:00,297-Speed 127.68 samples/sec Loss 10.5890 Epoch: 9 Global Step: 99095 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:19:20,178-Speed 128.77 samples/sec Loss 10.8059 Epoch: 9 Global Step: 99100 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:19:40,031-Speed 128.98 samples/sec Loss 10.7274 Epoch: 9 Global Step: 99105 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:19:59,773-Speed 129.68 samples/sec Loss 10.3851 Epoch: 9 Global Step: 99110 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:20:19,794-Speed 127.87 samples/sec Loss 10.6323 Epoch: 9 Global Step: 99115 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:20:39,658-Speed 128.88 samples/sec Loss 10.8373 Epoch: 9 Global Step: 99120 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:20:59,584-Speed 128.49 samples/sec Loss 10.5947 Epoch: 9 Global Step: 99125 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:21:19,370-Speed 129.39 samples/sec Loss 10.6520 Epoch: 9 Global Step: 99130 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:21:39,220-Speed 128.97 samples/sec Loss 10.6922 Epoch: 9 Global Step: 99135 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:21:59,175-Speed 128.30 samples/sec Loss 10.7033 Epoch: 9 Global Step: 99140 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:22:19,063-Speed 128.73 samples/sec Loss 10.7892 Epoch: 9 Global Step: 99145 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:22:38,880-Speed 129.19 samples/sec Loss 10.4499 Epoch: 9 Global Step: 99150 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:22:58,622-Speed 129.68 samples/sec Loss 10.5506 Epoch: 9 Global Step: 99155 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:23:18,662-Speed 127.76 samples/sec Loss 10.5102 Epoch: 9 Global Step: 99160 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:23:38,669-Speed 127.96 samples/sec Loss 10.3564 Epoch: 9 Global Step: 99165 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:23:58,507-Speed 129.06 samples/sec Loss 10.4978 Epoch: 9 Global Step: 99170 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:24:18,271-Speed 129.53 samples/sec Loss 10.8625 Epoch: 9 Global Step: 99175 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:24:38,131-Speed 128.91 samples/sec Loss 10.5898 Epoch: 9 Global Step: 99180 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:24:58,068-Speed 128.41 samples/sec Loss 10.3649 Epoch: 9 Global Step: 99185 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:25:17,953-Speed 128.74 samples/sec Loss 10.6049 Epoch: 9 Global Step: 99190 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:25:37,863-Speed 128.59 samples/sec Loss 10.4921 Epoch: 9 Global Step: 99195 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:25:57,529-Speed 130.18 samples/sec Loss 11.0060 Epoch: 9 Global Step: 99200 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:26:17,653-Speed 127.21 samples/sec Loss 10.6224 Epoch: 9 Global Step: 99205 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:26:37,493-Speed 129.04 samples/sec Loss 10.7731 Epoch: 9 Global Step: 99210 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:26:57,316-Speed 129.15 samples/sec Loss 11.0340 Epoch: 9 Global Step: 99215 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:27:17,035-Speed 129.83 samples/sec Loss 10.8320 Epoch: 9 Global Step: 99220 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:27:36,879-Speed 129.02 samples/sec Loss 10.8544 Epoch: 9 Global Step: 99225 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:27:56,927-Speed 127.70 samples/sec Loss 10.5893 Epoch: 9 Global Step: 99230 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:28:16,845-Speed 128.53 samples/sec Loss 10.4708 Epoch: 9 Global Step: 99235 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:28:37,118-Speed 126.28 samples/sec Loss 10.8941 Epoch: 9 Global Step: 99240 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:28:56,870-Speed 129.61 samples/sec Loss 10.8473 Epoch: 9 Global Step: 99245 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:29:17,142-Speed 126.29 samples/sec Loss 10.3182 Epoch: 9 Global Step: 99250 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:29:37,160-Speed 127.89 samples/sec Loss 10.6113 Epoch: 9 Global Step: 99255 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:29:56,846-Speed 130.04 samples/sec Loss 10.6505 Epoch: 9 Global Step: 99260 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:30:16,566-Speed 129.83 samples/sec Loss 10.6583 Epoch: 9 Global Step: 99265 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:30:36,490-Speed 128.49 samples/sec Loss 10.5766 Epoch: 9 Global Step: 99270 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:30:56,556-Speed 127.58 samples/sec Loss 10.5384 Epoch: 9 Global Step: 99275 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:31:16,479-Speed 128.50 samples/sec Loss 10.5118 Epoch: 9 Global Step: 99280 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:31:36,238-Speed 129.57 samples/sec Loss 10.8294 Epoch: 9 Global Step: 99285 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:31:56,040-Speed 129.29 samples/sec Loss 10.5898 Epoch: 9 Global Step: 99290 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:32:16,192-Speed 127.04 samples/sec Loss 10.6973 Epoch: 9 Global Step: 99295 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:32:36,058-Speed 128.87 samples/sec Loss 11.0655 Epoch: 9 Global Step: 99300 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:32:55,698-Speed 130.36 samples/sec Loss 10.6602 Epoch: 9 Global Step: 99305 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:33:15,648-Speed 128.32 samples/sec Loss 10.2176 Epoch: 9 Global Step: 99310 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:33:35,407-Speed 129.57 samples/sec Loss 10.7793 Epoch: 9 Global Step: 99315 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:33:55,566-Speed 126.99 samples/sec Loss 10.5872 Epoch: 9 Global Step: 99320 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:34:15,447-Speed 128.77 samples/sec Loss 11.0876 Epoch: 9 Global Step: 99325 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:34:35,189-Speed 129.68 samples/sec Loss 11.0211 Epoch: 9 Global Step: 99330 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:34:55,269-Speed 127.50 samples/sec Loss 10.8758 Epoch: 9 Global Step: 99335 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:35:15,239-Speed 128.20 samples/sec Loss 10.4550 Epoch: 9 Global Step: 99340 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:35:35,140-Speed 128.64 samples/sec Loss 10.5555 Epoch: 9 Global Step: 99345 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:35:54,885-Speed 129.66 samples/sec Loss 10.9226 Epoch: 9 Global Step: 99350 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:36:14,855-Speed 128.20 samples/sec Loss 11.2206 Epoch: 9 Global Step: 99355 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:36:34,917-Speed 127.61 samples/sec Loss 10.3096 Epoch: 9 Global Step: 99360 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:36:55,102-Speed 126.83 samples/sec Loss 10.7681 Epoch: 9 Global Step: 99365 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:37:14,874-Speed 129.48 samples/sec Loss 10.6844 Epoch: 9 Global Step: 99370 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:37:34,673-Speed 129.31 samples/sec Loss 10.2594 Epoch: 9 Global Step: 99375 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:37:54,672-Speed 128.02 samples/sec Loss 10.9247 Epoch: 9 Global Step: 99380 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:38:14,795-Speed 127.23 samples/sec Loss 10.7066 Epoch: 9 Global Step: 99385 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:38:34,559-Speed 129.53 samples/sec Loss 10.8020 Epoch: 9 Global Step: 99390 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:38:54,353-Speed 129.34 samples/sec Loss 10.5639 Epoch: 9 Global Step: 99395 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:39:14,352-Speed 128.01 samples/sec Loss 10.5651 Epoch: 9 Global Step: 99400 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:39:34,053-Speed 129.95 samples/sec Loss 10.4230 Epoch: 9 Global Step: 99405 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:39:54,245-Speed 126.78 samples/sec Loss 10.6307 Epoch: 9 Global Step: 99410 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:40:14,006-Speed 129.56 samples/sec Loss 10.4261 Epoch: 9 Global Step: 99415 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:40:34,002-Speed 128.03 samples/sec Loss 10.6359 Epoch: 9 Global Step: 99420 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:40:53,749-Speed 129.64 samples/sec Loss 10.7650 Epoch: 9 Global Step: 99425 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:41:13,935-Speed 126.83 samples/sec Loss 10.4923 Epoch: 9 Global Step: 99430 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:41:33,595-Speed 130.22 samples/sec Loss 10.6954 Epoch: 9 Global Step: 99435 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:41:53,541-Speed 128.35 samples/sec Loss 10.7553 Epoch: 9 Global Step: 99440 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:42:13,341-Speed 129.30 samples/sec Loss 10.1169 Epoch: 9 Global Step: 99445 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:42:33,321-Speed 128.13 samples/sec Loss 10.7166 Epoch: 9 Global Step: 99450 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:42:53,292-Speed 128.19 samples/sec Loss 10.7045 Epoch: 9 Global Step: 99455 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:43:13,164-Speed 128.83 samples/sec Loss 10.4563 Epoch: 9 Global Step: 99460 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:43:32,995-Speed 129.10 samples/sec Loss 10.8013 Epoch: 9 Global Step: 99465 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:43:52,854-Speed 128.92 samples/sec Loss 10.5259 Epoch: 9 Global Step: 99470 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:44:12,878-Speed 127.85 samples/sec Loss 10.3843 Epoch: 9 Global Step: 99475 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:44:32,818-Speed 128.39 samples/sec Loss 10.3131 Epoch: 9 Global Step: 99480 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:44:52,632-Speed 129.21 samples/sec Loss 10.4809 Epoch: 9 Global Step: 99485 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:45:12,430-Speed 129.31 samples/sec Loss 10.4781 Epoch: 9 Global Step: 99490 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:45:32,333-Speed 128.63 samples/sec Loss 10.2727 Epoch: 9 Global Step: 99495 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:45:52,235-Speed 128.63 samples/sec Loss 10.5762 Epoch: 9 Global Step: 99500 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:46:12,252-Speed 127.90 samples/sec Loss 10.4109 Epoch: 9 Global Step: 99505 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:46:32,507-Speed 126.39 samples/sec Loss 10.8419 Epoch: 9 Global Step: 99510 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:46:52,223-Speed 129.85 samples/sec Loss 10.9216 Epoch: 9 Global Step: 99515 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:47:12,213-Speed 128.07 samples/sec Loss 10.5383 Epoch: 9 Global Step: 99520 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:47:32,238-Speed 127.84 samples/sec Loss 10.7686 Epoch: 9 Global Step: 99525 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:47:51,898-Speed 130.22 samples/sec Loss 10.2742 Epoch: 9 Global Step: 99530 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:48:11,846-Speed 128.34 samples/sec Loss 10.6940 Epoch: 9 Global Step: 99535 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:48:31,491-Speed 130.32 samples/sec Loss 10.8014 Epoch: 9 Global Step: 99540 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:48:51,636-Speed 127.08 samples/sec Loss 10.8172 Epoch: 9 Global Step: 99545 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:49:11,527-Speed 128.71 samples/sec Loss 10.6349 Epoch: 9 Global Step: 99550 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:49:31,376-Speed 128.98 samples/sec Loss 10.5919 Epoch: 9 Global Step: 99555 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:49:51,250-Speed 128.82 samples/sec Loss 10.6914 Epoch: 9 Global Step: 99560 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:50:11,215-Speed 128.23 samples/sec Loss 10.7325 Epoch: 9 Global Step: 99565 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:50:31,166-Speed 128.32 samples/sec Loss 10.5917 Epoch: 9 Global Step: 99570 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:50:50,910-Speed 129.67 samples/sec Loss 10.8188 Epoch: 9 Global Step: 99575 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:51:10,759-Speed 128.98 samples/sec Loss 10.4442 Epoch: 9 Global Step: 99580 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:51:30,504-Speed 129.66 samples/sec Loss 10.8302 Epoch: 9 Global Step: 99585 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:51:50,598-Speed 127.41 samples/sec Loss 10.4826 Epoch: 9 Global Step: 99590 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:52:10,507-Speed 128.60 samples/sec Loss 10.7362 Epoch: 9 Global Step: 99595 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:52:30,205-Speed 129.97 samples/sec Loss 10.8078 Epoch: 9 Global Step: 99600 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:52:50,138-Speed 128.44 samples/sec Loss 10.2018 Epoch: 9 Global Step: 99605 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:53:10,034-Speed 128.68 samples/sec Loss 10.6136 Epoch: 9 Global Step: 99610 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:53:30,058-Speed 127.85 samples/sec Loss 10.6678 Epoch: 9 Global Step: 99615 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:53:49,820-Speed 129.54 samples/sec Loss 10.6659 Epoch: 9 Global Step: 99620 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:54:09,763-Speed 128.38 samples/sec Loss 10.9158 Epoch: 9 Global Step: 99625 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:54:29,697-Speed 128.43 samples/sec Loss 10.7394 Epoch: 9 Global Step: 99630 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:54:49,681-Speed 128.11 samples/sec Loss 10.4440 Epoch: 9 Global Step: 99635 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:55:09,389-Speed 129.90 samples/sec Loss 10.5984 Epoch: 9 Global Step: 99640 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:55:44,592-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-11 16:56:11,540-Speed 41.19 samples/sec Loss 10.0564 Epoch: 10 Global Step: 99645 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:56:31,621-Speed 127.49 samples/sec Loss 9.6237 Epoch: 10 Global Step: 99650 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:56:50,769-Speed 133.70 samples/sec Loss 9.5609 Epoch: 10 Global Step: 99655 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:57:09,963-Speed 133.38 samples/sec Loss 9.2046 Epoch: 10 Global Step: 99660 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:57:29,220-Speed 132.95 samples/sec Loss 8.8760 Epoch: 10 Global Step: 99665 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:57:48,618-Speed 131.98 samples/sec Loss 8.7560 Epoch: 10 Global Step: 99670 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:58:07,481-Speed 135.72 samples/sec Loss 8.3292 Epoch: 10 Global Step: 99675 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:58:26,848-Speed 132.19 samples/sec Loss 8.6799 Epoch: 10 Global Step: 99680 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:58:46,166-Speed 132.53 samples/sec Loss 8.2956 Epoch: 10 Global Step: 99685 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:59:05,175-Speed 134.68 samples/sec Loss 8.1792 Epoch: 10 Global Step: 99690 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:59:24,620-Speed 131.66 samples/sec Loss 8.3523 Epoch: 10 Global Step: 99695 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 16:59:43,669-Speed 134.40 samples/sec Loss 8.2634 Epoch: 10 Global Step: 99700 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:00:02,895-Speed 133.16 samples/sec Loss 8.3669 Epoch: 10 Global Step: 99705 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:00:22,095-Speed 133.34 samples/sec Loss 8.3760 Epoch: 10 Global Step: 99710 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:00:41,478-Speed 132.08 samples/sec Loss 8.4168 Epoch: 10 Global Step: 99715 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:01:00,332-Speed 135.79 samples/sec Loss 8.1631 Epoch: 10 Global Step: 99720 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:01:19,721-Speed 132.04 samples/sec Loss 7.9859 Epoch: 10 Global Step: 99725 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:01:39,046-Speed 132.48 samples/sec Loss 8.3569 Epoch: 10 Global Step: 99730 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:01:58,046-Speed 134.75 samples/sec Loss 8.3541 Epoch: 10 Global Step: 99735 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:02:17,291-Speed 133.02 samples/sec Loss 7.9015 Epoch: 10 Global Step: 99740 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:02:36,356-Speed 134.29 samples/sec Loss 7.8029 Epoch: 10 Global Step: 99745 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:02:55,574-Speed 133.21 samples/sec Loss 7.8050 Epoch: 10 Global Step: 99750 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:03:14,788-Speed 133.25 samples/sec Loss 7.7219 Epoch: 10 Global Step: 99755 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:03:34,176-Speed 132.04 samples/sec Loss 7.8395 Epoch: 10 Global Step: 99760 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:03:53,066-Speed 135.52 samples/sec Loss 7.5501 Epoch: 10 Global Step: 99765 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:04:12,519-Speed 131.61 samples/sec Loss 7.8468 Epoch: 10 Global Step: 99770 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:04:31,704-Speed 133.45 samples/sec Loss 7.9678 Epoch: 10 Global Step: 99775 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:04:50,696-Speed 134.80 samples/sec Loss 7.6631 Epoch: 10 Global Step: 99780 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:05:09,954-Speed 132.94 samples/sec Loss 7.6369 Epoch: 10 Global Step: 99785 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:05:28,976-Speed 134.59 samples/sec Loss 7.7801 Epoch: 10 Global Step: 99790 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:05:48,250-Speed 132.83 samples/sec Loss 8.1826 Epoch: 10 Global Step: 99795 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:06:07,445-Speed 133.37 samples/sec Loss 7.5526 Epoch: 10 Global Step: 99800 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:06:26,779-Speed 132.42 samples/sec Loss 7.8234 Epoch: 10 Global Step: 99805 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:06:45,709-Speed 135.24 samples/sec Loss 7.7388 Epoch: 10 Global Step: 99810 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:07:05,094-Speed 132.06 samples/sec Loss 7.5600 Epoch: 10 Global Step: 99815 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:07:24,206-Speed 133.96 samples/sec Loss 7.8070 Epoch: 10 Global Step: 99820 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:07:43,647-Speed 131.69 samples/sec Loss 7.6995 Epoch: 10 Global Step: 99825 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:08:03,024-Speed 132.12 samples/sec Loss 7.6814 Epoch: 10 Global Step: 99830 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:08:22,052-Speed 134.55 samples/sec Loss 7.7320 Epoch: 10 Global Step: 99835 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:08:41,290-Speed 133.07 samples/sec Loss 7.6456 Epoch: 10 Global Step: 99840 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:09:00,448-Speed 133.63 samples/sec Loss 7.5636 Epoch: 10 Global Step: 99845 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 17:09:19,796-Speed 132.32 samples/sec Loss 7.4001 Epoch: 10 Global Step: 99850 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:09:38,679-Speed 135.58 samples/sec Loss 7.5314 Epoch: 10 Global Step: 99855 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:09:58,068-Speed 132.04 samples/sec Loss 7.6386 Epoch: 10 Global Step: 99860 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:10:17,233-Speed 133.58 samples/sec Loss 7.4244 Epoch: 10 Global Step: 99865 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:10:36,225-Speed 134.81 samples/sec Loss 7.5059 Epoch: 10 Global Step: 99870 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:10:55,604-Speed 132.11 samples/sec Loss 7.3121 Epoch: 10 Global Step: 99875 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:11:14,643-Speed 134.47 samples/sec Loss 7.3494 Epoch: 10 Global Step: 99880 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:11:33,882-Speed 133.07 samples/sec Loss 7.4917 Epoch: 10 Global Step: 99885 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:11:53,089-Speed 133.30 samples/sec Loss 7.3495 Epoch: 10 Global Step: 99890 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:12:12,432-Speed 132.35 samples/sec Loss 7.3512 Epoch: 10 Global Step: 99895 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:12:31,336-Speed 135.42 samples/sec Loss 7.5310 Epoch: 10 Global Step: 99900 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:12:50,724-Speed 132.05 samples/sec Loss 7.3009 Epoch: 10 Global Step: 99905 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:13:09,902-Speed 133.50 samples/sec Loss 7.3509 Epoch: 10 Global Step: 99910 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:13:28,903-Speed 134.74 samples/sec Loss 7.4014 Epoch: 10 Global Step: 99915 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:13:48,214-Speed 132.58 samples/sec Loss 7.3835 Epoch: 10 Global Step: 99920 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:14:07,219-Speed 134.71 samples/sec Loss 7.4160 Epoch: 10 Global Step: 99925 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:14:26,464-Speed 133.03 samples/sec Loss 7.2594 Epoch: 10 Global Step: 99930 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:14:45,664-Speed 133.34 samples/sec Loss 7.0983 Epoch: 10 Global Step: 99935 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:15:04,998-Speed 132.42 samples/sec Loss 7.5481 Epoch: 10 Global Step: 99940 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:15:23,866-Speed 135.69 samples/sec Loss 7.6707 Epoch: 10 Global Step: 99945 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:15:43,250-Speed 132.07 samples/sec Loss 7.1666 Epoch: 10 Global Step: 99950 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:16:02,405-Speed 133.66 samples/sec Loss 7.1636 Epoch: 10 Global Step: 99955 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:16:21,326-Speed 135.31 samples/sec Loss 7.0725 Epoch: 10 Global Step: 99960 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:16:40,667-Speed 132.37 samples/sec Loss 7.4174 Epoch: 10 Global Step: 99965 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:16:59,719-Speed 134.38 samples/sec Loss 7.0472 Epoch: 10 Global Step: 99970 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:17:18,983-Speed 132.90 samples/sec Loss 7.3094 Epoch: 10 Global Step: 99975 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:17:38,140-Speed 133.64 samples/sec Loss 6.9479 Epoch: 10 Global Step: 99980 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:17:57,405-Speed 132.89 samples/sec Loss 7.4995 Epoch: 10 Global Step: 99985 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:18:16,275-Speed 135.67 samples/sec Loss 6.8718 Epoch: 10 Global Step: 99990 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:18:35,650-Speed 132.13 samples/sec Loss 7.1799 Epoch: 10 Global Step: 99995 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:18:54,823-Speed 133.53 samples/sec Loss 7.2836 Epoch: 10 Global Step: 100000 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 17:33:41,126-[lfw][100000]XNorm: 22.845649 Training: 2021-06-11 17:33:41,127-[lfw][100000]Accuracy-Flip: 0.99800+-0.00233 Training: 2021-06-11 17:33:41,127-[lfw][100000]Accuracy-Highest: 0.99800 Training: 2021-06-11 17:50:54,738-[cfp_fp][100000]XNorm: 21.324798 Training: 2021-06-11 17:50:54,738-[cfp_fp][100000]Accuracy-Flip: 0.98700+-0.00207 Training: 2021-06-11 17:50:54,738-[cfp_fp][100000]Accuracy-Highest: 0.98700 Training: 2021-06-11 18:05:43,880-[agedb_30][100000]XNorm: 22.909289 Training: 2021-06-11 18:05:43,880-[agedb_30][100000]Accuracy-Flip: 0.97817+-0.00762 Training: 2021-06-11 18:05:43,880-[agedb_30][100000]Accuracy-Highest: 0.97817 Training: 2021-06-11 18:06:03,044-Speed 0.91 samples/sec Loss 7.0908 Epoch: 10 Global Step: 100005 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:06:22,073-Speed 134.54 samples/sec Loss 7.0323 Epoch: 10 Global Step: 100010 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:06:40,951-Speed 135.61 samples/sec Loss 7.2930 Epoch: 10 Global Step: 100015 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:07:00,069-Speed 133.91 samples/sec Loss 7.2528 Epoch: 10 Global Step: 100020 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:07:19,060-Speed 134.81 samples/sec Loss 6.9553 Epoch: 10 Global Step: 100025 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:07:38,552-Speed 131.34 samples/sec Loss 7.3605 Epoch: 10 Global Step: 100030 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:07:57,440-Speed 135.54 samples/sec Loss 7.0606 Epoch: 10 Global Step: 100035 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:08:16,466-Speed 134.56 samples/sec Loss 6.8253 Epoch: 10 Global Step: 100040 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:08:35,495-Speed 134.54 samples/sec Loss 6.7612 Epoch: 10 Global Step: 100045 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:08:54,421-Speed 135.28 samples/sec Loss 7.1800 Epoch: 10 Global Step: 100050 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:09:13,515-Speed 134.08 samples/sec Loss 6.9246 Epoch: 10 Global Step: 100055 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:09:32,397-Speed 135.59 samples/sec Loss 7.0555 Epoch: 10 Global Step: 100060 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:09:51,313-Speed 135.34 samples/sec Loss 7.0490 Epoch: 10 Global Step: 100065 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:10:10,326-Speed 134.65 samples/sec Loss 6.6819 Epoch: 10 Global Step: 100070 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:10:29,658-Speed 132.43 samples/sec Loss 6.7429 Epoch: 10 Global Step: 100075 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:10:48,695-Speed 134.48 samples/sec Loss 7.1040 Epoch: 10 Global Step: 100080 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:11:07,808-Speed 133.95 samples/sec Loss 6.9503 Epoch: 10 Global Step: 100085 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:11:26,835-Speed 134.55 samples/sec Loss 7.1807 Epoch: 10 Global Step: 100090 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:11:45,723-Speed 135.54 samples/sec Loss 6.7937 Epoch: 10 Global Step: 100095 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:12:04,773-Speed 134.39 samples/sec Loss 6.7872 Epoch: 10 Global Step: 100100 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:12:23,674-Speed 135.45 samples/sec Loss 7.0243 Epoch: 10 Global Step: 100105 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:12:42,728-Speed 134.36 samples/sec Loss 6.9452 Epoch: 10 Global Step: 100110 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:13:01,757-Speed 134.54 samples/sec Loss 6.7044 Epoch: 10 Global Step: 100115 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:13:21,133-Speed 132.13 samples/sec Loss 7.0734 Epoch: 10 Global Step: 100120 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:13:40,082-Speed 135.11 samples/sec Loss 6.7088 Epoch: 10 Global Step: 100125 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:13:59,276-Speed 133.39 samples/sec Loss 6.8571 Epoch: 10 Global Step: 100130 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:14:18,273-Speed 134.76 samples/sec Loss 6.8848 Epoch: 10 Global Step: 100135 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:14:37,295-Speed 134.59 samples/sec Loss 6.5874 Epoch: 10 Global Step: 100140 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:14:56,355-Speed 134.32 samples/sec Loss 6.6412 Epoch: 10 Global Step: 100145 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:15:15,360-Speed 134.71 samples/sec Loss 6.7998 Epoch: 10 Global Step: 100150 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:15:34,427-Speed 134.27 samples/sec Loss 6.9042 Epoch: 10 Global Step: 100155 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:15:53,450-Speed 134.59 samples/sec Loss 6.5452 Epoch: 10 Global Step: 100160 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:16:12,796-Speed 132.33 samples/sec Loss 6.7360 Epoch: 10 Global Step: 100165 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:16:31,673-Speed 135.62 samples/sec Loss 6.6428 Epoch: 10 Global Step: 100170 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:16:50,836-Speed 133.60 samples/sec Loss 6.3393 Epoch: 10 Global Step: 100175 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:17:09,886-Speed 134.39 samples/sec Loss 6.7117 Epoch: 10 Global Step: 100180 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:17:28,771-Speed 135.57 samples/sec Loss 6.3745 Epoch: 10 Global Step: 100185 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:17:47,816-Speed 134.42 samples/sec Loss 6.5469 Epoch: 10 Global Step: 100190 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:18:06,700-Speed 135.58 samples/sec Loss 6.6833 Epoch: 10 Global Step: 100195 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:18:25,760-Speed 134.32 samples/sec Loss 6.7788 Epoch: 10 Global Step: 100200 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:18:44,790-Speed 134.54 samples/sec Loss 6.4038 Epoch: 10 Global Step: 100205 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:19:04,102-Speed 132.57 samples/sec Loss 7.0439 Epoch: 10 Global Step: 100210 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:19:23,074-Speed 134.94 samples/sec Loss 6.7656 Epoch: 10 Global Step: 100215 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:19:42,163-Speed 134.11 samples/sec Loss 6.5775 Epoch: 10 Global Step: 100220 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:20:01,219-Speed 134.35 samples/sec Loss 6.5875 Epoch: 10 Global Step: 100225 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:20:20,106-Speed 135.55 samples/sec Loss 6.4804 Epoch: 10 Global Step: 100230 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:20:39,161-Speed 134.35 samples/sec Loss 6.3673 Epoch: 10 Global Step: 100235 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:20:58,060-Speed 135.47 samples/sec Loss 6.8460 Epoch: 10 Global Step: 100240 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:21:17,134-Speed 134.22 samples/sec Loss 6.8355 Epoch: 10 Global Step: 100245 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:21:36,194-Speed 134.32 samples/sec Loss 6.5826 Epoch: 10 Global Step: 100250 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:21:55,323-Speed 133.84 samples/sec Loss 6.2767 Epoch: 10 Global Step: 100255 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:22:14,383-Speed 134.32 samples/sec Loss 6.3349 Epoch: 10 Global Step: 100260 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:22:33,461-Speed 134.19 samples/sec Loss 6.1666 Epoch: 10 Global Step: 100265 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:22:52,518-Speed 134.34 samples/sec Loss 6.5357 Epoch: 10 Global Step: 100270 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:23:11,401-Speed 135.58 samples/sec Loss 6.4199 Epoch: 10 Global Step: 100275 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:23:30,396-Speed 134.77 samples/sec Loss 6.5219 Epoch: 10 Global Step: 100280 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:23:49,424-Speed 134.55 samples/sec Loss 6.6559 Epoch: 10 Global Step: 100285 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:24:08,493-Speed 134.26 samples/sec Loss 6.4033 Epoch: 10 Global Step: 100290 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:24:27,564-Speed 134.25 samples/sec Loss 6.4649 Epoch: 10 Global Step: 100295 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:24:46,709-Speed 133.72 samples/sec Loss 6.8674 Epoch: 10 Global Step: 100300 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:25:05,817-Speed 133.98 samples/sec Loss 6.3893 Epoch: 10 Global Step: 100305 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:25:24,960-Speed 133.73 samples/sec Loss 6.3070 Epoch: 10 Global Step: 100310 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:25:44,032-Speed 134.23 samples/sec Loss 6.5448 Epoch: 10 Global Step: 100315 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:26:02,933-Speed 135.45 samples/sec Loss 6.4515 Epoch: 10 Global Step: 100320 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:26:21,924-Speed 134.81 samples/sec Loss 6.2574 Epoch: 10 Global Step: 100325 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:26:40,954-Speed 134.53 samples/sec Loss 6.0679 Epoch: 10 Global Step: 100330 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:26:59,976-Speed 134.59 samples/sec Loss 6.4532 Epoch: 10 Global Step: 100335 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:27:19,051-Speed 134.21 samples/sec Loss 6.3997 Epoch: 10 Global Step: 100340 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:27:38,274-Speed 133.18 samples/sec Loss 6.2732 Epoch: 10 Global Step: 100345 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:27:57,282-Speed 134.68 samples/sec Loss 6.4690 Epoch: 10 Global Step: 100350 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:28:16,285-Speed 134.73 samples/sec Loss 6.1395 Epoch: 10 Global Step: 100355 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:28:35,392-Speed 133.99 samples/sec Loss 6.4798 Epoch: 10 Global Step: 100360 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:28:54,313-Speed 135.31 samples/sec Loss 6.0614 Epoch: 10 Global Step: 100365 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:29:13,582-Speed 132.87 samples/sec Loss 6.5251 Epoch: 10 Global Step: 100370 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:29:32,741-Speed 133.63 samples/sec Loss 6.4815 Epoch: 10 Global Step: 100375 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:29:51,737-Speed 134.77 samples/sec Loss 6.1459 Epoch: 10 Global Step: 100380 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:30:10,794-Speed 134.35 samples/sec Loss 6.3685 Epoch: 10 Global Step: 100385 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:30:30,039-Speed 133.03 samples/sec Loss 6.2774 Epoch: 10 Global Step: 100390 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:30:49,110-Speed 134.25 samples/sec Loss 6.2847 Epoch: 10 Global Step: 100395 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:31:08,145-Speed 134.49 samples/sec Loss 6.2257 Epoch: 10 Global Step: 100400 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:31:27,220-Speed 134.21 samples/sec Loss 6.2588 Epoch: 10 Global Step: 100405 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:31:46,221-Speed 134.74 samples/sec Loss 6.1922 Epoch: 10 Global Step: 100410 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:32:05,212-Speed 134.81 samples/sec Loss 6.3845 Epoch: 10 Global Step: 100415 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:32:24,262-Speed 134.39 samples/sec Loss 6.1035 Epoch: 10 Global Step: 100420 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:32:43,279-Speed 134.62 samples/sec Loss 6.1679 Epoch: 10 Global Step: 100425 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:33:02,636-Speed 132.26 samples/sec Loss 6.0075 Epoch: 10 Global Step: 100430 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:33:21,835-Speed 133.35 samples/sec Loss 6.1039 Epoch: 10 Global Step: 100435 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:33:40,936-Speed 134.03 samples/sec Loss 6.1086 Epoch: 10 Global Step: 100440 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:33:59,971-Speed 134.49 samples/sec Loss 6.2758 Epoch: 10 Global Step: 100445 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:34:19,025-Speed 134.36 samples/sec Loss 5.8840 Epoch: 10 Global Step: 100450 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:34:38,039-Speed 134.65 samples/sec Loss 6.5311 Epoch: 10 Global Step: 100455 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:34:57,071-Speed 134.52 samples/sec Loss 6.3493 Epoch: 10 Global Step: 100460 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:35:16,153-Speed 134.17 samples/sec Loss 6.1212 Epoch: 10 Global Step: 100465 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:35:35,178-Speed 134.57 samples/sec Loss 6.0229 Epoch: 10 Global Step: 100470 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:35:54,156-Speed 134.90 samples/sec Loss 6.2283 Epoch: 10 Global Step: 100475 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:36:13,378-Speed 133.19 samples/sec Loss 6.2034 Epoch: 10 Global Step: 100480 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:36:32,451-Speed 134.23 samples/sec Loss 5.9966 Epoch: 10 Global Step: 100485 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:36:51,523-Speed 134.24 samples/sec Loss 6.4565 Epoch: 10 Global Step: 100490 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:37:10,474-Speed 135.09 samples/sec Loss 6.0380 Epoch: 10 Global Step: 100495 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:37:29,516-Speed 134.45 samples/sec Loss 6.2170 Epoch: 10 Global Step: 100500 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:37:48,443-Speed 135.26 samples/sec Loss 6.3685 Epoch: 10 Global Step: 100505 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:38:07,556-Speed 133.95 samples/sec Loss 6.0686 Epoch: 10 Global Step: 100510 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:38:26,477-Speed 135.31 samples/sec Loss 5.9930 Epoch: 10 Global Step: 100515 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:38:45,465-Speed 134.83 samples/sec Loss 6.0050 Epoch: 10 Global Step: 100520 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:39:04,666-Speed 133.33 samples/sec Loss 5.8362 Epoch: 10 Global Step: 100525 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:39:23,726-Speed 134.32 samples/sec Loss 6.0254 Epoch: 10 Global Step: 100530 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:39:42,692-Speed 134.98 samples/sec Loss 6.1832 Epoch: 10 Global Step: 100535 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:40:01,720-Speed 134.55 samples/sec Loss 5.7323 Epoch: 10 Global Step: 100540 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:40:20,790-Speed 134.25 samples/sec Loss 5.8485 Epoch: 10 Global Step: 100545 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:40:39,723-Speed 135.22 samples/sec Loss 5.7718 Epoch: 10 Global Step: 100550 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:40:58,902-Speed 133.49 samples/sec Loss 5.9250 Epoch: 10 Global Step: 100555 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:41:17,889-Speed 134.84 samples/sec Loss 5.9152 Epoch: 10 Global Step: 100560 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:41:36,807-Speed 135.33 samples/sec Loss 6.0581 Epoch: 10 Global Step: 100565 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:41:56,014-Speed 133.29 samples/sec Loss 5.8093 Epoch: 10 Global Step: 100570 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:42:15,168-Speed 133.66 samples/sec Loss 6.2149 Epoch: 10 Global Step: 100575 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:42:34,181-Speed 134.65 samples/sec Loss 6.3748 Epoch: 10 Global Step: 100580 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:42:53,120-Speed 135.17 samples/sec Loss 6.0201 Epoch: 10 Global Step: 100585 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:43:12,249-Speed 133.84 samples/sec Loss 5.9684 Epoch: 10 Global Step: 100590 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:43:31,275-Speed 134.57 samples/sec Loss 5.8631 Epoch: 10 Global Step: 100595 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:43:50,334-Speed 134.32 samples/sec Loss 5.7556 Epoch: 10 Global Step: 100600 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:44:09,398-Speed 134.30 samples/sec Loss 6.1524 Epoch: 10 Global Step: 100605 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:44:28,293-Speed 135.49 samples/sec Loss 5.9670 Epoch: 10 Global Step: 100610 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:44:47,538-Speed 133.03 samples/sec Loss 6.2881 Epoch: 10 Global Step: 100615 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:45:06,730-Speed 133.40 samples/sec Loss 6.0807 Epoch: 10 Global Step: 100620 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:45:25,746-Speed 134.63 samples/sec Loss 5.9215 Epoch: 10 Global Step: 100625 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:45:44,670-Speed 135.29 samples/sec Loss 5.7726 Epoch: 10 Global Step: 100630 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:46:03,700-Speed 134.53 samples/sec Loss 5.6454 Epoch: 10 Global Step: 100635 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:46:22,709-Speed 134.68 samples/sec Loss 5.9022 Epoch: 10 Global Step: 100640 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:46:41,747-Speed 134.48 samples/sec Loss 6.1562 Epoch: 10 Global Step: 100645 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:47:00,753-Speed 134.70 samples/sec Loss 5.7811 Epoch: 10 Global Step: 100650 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:47:19,650-Speed 135.48 samples/sec Loss 5.9964 Epoch: 10 Global Step: 100655 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:47:38,905-Speed 132.96 samples/sec Loss 5.7524 Epoch: 10 Global Step: 100660 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:47:57,948-Speed 134.44 samples/sec Loss 6.0491 Epoch: 10 Global Step: 100665 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:48:17,102-Speed 133.66 samples/sec Loss 5.8313 Epoch: 10 Global Step: 100670 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:48:35,992-Speed 135.53 samples/sec Loss 5.7400 Epoch: 10 Global Step: 100675 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:48:55,059-Speed 134.27 samples/sec Loss 5.9911 Epoch: 10 Global Step: 100680 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:49:14,097-Speed 134.48 samples/sec Loss 6.1425 Epoch: 10 Global Step: 100685 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:49:33,064-Speed 134.97 samples/sec Loss 5.7341 Epoch: 10 Global Step: 100690 Fp16 Grad Scale: 8192 Required: 177 hours Training: 2021-06-11 18:49:52,083-Speed 134.61 samples/sec Loss 6.0812 Epoch: 10 Global Step: 100695 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:50:10,968-Speed 135.56 samples/sec Loss 5.9445 Epoch: 10 Global Step: 100700 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:50:30,234-Speed 132.88 samples/sec Loss 5.8790 Epoch: 10 Global Step: 100705 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:50:49,341-Speed 133.99 samples/sec Loss 5.9033 Epoch: 10 Global Step: 100710 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:51:08,381-Speed 134.46 samples/sec Loss 5.6129 Epoch: 10 Global Step: 100715 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:51:27,287-Speed 135.41 samples/sec Loss 5.9465 Epoch: 10 Global Step: 100720 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:51:46,347-Speed 134.32 samples/sec Loss 6.0480 Epoch: 10 Global Step: 100725 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:52:05,500-Speed 133.67 samples/sec Loss 5.8464 Epoch: 10 Global Step: 100730 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:52:24,425-Speed 135.28 samples/sec Loss 5.9748 Epoch: 10 Global Step: 100735 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:52:43,515-Speed 134.11 samples/sec Loss 5.9493 Epoch: 10 Global Step: 100740 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:53:02,398-Speed 135.58 samples/sec Loss 5.9121 Epoch: 10 Global Step: 100745 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:53:21,654-Speed 132.95 samples/sec Loss 5.7307 Epoch: 10 Global Step: 100750 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:53:40,711-Speed 134.35 samples/sec Loss 5.8559 Epoch: 10 Global Step: 100755 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:53:59,895-Speed 133.45 samples/sec Loss 5.7866 Epoch: 10 Global Step: 100760 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:54:18,778-Speed 135.58 samples/sec Loss 5.9839 Epoch: 10 Global Step: 100765 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:54:37,813-Speed 134.50 samples/sec Loss 5.6407 Epoch: 10 Global Step: 100770 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:54:56,859-Speed 134.42 samples/sec Loss 5.5750 Epoch: 10 Global Step: 100775 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:55:15,806-Speed 135.12 samples/sec Loss 5.7946 Epoch: 10 Global Step: 100780 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:55:34,861-Speed 134.35 samples/sec Loss 5.6816 Epoch: 10 Global Step: 100785 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:55:53,793-Speed 135.23 samples/sec Loss 5.4110 Epoch: 10 Global Step: 100790 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:56:13,151-Speed 132.25 samples/sec Loss 5.7290 Epoch: 10 Global Step: 100795 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:56:32,023-Speed 135.66 samples/sec Loss 5.7390 Epoch: 10 Global Step: 100800 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:56:51,125-Speed 134.02 samples/sec Loss 5.6571 Epoch: 10 Global Step: 100805 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:57:10,125-Speed 134.74 samples/sec Loss 5.5165 Epoch: 10 Global Step: 100810 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:57:29,061-Speed 135.20 samples/sec Loss 5.4092 Epoch: 10 Global Step: 100815 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:57:48,120-Speed 134.33 samples/sec Loss 5.9239 Epoch: 10 Global Step: 100820 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:58:07,125-Speed 134.71 samples/sec Loss 5.4841 Epoch: 10 Global Step: 100825 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:58:26,188-Speed 134.30 samples/sec Loss 5.8657 Epoch: 10 Global Step: 100830 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:58:45,288-Speed 134.04 samples/sec Loss 6.0902 Epoch: 10 Global Step: 100835 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:59:04,652-Speed 132.21 samples/sec Loss 5.6805 Epoch: 10 Global Step: 100840 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:59:23,557-Speed 135.42 samples/sec Loss 5.7804 Epoch: 10 Global Step: 100845 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 18:59:42,733-Speed 133.51 samples/sec Loss 5.5825 Epoch: 10 Global Step: 100850 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:00:01,722-Speed 134.82 samples/sec Loss 5.6487 Epoch: 10 Global Step: 100855 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:00:20,726-Speed 134.72 samples/sec Loss 5.4820 Epoch: 10 Global Step: 100860 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:00:39,898-Speed 133.54 samples/sec Loss 5.8051 Epoch: 10 Global Step: 100865 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:00:58,861-Speed 135.01 samples/sec Loss 5.5636 Epoch: 10 Global Step: 100870 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:01:17,871-Speed 134.67 samples/sec Loss 5.8936 Epoch: 10 Global Step: 100875 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:01:36,955-Speed 134.15 samples/sec Loss 5.5971 Epoch: 10 Global Step: 100880 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:01:56,351-Speed 131.99 samples/sec Loss 5.5027 Epoch: 10 Global Step: 100885 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:02:15,247-Speed 135.49 samples/sec Loss 5.7063 Epoch: 10 Global Step: 100890 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:02:34,284-Speed 134.49 samples/sec Loss 5.6930 Epoch: 10 Global Step: 100895 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:02:53,383-Speed 134.04 samples/sec Loss 5.7332 Epoch: 10 Global Step: 100900 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:03:12,432-Speed 134.40 samples/sec Loss 5.6665 Epoch: 10 Global Step: 100905 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:03:31,489-Speed 134.34 samples/sec Loss 5.6621 Epoch: 10 Global Step: 100910 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:03:50,407-Speed 135.33 samples/sec Loss 5.4923 Epoch: 10 Global Step: 100915 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:04:09,497-Speed 134.11 samples/sec Loss 5.6933 Epoch: 10 Global Step: 100920 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:04:28,479-Speed 134.87 samples/sec Loss 5.5103 Epoch: 10 Global Step: 100925 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:04:47,902-Speed 131.81 samples/sec Loss 5.4597 Epoch: 10 Global Step: 100930 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:05:06,770-Speed 135.69 samples/sec Loss 5.5482 Epoch: 10 Global Step: 100935 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:05:25,783-Speed 134.65 samples/sec Loss 5.4798 Epoch: 10 Global Step: 100940 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:05:44,789-Speed 134.70 samples/sec Loss 5.4650 Epoch: 10 Global Step: 100945 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:06:03,728-Speed 135.18 samples/sec Loss 5.4351 Epoch: 10 Global Step: 100950 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:06:22,893-Speed 133.59 samples/sec Loss 5.6504 Epoch: 10 Global Step: 100955 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:06:41,898-Speed 134.70 samples/sec Loss 5.6372 Epoch: 10 Global Step: 100960 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:07:00,982-Speed 134.16 samples/sec Loss 5.6951 Epoch: 10 Global Step: 100965 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:07:20,016-Speed 134.50 samples/sec Loss 5.5936 Epoch: 10 Global Step: 100970 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:07:39,470-Speed 131.60 samples/sec Loss 5.4602 Epoch: 10 Global Step: 100975 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:07:58,346-Speed 135.63 samples/sec Loss 5.7200 Epoch: 10 Global Step: 100980 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:08:17,469-Speed 133.87 samples/sec Loss 5.7203 Epoch: 10 Global Step: 100985 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:08:36,469-Speed 134.75 samples/sec Loss 5.5885 Epoch: 10 Global Step: 100990 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:08:55,534-Speed 134.29 samples/sec Loss 5.3982 Epoch: 10 Global Step: 100995 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:09:14,699-Speed 133.58 samples/sec Loss 5.3932 Epoch: 10 Global Step: 101000 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:09:33,708-Speed 134.68 samples/sec Loss 5.6541 Epoch: 10 Global Step: 101005 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:09:52,749-Speed 134.45 samples/sec Loss 5.6663 Epoch: 10 Global Step: 101010 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:10:11,825-Speed 134.21 samples/sec Loss 5.2562 Epoch: 10 Global Step: 101015 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:10:31,269-Speed 131.67 samples/sec Loss 5.4892 Epoch: 10 Global Step: 101020 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:10:50,159-Speed 135.53 samples/sec Loss 5.4365 Epoch: 10 Global Step: 101025 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:11:09,189-Speed 134.53 samples/sec Loss 5.2633 Epoch: 10 Global Step: 101030 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:11:28,317-Speed 133.84 samples/sec Loss 5.6900 Epoch: 10 Global Step: 101035 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:11:47,364-Speed 134.42 samples/sec Loss 5.5318 Epoch: 10 Global Step: 101040 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:12:06,432-Speed 134.26 samples/sec Loss 5.5667 Epoch: 10 Global Step: 101045 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:12:25,305-Speed 135.65 samples/sec Loss 5.5490 Epoch: 10 Global Step: 101050 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:12:44,332-Speed 134.55 samples/sec Loss 5.5319 Epoch: 10 Global Step: 101055 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:13:03,346-Speed 134.65 samples/sec Loss 5.8175 Epoch: 10 Global Step: 101060 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:13:22,781-Speed 131.73 samples/sec Loss 5.5990 Epoch: 10 Global Step: 101065 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:13:41,656-Speed 135.64 samples/sec Loss 5.2374 Epoch: 10 Global Step: 101070 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:14:00,651-Speed 134.77 samples/sec Loss 5.7323 Epoch: 10 Global Step: 101075 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:14:19,648-Speed 134.76 samples/sec Loss 5.5657 Epoch: 10 Global Step: 101080 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:14:38,696-Speed 134.41 samples/sec Loss 5.4110 Epoch: 10 Global Step: 101085 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:14:57,777-Speed 134.17 samples/sec Loss 5.3527 Epoch: 10 Global Step: 101090 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:15:16,664-Speed 135.55 samples/sec Loss 5.5104 Epoch: 10 Global Step: 101095 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:15:35,664-Speed 134.74 samples/sec Loss 5.4705 Epoch: 10 Global Step: 101100 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:15:54,691-Speed 134.56 samples/sec Loss 5.4300 Epoch: 10 Global Step: 101105 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:16:14,132-Speed 131.69 samples/sec Loss 5.3064 Epoch: 10 Global Step: 101110 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:16:33,005-Speed 135.65 samples/sec Loss 5.2284 Epoch: 10 Global Step: 101115 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:16:52,004-Speed 134.75 samples/sec Loss 5.5667 Epoch: 10 Global Step: 101120 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:17:11,535-Speed 131.08 samples/sec Loss 5.5827 Epoch: 10 Global Step: 101125 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:17:30,637-Speed 134.03 samples/sec Loss 5.2375 Epoch: 10 Global Step: 101130 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:17:49,785-Speed 133.70 samples/sec Loss 5.5450 Epoch: 10 Global Step: 101135 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:18:08,862-Speed 134.20 samples/sec Loss 5.6441 Epoch: 10 Global Step: 101140 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:18:27,951-Speed 134.12 samples/sec Loss 5.4775 Epoch: 10 Global Step: 101145 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:18:46,913-Speed 135.01 samples/sec Loss 5.4136 Epoch: 10 Global Step: 101150 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:19:06,358-Speed 131.67 samples/sec Loss 5.5116 Epoch: 10 Global Step: 101155 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:19:25,352-Speed 134.78 samples/sec Loss 5.4419 Epoch: 10 Global Step: 101160 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:19:44,449-Speed 134.06 samples/sec Loss 5.6092 Epoch: 10 Global Step: 101165 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:20:03,489-Speed 134.46 samples/sec Loss 5.3161 Epoch: 10 Global Step: 101170 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:20:22,538-Speed 134.40 samples/sec Loss 5.2500 Epoch: 10 Global Step: 101175 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:20:41,601-Speed 134.30 samples/sec Loss 5.4241 Epoch: 10 Global Step: 101180 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:21:00,577-Speed 134.92 samples/sec Loss 5.1737 Epoch: 10 Global Step: 101185 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:21:19,591-Speed 134.64 samples/sec Loss 5.1820 Epoch: 10 Global Step: 101190 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:21:38,626-Speed 134.50 samples/sec Loss 5.3325 Epoch: 10 Global Step: 101195 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:21:58,074-Speed 131.63 samples/sec Loss 5.2908 Epoch: 10 Global Step: 101200 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:22:17,108-Speed 134.50 samples/sec Loss 5.4477 Epoch: 10 Global Step: 101205 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:22:36,238-Speed 133.84 samples/sec Loss 5.3519 Epoch: 10 Global Step: 101210 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:22:55,279-Speed 134.45 samples/sec Loss 5.2590 Epoch: 10 Global Step: 101215 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:23:14,315-Speed 134.49 samples/sec Loss 5.4555 Epoch: 10 Global Step: 101220 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:23:33,371-Speed 134.35 samples/sec Loss 5.4004 Epoch: 10 Global Step: 101225 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:23:52,359-Speed 134.83 samples/sec Loss 5.0780 Epoch: 10 Global Step: 101230 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:24:11,570-Speed 133.26 samples/sec Loss 5.6213 Epoch: 10 Global Step: 101235 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:24:30,537-Speed 134.98 samples/sec Loss 5.6192 Epoch: 10 Global Step: 101240 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:24:49,871-Speed 132.42 samples/sec Loss 5.2004 Epoch: 10 Global Step: 101245 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:25:08,872-Speed 134.73 samples/sec Loss 5.4974 Epoch: 10 Global Step: 101250 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:25:27,955-Speed 134.16 samples/sec Loss 5.3785 Epoch: 10 Global Step: 101255 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:25:47,106-Speed 133.68 samples/sec Loss 5.3827 Epoch: 10 Global Step: 101260 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:26:06,046-Speed 135.17 samples/sec Loss 5.4887 Epoch: 10 Global Step: 101265 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:26:25,110-Speed 134.29 samples/sec Loss 5.2397 Epoch: 10 Global Step: 101270 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:26:43,989-Speed 135.61 samples/sec Loss 5.1057 Epoch: 10 Global Step: 101275 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:27:02,988-Speed 134.75 samples/sec Loss 5.2398 Epoch: 10 Global Step: 101280 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:27:22,060-Speed 134.24 samples/sec Loss 5.1698 Epoch: 10 Global Step: 101285 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:27:41,337-Speed 132.81 samples/sec Loss 5.5122 Epoch: 10 Global Step: 101290 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:28:00,638-Speed 132.64 samples/sec Loss 5.1231 Epoch: 10 Global Step: 101295 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:28:19,624-Speed 134.84 samples/sec Loss 5.1657 Epoch: 10 Global Step: 101300 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:28:38,807-Speed 133.46 samples/sec Loss 5.2444 Epoch: 10 Global Step: 101305 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:28:57,704-Speed 135.48 samples/sec Loss 5.1067 Epoch: 10 Global Step: 101310 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:29:16,889-Speed 133.44 samples/sec Loss 5.1478 Epoch: 10 Global Step: 101315 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:29:35,941-Speed 134.38 samples/sec Loss 5.2787 Epoch: 10 Global Step: 101320 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:29:54,952-Speed 134.67 samples/sec Loss 5.1472 Epoch: 10 Global Step: 101325 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:30:14,129-Speed 133.50 samples/sec Loss 5.3375 Epoch: 10 Global Step: 101330 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:30:33,419-Speed 132.72 samples/sec Loss 5.1913 Epoch: 10 Global Step: 101335 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:30:52,416-Speed 134.76 samples/sec Loss 5.0383 Epoch: 10 Global Step: 101340 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:31:11,511-Speed 134.07 samples/sec Loss 5.0065 Epoch: 10 Global Step: 101345 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:31:30,564-Speed 134.37 samples/sec Loss 5.0535 Epoch: 10 Global Step: 101350 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:31:49,472-Speed 135.40 samples/sec Loss 4.8494 Epoch: 10 Global Step: 101355 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:32:08,469-Speed 134.76 samples/sec Loss 5.0469 Epoch: 10 Global Step: 101360 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:32:27,484-Speed 134.64 samples/sec Loss 5.1457 Epoch: 10 Global Step: 101365 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:32:46,755-Speed 132.85 samples/sec Loss 5.3980 Epoch: 10 Global Step: 101370 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:33:06,499-Speed 129.67 samples/sec Loss 4.9558 Epoch: 10 Global Step: 101375 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:33:25,750-Speed 132.99 samples/sec Loss 5.0399 Epoch: 10 Global Step: 101380 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:33:44,932-Speed 133.46 samples/sec Loss 5.1751 Epoch: 10 Global Step: 101385 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:34:03,939-Speed 134.70 samples/sec Loss 5.0012 Epoch: 10 Global Step: 101390 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:34:23,015-Speed 134.20 samples/sec Loss 5.1531 Epoch: 10 Global Step: 101395 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:34:41,946-Speed 135.24 samples/sec Loss 4.9527 Epoch: 10 Global Step: 101400 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:35:01,003-Speed 134.34 samples/sec Loss 4.9909 Epoch: 10 Global Step: 101405 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:35:20,034-Speed 134.52 samples/sec Loss 5.3070 Epoch: 10 Global Step: 101410 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:35:39,062-Speed 134.54 samples/sec Loss 5.0170 Epoch: 10 Global Step: 101415 Fp16 Grad Scale: 8192 Required: 176 hours Training: 2021-06-11 19:35:58,125-Speed 134.30 samples/sec Loss 4.9729 Epoch: 10 Global Step: 101420 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:36:17,327-Speed 133.32 samples/sec Loss 4.9908 Epoch: 10 Global Step: 101425 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:36:36,443-Speed 133.93 samples/sec Loss 4.7386 Epoch: 10 Global Step: 101430 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:36:55,671-Speed 133.15 samples/sec Loss 5.1405 Epoch: 10 Global Step: 101435 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:37:14,739-Speed 134.26 samples/sec Loss 4.8489 Epoch: 10 Global Step: 101440 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:37:33,731-Speed 134.80 samples/sec Loss 4.9500 Epoch: 10 Global Step: 101445 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:37:52,769-Speed 134.48 samples/sec Loss 4.7155 Epoch: 10 Global Step: 101450 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:38:11,926-Speed 133.64 samples/sec Loss 5.1072 Epoch: 10 Global Step: 101455 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:38:30,933-Speed 134.70 samples/sec Loss 5.1965 Epoch: 10 Global Step: 101460 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:38:50,090-Speed 133.64 samples/sec Loss 5.1849 Epoch: 10 Global Step: 101465 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:39:09,278-Speed 133.42 samples/sec Loss 5.1378 Epoch: 10 Global Step: 101470 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:39:28,479-Speed 133.33 samples/sec Loss 5.1658 Epoch: 10 Global Step: 101475 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:39:47,591-Speed 133.95 samples/sec Loss 5.0691 Epoch: 10 Global Step: 101480 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:40:06,659-Speed 134.27 samples/sec Loss 4.9790 Epoch: 10 Global Step: 101485 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:40:25,665-Speed 134.70 samples/sec Loss 4.9368 Epoch: 10 Global Step: 101490 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:40:44,907-Speed 133.05 samples/sec Loss 5.1436 Epoch: 10 Global Step: 101495 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:41:04,148-Speed 133.06 samples/sec Loss 5.1429 Epoch: 10 Global Step: 101500 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:41:23,419-Speed 132.85 samples/sec Loss 5.0710 Epoch: 10 Global Step: 101505 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:41:42,434-Speed 134.64 samples/sec Loss 4.7424 Epoch: 10 Global Step: 101510 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:42:01,621-Speed 133.43 samples/sec Loss 5.0346 Epoch: 10 Global Step: 101515 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:42:20,788-Speed 133.57 samples/sec Loss 5.2885 Epoch: 10 Global Step: 101520 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:42:39,757-Speed 134.97 samples/sec Loss 4.8581 Epoch: 10 Global Step: 101525 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:42:58,726-Speed 134.96 samples/sec Loss 4.8303 Epoch: 10 Global Step: 101530 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:43:17,750-Speed 134.58 samples/sec Loss 4.8778 Epoch: 10 Global Step: 101535 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:43:36,860-Speed 133.97 samples/sec Loss 5.1629 Epoch: 10 Global Step: 101540 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:43:55,913-Speed 134.37 samples/sec Loss 5.1506 Epoch: 10 Global Step: 101545 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:44:14,895-Speed 134.88 samples/sec Loss 4.9888 Epoch: 10 Global Step: 101550 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:44:33,852-Speed 135.05 samples/sec Loss 4.8520 Epoch: 10 Global Step: 101555 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:44:53,172-Speed 132.51 samples/sec Loss 4.9778 Epoch: 10 Global Step: 101560 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:45:12,341-Speed 133.55 samples/sec Loss 4.9998 Epoch: 10 Global Step: 101565 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:45:31,368-Speed 134.56 samples/sec Loss 4.9171 Epoch: 10 Global Step: 101570 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:45:50,401-Speed 134.51 samples/sec Loss 5.0176 Epoch: 10 Global Step: 101575 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:46:09,448-Speed 134.41 samples/sec Loss 4.8232 Epoch: 10 Global Step: 101580 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:46:28,575-Speed 133.84 samples/sec Loss 5.0449 Epoch: 10 Global Step: 101585 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:46:47,778-Speed 133.32 samples/sec Loss 4.9743 Epoch: 10 Global Step: 101590 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:47:06,740-Speed 135.02 samples/sec Loss 4.7531 Epoch: 10 Global Step: 101595 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:47:25,893-Speed 133.67 samples/sec Loss 5.1788 Epoch: 10 Global Step: 101600 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:47:45,227-Speed 132.41 samples/sec Loss 4.7816 Epoch: 10 Global Step: 101605 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:48:04,717-Speed 131.36 samples/sec Loss 5.0787 Epoch: 10 Global Step: 101610 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:48:23,763-Speed 134.42 samples/sec Loss 4.9301 Epoch: 10 Global Step: 101615 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:48:42,872-Speed 133.97 samples/sec Loss 5.0730 Epoch: 10 Global Step: 101620 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:49:02,112-Speed 133.07 samples/sec Loss 5.0670 Epoch: 10 Global Step: 101625 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:49:21,318-Speed 133.29 samples/sec Loss 5.4420 Epoch: 10 Global Step: 101630 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:49:40,492-Speed 133.52 samples/sec Loss 4.9364 Epoch: 10 Global Step: 101635 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:49:59,654-Speed 133.61 samples/sec Loss 5.0525 Epoch: 10 Global Step: 101640 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:50:18,714-Speed 134.32 samples/sec Loss 4.6549 Epoch: 10 Global Step: 101645 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:50:37,904-Speed 133.42 samples/sec Loss 4.9189 Epoch: 10 Global Step: 101650 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:50:57,074-Speed 133.55 samples/sec Loss 4.5528 Epoch: 10 Global Step: 101655 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:51:16,116-Speed 134.44 samples/sec Loss 4.7761 Epoch: 10 Global Step: 101660 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:51:35,349-Speed 133.11 samples/sec Loss 4.9177 Epoch: 10 Global Step: 101665 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:51:54,581-Speed 133.13 samples/sec Loss 5.0407 Epoch: 10 Global Step: 101670 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:52:13,792-Speed 133.26 samples/sec Loss 5.1243 Epoch: 10 Global Step: 101675 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:52:32,960-Speed 133.57 samples/sec Loss 4.9769 Epoch: 10 Global Step: 101680 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:52:51,955-Speed 134.78 samples/sec Loss 4.8227 Epoch: 10 Global Step: 101685 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:53:11,016-Speed 134.31 samples/sec Loss 4.9785 Epoch: 10 Global Step: 101690 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:53:30,226-Speed 133.28 samples/sec Loss 4.8422 Epoch: 10 Global Step: 101695 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:53:49,496-Speed 132.86 samples/sec Loss 5.1231 Epoch: 10 Global Step: 101700 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:54:08,476-Speed 134.89 samples/sec Loss 4.8392 Epoch: 10 Global Step: 101705 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:54:27,494-Speed 134.62 samples/sec Loss 4.8407 Epoch: 10 Global Step: 101710 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:54:46,711-Speed 133.22 samples/sec Loss 4.8351 Epoch: 10 Global Step: 101715 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:55:05,697-Speed 134.85 samples/sec Loss 4.8360 Epoch: 10 Global Step: 101720 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:55:24,980-Speed 132.76 samples/sec Loss 4.7319 Epoch: 10 Global Step: 101725 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:55:44,079-Speed 134.05 samples/sec Loss 4.9107 Epoch: 10 Global Step: 101730 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:56:03,195-Speed 133.93 samples/sec Loss 4.7927 Epoch: 10 Global Step: 101735 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:56:22,406-Speed 133.27 samples/sec Loss 4.8063 Epoch: 10 Global Step: 101740 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:56:41,790-Speed 132.07 samples/sec Loss 4.9029 Epoch: 10 Global Step: 101745 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:57:00,900-Speed 133.97 samples/sec Loss 4.8682 Epoch: 10 Global Step: 101750 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:57:19,904-Speed 134.71 samples/sec Loss 4.8551 Epoch: 10 Global Step: 101755 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:57:39,162-Speed 132.94 samples/sec Loss 5.0774 Epoch: 10 Global Step: 101760 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:57:58,164-Speed 134.73 samples/sec Loss 4.8571 Epoch: 10 Global Step: 101765 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:58:17,341-Speed 133.50 samples/sec Loss 4.8164 Epoch: 10 Global Step: 101770 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:58:36,544-Speed 133.32 samples/sec Loss 5.0874 Epoch: 10 Global Step: 101775 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:58:55,669-Speed 133.86 samples/sec Loss 4.8142 Epoch: 10 Global Step: 101780 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:59:14,923-Speed 132.97 samples/sec Loss 4.6168 Epoch: 10 Global Step: 101785 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:59:34,210-Speed 132.74 samples/sec Loss 4.7914 Epoch: 10 Global Step: 101790 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 19:59:53,365-Speed 133.65 samples/sec Loss 5.0381 Epoch: 10 Global Step: 101795 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:00:12,385-Speed 134.61 samples/sec Loss 4.7363 Epoch: 10 Global Step: 101800 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:00:31,536-Speed 133.68 samples/sec Loss 4.7236 Epoch: 10 Global Step: 101805 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:00:50,550-Speed 134.65 samples/sec Loss 4.6375 Epoch: 10 Global Step: 101810 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:01:09,480-Speed 135.24 samples/sec Loss 4.6479 Epoch: 10 Global Step: 101815 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:01:28,868-Speed 132.05 samples/sec Loss 4.9160 Epoch: 10 Global Step: 101820 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:01:48,152-Speed 132.76 samples/sec Loss 4.8137 Epoch: 10 Global Step: 101825 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:02:07,529-Speed 132.13 samples/sec Loss 4.8628 Epoch: 10 Global Step: 101830 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:02:26,598-Speed 134.25 samples/sec Loss 4.8172 Epoch: 10 Global Step: 101835 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:02:45,960-Speed 132.23 samples/sec Loss 4.6423 Epoch: 10 Global Step: 101840 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:03:04,981-Speed 134.59 samples/sec Loss 4.8430 Epoch: 10 Global Step: 101845 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:03:24,052-Speed 134.24 samples/sec Loss 4.5354 Epoch: 10 Global Step: 101850 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:03:45,723-Speed 118.14 samples/sec Loss 4.7520 Epoch: 10 Global Step: 101855 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:04:16,479-Speed 83.24 samples/sec Loss 4.8322 Epoch: 10 Global Step: 101860 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:04:35,657-Speed 133.49 samples/sec Loss 4.9007 Epoch: 10 Global Step: 101865 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:04:54,826-Speed 133.55 samples/sec Loss 4.5433 Epoch: 10 Global Step: 101870 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:05:14,076-Speed 133.00 samples/sec Loss 4.8173 Epoch: 10 Global Step: 101875 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:05:33,503-Speed 131.78 samples/sec Loss 4.7297 Epoch: 10 Global Step: 101880 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:05:52,614-Speed 133.96 samples/sec Loss 4.6651 Epoch: 10 Global Step: 101885 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:06:11,736-Speed 133.88 samples/sec Loss 4.6885 Epoch: 10 Global Step: 101890 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:06:31,038-Speed 132.64 samples/sec Loss 4.9986 Epoch: 10 Global Step: 101895 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:06:50,242-Speed 133.31 samples/sec Loss 4.8735 Epoch: 10 Global Step: 101900 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:07:09,292-Speed 134.39 samples/sec Loss 4.7341 Epoch: 10 Global Step: 101905 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:07:28,442-Speed 133.69 samples/sec Loss 4.9376 Epoch: 10 Global Step: 101910 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:07:47,681-Speed 133.07 samples/sec Loss 4.7749 Epoch: 10 Global Step: 101915 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:08:07,139-Speed 131.57 samples/sec Loss 4.6325 Epoch: 10 Global Step: 101920 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:08:26,352-Speed 133.25 samples/sec Loss 4.7534 Epoch: 10 Global Step: 101925 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:08:45,227-Speed 135.63 samples/sec Loss 4.6460 Epoch: 10 Global Step: 101930 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:09:04,352-Speed 133.87 samples/sec Loss 4.6068 Epoch: 10 Global Step: 101935 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:09:23,592-Speed 133.07 samples/sec Loss 4.5471 Epoch: 10 Global Step: 101940 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:09:42,810-Speed 133.22 samples/sec Loss 4.8287 Epoch: 10 Global Step: 101945 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:10:01,884-Speed 134.22 samples/sec Loss 4.5361 Epoch: 10 Global Step: 101950 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:10:21,314-Speed 131.76 samples/sec Loss 5.1083 Epoch: 10 Global Step: 101955 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:10:40,437-Speed 133.88 samples/sec Loss 4.7051 Epoch: 10 Global Step: 101960 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:10:59,804-Speed 132.19 samples/sec Loss 4.7897 Epoch: 10 Global Step: 101965 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:11:19,164-Speed 132.24 samples/sec Loss 4.5636 Epoch: 10 Global Step: 101970 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:11:38,475-Speed 132.57 samples/sec Loss 5.0384 Epoch: 10 Global Step: 101975 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:11:57,599-Speed 133.87 samples/sec Loss 4.6005 Epoch: 10 Global Step: 101980 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:12:16,824-Speed 133.17 samples/sec Loss 4.6844 Epoch: 10 Global Step: 101985 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:12:35,901-Speed 134.20 samples/sec Loss 4.7151 Epoch: 10 Global Step: 101990 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:12:55,396-Speed 131.33 samples/sec Loss 4.7043 Epoch: 10 Global Step: 101995 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:13:14,403-Speed 134.69 samples/sec Loss 4.4927 Epoch: 10 Global Step: 102000 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:13:33,390-Speed 134.84 samples/sec Loss 4.6204 Epoch: 10 Global Step: 102005 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:13:53,006-Speed 130.51 samples/sec Loss 4.6036 Epoch: 10 Global Step: 102010 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:14:12,542-Speed 131.05 samples/sec Loss 4.7389 Epoch: 10 Global Step: 102015 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:14:31,729-Speed 133.42 samples/sec Loss 4.6252 Epoch: 10 Global Step: 102020 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:14:50,974-Speed 133.03 samples/sec Loss 4.6516 Epoch: 10 Global Step: 102025 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:15:10,105-Speed 133.82 samples/sec Loss 4.6025 Epoch: 10 Global Step: 102030 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:15:29,458-Speed 132.28 samples/sec Loss 5.0042 Epoch: 10 Global Step: 102035 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:15:48,835-Speed 132.12 samples/sec Loss 4.7375 Epoch: 10 Global Step: 102040 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:16:08,148-Speed 132.56 samples/sec Loss 4.6604 Epoch: 10 Global Step: 102045 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:16:27,371-Speed 133.18 samples/sec Loss 4.6472 Epoch: 10 Global Step: 102050 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:16:46,712-Speed 132.37 samples/sec Loss 4.5475 Epoch: 10 Global Step: 102055 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:17:06,166-Speed 131.60 samples/sec Loss 4.6954 Epoch: 10 Global Step: 102060 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:17:25,518-Speed 132.29 samples/sec Loss 4.8058 Epoch: 10 Global Step: 102065 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:17:44,748-Speed 133.14 samples/sec Loss 4.6530 Epoch: 10 Global Step: 102070 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:18:04,047-Speed 132.65 samples/sec Loss 4.5955 Epoch: 10 Global Step: 102075 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:18:23,432-Speed 132.07 samples/sec Loss 4.7584 Epoch: 10 Global Step: 102080 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:18:42,758-Speed 132.47 samples/sec Loss 4.6296 Epoch: 10 Global Step: 102085 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:19:02,161-Speed 131.94 samples/sec Loss 4.7703 Epoch: 10 Global Step: 102090 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:19:21,425-Speed 132.90 samples/sec Loss 4.4544 Epoch: 10 Global Step: 102095 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:19:40,683-Speed 132.94 samples/sec Loss 4.5532 Epoch: 10 Global Step: 102100 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:19:59,779-Speed 134.07 samples/sec Loss 4.5730 Epoch: 10 Global Step: 102105 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:20:18,918-Speed 133.77 samples/sec Loss 4.4912 Epoch: 10 Global Step: 102110 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:20:38,124-Speed 133.30 samples/sec Loss 4.5814 Epoch: 10 Global Step: 102115 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:20:57,525-Speed 131.96 samples/sec Loss 4.8590 Epoch: 10 Global Step: 102120 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:21:16,733-Speed 133.29 samples/sec Loss 4.5262 Epoch: 10 Global Step: 102125 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:21:35,738-Speed 134.71 samples/sec Loss 4.5416 Epoch: 10 Global Step: 102130 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:21:54,868-Speed 133.83 samples/sec Loss 4.5856 Epoch: 10 Global Step: 102135 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:22:13,968-Speed 134.04 samples/sec Loss 4.5258 Epoch: 10 Global Step: 102140 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:22:33,668-Speed 129.96 samples/sec Loss 4.8285 Epoch: 10 Global Step: 102145 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:22:52,985-Speed 132.53 samples/sec Loss 4.6553 Epoch: 10 Global Step: 102150 Fp16 Grad Scale: 8192 Required: 175 hours Training: 2021-06-11 20:23:12,452-Speed 131.51 samples/sec Loss 4.2866 Epoch: 10 Global Step: 102155 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:23:31,453-Speed 134.74 samples/sec Loss 4.5517 Epoch: 10 Global Step: 102160 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:23:50,736-Speed 132.77 samples/sec Loss 4.4116 Epoch: 10 Global Step: 102165 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:24:10,227-Speed 131.35 samples/sec Loss 4.8371 Epoch: 10 Global Step: 102170 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:24:29,592-Speed 132.20 samples/sec Loss 4.8395 Epoch: 10 Global Step: 102175 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:24:49,061-Speed 131.50 samples/sec Loss 4.6327 Epoch: 10 Global Step: 102180 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:25:08,404-Speed 132.36 samples/sec Loss 4.7087 Epoch: 10 Global Step: 102185 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:25:27,867-Speed 131.54 samples/sec Loss 4.5690 Epoch: 10 Global Step: 102190 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:25:47,317-Speed 131.63 samples/sec Loss 4.7278 Epoch: 10 Global Step: 102195 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:26:06,562-Speed 133.03 samples/sec Loss 4.4860 Epoch: 10 Global Step: 102200 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:26:25,759-Speed 133.36 samples/sec Loss 4.4021 Epoch: 10 Global Step: 102205 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:26:44,908-Speed 133.69 samples/sec Loss 4.3274 Epoch: 10 Global Step: 102210 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:27:04,157-Speed 133.00 samples/sec Loss 4.6688 Epoch: 10 Global Step: 102215 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:27:23,423-Speed 132.88 samples/sec Loss 4.5831 Epoch: 10 Global Step: 102220 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:27:42,659-Speed 133.09 samples/sec Loss 4.7114 Epoch: 10 Global Step: 102225 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:28:02,227-Speed 130.83 samples/sec Loss 4.6058 Epoch: 10 Global Step: 102230 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:28:21,636-Speed 131.90 samples/sec Loss 4.4174 Epoch: 10 Global Step: 102235 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:28:41,233-Speed 130.64 samples/sec Loss 4.4952 Epoch: 10 Global Step: 102240 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:29:01,135-Speed 128.64 samples/sec Loss 4.5835 Epoch: 10 Global Step: 102245 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:29:20,177-Speed 134.44 samples/sec Loss 4.4567 Epoch: 10 Global Step: 102250 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:29:39,422-Speed 133.03 samples/sec Loss 4.2622 Epoch: 10 Global Step: 102255 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:29:58,594-Speed 133.54 samples/sec Loss 4.4463 Epoch: 10 Global Step: 102260 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:30:17,880-Speed 132.75 samples/sec Loss 4.5735 Epoch: 10 Global Step: 102265 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:30:36,796-Speed 135.34 samples/sec Loss 4.4968 Epoch: 10 Global Step: 102270 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:30:56,159-Speed 132.22 samples/sec Loss 4.7521 Epoch: 10 Global Step: 102275 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:31:15,390-Speed 133.13 samples/sec Loss 4.2083 Epoch: 10 Global Step: 102280 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:31:34,975-Speed 130.72 samples/sec Loss 4.4952 Epoch: 10 Global Step: 102285 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:31:54,066-Speed 134.10 samples/sec Loss 4.4185 Epoch: 10 Global Step: 102290 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:32:13,448-Speed 132.09 samples/sec Loss 4.3676 Epoch: 10 Global Step: 102295 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:32:32,667-Speed 133.21 samples/sec Loss 4.3242 Epoch: 10 Global Step: 102300 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:32:52,085-Speed 131.85 samples/sec Loss 4.5957 Epoch: 10 Global Step: 102305 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:33:11,435-Speed 132.31 samples/sec Loss 4.6971 Epoch: 10 Global Step: 102310 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:33:30,542-Speed 133.99 samples/sec Loss 4.5897 Epoch: 10 Global Step: 102315 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:33:49,920-Speed 132.11 samples/sec Loss 4.7350 Epoch: 10 Global Step: 102320 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:34:09,380-Speed 131.56 samples/sec Loss 4.5124 Epoch: 10 Global Step: 102325 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:34:28,804-Speed 131.80 samples/sec Loss 4.3798 Epoch: 10 Global Step: 102330 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:34:48,027-Speed 133.19 samples/sec Loss 4.5608 Epoch: 10 Global Step: 102335 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:35:07,277-Speed 132.99 samples/sec Loss 4.3046 Epoch: 10 Global Step: 102340 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:35:26,321-Speed 134.43 samples/sec Loss 4.3181 Epoch: 10 Global Step: 102345 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:35:45,332-Speed 134.67 samples/sec Loss 4.6174 Epoch: 10 Global Step: 102350 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:36:04,758-Speed 131.79 samples/sec Loss 4.2846 Epoch: 10 Global Step: 102355 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:36:24,079-Speed 132.50 samples/sec Loss 4.4313 Epoch: 10 Global Step: 102360 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:36:43,366-Speed 132.75 samples/sec Loss 4.5140 Epoch: 10 Global Step: 102365 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:37:02,870-Speed 131.26 samples/sec Loss 4.5658 Epoch: 10 Global Step: 102370 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:37:22,376-Speed 131.25 samples/sec Loss 4.5970 Epoch: 10 Global Step: 102375 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:37:41,705-Speed 132.45 samples/sec Loss 4.5788 Epoch: 10 Global Step: 102380 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:38:01,093-Speed 132.05 samples/sec Loss 4.4668 Epoch: 10 Global Step: 102385 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:38:20,517-Speed 131.80 samples/sec Loss 4.4732 Epoch: 10 Global Step: 102390 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:38:39,720-Speed 133.32 samples/sec Loss 4.3711 Epoch: 10 Global Step: 102395 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:38:59,334-Speed 130.53 samples/sec Loss 4.6563 Epoch: 10 Global Step: 102400 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:39:18,334-Speed 134.75 samples/sec Loss 4.1427 Epoch: 10 Global Step: 102405 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:39:37,499-Speed 133.58 samples/sec Loss 4.2382 Epoch: 10 Global Step: 102410 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:39:57,120-Speed 130.48 samples/sec Loss 4.5444 Epoch: 10 Global Step: 102415 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:40:16,582-Speed 131.55 samples/sec Loss 4.1219 Epoch: 10 Global Step: 102420 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:40:35,776-Speed 133.39 samples/sec Loss 4.3455 Epoch: 10 Global Step: 102425 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:40:54,958-Speed 133.47 samples/sec Loss 4.3711 Epoch: 10 Global Step: 102430 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:41:14,298-Speed 132.37 samples/sec Loss 4.4787 Epoch: 10 Global Step: 102435 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:41:33,531-Speed 133.12 samples/sec Loss 4.5498 Epoch: 10 Global Step: 102440 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:41:52,777-Speed 133.02 samples/sec Loss 4.3092 Epoch: 10 Global Step: 102445 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:42:12,177-Speed 131.96 samples/sec Loss 4.3667 Epoch: 10 Global Step: 102450 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:42:31,557-Speed 132.10 samples/sec Loss 4.5336 Epoch: 10 Global Step: 102455 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:42:51,184-Speed 130.44 samples/sec Loss 4.4880 Epoch: 10 Global Step: 102460 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:43:10,671-Speed 131.37 samples/sec Loss 4.4107 Epoch: 10 Global Step: 102465 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:43:29,947-Speed 132.82 samples/sec Loss 4.4069 Epoch: 10 Global Step: 102470 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:43:49,358-Speed 131.89 samples/sec Loss 4.4631 Epoch: 10 Global Step: 102475 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:44:08,537-Speed 133.49 samples/sec Loss 4.1846 Epoch: 10 Global Step: 102480 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:44:27,869-Speed 132.43 samples/sec Loss 4.4795 Epoch: 10 Global Step: 102485 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:44:47,119-Speed 132.99 samples/sec Loss 4.3142 Epoch: 10 Global Step: 102490 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:45:06,428-Speed 132.59 samples/sec Loss 4.4973 Epoch: 10 Global Step: 102495 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:45:25,707-Speed 132.79 samples/sec Loss 4.3657 Epoch: 10 Global Step: 102500 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:45:45,272-Speed 130.85 samples/sec Loss 4.5320 Epoch: 10 Global Step: 102505 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:46:04,712-Speed 131.69 samples/sec Loss 4.6047 Epoch: 10 Global Step: 102510 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:46:24,132-Speed 131.83 samples/sec Loss 4.4649 Epoch: 10 Global Step: 102515 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:46:43,535-Speed 131.95 samples/sec Loss 4.4617 Epoch: 10 Global Step: 102520 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:47:02,969-Speed 131.74 samples/sec Loss 4.4022 Epoch: 10 Global Step: 102525 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:47:22,407-Speed 131.70 samples/sec Loss 4.4997 Epoch: 10 Global Step: 102530 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:47:41,609-Speed 133.33 samples/sec Loss 4.2992 Epoch: 10 Global Step: 102535 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:48:01,076-Speed 131.51 samples/sec Loss 4.3491 Epoch: 10 Global Step: 102540 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:48:20,455-Speed 132.11 samples/sec Loss 4.4900 Epoch: 10 Global Step: 102545 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:48:40,009-Speed 130.93 samples/sec Loss 4.3217 Epoch: 10 Global Step: 102550 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:48:59,389-Speed 132.10 samples/sec Loss 4.3559 Epoch: 10 Global Step: 102555 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:49:18,822-Speed 131.74 samples/sec Loss 4.3760 Epoch: 10 Global Step: 102560 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:49:38,417-Speed 130.65 samples/sec Loss 4.4766 Epoch: 10 Global Step: 102565 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:49:57,832-Speed 131.87 samples/sec Loss 4.7812 Epoch: 10 Global Step: 102570 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:50:16,990-Speed 133.63 samples/sec Loss 4.3694 Epoch: 10 Global Step: 102575 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:50:36,152-Speed 133.61 samples/sec Loss 4.3619 Epoch: 10 Global Step: 102580 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:50:55,626-Speed 131.46 samples/sec Loss 4.5494 Epoch: 10 Global Step: 102585 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:51:15,227-Speed 130.61 samples/sec Loss 4.6050 Epoch: 10 Global Step: 102590 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:51:34,886-Speed 130.23 samples/sec Loss 4.3058 Epoch: 10 Global Step: 102595 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:51:54,357-Speed 131.48 samples/sec Loss 4.3839 Epoch: 10 Global Step: 102600 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:52:13,668-Speed 132.57 samples/sec Loss 4.3572 Epoch: 10 Global Step: 102605 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:52:32,952-Speed 132.76 samples/sec Loss 4.2121 Epoch: 10 Global Step: 102610 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:52:52,266-Speed 132.55 samples/sec Loss 4.4931 Epoch: 10 Global Step: 102615 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:53:11,503-Speed 133.09 samples/sec Loss 4.3556 Epoch: 10 Global Step: 102620 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:53:30,875-Speed 132.16 samples/sec Loss 4.5442 Epoch: 10 Global Step: 102625 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:53:50,096-Speed 133.19 samples/sec Loss 4.0649 Epoch: 10 Global Step: 102630 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:54:09,493-Speed 131.99 samples/sec Loss 4.2135 Epoch: 10 Global Step: 102635 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:54:29,257-Speed 129.54 samples/sec Loss 4.3576 Epoch: 10 Global Step: 102640 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:54:48,487-Speed 133.13 samples/sec Loss 4.2173 Epoch: 10 Global Step: 102645 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:55:07,908-Speed 131.83 samples/sec Loss 4.3533 Epoch: 10 Global Step: 102650 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:55:27,296-Speed 132.05 samples/sec Loss 4.0818 Epoch: 10 Global Step: 102655 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:55:46,510-Speed 133.25 samples/sec Loss 4.0335 Epoch: 10 Global Step: 102660 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:56:05,827-Speed 132.53 samples/sec Loss 4.2845 Epoch: 10 Global Step: 102665 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:56:25,200-Speed 132.15 samples/sec Loss 4.3126 Epoch: 10 Global Step: 102670 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:56:44,409-Speed 133.28 samples/sec Loss 4.1303 Epoch: 10 Global Step: 102675 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:57:03,681-Speed 132.84 samples/sec Loss 4.3367 Epoch: 10 Global Step: 102680 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:57:23,466-Speed 129.40 samples/sec Loss 4.3106 Epoch: 10 Global Step: 102685 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:57:42,895-Speed 131.77 samples/sec Loss 4.3765 Epoch: 10 Global Step: 102690 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:58:02,137-Speed 133.05 samples/sec Loss 4.2966 Epoch: 10 Global Step: 102695 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:58:21,400-Speed 132.90 samples/sec Loss 4.3709 Epoch: 10 Global Step: 102700 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:58:40,705-Speed 132.62 samples/sec Loss 4.4435 Epoch: 10 Global Step: 102705 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:59:00,282-Speed 130.77 samples/sec Loss 4.2803 Epoch: 10 Global Step: 102710 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:59:19,570-Speed 132.73 samples/sec Loss 4.1245 Epoch: 10 Global Step: 102715 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:59:39,099-Speed 131.09 samples/sec Loss 4.2230 Epoch: 10 Global Step: 102720 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 20:59:58,384-Speed 132.75 samples/sec Loss 4.1703 Epoch: 10 Global Step: 102725 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:00:18,157-Speed 129.47 samples/sec Loss 4.3336 Epoch: 10 Global Step: 102730 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:00:37,264-Speed 133.99 samples/sec Loss 4.2543 Epoch: 10 Global Step: 102735 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:00:56,659-Speed 132.00 samples/sec Loss 4.5440 Epoch: 10 Global Step: 102740 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:01:16,158-Speed 131.29 samples/sec Loss 4.1397 Epoch: 10 Global Step: 102745 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:01:35,701-Speed 131.01 samples/sec Loss 4.1199 Epoch: 10 Global Step: 102750 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:01:55,067-Speed 132.20 samples/sec Loss 4.2416 Epoch: 10 Global Step: 102755 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:02:14,680-Speed 130.53 samples/sec Loss 4.3755 Epoch: 10 Global Step: 102760 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:02:33,700-Speed 134.61 samples/sec Loss 4.2370 Epoch: 10 Global Step: 102765 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:02:53,087-Speed 132.05 samples/sec Loss 4.1716 Epoch: 10 Global Step: 102770 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:03:12,717-Speed 130.42 samples/sec Loss 4.1952 Epoch: 10 Global Step: 102775 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:03:32,005-Speed 132.73 samples/sec Loss 4.2202 Epoch: 10 Global Step: 102780 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:03:51,434-Speed 131.77 samples/sec Loss 4.0869 Epoch: 10 Global Step: 102785 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:04:11,051-Speed 130.50 samples/sec Loss 4.3121 Epoch: 10 Global Step: 102790 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:04:30,415-Speed 132.21 samples/sec Loss 4.3849 Epoch: 10 Global Step: 102795 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:04:50,011-Speed 130.64 samples/sec Loss 4.3178 Epoch: 10 Global Step: 102800 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:05:09,612-Speed 130.61 samples/sec Loss 4.2240 Epoch: 10 Global Step: 102805 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:05:28,969-Speed 132.26 samples/sec Loss 4.4402 Epoch: 10 Global Step: 102810 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:05:48,563-Speed 130.66 samples/sec Loss 4.2881 Epoch: 10 Global Step: 102815 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:06:08,190-Speed 130.44 samples/sec Loss 4.1656 Epoch: 10 Global Step: 102820 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:06:27,690-Speed 131.29 samples/sec Loss 4.1307 Epoch: 10 Global Step: 102825 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:06:47,193-Speed 131.27 samples/sec Loss 4.4528 Epoch: 10 Global Step: 102830 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:07:06,698-Speed 131.26 samples/sec Loss 4.4435 Epoch: 10 Global Step: 102835 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:07:26,239-Speed 131.01 samples/sec Loss 4.3393 Epoch: 10 Global Step: 102840 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:07:45,973-Speed 129.74 samples/sec Loss 4.1080 Epoch: 10 Global Step: 102845 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:08:05,450-Speed 131.45 samples/sec Loss 4.4137 Epoch: 10 Global Step: 102850 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:08:25,027-Speed 130.77 samples/sec Loss 4.1431 Epoch: 10 Global Step: 102855 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:08:44,549-Speed 131.14 samples/sec Loss 4.5139 Epoch: 10 Global Step: 102860 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:09:04,209-Speed 130.22 samples/sec Loss 4.1751 Epoch: 10 Global Step: 102865 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:09:23,717-Speed 131.24 samples/sec Loss 4.2385 Epoch: 10 Global Step: 102870 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:09:43,260-Speed 130.99 samples/sec Loss 4.4295 Epoch: 10 Global Step: 102875 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:10:02,796-Speed 131.05 samples/sec Loss 4.2060 Epoch: 10 Global Step: 102880 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:10:22,282-Speed 131.38 samples/sec Loss 4.1055 Epoch: 10 Global Step: 102885 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:10:41,765-Speed 131.41 samples/sec Loss 4.1448 Epoch: 10 Global Step: 102890 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:11:01,242-Speed 131.45 samples/sec Loss 4.2545 Epoch: 10 Global Step: 102895 Fp16 Grad Scale: 8192 Required: 174 hours Training: 2021-06-11 21:11:20,672-Speed 131.76 samples/sec Loss 4.1159 Epoch: 10 Global Step: 102900 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:11:40,178-Speed 131.25 samples/sec Loss 4.3340 Epoch: 10 Global Step: 102905 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:11:59,593-Speed 131.87 samples/sec Loss 3.9878 Epoch: 10 Global Step: 102910 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:12:19,187-Speed 130.65 samples/sec Loss 4.2391 Epoch: 10 Global Step: 102915 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:12:38,706-Speed 131.17 samples/sec Loss 4.2633 Epoch: 10 Global Step: 102920 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:12:58,342-Speed 130.37 samples/sec Loss 4.3527 Epoch: 10 Global Step: 102925 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:13:17,658-Speed 132.54 samples/sec Loss 4.0762 Epoch: 10 Global Step: 102930 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:13:36,725-Speed 134.27 samples/sec Loss 4.0724 Epoch: 10 Global Step: 102935 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:13:56,089-Speed 132.21 samples/sec Loss 4.2007 Epoch: 10 Global Step: 102940 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:14:15,428-Speed 132.38 samples/sec Loss 4.0941 Epoch: 10 Global Step: 102945 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:14:34,924-Speed 131.32 samples/sec Loss 4.2170 Epoch: 10 Global Step: 102950 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:14:54,586-Speed 130.21 samples/sec Loss 4.1027 Epoch: 10 Global Step: 102955 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:15:14,201-Speed 130.52 samples/sec Loss 4.3120 Epoch: 10 Global Step: 102960 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:15:33,540-Speed 132.38 samples/sec Loss 4.2044 Epoch: 10 Global Step: 102965 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:15:53,032-Speed 131.34 samples/sec Loss 3.9433 Epoch: 10 Global Step: 102970 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:16:12,259-Speed 133.15 samples/sec Loss 4.0602 Epoch: 10 Global Step: 102975 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:16:31,486-Speed 133.16 samples/sec Loss 4.1292 Epoch: 10 Global Step: 102980 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:16:51,061-Speed 130.78 samples/sec Loss 4.2472 Epoch: 10 Global Step: 102985 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:17:10,566-Speed 131.26 samples/sec Loss 4.4991 Epoch: 10 Global Step: 102990 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:17:30,077-Speed 131.22 samples/sec Loss 4.1718 Epoch: 10 Global Step: 102995 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:17:49,688-Speed 130.54 samples/sec Loss 4.1316 Epoch: 10 Global Step: 103000 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:18:09,128-Speed 131.70 samples/sec Loss 4.1621 Epoch: 10 Global Step: 103005 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:18:28,412-Speed 132.76 samples/sec Loss 4.1012 Epoch: 10 Global Step: 103010 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:18:48,041-Speed 130.43 samples/sec Loss 4.0473 Epoch: 10 Global Step: 103015 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:19:07,473-Speed 131.74 samples/sec Loss 4.0100 Epoch: 10 Global Step: 103020 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:19:26,959-Speed 131.39 samples/sec Loss 4.1612 Epoch: 10 Global Step: 103025 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:19:46,445-Speed 131.39 samples/sec Loss 4.1924 Epoch: 10 Global Step: 103030 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:20:05,883-Speed 131.71 samples/sec Loss 4.1831 Epoch: 10 Global Step: 103035 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:20:25,677-Speed 129.34 samples/sec Loss 4.2394 Epoch: 10 Global Step: 103040 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:20:45,290-Speed 130.53 samples/sec Loss 3.8461 Epoch: 10 Global Step: 103045 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:21:04,862-Speed 130.81 samples/sec Loss 4.1920 Epoch: 10 Global Step: 103050 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:21:24,299-Speed 131.72 samples/sec Loss 4.2361 Epoch: 10 Global Step: 103055 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:21:43,913-Speed 130.53 samples/sec Loss 4.3435 Epoch: 10 Global Step: 103060 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:22:03,346-Speed 131.74 samples/sec Loss 4.2678 Epoch: 10 Global Step: 103065 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:22:22,890-Speed 131.00 samples/sec Loss 4.3057 Epoch: 10 Global Step: 103070 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:22:42,230-Speed 132.38 samples/sec Loss 3.9830 Epoch: 10 Global Step: 103075 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:23:01,688-Speed 131.57 samples/sec Loss 4.1075 Epoch: 10 Global Step: 103080 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:23:21,084-Speed 131.99 samples/sec Loss 3.9817 Epoch: 10 Global Step: 103085 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:23:40,700-Speed 130.52 samples/sec Loss 4.2202 Epoch: 10 Global Step: 103090 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:24:00,258-Speed 130.90 samples/sec Loss 4.0696 Epoch: 10 Global Step: 103095 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:24:19,575-Speed 132.53 samples/sec Loss 4.0936 Epoch: 10 Global Step: 103100 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:24:39,195-Speed 130.49 samples/sec Loss 4.1517 Epoch: 10 Global Step: 103105 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:24:58,508-Speed 132.56 samples/sec Loss 3.9703 Epoch: 10 Global Step: 103110 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:25:18,111-Speed 130.59 samples/sec Loss 4.2082 Epoch: 10 Global Step: 103115 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:25:37,702-Speed 130.70 samples/sec Loss 4.1976 Epoch: 10 Global Step: 103120 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:25:57,259-Speed 130.91 samples/sec Loss 4.2251 Epoch: 10 Global Step: 103125 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:26:16,768-Speed 131.22 samples/sec Loss 4.2513 Epoch: 10 Global Step: 103130 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:26:36,680-Speed 128.57 samples/sec Loss 4.1139 Epoch: 10 Global Step: 103135 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:26:56,122-Speed 131.68 samples/sec Loss 4.0739 Epoch: 10 Global Step: 103140 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:27:15,631-Speed 131.23 samples/sec Loss 4.0835 Epoch: 10 Global Step: 103145 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:27:35,057-Speed 131.79 samples/sec Loss 3.9497 Epoch: 10 Global Step: 103150 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:27:54,480-Speed 131.81 samples/sec Loss 4.1746 Epoch: 10 Global Step: 103155 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:28:14,129-Speed 130.30 samples/sec Loss 4.1617 Epoch: 10 Global Step: 103160 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:28:33,634-Speed 131.25 samples/sec Loss 4.1052 Epoch: 10 Global Step: 103165 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:28:53,124-Speed 131.36 samples/sec Loss 3.9626 Epoch: 10 Global Step: 103170 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:29:12,667-Speed 131.00 samples/sec Loss 4.1707 Epoch: 10 Global Step: 103175 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:29:32,362-Speed 129.99 samples/sec Loss 4.2342 Epoch: 10 Global Step: 103180 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:29:51,677-Speed 132.54 samples/sec Loss 4.0587 Epoch: 10 Global Step: 103185 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:30:11,332-Speed 130.25 samples/sec Loss 4.2001 Epoch: 10 Global Step: 103190 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:30:30,761-Speed 131.77 samples/sec Loss 3.8124 Epoch: 10 Global Step: 103195 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:30:50,358-Speed 130.64 samples/sec Loss 3.9961 Epoch: 10 Global Step: 103200 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:31:09,681-Speed 132.49 samples/sec Loss 3.9212 Epoch: 10 Global Step: 103205 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:31:29,210-Speed 131.10 samples/sec Loss 4.1074 Epoch: 10 Global Step: 103210 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:31:48,698-Speed 131.37 samples/sec Loss 3.9670 Epoch: 10 Global Step: 103215 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:32:08,108-Speed 131.90 samples/sec Loss 3.9183 Epoch: 10 Global Step: 103220 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:32:27,949-Speed 129.04 samples/sec Loss 3.9949 Epoch: 10 Global Step: 103225 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:32:47,518-Speed 130.82 samples/sec Loss 4.2616 Epoch: 10 Global Step: 103230 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:33:07,181-Speed 130.20 samples/sec Loss 4.0657 Epoch: 10 Global Step: 103235 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:33:26,758-Speed 130.77 samples/sec Loss 4.0001 Epoch: 10 Global Step: 103240 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:33:46,143-Speed 132.07 samples/sec Loss 3.9359 Epoch: 10 Global Step: 103245 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:34:05,677-Speed 131.06 samples/sec Loss 4.0890 Epoch: 10 Global Step: 103250 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:34:25,217-Speed 131.02 samples/sec Loss 4.1716 Epoch: 10 Global Step: 103255 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:34:44,873-Speed 130.25 samples/sec Loss 4.0685 Epoch: 10 Global Step: 103260 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:35:04,237-Speed 132.22 samples/sec Loss 4.2144 Epoch: 10 Global Step: 103265 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:35:24,055-Speed 129.18 samples/sec Loss 4.0009 Epoch: 10 Global Step: 103270 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:35:43,741-Speed 130.05 samples/sec Loss 3.9352 Epoch: 10 Global Step: 103275 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:36:03,193-Speed 131.61 samples/sec Loss 4.0821 Epoch: 10 Global Step: 103280 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:36:22,536-Speed 132.36 samples/sec Loss 3.8799 Epoch: 10 Global Step: 103285 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:36:41,996-Speed 131.56 samples/sec Loss 3.9232 Epoch: 10 Global Step: 103290 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:37:01,372-Speed 132.13 samples/sec Loss 3.8906 Epoch: 10 Global Step: 103295 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:37:20,898-Speed 131.12 samples/sec Loss 3.9844 Epoch: 10 Global Step: 103300 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:37:40,346-Speed 131.64 samples/sec Loss 4.0393 Epoch: 10 Global Step: 103305 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:37:59,866-Speed 131.15 samples/sec Loss 4.2282 Epoch: 10 Global Step: 103310 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:38:19,637-Speed 129.49 samples/sec Loss 3.9726 Epoch: 10 Global Step: 103315 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:38:39,276-Speed 130.36 samples/sec Loss 4.1106 Epoch: 10 Global Step: 103320 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:38:58,599-Speed 132.49 samples/sec Loss 3.8929 Epoch: 10 Global Step: 103325 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:39:18,050-Speed 131.62 samples/sec Loss 4.0419 Epoch: 10 Global Step: 103330 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:39:37,626-Speed 130.78 samples/sec Loss 4.0245 Epoch: 10 Global Step: 103335 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:39:56,892-Speed 132.88 samples/sec Loss 3.8045 Epoch: 10 Global Step: 103340 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:40:16,390-Speed 131.30 samples/sec Loss 4.1534 Epoch: 10 Global Step: 103345 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:40:35,944-Speed 130.93 samples/sec Loss 4.0434 Epoch: 10 Global Step: 103350 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:40:55,507-Speed 130.87 samples/sec Loss 4.0502 Epoch: 10 Global Step: 103355 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:41:15,051-Speed 130.99 samples/sec Loss 4.0116 Epoch: 10 Global Step: 103360 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:41:34,633-Speed 130.74 samples/sec Loss 3.7357 Epoch: 10 Global Step: 103365 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:41:54,098-Speed 131.53 samples/sec Loss 3.8306 Epoch: 10 Global Step: 103370 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:42:13,606-Speed 131.23 samples/sec Loss 3.9083 Epoch: 10 Global Step: 103375 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:42:32,970-Speed 132.21 samples/sec Loss 3.9400 Epoch: 10 Global Step: 103380 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:42:52,529-Speed 130.90 samples/sec Loss 4.1955 Epoch: 10 Global Step: 103385 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:43:12,155-Speed 130.45 samples/sec Loss 4.3225 Epoch: 10 Global Step: 103390 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:43:31,700-Speed 130.99 samples/sec Loss 4.0841 Epoch: 10 Global Step: 103395 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:43:51,297-Speed 130.64 samples/sec Loss 4.1080 Epoch: 10 Global Step: 103400 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:44:11,262-Speed 128.23 samples/sec Loss 4.0844 Epoch: 10 Global Step: 103405 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:44:30,861-Speed 130.63 samples/sec Loss 4.1427 Epoch: 10 Global Step: 103410 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:44:50,416-Speed 130.92 samples/sec Loss 4.0260 Epoch: 10 Global Step: 103415 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:45:10,017-Speed 130.61 samples/sec Loss 3.8886 Epoch: 10 Global Step: 103420 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:45:29,629-Speed 130.54 samples/sec Loss 4.1834 Epoch: 10 Global Step: 103425 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:45:49,213-Speed 130.73 samples/sec Loss 4.0090 Epoch: 10 Global Step: 103430 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:46:08,769-Speed 130.92 samples/sec Loss 4.2116 Epoch: 10 Global Step: 103435 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:46:28,638-Speed 128.85 samples/sec Loss 4.0376 Epoch: 10 Global Step: 103440 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:46:48,244-Speed 130.58 samples/sec Loss 4.1254 Epoch: 10 Global Step: 103445 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:47:08,146-Speed 128.64 samples/sec Loss 4.0667 Epoch: 10 Global Step: 103450 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:47:27,612-Speed 131.52 samples/sec Loss 3.9315 Epoch: 10 Global Step: 103455 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:47:47,167-Speed 130.91 samples/sec Loss 3.9071 Epoch: 10 Global Step: 103460 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:48:06,779-Speed 130.54 samples/sec Loss 3.7447 Epoch: 10 Global Step: 103465 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:48:26,208-Speed 131.77 samples/sec Loss 3.9333 Epoch: 10 Global Step: 103470 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:48:45,734-Speed 131.12 samples/sec Loss 4.1041 Epoch: 10 Global Step: 103475 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:49:05,210-Speed 131.45 samples/sec Loss 4.0208 Epoch: 10 Global Step: 103480 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:49:24,729-Speed 131.16 samples/sec Loss 3.9833 Epoch: 10 Global Step: 103485 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:49:44,323-Speed 130.66 samples/sec Loss 4.1862 Epoch: 10 Global Step: 103490 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:50:03,980-Speed 130.24 samples/sec Loss 3.9917 Epoch: 10 Global Step: 103495 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:50:23,524-Speed 131.00 samples/sec Loss 4.0030 Epoch: 10 Global Step: 103500 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:50:43,051-Speed 131.10 samples/sec Loss 3.8992 Epoch: 10 Global Step: 103505 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:51:02,594-Speed 131.00 samples/sec Loss 3.6520 Epoch: 10 Global Step: 103510 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:51:21,812-Speed 133.22 samples/sec Loss 3.8929 Epoch: 10 Global Step: 103515 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:51:41,409-Speed 130.64 samples/sec Loss 3.9674 Epoch: 10 Global Step: 103520 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:52:00,856-Speed 131.65 samples/sec Loss 4.0140 Epoch: 10 Global Step: 103525 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:52:20,342-Speed 131.38 samples/sec Loss 3.7751 Epoch: 10 Global Step: 103530 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:52:39,912-Speed 130.82 samples/sec Loss 4.0880 Epoch: 10 Global Step: 103535 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:52:59,820-Speed 128.60 samples/sec Loss 3.9068 Epoch: 10 Global Step: 103540 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:53:19,257-Speed 131.72 samples/sec Loss 4.1443 Epoch: 10 Global Step: 103545 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:53:38,789-Speed 131.07 samples/sec Loss 3.9460 Epoch: 10 Global Step: 103550 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:53:58,344-Speed 130.93 samples/sec Loss 3.6719 Epoch: 10 Global Step: 103555 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:54:17,918-Speed 130.79 samples/sec Loss 4.0199 Epoch: 10 Global Step: 103560 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:54:38,007-Speed 127.44 samples/sec Loss 4.1770 Epoch: 10 Global Step: 103565 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:54:57,436-Speed 131.77 samples/sec Loss 4.0478 Epoch: 10 Global Step: 103570 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:55:16,933-Speed 131.31 samples/sec Loss 3.9291 Epoch: 10 Global Step: 103575 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:55:36,259-Speed 132.47 samples/sec Loss 3.8488 Epoch: 10 Global Step: 103580 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:55:56,142-Speed 128.76 samples/sec Loss 3.9306 Epoch: 10 Global Step: 103585 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:56:15,566-Speed 131.80 samples/sec Loss 3.8181 Epoch: 10 Global Step: 103590 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:56:35,311-Speed 129.66 samples/sec Loss 4.0116 Epoch: 10 Global Step: 103595 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:56:54,740-Speed 131.77 samples/sec Loss 3.8706 Epoch: 10 Global Step: 103600 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:57:14,330-Speed 130.69 samples/sec Loss 4.1421 Epoch: 10 Global Step: 103605 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:57:33,781-Speed 131.62 samples/sec Loss 3.8667 Epoch: 10 Global Step: 103610 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:57:53,330-Speed 130.96 samples/sec Loss 3.9791 Epoch: 10 Global Step: 103615 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:58:13,082-Speed 129.62 samples/sec Loss 3.8720 Epoch: 10 Global Step: 103620 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:58:32,705-Speed 130.46 samples/sec Loss 3.9294 Epoch: 10 Global Step: 103625 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:58:52,503-Speed 129.31 samples/sec Loss 3.9724 Epoch: 10 Global Step: 103630 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:59:12,118-Speed 130.52 samples/sec Loss 3.7311 Epoch: 10 Global Step: 103635 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:59:31,617-Speed 131.30 samples/sec Loss 3.8927 Epoch: 10 Global Step: 103640 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 21:59:51,170-Speed 130.93 samples/sec Loss 3.8270 Epoch: 10 Global Step: 103645 Fp16 Grad Scale: 8192 Required: 173 hours Training: 2021-06-11 22:00:10,886-Speed 129.85 samples/sec Loss 3.8712 Epoch: 10 Global Step: 103650 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:00:30,302-Speed 131.85 samples/sec Loss 3.8876 Epoch: 10 Global Step: 103655 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:00:49,771-Speed 131.50 samples/sec Loss 3.7763 Epoch: 10 Global Step: 103660 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:01:09,378-Speed 130.57 samples/sec Loss 3.6714 Epoch: 10 Global Step: 103665 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:01:28,999-Speed 130.48 samples/sec Loss 4.0915 Epoch: 10 Global Step: 103670 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:01:48,792-Speed 129.34 samples/sec Loss 4.0141 Epoch: 10 Global Step: 103675 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:02:08,377-Speed 130.72 samples/sec Loss 3.8245 Epoch: 10 Global Step: 103680 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:02:27,937-Speed 130.89 samples/sec Loss 3.7739 Epoch: 10 Global Step: 103685 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:02:47,471-Speed 131.06 samples/sec Loss 4.2147 Epoch: 10 Global Step: 103690 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:03:07,037-Speed 130.85 samples/sec Loss 3.8607 Epoch: 10 Global Step: 103695 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:03:26,457-Speed 131.83 samples/sec Loss 3.9802 Epoch: 10 Global Step: 103700 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:03:46,086-Speed 130.43 samples/sec Loss 3.7049 Epoch: 10 Global Step: 103705 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:04:05,636-Speed 130.95 samples/sec Loss 3.6766 Epoch: 10 Global Step: 103710 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:04:25,098-Speed 131.55 samples/sec Loss 3.9049 Epoch: 10 Global Step: 103715 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:04:45,032-Speed 128.43 samples/sec Loss 3.7287 Epoch: 10 Global Step: 103720 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:05:04,638-Speed 130.58 samples/sec Loss 3.7767 Epoch: 10 Global Step: 103725 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:05:24,177-Speed 131.03 samples/sec Loss 3.9524 Epoch: 10 Global Step: 103730 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:05:43,709-Speed 131.08 samples/sec Loss 3.9502 Epoch: 10 Global Step: 103735 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:06:03,284-Speed 130.78 samples/sec Loss 3.9112 Epoch: 10 Global Step: 103740 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:06:22,817-Speed 131.07 samples/sec Loss 3.9782 Epoch: 10 Global Step: 103745 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:06:42,408-Speed 130.68 samples/sec Loss 3.8459 Epoch: 10 Global Step: 103750 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:07:02,224-Speed 129.20 samples/sec Loss 3.7991 Epoch: 10 Global Step: 103755 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:07:21,752-Speed 131.10 samples/sec Loss 3.7428 Epoch: 10 Global Step: 103760 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:07:41,629-Speed 128.81 samples/sec Loss 3.9629 Epoch: 10 Global Step: 103765 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:08:01,413-Speed 129.40 samples/sec Loss 3.9500 Epoch: 10 Global Step: 103770 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:08:20,840-Speed 131.78 samples/sec Loss 3.8682 Epoch: 10 Global Step: 103775 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:08:40,455-Speed 130.52 samples/sec Loss 3.8663 Epoch: 10 Global Step: 103780 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:08:59,979-Speed 131.13 samples/sec Loss 3.7011 Epoch: 10 Global Step: 103785 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:09:19,596-Speed 130.51 samples/sec Loss 3.7786 Epoch: 10 Global Step: 103790 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:09:39,193-Speed 130.64 samples/sec Loss 3.9191 Epoch: 10 Global Step: 103795 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:09:58,709-Speed 131.18 samples/sec Loss 3.9255 Epoch: 10 Global Step: 103800 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:10:18,314-Speed 130.58 samples/sec Loss 3.8547 Epoch: 10 Global Step: 103805 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:10:38,231-Speed 128.54 samples/sec Loss 3.9154 Epoch: 10 Global Step: 103810 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:10:57,674-Speed 131.67 samples/sec Loss 4.0783 Epoch: 10 Global Step: 103815 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:11:17,337-Speed 130.20 samples/sec Loss 3.7462 Epoch: 10 Global Step: 103820 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:11:36,932-Speed 130.65 samples/sec Loss 3.7676 Epoch: 10 Global Step: 103825 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:11:56,471-Speed 131.03 samples/sec Loss 4.0310 Epoch: 10 Global Step: 103830 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:12:16,205-Speed 129.74 samples/sec Loss 3.8865 Epoch: 10 Global Step: 103835 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:12:36,137-Speed 128.44 samples/sec Loss 3.6445 Epoch: 10 Global Step: 103840 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:12:55,619-Speed 131.41 samples/sec Loss 3.9557 Epoch: 10 Global Step: 103845 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:13:15,071-Speed 131.61 samples/sec Loss 3.8734 Epoch: 10 Global Step: 103850 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:13:35,168-Speed 127.39 samples/sec Loss 4.0938 Epoch: 10 Global Step: 103855 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:13:54,614-Speed 131.66 samples/sec Loss 4.0268 Epoch: 10 Global Step: 103860 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:14:14,151-Speed 131.04 samples/sec Loss 4.0772 Epoch: 10 Global Step: 103865 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:14:33,676-Speed 131.12 samples/sec Loss 3.8395 Epoch: 10 Global Step: 103870 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:14:53,189-Speed 131.20 samples/sec Loss 3.7341 Epoch: 10 Global Step: 103875 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:15:12,811-Speed 130.47 samples/sec Loss 3.7812 Epoch: 10 Global Step: 103880 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:15:32,352-Speed 131.02 samples/sec Loss 3.8850 Epoch: 10 Global Step: 103885 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:15:51,947-Speed 130.65 samples/sec Loss 3.8317 Epoch: 10 Global Step: 103890 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:16:11,485-Speed 131.03 samples/sec Loss 3.7458 Epoch: 10 Global Step: 103895 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:16:31,335-Speed 128.98 samples/sec Loss 3.6582 Epoch: 10 Global Step: 103900 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:16:50,880-Speed 130.99 samples/sec Loss 3.7617 Epoch: 10 Global Step: 103905 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:17:10,474-Speed 130.66 samples/sec Loss 3.9392 Epoch: 10 Global Step: 103910 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:17:30,077-Speed 130.60 samples/sec Loss 3.6667 Epoch: 10 Global Step: 103915 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:17:49,600-Speed 131.13 samples/sec Loss 3.6079 Epoch: 10 Global Step: 103920 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:18:09,157-Speed 130.91 samples/sec Loss 3.7297 Epoch: 10 Global Step: 103925 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:18:28,702-Speed 130.99 samples/sec Loss 3.8693 Epoch: 10 Global Step: 103930 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:18:48,965-Speed 126.34 samples/sec Loss 3.9143 Epoch: 10 Global Step: 103935 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:19:08,605-Speed 130.35 samples/sec Loss 3.8311 Epoch: 10 Global Step: 103940 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:19:28,482-Speed 128.80 samples/sec Loss 3.7515 Epoch: 10 Global Step: 103945 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:19:47,911-Speed 131.77 samples/sec Loss 3.8553 Epoch: 10 Global Step: 103950 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:20:07,537-Speed 130.45 samples/sec Loss 3.9390 Epoch: 10 Global Step: 103955 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:20:27,140-Speed 130.59 samples/sec Loss 4.0117 Epoch: 10 Global Step: 103960 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:20:46,685-Speed 130.99 samples/sec Loss 3.7686 Epoch: 10 Global Step: 103965 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:21:06,212-Speed 131.11 samples/sec Loss 3.9879 Epoch: 10 Global Step: 103970 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:21:25,689-Speed 131.45 samples/sec Loss 3.8366 Epoch: 10 Global Step: 103975 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:21:45,299-Speed 130.56 samples/sec Loss 3.7822 Epoch: 10 Global Step: 103980 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:22:04,735-Speed 131.72 samples/sec Loss 3.7742 Epoch: 10 Global Step: 103985 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:22:24,759-Speed 127.85 samples/sec Loss 3.8038 Epoch: 10 Global Step: 103990 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:22:44,846-Speed 127.45 samples/sec Loss 3.8030 Epoch: 10 Global Step: 103995 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:23:04,326-Speed 131.43 samples/sec Loss 3.7693 Epoch: 10 Global Step: 104000 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:23:23,865-Speed 131.03 samples/sec Loss 3.8570 Epoch: 10 Global Step: 104005 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:23:43,482-Speed 130.51 samples/sec Loss 3.7550 Epoch: 10 Global Step: 104010 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:24:03,014-Speed 131.07 samples/sec Loss 3.9196 Epoch: 10 Global Step: 104015 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:24:22,531-Speed 131.17 samples/sec Loss 3.7140 Epoch: 10 Global Step: 104020 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:24:42,137-Speed 130.58 samples/sec Loss 3.7262 Epoch: 10 Global Step: 104025 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:25:01,695-Speed 130.90 samples/sec Loss 3.8138 Epoch: 10 Global Step: 104030 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:25:21,474-Speed 129.44 samples/sec Loss 3.7221 Epoch: 10 Global Step: 104035 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:25:41,224-Speed 129.62 samples/sec Loss 3.8285 Epoch: 10 Global Step: 104040 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:26:00,850-Speed 130.45 samples/sec Loss 3.8779 Epoch: 10 Global Step: 104045 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:26:20,344-Speed 131.33 samples/sec Loss 3.8072 Epoch: 10 Global Step: 104050 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:26:39,934-Speed 130.69 samples/sec Loss 3.7751 Epoch: 10 Global Step: 104055 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:26:59,257-Speed 132.49 samples/sec Loss 3.6775 Epoch: 10 Global Step: 104060 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:27:18,759-Speed 131.27 samples/sec Loss 3.5810 Epoch: 10 Global Step: 104065 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:27:38,330-Speed 130.82 samples/sec Loss 3.9214 Epoch: 10 Global Step: 104070 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:27:57,855-Speed 131.12 samples/sec Loss 3.7367 Epoch: 10 Global Step: 104075 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:28:17,636-Speed 129.42 samples/sec Loss 3.9372 Epoch: 10 Global Step: 104080 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:28:37,238-Speed 130.61 samples/sec Loss 3.6672 Epoch: 10 Global Step: 104085 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:28:56,793-Speed 130.92 samples/sec Loss 3.9410 Epoch: 10 Global Step: 104090 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:29:16,342-Speed 130.96 samples/sec Loss 3.6731 Epoch: 10 Global Step: 104095 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:29:35,950-Speed 130.56 samples/sec Loss 3.7864 Epoch: 10 Global Step: 104100 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:29:55,839-Speed 128.72 samples/sec Loss 3.6981 Epoch: 10 Global Step: 104105 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:30:15,570-Speed 129.75 samples/sec Loss 3.7977 Epoch: 10 Global Step: 104110 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:30:35,256-Speed 130.05 samples/sec Loss 3.8200 Epoch: 10 Global Step: 104115 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:30:54,812-Speed 130.92 samples/sec Loss 3.7962 Epoch: 10 Global Step: 104120 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:31:14,557-Speed 129.66 samples/sec Loss 3.7774 Epoch: 10 Global Step: 104125 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:31:34,296-Speed 129.70 samples/sec Loss 3.7621 Epoch: 10 Global Step: 104130 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:31:53,866-Speed 130.82 samples/sec Loss 3.7674 Epoch: 10 Global Step: 104135 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:32:13,460-Speed 130.66 samples/sec Loss 3.8562 Epoch: 10 Global Step: 104140 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:32:32,954-Speed 131.33 samples/sec Loss 3.7628 Epoch: 10 Global Step: 104145 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:32:52,571-Speed 130.51 samples/sec Loss 3.9537 Epoch: 10 Global Step: 104150 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:33:12,115-Speed 130.99 samples/sec Loss 3.6966 Epoch: 10 Global Step: 104155 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:33:31,759-Speed 130.33 samples/sec Loss 3.6126 Epoch: 10 Global Step: 104160 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:33:51,563-Speed 129.28 samples/sec Loss 3.8077 Epoch: 10 Global Step: 104165 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:34:11,449-Speed 128.74 samples/sec Loss 3.6391 Epoch: 10 Global Step: 104170 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:34:31,200-Speed 129.62 samples/sec Loss 3.9487 Epoch: 10 Global Step: 104175 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:34:50,740-Speed 131.02 samples/sec Loss 3.8111 Epoch: 10 Global Step: 104180 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:35:10,364-Speed 130.46 samples/sec Loss 3.9098 Epoch: 10 Global Step: 104185 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:35:29,839-Speed 131.46 samples/sec Loss 3.8526 Epoch: 10 Global Step: 104190 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:35:49,590-Speed 129.62 samples/sec Loss 3.6407 Epoch: 10 Global Step: 104195 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:36:09,148-Speed 130.90 samples/sec Loss 3.7099 Epoch: 10 Global Step: 104200 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:36:29,436-Speed 126.19 samples/sec Loss 3.5602 Epoch: 10 Global Step: 104205 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:36:49,121-Speed 130.06 samples/sec Loss 3.5272 Epoch: 10 Global Step: 104210 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:37:09,113-Speed 128.06 samples/sec Loss 3.9452 Epoch: 10 Global Step: 104215 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:37:28,557-Speed 131.66 samples/sec Loss 3.6168 Epoch: 10 Global Step: 104220 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:37:48,150-Speed 130.67 samples/sec Loss 3.7681 Epoch: 10 Global Step: 104225 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:38:07,740-Speed 130.68 samples/sec Loss 3.8557 Epoch: 10 Global Step: 104230 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:38:27,362-Speed 130.48 samples/sec Loss 3.7339 Epoch: 10 Global Step: 104235 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:38:46,987-Speed 130.45 samples/sec Loss 3.8963 Epoch: 10 Global Step: 104240 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:39:06,540-Speed 130.93 samples/sec Loss 3.7778 Epoch: 10 Global Step: 104245 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:39:26,093-Speed 130.94 samples/sec Loss 3.6530 Epoch: 10 Global Step: 104250 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:39:45,662-Speed 130.83 samples/sec Loss 3.5941 Epoch: 10 Global Step: 104255 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:40:05,644-Speed 128.12 samples/sec Loss 3.7178 Epoch: 10 Global Step: 104260 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:40:25,086-Speed 131.68 samples/sec Loss 3.8577 Epoch: 10 Global Step: 104265 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:40:44,687-Speed 130.61 samples/sec Loss 3.7446 Epoch: 10 Global Step: 104270 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:41:04,167-Speed 131.43 samples/sec Loss 3.4985 Epoch: 10 Global Step: 104275 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:41:23,707-Speed 131.02 samples/sec Loss 3.7617 Epoch: 10 Global Step: 104280 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:41:43,319-Speed 130.54 samples/sec Loss 3.5539 Epoch: 10 Global Step: 104285 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:42:02,850-Speed 131.08 samples/sec Loss 3.7005 Epoch: 10 Global Step: 104290 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:42:22,346-Speed 131.32 samples/sec Loss 3.5562 Epoch: 10 Global Step: 104295 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:42:41,891-Speed 130.99 samples/sec Loss 3.6343 Epoch: 10 Global Step: 104300 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:43:01,820-Speed 128.46 samples/sec Loss 3.7035 Epoch: 10 Global Step: 104305 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:43:21,253-Speed 131.74 samples/sec Loss 3.6364 Epoch: 10 Global Step: 104310 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:43:40,880-Speed 130.44 samples/sec Loss 3.7549 Epoch: 10 Global Step: 104315 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:44:00,485-Speed 130.59 samples/sec Loss 3.6512 Epoch: 10 Global Step: 104320 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:44:19,944-Speed 131.57 samples/sec Loss 3.5830 Epoch: 10 Global Step: 104325 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:44:39,620-Speed 130.11 samples/sec Loss 3.5516 Epoch: 10 Global Step: 104330 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:44:59,090-Speed 131.49 samples/sec Loss 3.5918 Epoch: 10 Global Step: 104335 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:45:18,707-Speed 130.50 samples/sec Loss 3.7460 Epoch: 10 Global Step: 104340 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:45:38,334-Speed 130.44 samples/sec Loss 3.7152 Epoch: 10 Global Step: 104345 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:45:58,120-Speed 129.39 samples/sec Loss 3.6649 Epoch: 10 Global Step: 104350 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:46:17,691-Speed 130.82 samples/sec Loss 3.8733 Epoch: 10 Global Step: 104355 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:46:37,291-Speed 130.62 samples/sec Loss 3.6218 Epoch: 10 Global Step: 104360 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:46:56,856-Speed 130.86 samples/sec Loss 3.8195 Epoch: 10 Global Step: 104365 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:47:16,353-Speed 131.31 samples/sec Loss 3.4464 Epoch: 10 Global Step: 104370 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:47:35,877-Speed 131.13 samples/sec Loss 3.7437 Epoch: 10 Global Step: 104375 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:47:55,305-Speed 131.78 samples/sec Loss 3.7211 Epoch: 10 Global Step: 104380 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:48:14,907-Speed 130.61 samples/sec Loss 3.6072 Epoch: 10 Global Step: 104385 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:48:34,582-Speed 130.12 samples/sec Loss 3.7121 Epoch: 10 Global Step: 104390 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:48:54,387-Speed 129.27 samples/sec Loss 3.4979 Epoch: 10 Global Step: 104395 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:49:13,865-Speed 131.43 samples/sec Loss 3.6907 Epoch: 10 Global Step: 104400 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:49:33,489-Speed 130.46 samples/sec Loss 3.7632 Epoch: 10 Global Step: 104405 Fp16 Grad Scale: 8192 Required: 172 hours Training: 2021-06-11 22:49:53,044-Speed 130.91 samples/sec Loss 3.6761 Epoch: 10 Global Step: 104410 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:50:12,944-Speed 128.65 samples/sec Loss 3.6608 Epoch: 10 Global Step: 104415 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:50:32,489-Speed 130.99 samples/sec Loss 3.6768 Epoch: 10 Global Step: 104420 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:50:52,008-Speed 131.16 samples/sec Loss 3.6717 Epoch: 10 Global Step: 104425 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:51:11,601-Speed 130.67 samples/sec Loss 3.6651 Epoch: 10 Global Step: 104430 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:51:31,149-Speed 130.97 samples/sec Loss 3.6254 Epoch: 10 Global Step: 104435 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:51:50,952-Speed 129.29 samples/sec Loss 3.6694 Epoch: 10 Global Step: 104440 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:52:10,458-Speed 131.25 samples/sec Loss 3.8347 Epoch: 10 Global Step: 104445 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:52:30,197-Speed 129.70 samples/sec Loss 3.7891 Epoch: 10 Global Step: 104450 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:52:49,737-Speed 131.02 samples/sec Loss 3.8958 Epoch: 10 Global Step: 104455 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:53:09,495-Speed 129.58 samples/sec Loss 3.7233 Epoch: 10 Global Step: 104460 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:53:28,972-Speed 131.46 samples/sec Loss 3.8104 Epoch: 10 Global Step: 104465 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:53:48,522-Speed 130.96 samples/sec Loss 3.8514 Epoch: 10 Global Step: 104470 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:54:08,087-Speed 130.85 samples/sec Loss 3.6357 Epoch: 10 Global Step: 104475 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:54:27,639-Speed 130.94 samples/sec Loss 3.6843 Epoch: 10 Global Step: 104480 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:54:47,487-Speed 128.99 samples/sec Loss 3.9672 Epoch: 10 Global Step: 104485 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:55:07,098-Speed 130.55 samples/sec Loss 3.5862 Epoch: 10 Global Step: 104490 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:55:26,555-Speed 131.58 samples/sec Loss 3.5913 Epoch: 10 Global Step: 104495 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:55:46,183-Speed 130.43 samples/sec Loss 3.7199 Epoch: 10 Global Step: 104500 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:56:05,752-Speed 130.84 samples/sec Loss 3.9239 Epoch: 10 Global Step: 104505 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:56:25,302-Speed 130.95 samples/sec Loss 3.7085 Epoch: 10 Global Step: 104510 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:56:44,888-Speed 130.72 samples/sec Loss 3.8684 Epoch: 10 Global Step: 104515 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:57:04,438-Speed 130.95 samples/sec Loss 3.8311 Epoch: 10 Global Step: 104520 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:57:24,058-Speed 130.49 samples/sec Loss 3.6638 Epoch: 10 Global Step: 104525 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:57:44,006-Speed 128.34 samples/sec Loss 3.6969 Epoch: 10 Global Step: 104530 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:58:03,448-Speed 131.68 samples/sec Loss 3.7596 Epoch: 10 Global Step: 104535 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:58:23,171-Speed 129.80 samples/sec Loss 3.6472 Epoch: 10 Global Step: 104540 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:58:42,752-Speed 130.75 samples/sec Loss 3.6750 Epoch: 10 Global Step: 104545 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:59:02,253-Speed 131.28 samples/sec Loss 3.7512 Epoch: 10 Global Step: 104550 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:59:21,851-Speed 130.63 samples/sec Loss 3.6723 Epoch: 10 Global Step: 104555 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 22:59:41,459-Speed 130.57 samples/sec Loss 3.4199 Epoch: 10 Global Step: 104560 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:00:01,009-Speed 130.95 samples/sec Loss 3.7094 Epoch: 10 Global Step: 104565 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:00:20,661-Speed 130.28 samples/sec Loss 3.8040 Epoch: 10 Global Step: 104570 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:00:40,474-Speed 129.22 samples/sec Loss 3.6588 Epoch: 10 Global Step: 104575 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:00:59,908-Speed 131.73 samples/sec Loss 3.6868 Epoch: 10 Global Step: 104580 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:01:19,660-Speed 129.61 samples/sec Loss 3.7828 Epoch: 10 Global Step: 104585 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:01:39,194-Speed 131.06 samples/sec Loss 3.8499 Epoch: 10 Global Step: 104590 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:01:58,743-Speed 130.96 samples/sec Loss 3.6627 Epoch: 10 Global Step: 104595 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:02:18,311-Speed 130.83 samples/sec Loss 3.7913 Epoch: 10 Global Step: 104600 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:02:37,858-Speed 130.98 samples/sec Loss 3.5426 Epoch: 10 Global Step: 104605 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:02:57,479-Speed 130.48 samples/sec Loss 3.8203 Epoch: 10 Global Step: 104610 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:03:17,041-Speed 130.87 samples/sec Loss 3.4771 Epoch: 10 Global Step: 104615 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:03:37,016-Speed 128.16 samples/sec Loss 3.6443 Epoch: 10 Global Step: 104620 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:03:56,447-Speed 131.76 samples/sec Loss 3.6005 Epoch: 10 Global Step: 104625 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:04:16,140-Speed 130.00 samples/sec Loss 3.5370 Epoch: 10 Global Step: 104630 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:04:35,708-Speed 130.83 samples/sec Loss 3.5097 Epoch: 10 Global Step: 104635 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:04:55,322-Speed 130.53 samples/sec Loss 3.3755 Epoch: 10 Global Step: 104640 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:05:14,921-Speed 130.63 samples/sec Loss 3.7038 Epoch: 10 Global Step: 104645 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:05:34,480-Speed 130.89 samples/sec Loss 3.6047 Epoch: 10 Global Step: 104650 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:05:54,041-Speed 130.88 samples/sec Loss 3.9318 Epoch: 10 Global Step: 104655 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:06:13,715-Speed 130.13 samples/sec Loss 3.6122 Epoch: 10 Global Step: 104660 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:06:33,519-Speed 129.28 samples/sec Loss 3.7166 Epoch: 10 Global Step: 104665 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:06:53,171-Speed 130.28 samples/sec Loss 3.7543 Epoch: 10 Global Step: 104670 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:07:12,797-Speed 130.44 samples/sec Loss 3.6854 Epoch: 10 Global Step: 104675 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:07:32,414-Speed 130.51 samples/sec Loss 3.4605 Epoch: 10 Global Step: 104680 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:07:52,038-Speed 130.46 samples/sec Loss 3.5766 Epoch: 10 Global Step: 104685 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:08:11,638-Speed 130.62 samples/sec Loss 3.6285 Epoch: 10 Global Step: 104690 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:08:31,188-Speed 130.96 samples/sec Loss 3.6206 Epoch: 10 Global Step: 104695 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:08:50,721-Speed 131.07 samples/sec Loss 3.6147 Epoch: 10 Global Step: 104700 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:09:10,252-Speed 131.08 samples/sec Loss 3.6668 Epoch: 10 Global Step: 104705 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:09:30,045-Speed 129.35 samples/sec Loss 3.6926 Epoch: 10 Global Step: 104710 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:09:49,715-Speed 130.16 samples/sec Loss 3.3490 Epoch: 10 Global Step: 104715 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:10:09,213-Speed 131.30 samples/sec Loss 3.6644 Epoch: 10 Global Step: 104720 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:10:28,750-Speed 131.04 samples/sec Loss 3.2800 Epoch: 10 Global Step: 104725 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:10:48,356-Speed 130.58 samples/sec Loss 3.6463 Epoch: 10 Global Step: 104730 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:11:07,902-Speed 130.98 samples/sec Loss 3.6297 Epoch: 10 Global Step: 104735 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:11:27,528-Speed 130.45 samples/sec Loss 3.5490 Epoch: 10 Global Step: 104740 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:11:47,086-Speed 130.90 samples/sec Loss 3.6753 Epoch: 10 Global Step: 104745 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:12:06,631-Speed 130.99 samples/sec Loss 3.3120 Epoch: 10 Global Step: 104750 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:12:26,464-Speed 129.08 samples/sec Loss 3.5079 Epoch: 10 Global Step: 104755 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:12:46,317-Speed 128.96 samples/sec Loss 3.8649 Epoch: 10 Global Step: 104760 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:13:05,870-Speed 130.93 samples/sec Loss 3.7786 Epoch: 10 Global Step: 104765 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:13:25,406-Speed 131.05 samples/sec Loss 3.5312 Epoch: 10 Global Step: 104770 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:13:45,027-Speed 130.48 samples/sec Loss 3.6389 Epoch: 10 Global Step: 104775 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:14:04,580-Speed 130.94 samples/sec Loss 3.4953 Epoch: 10 Global Step: 104780 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:14:24,181-Speed 130.61 samples/sec Loss 3.6791 Epoch: 10 Global Step: 104785 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:14:43,710-Speed 131.09 samples/sec Loss 3.7185 Epoch: 10 Global Step: 104790 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:15:03,248-Speed 131.03 samples/sec Loss 3.6863 Epoch: 10 Global Step: 104795 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:15:23,038-Speed 129.37 samples/sec Loss 3.5983 Epoch: 10 Global Step: 104800 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:15:42,798-Speed 129.56 samples/sec Loss 3.6369 Epoch: 10 Global Step: 104805 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:16:02,346-Speed 130.97 samples/sec Loss 3.4199 Epoch: 10 Global Step: 104810 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:16:21,929-Speed 130.73 samples/sec Loss 3.4552 Epoch: 10 Global Step: 104815 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:16:42,093-Speed 126.97 samples/sec Loss 3.6998 Epoch: 10 Global Step: 104820 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:17:01,538-Speed 131.66 samples/sec Loss 3.6519 Epoch: 10 Global Step: 104825 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:17:21,142-Speed 130.60 samples/sec Loss 3.5896 Epoch: 10 Global Step: 104830 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:17:40,682-Speed 131.02 samples/sec Loss 3.6745 Epoch: 10 Global Step: 104835 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:18:00,348-Speed 130.18 samples/sec Loss 3.7483 Epoch: 10 Global Step: 104840 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:18:20,103-Speed 129.60 samples/sec Loss 3.9178 Epoch: 10 Global Step: 104845 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:18:39,722-Speed 130.50 samples/sec Loss 3.6629 Epoch: 10 Global Step: 104850 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:18:59,193-Speed 131.48 samples/sec Loss 3.7745 Epoch: 10 Global Step: 104855 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:19:18,817-Speed 130.46 samples/sec Loss 3.5779 Epoch: 10 Global Step: 104860 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:19:38,365-Speed 130.97 samples/sec Loss 3.6397 Epoch: 10 Global Step: 104865 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:19:57,829-Speed 131.53 samples/sec Loss 3.2490 Epoch: 10 Global Step: 104870 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:20:17,532-Speed 129.94 samples/sec Loss 3.6464 Epoch: 10 Global Step: 104875 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:20:37,076-Speed 131.00 samples/sec Loss 3.8202 Epoch: 10 Global Step: 104880 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:20:56,673-Speed 130.64 samples/sec Loss 3.7025 Epoch: 10 Global Step: 104885 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:21:16,399-Speed 129.78 samples/sec Loss 3.5207 Epoch: 10 Global Step: 104890 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:21:36,149-Speed 129.63 samples/sec Loss 3.4508 Epoch: 10 Global Step: 104895 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:21:55,700-Speed 130.95 samples/sec Loss 3.8126 Epoch: 10 Global Step: 104900 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:22:15,302-Speed 130.60 samples/sec Loss 3.5167 Epoch: 10 Global Step: 104905 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:22:34,882-Speed 130.75 samples/sec Loss 3.3714 Epoch: 10 Global Step: 104910 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:22:54,427-Speed 130.99 samples/sec Loss 3.4198 Epoch: 10 Global Step: 104915 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:23:14,012-Speed 130.72 samples/sec Loss 3.5207 Epoch: 10 Global Step: 104920 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:23:33,733-Speed 129.82 samples/sec Loss 3.5449 Epoch: 10 Global Step: 104925 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:23:53,350-Speed 130.51 samples/sec Loss 3.5063 Epoch: 10 Global Step: 104930 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:24:13,084-Speed 129.73 samples/sec Loss 3.4834 Epoch: 10 Global Step: 104935 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:24:32,642-Speed 130.91 samples/sec Loss 3.5710 Epoch: 10 Global Step: 104940 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:24:52,298-Speed 130.25 samples/sec Loss 3.5358 Epoch: 10 Global Step: 104945 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:25:11,926-Speed 130.43 samples/sec Loss 3.4071 Epoch: 10 Global Step: 104950 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:25:31,483-Speed 130.91 samples/sec Loss 3.4611 Epoch: 10 Global Step: 104955 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:25:51,038-Speed 130.92 samples/sec Loss 3.5428 Epoch: 10 Global Step: 104960 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:26:10,578-Speed 131.02 samples/sec Loss 3.5943 Epoch: 10 Global Step: 104965 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:26:30,199-Speed 130.49 samples/sec Loss 3.6504 Epoch: 10 Global Step: 104970 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:26:49,804-Speed 130.58 samples/sec Loss 3.4170 Epoch: 10 Global Step: 104975 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:27:09,699-Speed 128.68 samples/sec Loss 3.6189 Epoch: 10 Global Step: 104980 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:27:29,129-Speed 131.76 samples/sec Loss 3.6086 Epoch: 10 Global Step: 104985 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:27:48,855-Speed 129.78 samples/sec Loss 3.4964 Epoch: 10 Global Step: 104990 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:28:08,463-Speed 130.57 samples/sec Loss 3.7663 Epoch: 10 Global Step: 104995 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:28:27,899-Speed 131.72 samples/sec Loss 3.7386 Epoch: 10 Global Step: 105000 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:28:47,486-Speed 130.71 samples/sec Loss 3.7263 Epoch: 10 Global Step: 105005 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:29:07,116-Speed 130.42 samples/sec Loss 3.6122 Epoch: 10 Global Step: 105010 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:29:26,733-Speed 130.51 samples/sec Loss 3.5740 Epoch: 10 Global Step: 105015 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:29:46,347-Speed 130.53 samples/sec Loss 3.6826 Epoch: 10 Global Step: 105020 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:30:06,161-Speed 129.21 samples/sec Loss 3.4809 Epoch: 10 Global Step: 105025 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:30:25,611-Speed 131.63 samples/sec Loss 3.5622 Epoch: 10 Global Step: 105030 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:30:45,345-Speed 129.73 samples/sec Loss 3.7309 Epoch: 10 Global Step: 105035 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:31:04,933-Speed 130.70 samples/sec Loss 3.3927 Epoch: 10 Global Step: 105040 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:31:24,399-Speed 131.52 samples/sec Loss 3.7426 Epoch: 10 Global Step: 105045 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:31:44,000-Speed 130.61 samples/sec Loss 3.5350 Epoch: 10 Global Step: 105050 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:32:03,601-Speed 130.61 samples/sec Loss 3.5260 Epoch: 10 Global Step: 105055 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:32:23,283-Speed 130.07 samples/sec Loss 3.4261 Epoch: 10 Global Step: 105060 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:32:42,849-Speed 130.85 samples/sec Loss 3.4938 Epoch: 10 Global Step: 105065 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:33:02,750-Speed 128.64 samples/sec Loss 3.6655 Epoch: 10 Global Step: 105070 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:33:22,201-Speed 131.62 samples/sec Loss 3.5035 Epoch: 10 Global Step: 105075 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:33:41,894-Speed 130.00 samples/sec Loss 3.5588 Epoch: 10 Global Step: 105080 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:34:01,641-Speed 129.65 samples/sec Loss 3.5310 Epoch: 10 Global Step: 105085 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:34:21,196-Speed 130.92 samples/sec Loss 3.5088 Epoch: 10 Global Step: 105090 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:34:40,787-Speed 130.68 samples/sec Loss 3.4761 Epoch: 10 Global Step: 105095 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:35:00,389-Speed 130.61 samples/sec Loss 3.4912 Epoch: 10 Global Step: 105100 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:35:20,092-Speed 129.94 samples/sec Loss 3.3882 Epoch: 10 Global Step: 105105 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:35:39,672-Speed 130.75 samples/sec Loss 3.3723 Epoch: 10 Global Step: 105110 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:35:59,566-Speed 128.69 samples/sec Loss 3.4811 Epoch: 10 Global Step: 105115 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:36:19,050-Speed 131.40 samples/sec Loss 3.6914 Epoch: 10 Global Step: 105120 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:36:38,959-Speed 128.59 samples/sec Loss 3.2799 Epoch: 10 Global Step: 105125 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:36:58,382-Speed 131.81 samples/sec Loss 3.5293 Epoch: 10 Global Step: 105130 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:37:17,952-Speed 130.82 samples/sec Loss 3.4882 Epoch: 10 Global Step: 105135 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:37:37,505-Speed 130.93 samples/sec Loss 3.6293 Epoch: 10 Global Step: 105140 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:37:57,079-Speed 130.79 samples/sec Loss 3.7621 Epoch: 10 Global Step: 105145 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:38:16,820-Speed 129.69 samples/sec Loss 3.8539 Epoch: 10 Global Step: 105150 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:38:36,378-Speed 130.90 samples/sec Loss 3.3344 Epoch: 10 Global Step: 105155 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:38:56,207-Speed 129.11 samples/sec Loss 3.5943 Epoch: 10 Global Step: 105160 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:39:15,730-Speed 131.13 samples/sec Loss 3.5304 Epoch: 10 Global Step: 105165 Fp16 Grad Scale: 8192 Required: 171 hours Training: 2021-06-11 23:39:35,491-Speed 129.55 samples/sec Loss 3.4662 Epoch: 10 Global Step: 105170 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:39:55,044-Speed 130.94 samples/sec Loss 3.5953 Epoch: 10 Global Step: 105175 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:40:14,588-Speed 130.99 samples/sec Loss 3.7770 Epoch: 10 Global Step: 105180 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:40:34,209-Speed 130.48 samples/sec Loss 3.4383 Epoch: 10 Global Step: 105185 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:40:54,370-Speed 126.99 samples/sec Loss 3.5134 Epoch: 10 Global Step: 105190 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:41:14,871-Speed 124.88 samples/sec Loss 3.4818 Epoch: 10 Global Step: 105195 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:41:34,443-Speed 130.81 samples/sec Loss 3.5670 Epoch: 10 Global Step: 105200 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:41:54,240-Speed 129.32 samples/sec Loss 3.7026 Epoch: 10 Global Step: 105205 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:42:13,837-Speed 130.64 samples/sec Loss 3.4557 Epoch: 10 Global Step: 105210 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:42:33,534-Speed 129.98 samples/sec Loss 3.3933 Epoch: 10 Global Step: 105215 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:42:53,078-Speed 130.99 samples/sec Loss 3.5528 Epoch: 10 Global Step: 105220 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:43:12,618-Speed 131.02 samples/sec Loss 3.3186 Epoch: 10 Global Step: 105225 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:43:32,168-Speed 130.95 samples/sec Loss 3.4542 Epoch: 10 Global Step: 105230 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:43:51,712-Speed 131.00 samples/sec Loss 3.3919 Epoch: 10 Global Step: 105235 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:44:11,288-Speed 130.78 samples/sec Loss 3.5702 Epoch: 10 Global Step: 105240 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:44:30,968-Speed 130.09 samples/sec Loss 3.2592 Epoch: 10 Global Step: 105245 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:44:50,765-Speed 129.32 samples/sec Loss 3.3822 Epoch: 10 Global Step: 105250 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:45:10,401-Speed 130.38 samples/sec Loss 3.3804 Epoch: 10 Global Step: 105255 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:45:29,940-Speed 131.03 samples/sec Loss 3.4992 Epoch: 10 Global Step: 105260 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:45:49,501-Speed 130.88 samples/sec Loss 3.4249 Epoch: 10 Global Step: 105265 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:46:09,104-Speed 130.60 samples/sec Loss 3.4297 Epoch: 10 Global Step: 105270 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:46:28,639-Speed 131.05 samples/sec Loss 3.3973 Epoch: 10 Global Step: 105275 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:46:48,253-Speed 130.52 samples/sec Loss 3.5609 Epoch: 10 Global Step: 105280 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:47:07,803-Speed 130.96 samples/sec Loss 3.5695 Epoch: 10 Global Step: 105285 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:47:27,304-Speed 131.28 samples/sec Loss 3.0552 Epoch: 10 Global Step: 105290 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:47:47,093-Speed 129.37 samples/sec Loss 3.3896 Epoch: 10 Global Step: 105295 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:48:06,812-Speed 129.83 samples/sec Loss 3.6862 Epoch: 10 Global Step: 105300 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:48:26,487-Speed 130.12 samples/sec Loss 3.6861 Epoch: 10 Global Step: 105305 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:48:45,955-Speed 131.51 samples/sec Loss 3.4726 Epoch: 10 Global Step: 105310 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:49:05,564-Speed 130.56 samples/sec Loss 3.5534 Epoch: 10 Global Step: 105315 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:49:25,467-Speed 128.63 samples/sec Loss 3.3743 Epoch: 10 Global Step: 105320 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:49:45,077-Speed 130.55 samples/sec Loss 3.3176 Epoch: 10 Global Step: 105325 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:50:04,622-Speed 130.98 samples/sec Loss 3.5222 Epoch: 10 Global Step: 105330 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:50:24,122-Speed 131.29 samples/sec Loss 3.4822 Epoch: 10 Global Step: 105335 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:50:43,870-Speed 129.64 samples/sec Loss 3.4000 Epoch: 10 Global Step: 105340 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:51:03,682-Speed 129.22 samples/sec Loss 3.6261 Epoch: 10 Global Step: 105345 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:51:23,236-Speed 130.92 samples/sec Loss 3.5985 Epoch: 10 Global Step: 105350 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:51:42,852-Speed 130.51 samples/sec Loss 3.6226 Epoch: 10 Global Step: 105355 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:52:02,284-Speed 131.75 samples/sec Loss 3.4189 Epoch: 10 Global Step: 105360 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:52:21,955-Speed 130.15 samples/sec Loss 3.4172 Epoch: 10 Global Step: 105365 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:52:41,539-Speed 130.73 samples/sec Loss 3.6695 Epoch: 10 Global Step: 105370 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:53:01,049-Speed 131.22 samples/sec Loss 3.4579 Epoch: 10 Global Step: 105375 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:53:20,664-Speed 130.52 samples/sec Loss 3.5508 Epoch: 10 Global Step: 105380 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:53:40,556-Speed 128.70 samples/sec Loss 3.5645 Epoch: 10 Global Step: 105385 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:54:00,129-Speed 130.80 samples/sec Loss 3.0735 Epoch: 10 Global Step: 105390 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:54:19,595-Speed 131.52 samples/sec Loss 3.5011 Epoch: 10 Global Step: 105395 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:54:39,241-Speed 130.31 samples/sec Loss 3.4224 Epoch: 10 Global Step: 105400 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:54:58,806-Speed 130.86 samples/sec Loss 3.7116 Epoch: 10 Global Step: 105405 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:55:18,564-Speed 129.57 samples/sec Loss 3.7307 Epoch: 10 Global Step: 105410 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:55:38,098-Speed 131.06 samples/sec Loss 3.5040 Epoch: 10 Global Step: 105415 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:55:57,609-Speed 131.22 samples/sec Loss 3.6858 Epoch: 10 Global Step: 105420 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:56:17,180-Speed 130.81 samples/sec Loss 3.4621 Epoch: 10 Global Step: 105425 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:56:37,101-Speed 128.52 samples/sec Loss 3.4825 Epoch: 10 Global Step: 105430 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:56:56,530-Speed 131.77 samples/sec Loss 3.5471 Epoch: 10 Global Step: 105435 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:57:16,250-Speed 129.83 samples/sec Loss 3.4517 Epoch: 10 Global Step: 105440 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:57:35,782-Speed 131.07 samples/sec Loss 3.5545 Epoch: 10 Global Step: 105445 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:57:55,280-Speed 131.30 samples/sec Loss 3.4340 Epoch: 10 Global Step: 105450 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:58:14,833-Speed 130.93 samples/sec Loss 3.3586 Epoch: 10 Global Step: 105455 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:58:34,385-Speed 130.94 samples/sec Loss 3.5531 Epoch: 10 Global Step: 105460 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:58:53,974-Speed 130.69 samples/sec Loss 3.3422 Epoch: 10 Global Step: 105465 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:59:13,530-Speed 130.91 samples/sec Loss 3.5550 Epoch: 10 Global Step: 105470 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:59:33,367-Speed 129.06 samples/sec Loss 3.4022 Epoch: 10 Global Step: 105475 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-11 23:59:52,812-Speed 131.66 samples/sec Loss 3.4797 Epoch: 10 Global Step: 105480 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:00:12,544-Speed 129.75 samples/sec Loss 3.4075 Epoch: 10 Global Step: 105485 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:00:32,082-Speed 131.03 samples/sec Loss 3.4937 Epoch: 10 Global Step: 105490 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:00:51,707-Speed 130.45 samples/sec Loss 3.5355 Epoch: 10 Global Step: 105495 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:01:11,240-Speed 131.07 samples/sec Loss 3.4342 Epoch: 10 Global Step: 105500 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:01:30,823-Speed 130.73 samples/sec Loss 3.5124 Epoch: 10 Global Step: 105505 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:01:50,439-Speed 130.51 samples/sec Loss 3.6158 Epoch: 10 Global Step: 105510 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:02:09,980-Speed 131.01 samples/sec Loss 3.2891 Epoch: 10 Global Step: 105515 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:02:29,778-Speed 129.32 samples/sec Loss 3.3031 Epoch: 10 Global Step: 105520 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:02:49,332-Speed 130.93 samples/sec Loss 3.5899 Epoch: 10 Global Step: 105525 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:03:08,969-Speed 130.37 samples/sec Loss 3.3378 Epoch: 10 Global Step: 105530 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:03:28,515-Speed 130.98 samples/sec Loss 3.4327 Epoch: 10 Global Step: 105535 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:03:48,113-Speed 130.63 samples/sec Loss 3.4146 Epoch: 10 Global Step: 105540 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:04:07,575-Speed 131.55 samples/sec Loss 3.5273 Epoch: 10 Global Step: 105545 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:04:27,183-Speed 130.57 samples/sec Loss 3.4463 Epoch: 10 Global Step: 105550 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:04:46,744-Speed 130.88 samples/sec Loss 3.2553 Epoch: 10 Global Step: 105555 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:05:06,287-Speed 131.00 samples/sec Loss 3.5374 Epoch: 10 Global Step: 105560 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:05:26,071-Speed 129.41 samples/sec Loss 3.6514 Epoch: 10 Global Step: 105565 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:05:45,715-Speed 130.33 samples/sec Loss 3.4799 Epoch: 10 Global Step: 105570 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:06:05,144-Speed 131.77 samples/sec Loss 3.4159 Epoch: 10 Global Step: 105575 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:06:24,669-Speed 131.13 samples/sec Loss 3.5240 Epoch: 10 Global Step: 105580 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:06:44,205-Speed 131.05 samples/sec Loss 3.4079 Epoch: 10 Global Step: 105585 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:07:03,818-Speed 130.53 samples/sec Loss 3.5000 Epoch: 10 Global Step: 105590 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:07:23,424-Speed 130.58 samples/sec Loss 3.4933 Epoch: 10 Global Step: 105595 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:07:42,873-Speed 131.63 samples/sec Loss 3.3800 Epoch: 10 Global Step: 105600 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:08:02,497-Speed 130.46 samples/sec Loss 3.2542 Epoch: 10 Global Step: 105605 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:08:22,381-Speed 128.75 samples/sec Loss 3.5685 Epoch: 10 Global Step: 105610 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:08:42,098-Speed 129.84 samples/sec Loss 3.4361 Epoch: 10 Global Step: 105615 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:09:01,580-Speed 131.41 samples/sec Loss 3.3745 Epoch: 10 Global Step: 105620 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:09:21,191-Speed 130.55 samples/sec Loss 3.3748 Epoch: 10 Global Step: 105625 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:09:40,702-Speed 131.21 samples/sec Loss 3.4386 Epoch: 10 Global Step: 105630 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:10:00,314-Speed 130.54 samples/sec Loss 3.3678 Epoch: 10 Global Step: 105635 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:10:19,762-Speed 131.64 samples/sec Loss 3.3232 Epoch: 10 Global Step: 105640 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:10:39,307-Speed 130.99 samples/sec Loss 3.6185 Epoch: 10 Global Step: 105645 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:10:58,886-Speed 130.76 samples/sec Loss 3.4558 Epoch: 10 Global Step: 105650 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:11:18,829-Speed 128.37 samples/sec Loss 3.2505 Epoch: 10 Global Step: 105655 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:11:38,272-Speed 131.67 samples/sec Loss 3.3864 Epoch: 10 Global Step: 105660 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:11:57,804-Speed 131.08 samples/sec Loss 3.4672 Epoch: 10 Global Step: 105665 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:12:17,350-Speed 130.98 samples/sec Loss 3.2855 Epoch: 10 Global Step: 105670 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:12:36,844-Speed 131.34 samples/sec Loss 3.5523 Epoch: 10 Global Step: 105675 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:12:56,546-Speed 129.94 samples/sec Loss 3.2942 Epoch: 10 Global Step: 105680 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:13:16,010-Speed 131.53 samples/sec Loss 3.6602 Epoch: 10 Global Step: 105685 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:13:35,625-Speed 130.52 samples/sec Loss 3.4570 Epoch: 10 Global Step: 105690 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:13:55,172-Speed 130.97 samples/sec Loss 3.3084 Epoch: 10 Global Step: 105695 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:14:15,110-Speed 128.41 samples/sec Loss 3.5421 Epoch: 10 Global Step: 105700 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:14:34,544-Speed 131.73 samples/sec Loss 3.6084 Epoch: 10 Global Step: 105705 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:14:54,158-Speed 130.53 samples/sec Loss 3.6789 Epoch: 10 Global Step: 105710 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:15:13,682-Speed 131.12 samples/sec Loss 3.3967 Epoch: 10 Global Step: 105715 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:15:33,407-Speed 129.80 samples/sec Loss 3.5846 Epoch: 10 Global Step: 105720 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:15:53,013-Speed 130.58 samples/sec Loss 3.2939 Epoch: 10 Global Step: 105725 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:16:12,452-Speed 131.70 samples/sec Loss 3.4722 Epoch: 10 Global Step: 105730 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:16:32,064-Speed 130.54 samples/sec Loss 3.3297 Epoch: 10 Global Step: 105735 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:16:51,719-Speed 130.25 samples/sec Loss 3.3711 Epoch: 10 Global Step: 105740 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:17:11,522-Speed 129.29 samples/sec Loss 3.4128 Epoch: 10 Global Step: 105745 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:17:31,176-Speed 130.26 samples/sec Loss 3.2918 Epoch: 10 Global Step: 105750 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:17:50,724-Speed 130.97 samples/sec Loss 3.4651 Epoch: 10 Global Step: 105755 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:18:10,325-Speed 130.61 samples/sec Loss 3.6201 Epoch: 10 Global Step: 105760 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:18:29,927-Speed 130.61 samples/sec Loss 3.3949 Epoch: 10 Global Step: 105765 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:18:49,388-Speed 131.55 samples/sec Loss 3.3580 Epoch: 10 Global Step: 105770 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:19:08,968-Speed 130.76 samples/sec Loss 3.5371 Epoch: 10 Global Step: 105775 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:19:28,914-Speed 128.35 samples/sec Loss 3.4995 Epoch: 10 Global Step: 105780 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:19:48,534-Speed 130.49 samples/sec Loss 3.4467 Epoch: 10 Global Step: 105785 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:20:08,327-Speed 129.35 samples/sec Loss 3.2699 Epoch: 10 Global Step: 105790 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:20:27,951-Speed 130.46 samples/sec Loss 3.3066 Epoch: 10 Global Step: 105795 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:20:47,509-Speed 130.90 samples/sec Loss 3.4755 Epoch: 10 Global Step: 105800 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:21:07,120-Speed 130.54 samples/sec Loss 3.3464 Epoch: 10 Global Step: 105805 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:21:26,680-Speed 130.89 samples/sec Loss 3.2676 Epoch: 10 Global Step: 105810 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:21:46,102-Speed 131.82 samples/sec Loss 3.4091 Epoch: 10 Global Step: 105815 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:22:05,654-Speed 130.94 samples/sec Loss 3.2879 Epoch: 10 Global Step: 105820 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:22:25,220-Speed 130.85 samples/sec Loss 3.3129 Epoch: 10 Global Step: 105825 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:22:44,846-Speed 130.45 samples/sec Loss 3.3080 Epoch: 10 Global Step: 105830 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:23:04,588-Speed 129.68 samples/sec Loss 3.2222 Epoch: 10 Global Step: 105835 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:23:24,247-Speed 130.22 samples/sec Loss 3.4511 Epoch: 10 Global Step: 105840 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:23:43,780-Speed 131.07 samples/sec Loss 3.5702 Epoch: 10 Global Step: 105845 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:24:03,350-Speed 130.81 samples/sec Loss 3.2441 Epoch: 10 Global Step: 105850 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:24:22,930-Speed 130.76 samples/sec Loss 3.3657 Epoch: 10 Global Step: 105855 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:24:42,478-Speed 130.97 samples/sec Loss 3.3168 Epoch: 10 Global Step: 105860 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:25:02,080-Speed 130.60 samples/sec Loss 3.4846 Epoch: 10 Global Step: 105865 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:25:21,673-Speed 130.66 samples/sec Loss 3.3745 Epoch: 10 Global Step: 105870 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:25:41,202-Speed 131.10 samples/sec Loss 3.6017 Epoch: 10 Global Step: 105875 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:26:00,951-Speed 129.63 samples/sec Loss 3.5458 Epoch: 10 Global Step: 105880 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:26:20,604-Speed 130.27 samples/sec Loss 3.4771 Epoch: 10 Global Step: 105885 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:26:40,208-Speed 130.59 samples/sec Loss 3.5297 Epoch: 10 Global Step: 105890 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:26:59,818-Speed 130.56 samples/sec Loss 3.4705 Epoch: 10 Global Step: 105895 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:27:19,374-Speed 130.91 samples/sec Loss 3.4805 Epoch: 10 Global Step: 105900 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:27:38,973-Speed 130.62 samples/sec Loss 3.2887 Epoch: 10 Global Step: 105905 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:27:58,540-Speed 130.84 samples/sec Loss 3.5363 Epoch: 10 Global Step: 105910 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:28:18,159-Speed 130.49 samples/sec Loss 3.3290 Epoch: 10 Global Step: 105915 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:28:37,769-Speed 130.55 samples/sec Loss 3.4409 Epoch: 10 Global Step: 105920 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:28:57,655-Speed 128.74 samples/sec Loss 3.5939 Epoch: 10 Global Step: 105925 Fp16 Grad Scale: 8192 Required: 170 hours Training: 2021-06-12 00:29:17,102-Speed 131.64 samples/sec Loss 3.2681 Epoch: 10 Global Step: 105930 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:29:36,781-Speed 130.09 samples/sec Loss 3.3669 Epoch: 10 Global Step: 105935 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:29:56,336-Speed 130.92 samples/sec Loss 3.4630 Epoch: 10 Global Step: 105940 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:30:15,902-Speed 130.85 samples/sec Loss 3.3249 Epoch: 10 Global Step: 105945 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:30:35,509-Speed 130.57 samples/sec Loss 3.3354 Epoch: 10 Global Step: 105950 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:30:55,109-Speed 130.62 samples/sec Loss 3.3195 Epoch: 10 Global Step: 105955 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:31:14,697-Speed 130.70 samples/sec Loss 3.5498 Epoch: 10 Global Step: 105960 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:31:34,252-Speed 130.92 samples/sec Loss 3.3859 Epoch: 10 Global Step: 105965 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:31:54,151-Speed 128.66 samples/sec Loss 3.3881 Epoch: 10 Global Step: 105970 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:32:13,604-Speed 131.61 samples/sec Loss 3.4694 Epoch: 10 Global Step: 105975 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:32:33,418-Speed 129.21 samples/sec Loss 3.4407 Epoch: 10 Global Step: 105980 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:32:52,874-Speed 131.58 samples/sec Loss 3.6454 Epoch: 10 Global Step: 105985 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:33:12,416-Speed 131.01 samples/sec Loss 3.3599 Epoch: 10 Global Step: 105990 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:33:32,015-Speed 130.63 samples/sec Loss 3.4148 Epoch: 10 Global Step: 105995 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:33:51,449-Speed 131.73 samples/sec Loss 3.3573 Epoch: 10 Global Step: 106000 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:34:11,049-Speed 130.62 samples/sec Loss 3.4232 Epoch: 10 Global Step: 106005 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:34:30,483-Speed 131.74 samples/sec Loss 3.5056 Epoch: 10 Global Step: 106010 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:34:50,378-Speed 128.68 samples/sec Loss 3.4288 Epoch: 10 Global Step: 106015 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:35:10,151-Speed 129.47 samples/sec Loss 3.5166 Epoch: 10 Global Step: 106020 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:35:29,820-Speed 130.17 samples/sec Loss 3.3711 Epoch: 10 Global Step: 106025 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:35:49,347-Speed 131.11 samples/sec Loss 3.3741 Epoch: 10 Global Step: 106030 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:36:08,882-Speed 131.06 samples/sec Loss 3.4974 Epoch: 10 Global Step: 106035 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:36:28,490-Speed 130.56 samples/sec Loss 3.4356 Epoch: 10 Global Step: 106040 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:36:47,969-Speed 131.43 samples/sec Loss 3.6510 Epoch: 10 Global Step: 106045 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:37:07,525-Speed 130.92 samples/sec Loss 3.2918 Epoch: 10 Global Step: 106050 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:37:27,078-Speed 130.93 samples/sec Loss 3.1564 Epoch: 10 Global Step: 106055 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:37:46,875-Speed 129.32 samples/sec Loss 3.4242 Epoch: 10 Global Step: 106060 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:38:06,704-Speed 129.11 samples/sec Loss 3.3572 Epoch: 10 Global Step: 106065 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:38:26,144-Speed 131.69 samples/sec Loss 3.3933 Epoch: 10 Global Step: 106070 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:38:45,703-Speed 130.89 samples/sec Loss 3.0833 Epoch: 10 Global Step: 106075 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:39:05,272-Speed 130.82 samples/sec Loss 3.3842 Epoch: 10 Global Step: 106080 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:39:24,778-Speed 131.26 samples/sec Loss 3.3183 Epoch: 10 Global Step: 106085 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:39:44,389-Speed 130.54 samples/sec Loss 3.2500 Epoch: 10 Global Step: 106090 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:40:03,823-Speed 131.74 samples/sec Loss 3.5343 Epoch: 10 Global Step: 106095 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:40:23,363-Speed 131.02 samples/sec Loss 3.3872 Epoch: 10 Global Step: 106100 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:40:43,179-Speed 129.20 samples/sec Loss 3.4340 Epoch: 10 Global Step: 106105 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:41:02,879-Speed 129.96 samples/sec Loss 3.3213 Epoch: 10 Global Step: 106110 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:41:22,437-Speed 130.90 samples/sec Loss 3.4714 Epoch: 10 Global Step: 106115 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:41:41,963-Speed 131.12 samples/sec Loss 3.3855 Epoch: 10 Global Step: 106120 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:42:01,575-Speed 130.54 samples/sec Loss 3.5349 Epoch: 10 Global Step: 106125 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:42:21,110-Speed 131.05 samples/sec Loss 3.4075 Epoch: 10 Global Step: 106130 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:42:40,926-Speed 129.20 samples/sec Loss 3.2647 Epoch: 10 Global Step: 106135 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:43:00,483-Speed 130.91 samples/sec Loss 3.4387 Epoch: 10 Global Step: 106140 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:43:20,210-Speed 129.78 samples/sec Loss 3.6055 Epoch: 10 Global Step: 106145 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:43:39,967-Speed 129.58 samples/sec Loss 3.1869 Epoch: 10 Global Step: 106150 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:43:59,682-Speed 129.86 samples/sec Loss 3.5631 Epoch: 10 Global Step: 106155 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:44:19,241-Speed 130.89 samples/sec Loss 3.3110 Epoch: 10 Global Step: 106160 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:44:38,856-Speed 130.52 samples/sec Loss 3.2195 Epoch: 10 Global Step: 106165 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:44:58,307-Speed 131.62 samples/sec Loss 3.3476 Epoch: 10 Global Step: 106170 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:45:17,929-Speed 130.48 samples/sec Loss 3.5436 Epoch: 10 Global Step: 106175 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:45:37,518-Speed 130.70 samples/sec Loss 3.6433 Epoch: 10 Global Step: 106180 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:45:57,099-Speed 130.74 samples/sec Loss 3.3669 Epoch: 10 Global Step: 106185 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:46:16,689-Speed 130.69 samples/sec Loss 3.1488 Epoch: 10 Global Step: 106190 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:46:36,589-Speed 128.65 samples/sec Loss 3.2381 Epoch: 10 Global Step: 106195 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:46:56,525-Speed 128.42 samples/sec Loss 3.4263 Epoch: 10 Global Step: 106200 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:47:16,247-Speed 129.81 samples/sec Loss 3.4373 Epoch: 10 Global Step: 106205 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:47:35,916-Speed 130.16 samples/sec Loss 3.2790 Epoch: 10 Global Step: 106210 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:47:55,423-Speed 131.24 samples/sec Loss 3.2937 Epoch: 10 Global Step: 106215 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:48:14,981-Speed 130.90 samples/sec Loss 3.4353 Epoch: 10 Global Step: 106220 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:48:34,511-Speed 131.09 samples/sec Loss 3.2020 Epoch: 10 Global Step: 106225 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:48:54,109-Speed 130.64 samples/sec Loss 3.5243 Epoch: 10 Global Step: 106230 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:49:13,712-Speed 130.60 samples/sec Loss 3.3634 Epoch: 10 Global Step: 106235 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:49:33,534-Speed 129.15 samples/sec Loss 3.2315 Epoch: 10 Global Step: 106240 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:49:52,994-Speed 131.56 samples/sec Loss 3.4298 Epoch: 10 Global Step: 106245 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:50:12,710-Speed 129.85 samples/sec Loss 3.5108 Epoch: 10 Global Step: 106250 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:50:32,321-Speed 130.54 samples/sec Loss 3.5760 Epoch: 10 Global Step: 106255 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:50:51,804-Speed 131.40 samples/sec Loss 3.1643 Epoch: 10 Global Step: 106260 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:51:11,365-Speed 130.88 samples/sec Loss 3.3317 Epoch: 10 Global Step: 106265 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:51:30,834-Speed 131.50 samples/sec Loss 3.3435 Epoch: 10 Global Step: 106270 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:51:50,436-Speed 130.60 samples/sec Loss 3.2345 Epoch: 10 Global Step: 106275 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:52:10,003-Speed 130.84 samples/sec Loss 3.4415 Epoch: 10 Global Step: 106280 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:52:29,832-Speed 129.11 samples/sec Loss 3.3640 Epoch: 10 Global Step: 106285 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:52:49,513-Speed 130.08 samples/sec Loss 3.3202 Epoch: 10 Global Step: 106290 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:53:09,157-Speed 130.33 samples/sec Loss 3.4746 Epoch: 10 Global Step: 106295 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:53:28,691-Speed 131.06 samples/sec Loss 3.4967 Epoch: 10 Global Step: 106300 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:53:48,248-Speed 130.91 samples/sec Loss 3.2813 Epoch: 10 Global Step: 106305 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:54:07,802-Speed 130.92 samples/sec Loss 3.3156 Epoch: 10 Global Step: 106310 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:54:27,429-Speed 130.44 samples/sec Loss 3.3617 Epoch: 10 Global Step: 106315 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:54:47,048-Speed 130.50 samples/sec Loss 3.4992 Epoch: 10 Global Step: 106320 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:55:06,471-Speed 131.81 samples/sec Loss 3.2879 Epoch: 10 Global Step: 106325 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:55:26,301-Speed 129.10 samples/sec Loss 3.1791 Epoch: 10 Global Step: 106330 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:55:45,814-Speed 131.20 samples/sec Loss 3.2336 Epoch: 10 Global Step: 106335 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:56:05,527-Speed 129.87 samples/sec Loss 3.2875 Epoch: 10 Global Step: 106340 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:56:24,954-Speed 131.78 samples/sec Loss 3.3114 Epoch: 10 Global Step: 106345 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:56:44,581-Speed 130.44 samples/sec Loss 3.3354 Epoch: 10 Global Step: 106350 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:57:04,110-Speed 131.09 samples/sec Loss 3.3973 Epoch: 10 Global Step: 106355 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:57:23,652-Speed 131.01 samples/sec Loss 3.5176 Epoch: 10 Global Step: 106360 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:57:43,275-Speed 130.47 samples/sec Loss 3.3441 Epoch: 10 Global Step: 106365 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:58:02,711-Speed 131.73 samples/sec Loss 3.2456 Epoch: 10 Global Step: 106370 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:58:22,490-Speed 129.44 samples/sec Loss 3.2994 Epoch: 10 Global Step: 106375 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:58:42,210-Speed 129.83 samples/sec Loss 3.1946 Epoch: 10 Global Step: 106380 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:59:01,829-Speed 130.49 samples/sec Loss 3.2595 Epoch: 10 Global Step: 106385 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:59:21,341-Speed 131.21 samples/sec Loss 3.3191 Epoch: 10 Global Step: 106390 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 00:59:40,959-Speed 130.50 samples/sec Loss 3.4160 Epoch: 10 Global Step: 106395 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:00:00,489-Speed 131.09 samples/sec Loss 3.2388 Epoch: 10 Global Step: 106400 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:00:20,170-Speed 130.08 samples/sec Loss 3.4129 Epoch: 10 Global Step: 106405 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:00:39,744-Speed 130.79 samples/sec Loss 3.2329 Epoch: 10 Global Step: 106410 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:00:59,292-Speed 130.97 samples/sec Loss 3.3298 Epoch: 10 Global Step: 106415 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:01:19,083-Speed 129.36 samples/sec Loss 3.3668 Epoch: 10 Global Step: 106420 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:01:38,821-Speed 129.71 samples/sec Loss 3.4939 Epoch: 10 Global Step: 106425 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:01:58,269-Speed 131.64 samples/sec Loss 3.3822 Epoch: 10 Global Step: 106430 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:02:17,799-Speed 131.09 samples/sec Loss 3.6054 Epoch: 10 Global Step: 106435 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:02:37,362-Speed 130.87 samples/sec Loss 3.1186 Epoch: 10 Global Step: 106440 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:02:56,999-Speed 130.37 samples/sec Loss 3.2393 Epoch: 10 Global Step: 106445 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:03:16,632-Speed 130.40 samples/sec Loss 3.4211 Epoch: 10 Global Step: 106450 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:03:36,146-Speed 131.19 samples/sec Loss 3.1581 Epoch: 10 Global Step: 106455 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:03:55,690-Speed 131.00 samples/sec Loss 3.1630 Epoch: 10 Global Step: 106460 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:04:15,426-Speed 129.72 samples/sec Loss 3.1847 Epoch: 10 Global Step: 106465 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:04:35,962-Speed 124.66 samples/sec Loss 3.3641 Epoch: 10 Global Step: 106470 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:04:55,498-Speed 131.05 samples/sec Loss 3.4183 Epoch: 10 Global Step: 106475 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:05:15,131-Speed 130.40 samples/sec Loss 3.4146 Epoch: 10 Global Step: 106480 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:05:34,676-Speed 130.99 samples/sec Loss 3.3191 Epoch: 10 Global Step: 106485 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:05:54,218-Speed 131.01 samples/sec Loss 3.3552 Epoch: 10 Global Step: 106490 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:06:13,778-Speed 130.88 samples/sec Loss 3.1315 Epoch: 10 Global Step: 106495 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:06:33,393-Speed 130.53 samples/sec Loss 3.2019 Epoch: 10 Global Step: 106500 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:06:52,997-Speed 130.59 samples/sec Loss 3.3859 Epoch: 10 Global Step: 106505 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:07:12,757-Speed 129.56 samples/sec Loss 3.3500 Epoch: 10 Global Step: 106510 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:07:32,332-Speed 130.78 samples/sec Loss 3.3313 Epoch: 10 Global Step: 106515 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:07:51,821-Speed 131.37 samples/sec Loss 3.2728 Epoch: 10 Global Step: 106520 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:08:11,417-Speed 130.65 samples/sec Loss 3.0170 Epoch: 10 Global Step: 106525 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:08:30,959-Speed 131.01 samples/sec Loss 3.3443 Epoch: 10 Global Step: 106530 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:08:50,563-Speed 130.59 samples/sec Loss 3.0679 Epoch: 10 Global Step: 106535 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:09:10,104-Speed 131.01 samples/sec Loss 3.3412 Epoch: 10 Global Step: 106540 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:09:29,648-Speed 130.99 samples/sec Loss 3.1247 Epoch: 10 Global Step: 106545 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:09:49,285-Speed 130.38 samples/sec Loss 3.2486 Epoch: 10 Global Step: 106550 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:10:09,187-Speed 128.64 samples/sec Loss 3.3993 Epoch: 10 Global Step: 106555 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:10:28,624-Speed 131.71 samples/sec Loss 3.2560 Epoch: 10 Global Step: 106560 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:10:48,392-Speed 129.51 samples/sec Loss 3.3040 Epoch: 10 Global Step: 106565 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:11:07,950-Speed 130.90 samples/sec Loss 3.4518 Epoch: 10 Global Step: 106570 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:11:27,417-Speed 131.51 samples/sec Loss 3.4434 Epoch: 10 Global Step: 106575 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:11:47,014-Speed 130.64 samples/sec Loss 3.4963 Epoch: 10 Global Step: 106580 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:12:06,453-Speed 131.70 samples/sec Loss 3.2996 Epoch: 10 Global Step: 106585 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:12:26,064-Speed 130.54 samples/sec Loss 3.3581 Epoch: 10 Global Step: 106590 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:12:45,618-Speed 130.93 samples/sec Loss 3.1797 Epoch: 10 Global Step: 106595 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:13:05,470-Speed 128.96 samples/sec Loss 3.3826 Epoch: 10 Global Step: 106600 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:13:24,909-Speed 131.70 samples/sec Loss 3.3420 Epoch: 10 Global Step: 106605 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:13:44,524-Speed 130.52 samples/sec Loss 3.2155 Epoch: 10 Global Step: 106610 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:14:04,050-Speed 131.11 samples/sec Loss 3.3184 Epoch: 10 Global Step: 106615 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:14:23,720-Speed 130.16 samples/sec Loss 3.4745 Epoch: 10 Global Step: 106620 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:14:43,286-Speed 130.85 samples/sec Loss 3.4465 Epoch: 10 Global Step: 106625 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:15:02,843-Speed 130.91 samples/sec Loss 3.1813 Epoch: 10 Global Step: 106630 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:15:22,451-Speed 130.57 samples/sec Loss 3.2434 Epoch: 10 Global Step: 106635 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:15:41,974-Speed 131.13 samples/sec Loss 3.2984 Epoch: 10 Global Step: 106640 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:16:01,929-Speed 128.30 samples/sec Loss 3.1585 Epoch: 10 Global Step: 106645 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:16:21,347-Speed 131.84 samples/sec Loss 3.5939 Epoch: 10 Global Step: 106650 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:16:41,063-Speed 129.86 samples/sec Loss 3.1891 Epoch: 10 Global Step: 106655 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:17:00,584-Speed 131.14 samples/sec Loss 3.1974 Epoch: 10 Global Step: 106660 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:17:20,434-Speed 128.98 samples/sec Loss 3.3253 Epoch: 10 Global Step: 106665 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:17:40,341-Speed 128.60 samples/sec Loss 3.0511 Epoch: 10 Global Step: 106670 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:17:59,989-Speed 130.30 samples/sec Loss 3.3827 Epoch: 10 Global Step: 106675 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:18:19,582-Speed 130.67 samples/sec Loss 3.2511 Epoch: 10 Global Step: 106680 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:18:39,126-Speed 130.99 samples/sec Loss 3.2894 Epoch: 10 Global Step: 106685 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:18:58,963-Speed 129.06 samples/sec Loss 3.1024 Epoch: 10 Global Step: 106690 Fp16 Grad Scale: 8192 Required: 169 hours Training: 2021-06-12 01:19:18,497-Speed 131.06 samples/sec Loss 3.3570 Epoch: 10 Global Step: 106695 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:19:38,234-Speed 129.71 samples/sec Loss 3.3542 Epoch: 10 Global Step: 106700 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:19:57,965-Speed 129.75 samples/sec Loss 3.4011 Epoch: 10 Global Step: 106705 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:20:17,479-Speed 131.20 samples/sec Loss 3.3066 Epoch: 10 Global Step: 106710 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:20:37,109-Speed 130.42 samples/sec Loss 3.1463 Epoch: 10 Global Step: 106715 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:20:56,613-Speed 131.26 samples/sec Loss 3.2124 Epoch: 10 Global Step: 106720 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:21:16,232-Speed 130.50 samples/sec Loss 3.1705 Epoch: 10 Global Step: 106725 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:21:35,936-Speed 129.93 samples/sec Loss 3.4076 Epoch: 10 Global Step: 106730 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:21:55,735-Speed 129.31 samples/sec Loss 3.2638 Epoch: 10 Global Step: 106735 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:22:15,312-Speed 130.77 samples/sec Loss 3.4656 Epoch: 10 Global Step: 106740 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:22:34,937-Speed 130.46 samples/sec Loss 3.1383 Epoch: 10 Global Step: 106745 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:22:54,549-Speed 130.53 samples/sec Loss 3.1400 Epoch: 10 Global Step: 106750 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:23:14,171-Speed 130.48 samples/sec Loss 3.4354 Epoch: 10 Global Step: 106755 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:23:33,732-Speed 130.88 samples/sec Loss 3.3257 Epoch: 10 Global Step: 106760 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:23:53,295-Speed 130.87 samples/sec Loss 3.3263 Epoch: 10 Global Step: 106765 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:24:12,868-Speed 130.80 samples/sec Loss 3.3633 Epoch: 10 Global Step: 106770 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:24:32,481-Speed 130.53 samples/sec Loss 3.2331 Epoch: 10 Global Step: 106775 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:24:52,274-Speed 129.35 samples/sec Loss 3.3767 Epoch: 10 Global Step: 106780 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:25:12,008-Speed 129.73 samples/sec Loss 3.4005 Epoch: 10 Global Step: 106785 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:25:31,598-Speed 130.68 samples/sec Loss 3.3087 Epoch: 10 Global Step: 106790 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:25:51,149-Speed 130.95 samples/sec Loss 3.6777 Epoch: 10 Global Step: 106795 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:26:10,768-Speed 130.49 samples/sec Loss 3.2917 Epoch: 10 Global Step: 106800 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:26:30,332-Speed 130.86 samples/sec Loss 3.2884 Epoch: 10 Global Step: 106805 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:26:49,887-Speed 130.92 samples/sec Loss 3.1339 Epoch: 10 Global Step: 106810 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:27:09,508-Speed 130.48 samples/sec Loss 3.3035 Epoch: 10 Global Step: 106815 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:27:28,956-Speed 131.64 samples/sec Loss 3.3317 Epoch: 10 Global Step: 106820 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:27:48,766-Speed 129.23 samples/sec Loss 3.0439 Epoch: 10 Global Step: 106825 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:28:08,422-Speed 130.25 samples/sec Loss 3.1624 Epoch: 10 Global Step: 106830 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:28:28,028-Speed 130.57 samples/sec Loss 3.3335 Epoch: 10 Global Step: 106835 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:28:47,502-Speed 131.47 samples/sec Loss 3.2750 Epoch: 10 Global Step: 106840 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:29:07,073-Speed 130.81 samples/sec Loss 3.2280 Epoch: 10 Global Step: 106845 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:29:26,562-Speed 131.36 samples/sec Loss 3.3095 Epoch: 10 Global Step: 106850 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:29:46,182-Speed 130.49 samples/sec Loss 3.1004 Epoch: 10 Global Step: 106855 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:30:05,788-Speed 130.58 samples/sec Loss 3.1756 Epoch: 10 Global Step: 106860 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:30:25,257-Speed 131.50 samples/sec Loss 3.3654 Epoch: 10 Global Step: 106865 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:30:45,022-Speed 129.53 samples/sec Loss 3.4785 Epoch: 10 Global Step: 106870 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:31:04,724-Speed 129.94 samples/sec Loss 3.4040 Epoch: 10 Global Step: 106875 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:31:24,318-Speed 130.66 samples/sec Loss 3.3079 Epoch: 10 Global Step: 106880 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:31:43,795-Speed 131.45 samples/sec Loss 3.2580 Epoch: 10 Global Step: 106885 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:32:03,390-Speed 130.65 samples/sec Loss 3.2951 Epoch: 10 Global Step: 106890 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:32:23,162-Speed 129.49 samples/sec Loss 3.2215 Epoch: 10 Global Step: 106895 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:32:42,782-Speed 130.49 samples/sec Loss 3.5003 Epoch: 10 Global Step: 106900 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:33:02,581-Speed 129.31 samples/sec Loss 3.2784 Epoch: 10 Global Step: 106905 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:33:22,186-Speed 130.59 samples/sec Loss 3.1936 Epoch: 10 Global Step: 106910 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:33:41,982-Speed 129.32 samples/sec Loss 3.2690 Epoch: 10 Global Step: 106915 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:34:01,660-Speed 130.10 samples/sec Loss 3.2795 Epoch: 10 Global Step: 106920 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:34:21,223-Speed 130.86 samples/sec Loss 3.3102 Epoch: 10 Global Step: 106925 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:34:40,837-Speed 130.53 samples/sec Loss 3.3588 Epoch: 10 Global Step: 106930 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:35:00,402-Speed 130.85 samples/sec Loss 3.2016 Epoch: 10 Global Step: 106935 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:35:20,029-Speed 130.45 samples/sec Loss 3.4733 Epoch: 10 Global Step: 106940 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:35:39,570-Speed 131.01 samples/sec Loss 3.2293 Epoch: 10 Global Step: 106945 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:35:59,077-Speed 131.25 samples/sec Loss 3.0662 Epoch: 10 Global Step: 106950 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:36:18,669-Speed 130.67 samples/sec Loss 3.3871 Epoch: 10 Global Step: 106955 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:36:38,601-Speed 128.44 samples/sec Loss 3.0966 Epoch: 10 Global Step: 106960 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:36:58,046-Speed 131.66 samples/sec Loss 3.3159 Epoch: 10 Global Step: 106965 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:37:17,675-Speed 130.43 samples/sec Loss 3.4841 Epoch: 10 Global Step: 106970 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:37:37,248-Speed 130.80 samples/sec Loss 3.3082 Epoch: 10 Global Step: 106975 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:37:56,849-Speed 130.62 samples/sec Loss 3.3613 Epoch: 10 Global Step: 106980 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:38:16,447-Speed 130.63 samples/sec Loss 3.2242 Epoch: 10 Global Step: 106985 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:38:36,069-Speed 130.47 samples/sec Loss 3.2822 Epoch: 10 Global Step: 106990 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:38:55,657-Speed 130.70 samples/sec Loss 3.2390 Epoch: 10 Global Step: 106995 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:39:15,097-Speed 131.69 samples/sec Loss 3.3868 Epoch: 10 Global Step: 107000 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:39:35,017-Speed 128.52 samples/sec Loss 3.2842 Epoch: 10 Global Step: 107005 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:39:54,455-Speed 131.71 samples/sec Loss 3.1704 Epoch: 10 Global Step: 107010 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:40:14,231-Speed 129.46 samples/sec Loss 3.4078 Epoch: 10 Global Step: 107015 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:40:33,779-Speed 130.96 samples/sec Loss 3.2373 Epoch: 10 Global Step: 107020 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:40:53,387-Speed 130.57 samples/sec Loss 3.3053 Epoch: 10 Global Step: 107025 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:41:12,919-Speed 131.07 samples/sec Loss 3.2053 Epoch: 10 Global Step: 107030 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:41:32,539-Speed 130.49 samples/sec Loss 3.3885 Epoch: 10 Global Step: 107035 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:41:52,116-Speed 130.78 samples/sec Loss 3.2794 Epoch: 10 Global Step: 107040 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:42:11,807-Speed 130.01 samples/sec Loss 3.1910 Epoch: 10 Global Step: 107045 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:42:31,608-Speed 129.29 samples/sec Loss 3.2844 Epoch: 10 Global Step: 107050 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:42:51,323-Speed 129.86 samples/sec Loss 3.3768 Epoch: 10 Global Step: 107055 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:43:10,783-Speed 131.56 samples/sec Loss 3.0163 Epoch: 10 Global Step: 107060 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:43:30,345-Speed 130.87 samples/sec Loss 3.3846 Epoch: 10 Global Step: 107065 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:43:49,895-Speed 130.95 samples/sec Loss 3.3319 Epoch: 10 Global Step: 107070 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:44:09,436-Speed 131.01 samples/sec Loss 3.1681 Epoch: 10 Global Step: 107075 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:44:29,033-Speed 130.64 samples/sec Loss 3.3576 Epoch: 10 Global Step: 107080 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:44:48,556-Speed 131.13 samples/sec Loss 3.2224 Epoch: 10 Global Step: 107085 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:45:08,168-Speed 130.54 samples/sec Loss 3.2840 Epoch: 10 Global Step: 107090 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:45:27,863-Speed 129.99 samples/sec Loss 3.2683 Epoch: 10 Global Step: 107095 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:45:47,593-Speed 129.76 samples/sec Loss 3.5443 Epoch: 10 Global Step: 107100 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:46:07,138-Speed 130.99 samples/sec Loss 3.2008 Epoch: 10 Global Step: 107105 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:46:26,758-Speed 130.49 samples/sec Loss 3.0377 Epoch: 10 Global Step: 107110 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:46:46,216-Speed 131.57 samples/sec Loss 3.3384 Epoch: 10 Global Step: 107115 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:47:05,821-Speed 130.59 samples/sec Loss 3.3482 Epoch: 10 Global Step: 107120 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:47:25,389-Speed 130.83 samples/sec Loss 3.4803 Epoch: 10 Global Step: 107125 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:47:44,976-Speed 130.71 samples/sec Loss 3.3355 Epoch: 10 Global Step: 107130 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:48:04,590-Speed 130.53 samples/sec Loss 3.0508 Epoch: 10 Global Step: 107135 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:48:24,344-Speed 129.60 samples/sec Loss 3.2129 Epoch: 10 Global Step: 107140 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:48:44,074-Speed 129.76 samples/sec Loss 3.3390 Epoch: 10 Global Step: 107145 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:49:03,609-Speed 131.05 samples/sec Loss 3.3522 Epoch: 10 Global Step: 107150 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:49:23,220-Speed 130.55 samples/sec Loss 3.0703 Epoch: 10 Global Step: 107155 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:49:42,747-Speed 131.10 samples/sec Loss 3.2379 Epoch: 10 Global Step: 107160 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:50:02,380-Speed 130.40 samples/sec Loss 3.1851 Epoch: 10 Global Step: 107165 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:50:21,925-Speed 130.99 samples/sec Loss 3.2041 Epoch: 10 Global Step: 107170 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:50:41,473-Speed 130.96 samples/sec Loss 3.2736 Epoch: 10 Global Step: 107175 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:51:01,050-Speed 130.77 samples/sec Loss 3.2477 Epoch: 10 Global Step: 107180 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:51:21,020-Speed 128.20 samples/sec Loss 3.0939 Epoch: 10 Global Step: 107185 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:51:40,437-Speed 131.85 samples/sec Loss 3.2062 Epoch: 10 Global Step: 107190 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:52:00,005-Speed 130.83 samples/sec Loss 3.2915 Epoch: 10 Global Step: 107195 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:52:19,581-Speed 130.78 samples/sec Loss 3.2531 Epoch: 10 Global Step: 107200 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:52:39,184-Speed 130.60 samples/sec Loss 3.1883 Epoch: 10 Global Step: 107205 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:52:58,806-Speed 130.47 samples/sec Loss 3.1132 Epoch: 10 Global Step: 107210 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:53:18,427-Speed 130.48 samples/sec Loss 3.2309 Epoch: 10 Global Step: 107215 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:53:37,927-Speed 131.28 samples/sec Loss 3.2954 Epoch: 10 Global Step: 107220 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:53:57,491-Speed 130.87 samples/sec Loss 3.3378 Epoch: 10 Global Step: 107225 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:54:17,469-Speed 128.14 samples/sec Loss 3.4788 Epoch: 10 Global Step: 107230 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:54:36,913-Speed 131.67 samples/sec Loss 3.0854 Epoch: 10 Global Step: 107235 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:54:56,513-Speed 130.62 samples/sec Loss 3.3744 Epoch: 10 Global Step: 107240 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:55:16,091-Speed 130.77 samples/sec Loss 3.2865 Epoch: 10 Global Step: 107245 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:55:35,683-Speed 130.67 samples/sec Loss 3.2461 Epoch: 10 Global Step: 107250 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:55:55,287-Speed 130.59 samples/sec Loss 3.2641 Epoch: 10 Global Step: 107255 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:56:14,843-Speed 130.91 samples/sec Loss 3.5519 Epoch: 10 Global Step: 107260 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:56:34,443-Speed 130.62 samples/sec Loss 3.0534 Epoch: 10 Global Step: 107265 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:56:54,131-Speed 130.03 samples/sec Loss 3.2997 Epoch: 10 Global Step: 107270 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:57:13,996-Speed 128.88 samples/sec Loss 3.3188 Epoch: 10 Global Step: 107275 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:57:33,484-Speed 131.37 samples/sec Loss 3.2435 Epoch: 10 Global Step: 107280 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:57:53,188-Speed 129.93 samples/sec Loss 3.2389 Epoch: 10 Global Step: 107285 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:58:12,822-Speed 130.39 samples/sec Loss 3.3776 Epoch: 10 Global Step: 107290 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:58:32,362-Speed 131.02 samples/sec Loss 3.1487 Epoch: 10 Global Step: 107295 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:58:51,894-Speed 131.08 samples/sec Loss 3.1665 Epoch: 10 Global Step: 107300 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:59:11,472-Speed 130.76 samples/sec Loss 3.3051 Epoch: 10 Global Step: 107305 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:59:31,060-Speed 130.71 samples/sec Loss 3.1084 Epoch: 10 Global Step: 107310 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 01:59:50,677-Speed 130.50 samples/sec Loss 3.2689 Epoch: 10 Global Step: 107315 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:00:10,466-Speed 129.37 samples/sec Loss 3.3554 Epoch: 10 Global Step: 107320 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:00:29,955-Speed 131.36 samples/sec Loss 3.1084 Epoch: 10 Global Step: 107325 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:00:49,563-Speed 130.57 samples/sec Loss 3.2038 Epoch: 10 Global Step: 107330 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:01:09,270-Speed 129.91 samples/sec Loss 3.2028 Epoch: 10 Global Step: 107335 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:01:28,840-Speed 130.82 samples/sec Loss 3.2457 Epoch: 10 Global Step: 107340 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:01:48,275-Speed 131.73 samples/sec Loss 3.2066 Epoch: 10 Global Step: 107345 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:02:07,888-Speed 130.59 samples/sec Loss 3.2010 Epoch: 10 Global Step: 107350 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:02:27,550-Speed 130.20 samples/sec Loss 3.4333 Epoch: 10 Global Step: 107355 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:02:47,160-Speed 130.56 samples/sec Loss 3.2535 Epoch: 10 Global Step: 107360 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:03:06,910-Speed 129.63 samples/sec Loss 3.2922 Epoch: 10 Global Step: 107365 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:03:26,650-Speed 129.69 samples/sec Loss 3.2432 Epoch: 10 Global Step: 107370 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:03:46,180-Speed 131.09 samples/sec Loss 3.2679 Epoch: 10 Global Step: 107375 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:04:05,791-Speed 130.54 samples/sec Loss 3.3679 Epoch: 10 Global Step: 107380 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:04:25,378-Speed 130.70 samples/sec Loss 3.2168 Epoch: 10 Global Step: 107385 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:04:44,932-Speed 130.93 samples/sec Loss 3.2329 Epoch: 10 Global Step: 107390 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:05:04,538-Speed 130.58 samples/sec Loss 3.1532 Epoch: 10 Global Step: 107395 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:05:24,138-Speed 130.62 samples/sec Loss 3.1532 Epoch: 10 Global Step: 107400 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:05:43,868-Speed 129.76 samples/sec Loss 3.2469 Epoch: 10 Global Step: 107405 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:06:03,597-Speed 129.76 samples/sec Loss 3.2437 Epoch: 10 Global Step: 107410 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:06:23,263-Speed 130.18 samples/sec Loss 3.2685 Epoch: 10 Global Step: 107415 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:06:42,878-Speed 130.52 samples/sec Loss 3.2081 Epoch: 10 Global Step: 107420 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:07:02,477-Speed 130.62 samples/sec Loss 3.2864 Epoch: 10 Global Step: 107425 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:07:22,007-Speed 131.09 samples/sec Loss 3.3257 Epoch: 10 Global Step: 107430 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:07:41,481-Speed 131.47 samples/sec Loss 3.0623 Epoch: 10 Global Step: 107435 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:08:01,095-Speed 130.53 samples/sec Loss 2.9100 Epoch: 10 Global Step: 107440 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:08:20,708-Speed 130.54 samples/sec Loss 3.2485 Epoch: 10 Global Step: 107445 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:08:40,304-Speed 130.64 samples/sec Loss 3.0984 Epoch: 10 Global Step: 107450 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:09:00,081-Speed 129.45 samples/sec Loss 3.1645 Epoch: 10 Global Step: 107455 Fp16 Grad Scale: 8192 Required: 168 hours Training: 2021-06-12 02:09:19,713-Speed 130.41 samples/sec Loss 3.0961 Epoch: 10 Global Step: 107460 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:09:39,198-Speed 131.39 samples/sec Loss 3.3094 Epoch: 10 Global Step: 107465 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:09:58,778-Speed 130.76 samples/sec Loss 3.2585 Epoch: 10 Global Step: 107470 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:10:18,349-Speed 130.81 samples/sec Loss 3.1587 Epoch: 10 Global Step: 107475 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:10:37,915-Speed 130.85 samples/sec Loss 3.3821 Epoch: 10 Global Step: 107480 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:10:57,503-Speed 130.71 samples/sec Loss 3.0443 Epoch: 10 Global Step: 107485 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:11:17,073-Speed 130.82 samples/sec Loss 3.2493 Epoch: 10 Global Step: 107490 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:11:36,684-Speed 130.55 samples/sec Loss 3.1011 Epoch: 10 Global Step: 107495 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:11:56,530-Speed 129.00 samples/sec Loss 3.2474 Epoch: 10 Global Step: 107500 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:12:16,097-Speed 130.84 samples/sec Loss 3.3238 Epoch: 10 Global Step: 107505 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:12:35,706-Speed 130.55 samples/sec Loss 3.1980 Epoch: 10 Global Step: 107510 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:12:55,288-Speed 130.74 samples/sec Loss 3.1329 Epoch: 10 Global Step: 107515 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:13:14,824-Speed 131.05 samples/sec Loss 3.2330 Epoch: 10 Global Step: 107520 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:13:34,420-Speed 130.65 samples/sec Loss 3.0451 Epoch: 10 Global Step: 107525 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:13:54,018-Speed 130.63 samples/sec Loss 3.2441 Epoch: 10 Global Step: 107530 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:14:13,541-Speed 131.14 samples/sec Loss 3.1342 Epoch: 10 Global Step: 107535 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:14:33,119-Speed 130.76 samples/sec Loss 3.1078 Epoch: 10 Global Step: 107540 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:14:52,979-Speed 128.91 samples/sec Loss 3.1991 Epoch: 10 Global Step: 107545 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:15:12,768-Speed 129.37 samples/sec Loss 3.2508 Epoch: 10 Global Step: 107550 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:15:32,367-Speed 130.62 samples/sec Loss 3.3214 Epoch: 10 Global Step: 107555 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:15:51,926-Speed 130.89 samples/sec Loss 3.2601 Epoch: 10 Global Step: 107560 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:16:11,486-Speed 130.89 samples/sec Loss 3.2617 Epoch: 10 Global Step: 107565 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:16:31,101-Speed 130.51 samples/sec Loss 3.2637 Epoch: 10 Global Step: 107570 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:16:50,644-Speed 131.00 samples/sec Loss 3.3228 Epoch: 10 Global Step: 107575 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:17:10,188-Speed 130.99 samples/sec Loss 3.3393 Epoch: 10 Global Step: 107580 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:17:29,742-Speed 130.93 samples/sec Loss 3.3728 Epoch: 10 Global Step: 107585 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:17:49,683-Speed 128.39 samples/sec Loss 3.1567 Epoch: 10 Global Step: 107590 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:18:09,147-Speed 131.53 samples/sec Loss 3.2765 Epoch: 10 Global Step: 107595 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:18:28,687-Speed 131.03 samples/sec Loss 3.2217 Epoch: 10 Global Step: 107600 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:18:48,272-Speed 130.72 samples/sec Loss 3.2821 Epoch: 10 Global Step: 107605 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:19:07,816-Speed 130.99 samples/sec Loss 2.9934 Epoch: 10 Global Step: 107610 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:19:27,412-Speed 130.64 samples/sec Loss 3.3999 Epoch: 10 Global Step: 107615 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:19:46,882-Speed 131.49 samples/sec Loss 3.1700 Epoch: 10 Global Step: 107620 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:20:06,489-Speed 130.58 samples/sec Loss 3.4158 Epoch: 10 Global Step: 107625 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:20:26,053-Speed 130.86 samples/sec Loss 3.2196 Epoch: 10 Global Step: 107630 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:20:45,965-Speed 128.57 samples/sec Loss 3.2642 Epoch: 10 Global Step: 107635 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:21:05,473-Speed 131.23 samples/sec Loss 3.2211 Epoch: 10 Global Step: 107640 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:21:25,208-Speed 129.73 samples/sec Loss 3.1804 Epoch: 10 Global Step: 107645 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:21:44,753-Speed 130.99 samples/sec Loss 3.0199 Epoch: 10 Global Step: 107650 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:22:04,294-Speed 131.01 samples/sec Loss 3.0103 Epoch: 10 Global Step: 107655 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:22:23,971-Speed 130.11 samples/sec Loss 3.2888 Epoch: 10 Global Step: 107660 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:22:43,500-Speed 131.10 samples/sec Loss 3.1417 Epoch: 10 Global Step: 107665 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:23:03,078-Speed 130.77 samples/sec Loss 3.2606 Epoch: 10 Global Step: 107670 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:23:22,785-Speed 129.91 samples/sec Loss 3.4805 Epoch: 10 Global Step: 107675 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:23:42,599-Speed 129.21 samples/sec Loss 3.1355 Epoch: 10 Global Step: 107680 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:24:02,142-Speed 131.00 samples/sec Loss 3.0525 Epoch: 10 Global Step: 107685 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:24:21,710-Speed 130.83 samples/sec Loss 3.0867 Epoch: 10 Global Step: 107690 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:24:41,438-Speed 129.78 samples/sec Loss 3.2533 Epoch: 10 Global Step: 107695 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:25:01,385-Speed 128.34 samples/sec Loss 3.1690 Epoch: 10 Global Step: 107700 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:25:20,941-Speed 130.91 samples/sec Loss 3.4278 Epoch: 10 Global Step: 107705 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:25:40,552-Speed 130.55 samples/sec Loss 3.1853 Epoch: 10 Global Step: 107710 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:26:00,101-Speed 130.96 samples/sec Loss 3.2914 Epoch: 10 Global Step: 107715 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:26:19,617-Speed 131.19 samples/sec Loss 3.1164 Epoch: 10 Global Step: 107720 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:26:39,385-Speed 129.51 samples/sec Loss 3.2892 Epoch: 10 Global Step: 107725 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:26:59,084-Speed 129.96 samples/sec Loss 3.2176 Epoch: 10 Global Step: 107730 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:27:18,770-Speed 130.05 samples/sec Loss 3.3278 Epoch: 10 Global Step: 107735 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:27:38,293-Speed 131.13 samples/sec Loss 3.2907 Epoch: 10 Global Step: 107740 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:27:57,867-Speed 130.80 samples/sec Loss 3.3244 Epoch: 10 Global Step: 107745 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:28:17,421-Speed 130.92 samples/sec Loss 3.5474 Epoch: 10 Global Step: 107750 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:28:37,019-Speed 130.63 samples/sec Loss 3.2186 Epoch: 10 Global Step: 107755 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:28:56,541-Speed 131.14 samples/sec Loss 3.1866 Epoch: 10 Global Step: 107760 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:29:16,131-Speed 130.68 samples/sec Loss 3.0483 Epoch: 10 Global Step: 107765 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:29:35,883-Speed 129.61 samples/sec Loss 3.0968 Epoch: 10 Global Step: 107770 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:29:55,620-Speed 129.72 samples/sec Loss 3.0673 Epoch: 10 Global Step: 107775 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:30:15,166-Speed 130.98 samples/sec Loss 3.2696 Epoch: 10 Global Step: 107780 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:30:34,782-Speed 130.51 samples/sec Loss 3.4324 Epoch: 10 Global Step: 107785 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:30:54,310-Speed 131.11 samples/sec Loss 3.2720 Epoch: 10 Global Step: 107790 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:31:13,845-Speed 131.05 samples/sec Loss 3.0022 Epoch: 10 Global Step: 107795 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:31:33,432-Speed 130.71 samples/sec Loss 3.1379 Epoch: 10 Global Step: 107800 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:31:52,967-Speed 131.05 samples/sec Loss 3.2047 Epoch: 10 Global Step: 107805 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:32:12,627-Speed 130.22 samples/sec Loss 3.4775 Epoch: 10 Global Step: 107810 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:32:32,526-Speed 128.66 samples/sec Loss 2.9581 Epoch: 10 Global Step: 107815 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:32:52,082-Speed 130.91 samples/sec Loss 3.2627 Epoch: 10 Global Step: 107820 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:33:11,741-Speed 130.22 samples/sec Loss 3.3933 Epoch: 10 Global Step: 107825 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:33:31,336-Speed 130.66 samples/sec Loss 3.0453 Epoch: 10 Global Step: 107830 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:33:50,870-Speed 131.06 samples/sec Loss 3.2980 Epoch: 10 Global Step: 107835 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:34:10,466-Speed 130.65 samples/sec Loss 3.1964 Epoch: 10 Global Step: 107840 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:34:30,013-Speed 130.98 samples/sec Loss 3.3502 Epoch: 10 Global Step: 107845 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:34:49,613-Speed 130.62 samples/sec Loss 3.1273 Epoch: 10 Global Step: 107850 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:35:09,169-Speed 130.91 samples/sec Loss 3.2300 Epoch: 10 Global Step: 107855 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:35:29,040-Speed 128.84 samples/sec Loss 3.1716 Epoch: 10 Global Step: 107860 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:35:48,459-Speed 131.84 samples/sec Loss 3.1783 Epoch: 10 Global Step: 107865 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:36:08,173-Speed 129.87 samples/sec Loss 3.2992 Epoch: 10 Global Step: 107870 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:36:27,703-Speed 131.09 samples/sec Loss 3.1953 Epoch: 10 Global Step: 107875 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:36:47,207-Speed 131.26 samples/sec Loss 3.1880 Epoch: 10 Global Step: 107880 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:37:06,774-Speed 130.84 samples/sec Loss 3.3627 Epoch: 10 Global Step: 107885 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:37:26,337-Speed 130.86 samples/sec Loss 3.2323 Epoch: 10 Global Step: 107890 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:37:45,950-Speed 130.54 samples/sec Loss 3.2670 Epoch: 10 Global Step: 107895 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:38:05,625-Speed 130.12 samples/sec Loss 3.1859 Epoch: 10 Global Step: 107900 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:38:25,426-Speed 129.29 samples/sec Loss 3.0925 Epoch: 10 Global Step: 107905 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:38:44,882-Speed 131.59 samples/sec Loss 3.2705 Epoch: 10 Global Step: 107910 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:39:04,541-Speed 130.23 samples/sec Loss 3.2799 Epoch: 10 Global Step: 107915 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:39:24,053-Speed 131.21 samples/sec Loss 2.8869 Epoch: 10 Global Step: 107920 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:39:43,746-Speed 130.00 samples/sec Loss 3.1794 Epoch: 10 Global Step: 107925 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:40:03,291-Speed 130.99 samples/sec Loss 3.1731 Epoch: 10 Global Step: 107930 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:40:22,774-Speed 131.41 samples/sec Loss 3.2108 Epoch: 10 Global Step: 107935 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:40:42,437-Speed 130.20 samples/sec Loss 3.2833 Epoch: 10 Global Step: 107940 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:41:01,873-Speed 131.72 samples/sec Loss 3.1702 Epoch: 10 Global Step: 107945 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:41:21,668-Speed 129.34 samples/sec Loss 3.1077 Epoch: 10 Global Step: 107950 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:41:41,409-Speed 129.69 samples/sec Loss 3.3476 Epoch: 10 Global Step: 107955 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:42:01,022-Speed 130.53 samples/sec Loss 3.1465 Epoch: 10 Global Step: 107960 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:42:20,574-Speed 130.94 samples/sec Loss 3.2967 Epoch: 10 Global Step: 107965 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:42:40,182-Speed 130.56 samples/sec Loss 3.2101 Epoch: 10 Global Step: 107970 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:42:59,736-Speed 130.93 samples/sec Loss 3.3466 Epoch: 10 Global Step: 107975 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:43:19,339-Speed 130.60 samples/sec Loss 3.0166 Epoch: 10 Global Step: 107980 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:43:38,910-Speed 130.81 samples/sec Loss 3.3621 Epoch: 10 Global Step: 107985 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:43:58,405-Speed 131.32 samples/sec Loss 3.1486 Epoch: 10 Global Step: 107990 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:44:18,176-Speed 129.49 samples/sec Loss 3.2753 Epoch: 10 Global Step: 107995 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:44:37,836-Speed 130.22 samples/sec Loss 3.3150 Epoch: 10 Global Step: 108000 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:44:57,273-Speed 131.71 samples/sec Loss 3.0926 Epoch: 10 Global Step: 108005 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:45:16,753-Speed 131.43 samples/sec Loss 3.0522 Epoch: 10 Global Step: 108010 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:45:36,333-Speed 130.75 samples/sec Loss 3.2146 Epoch: 10 Global Step: 108015 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:45:55,874-Speed 131.01 samples/sec Loss 3.3522 Epoch: 10 Global Step: 108020 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:46:15,494-Speed 130.49 samples/sec Loss 3.2641 Epoch: 10 Global Step: 108025 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:46:34,933-Speed 131.70 samples/sec Loss 3.1621 Epoch: 10 Global Step: 108030 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:46:54,538-Speed 130.59 samples/sec Loss 3.4385 Epoch: 10 Global Step: 108035 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:47:14,305-Speed 129.51 samples/sec Loss 3.1746 Epoch: 10 Global Step: 108040 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:47:33,954-Speed 130.29 samples/sec Loss 3.2238 Epoch: 10 Global Step: 108045 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:47:53,395-Speed 131.69 samples/sec Loss 3.3343 Epoch: 10 Global Step: 108050 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:48:12,953-Speed 130.91 samples/sec Loss 3.3111 Epoch: 10 Global Step: 108055 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:48:32,522-Speed 130.83 samples/sec Loss 3.2985 Epoch: 10 Global Step: 108060 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:48:52,234-Speed 129.88 samples/sec Loss 3.2254 Epoch: 10 Global Step: 108065 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:49:11,864-Speed 130.42 samples/sec Loss 3.1642 Epoch: 10 Global Step: 108070 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:49:31,362-Speed 131.30 samples/sec Loss 3.0850 Epoch: 10 Global Step: 108075 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:49:50,969-Speed 130.58 samples/sec Loss 3.0179 Epoch: 10 Global Step: 108080 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:50:10,749-Speed 129.43 samples/sec Loss 3.3328 Epoch: 10 Global Step: 108085 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:50:30,380-Speed 130.41 samples/sec Loss 3.3335 Epoch: 10 Global Step: 108090 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:50:49,812-Speed 131.75 samples/sec Loss 3.1384 Epoch: 10 Global Step: 108095 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:51:09,427-Speed 130.52 samples/sec Loss 3.3084 Epoch: 10 Global Step: 108100 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:51:28,905-Speed 131.43 samples/sec Loss 3.3086 Epoch: 10 Global Step: 108105 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:51:48,524-Speed 130.50 samples/sec Loss 3.0704 Epoch: 10 Global Step: 108110 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:52:08,084-Speed 130.88 samples/sec Loss 3.2818 Epoch: 10 Global Step: 108115 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:52:27,630-Speed 130.98 samples/sec Loss 3.2638 Epoch: 10 Global Step: 108120 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:52:47,238-Speed 130.56 samples/sec Loss 3.4181 Epoch: 10 Global Step: 108125 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:53:07,003-Speed 129.53 samples/sec Loss 3.4037 Epoch: 10 Global Step: 108130 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:53:26,825-Speed 129.16 samples/sec Loss 3.0102 Epoch: 10 Global Step: 108135 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:53:46,369-Speed 131.00 samples/sec Loss 3.1278 Epoch: 10 Global Step: 108140 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:54:05,962-Speed 130.66 samples/sec Loss 3.3515 Epoch: 10 Global Step: 108145 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:54:25,525-Speed 130.87 samples/sec Loss 3.2316 Epoch: 10 Global Step: 108150 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:54:45,145-Speed 130.48 samples/sec Loss 3.4262 Epoch: 10 Global Step: 108155 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:55:04,733-Speed 130.70 samples/sec Loss 3.1023 Epoch: 10 Global Step: 108160 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:55:24,291-Speed 130.91 samples/sec Loss 3.1126 Epoch: 10 Global Step: 108165 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:55:43,904-Speed 130.53 samples/sec Loss 3.0956 Epoch: 10 Global Step: 108170 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:56:03,838-Speed 128.43 samples/sec Loss 3.2382 Epoch: 10 Global Step: 108175 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:56:23,270-Speed 131.75 samples/sec Loss 3.1148 Epoch: 10 Global Step: 108180 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:56:42,885-Speed 130.52 samples/sec Loss 3.1961 Epoch: 10 Global Step: 108185 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:57:03,091-Speed 126.70 samples/sec Loss 3.0814 Epoch: 10 Global Step: 108190 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:57:22,644-Speed 130.94 samples/sec Loss 3.0058 Epoch: 10 Global Step: 108195 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:57:42,240-Speed 130.64 samples/sec Loss 2.9318 Epoch: 10 Global Step: 108200 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:58:01,796-Speed 130.92 samples/sec Loss 3.2286 Epoch: 10 Global Step: 108205 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:58:21,419-Speed 130.47 samples/sec Loss 3.1005 Epoch: 10 Global Step: 108210 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:58:40,956-Speed 131.04 samples/sec Loss 3.0352 Epoch: 10 Global Step: 108215 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:59:00,910-Speed 128.30 samples/sec Loss 3.1889 Epoch: 10 Global Step: 108220 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:59:20,351-Speed 131.69 samples/sec Loss 3.1091 Epoch: 10 Global Step: 108225 Fp16 Grad Scale: 8192 Required: 167 hours Training: 2021-06-12 02:59:40,056-Speed 129.92 samples/sec Loss 3.0200 Epoch: 10 Global Step: 108230 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 02:59:59,589-Speed 131.07 samples/sec Loss 3.1268 Epoch: 10 Global Step: 108235 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:00:19,220-Speed 130.42 samples/sec Loss 3.1460 Epoch: 10 Global Step: 108240 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:00:38,794-Speed 130.79 samples/sec Loss 3.4931 Epoch: 10 Global Step: 108245 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:00:58,343-Speed 130.97 samples/sec Loss 3.0093 Epoch: 10 Global Step: 108250 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:01:17,938-Speed 130.65 samples/sec Loss 3.2304 Epoch: 10 Global Step: 108255 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:01:37,513-Speed 130.79 samples/sec Loss 3.3247 Epoch: 10 Global Step: 108260 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:01:57,303-Speed 129.37 samples/sec Loss 3.2401 Epoch: 10 Global Step: 108265 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:02:17,093-Speed 129.37 samples/sec Loss 3.3311 Epoch: 10 Global Step: 108270 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:02:36,739-Speed 130.31 samples/sec Loss 3.2317 Epoch: 10 Global Step: 108275 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:02:56,270-Speed 131.08 samples/sec Loss 3.1466 Epoch: 10 Global Step: 108280 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:03:15,795-Speed 131.12 samples/sec Loss 3.3485 Epoch: 10 Global Step: 108285 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:03:35,350-Speed 130.92 samples/sec Loss 3.2666 Epoch: 10 Global Step: 108290 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:03:54,815-Speed 131.53 samples/sec Loss 3.0921 Epoch: 10 Global Step: 108295 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:04:14,416-Speed 130.62 samples/sec Loss 3.0537 Epoch: 10 Global Step: 108300 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:04:33,962-Speed 130.98 samples/sec Loss 3.1951 Epoch: 10 Global Step: 108305 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:04:53,765-Speed 129.28 samples/sec Loss 3.4103 Epoch: 10 Global Step: 108310 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:05:13,334-Speed 130.83 samples/sec Loss 2.9837 Epoch: 10 Global Step: 108315 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:05:32,903-Speed 130.82 samples/sec Loss 3.0435 Epoch: 10 Global Step: 108320 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:05:52,542-Speed 130.36 samples/sec Loss 3.2339 Epoch: 10 Global Step: 108325 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:06:12,272-Speed 129.76 samples/sec Loss 3.1272 Epoch: 10 Global Step: 108330 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:06:31,915-Speed 130.34 samples/sec Loss 3.0785 Epoch: 10 Global Step: 108335 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:06:51,368-Speed 131.60 samples/sec Loss 3.1463 Epoch: 10 Global Step: 108340 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:07:11,078-Speed 129.89 samples/sec Loss 3.1879 Epoch: 10 Global Step: 108345 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:07:30,839-Speed 129.55 samples/sec Loss 3.1180 Epoch: 10 Global Step: 108350 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:07:50,667-Speed 129.12 samples/sec Loss 3.2292 Epoch: 10 Global Step: 108355 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:08:10,196-Speed 131.09 samples/sec Loss 3.4166 Epoch: 10 Global Step: 108360 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:08:29,861-Speed 130.19 samples/sec Loss 3.1788 Epoch: 10 Global Step: 108365 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:08:49,396-Speed 131.05 samples/sec Loss 3.1357 Epoch: 10 Global Step: 108370 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:09:09,022-Speed 130.45 samples/sec Loss 3.1388 Epoch: 10 Global Step: 108375 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:09:28,510-Speed 131.37 samples/sec Loss 3.1151 Epoch: 10 Global Step: 108380 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:09:48,042-Speed 131.07 samples/sec Loss 3.1838 Epoch: 10 Global Step: 108385 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:10:07,660-Speed 130.50 samples/sec Loss 3.1879 Epoch: 10 Global Step: 108390 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:10:27,384-Speed 129.80 samples/sec Loss 3.2001 Epoch: 10 Global Step: 108395 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:10:47,172-Speed 129.38 samples/sec Loss 3.1283 Epoch: 10 Global Step: 108400 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:11:06,668-Speed 131.31 samples/sec Loss 2.9416 Epoch: 10 Global Step: 108405 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:11:26,370-Speed 129.95 samples/sec Loss 3.2817 Epoch: 10 Global Step: 108410 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:11:45,911-Speed 131.02 samples/sec Loss 3.1292 Epoch: 10 Global Step: 108415 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:12:05,530-Speed 130.49 samples/sec Loss 3.3434 Epoch: 10 Global Step: 108420 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:12:24,942-Speed 131.88 samples/sec Loss 2.9715 Epoch: 10 Global Step: 108425 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:12:44,503-Speed 130.88 samples/sec Loss 3.0471 Epoch: 10 Global Step: 108430 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:13:04,087-Speed 130.73 samples/sec Loss 3.2835 Epoch: 10 Global Step: 108435 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:13:23,544-Speed 131.58 samples/sec Loss 3.2526 Epoch: 10 Global Step: 108440 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:13:43,325-Speed 129.43 samples/sec Loss 3.1485 Epoch: 10 Global Step: 108445 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:14:02,894-Speed 130.83 samples/sec Loss 3.0645 Epoch: 10 Global Step: 108450 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:14:22,472-Speed 130.76 samples/sec Loss 3.1535 Epoch: 10 Global Step: 108455 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:14:42,015-Speed 131.00 samples/sec Loss 3.1182 Epoch: 10 Global Step: 108460 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:15:01,584-Speed 130.83 samples/sec Loss 3.0773 Epoch: 10 Global Step: 108465 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:15:21,116-Speed 131.07 samples/sec Loss 3.1543 Epoch: 10 Global Step: 108470 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:15:40,733-Speed 130.51 samples/sec Loss 3.0247 Epoch: 10 Global Step: 108475 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:16:00,237-Speed 131.26 samples/sec Loss 3.1162 Epoch: 10 Global Step: 108480 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:16:19,710-Speed 131.47 samples/sec Loss 3.1835 Epoch: 10 Global Step: 108485 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:16:39,594-Speed 128.75 samples/sec Loss 3.2266 Epoch: 10 Global Step: 108490 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:16:59,448-Speed 128.95 samples/sec Loss 3.0946 Epoch: 10 Global Step: 108495 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:17:18,896-Speed 131.64 samples/sec Loss 3.1643 Epoch: 10 Global Step: 108500 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:17:38,522-Speed 130.44 samples/sec Loss 3.1876 Epoch: 10 Global Step: 108505 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:17:58,086-Speed 130.86 samples/sec Loss 3.1379 Epoch: 10 Global Step: 108510 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:18:17,628-Speed 131.01 samples/sec Loss 3.1707 Epoch: 10 Global Step: 108515 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:18:37,238-Speed 130.55 samples/sec Loss 3.1776 Epoch: 10 Global Step: 108520 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:18:56,856-Speed 130.50 samples/sec Loss 3.1754 Epoch: 10 Global Step: 108525 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:19:16,461-Speed 130.59 samples/sec Loss 3.0709 Epoch: 10 Global Step: 108530 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:19:36,245-Speed 129.41 samples/sec Loss 3.2193 Epoch: 10 Global Step: 108535 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:19:55,889-Speed 130.32 samples/sec Loss 3.3556 Epoch: 10 Global Step: 108540 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:20:15,428-Speed 131.03 samples/sec Loss 3.0781 Epoch: 10 Global Step: 108545 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:20:35,006-Speed 130.77 samples/sec Loss 2.9047 Epoch: 10 Global Step: 108550 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:20:54,555-Speed 130.96 samples/sec Loss 3.2991 Epoch: 10 Global Step: 108555 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:21:14,169-Speed 130.53 samples/sec Loss 3.0027 Epoch: 10 Global Step: 108560 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:21:33,646-Speed 131.44 samples/sec Loss 3.0695 Epoch: 10 Global Step: 108565 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:21:53,275-Speed 130.43 samples/sec Loss 3.2632 Epoch: 10 Global Step: 108570 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:22:12,829-Speed 130.93 samples/sec Loss 3.0379 Epoch: 10 Global Step: 108575 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:22:32,725-Speed 128.68 samples/sec Loss 3.0894 Epoch: 10 Global Step: 108580 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:22:52,232-Speed 131.24 samples/sec Loss 3.2748 Epoch: 10 Global Step: 108585 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:23:11,809-Speed 130.77 samples/sec Loss 2.9967 Epoch: 10 Global Step: 108590 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:23:31,390-Speed 130.75 samples/sec Loss 3.2403 Epoch: 10 Global Step: 108595 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:23:50,910-Speed 131.15 samples/sec Loss 3.2706 Epoch: 10 Global Step: 108600 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:24:10,528-Speed 130.51 samples/sec Loss 3.1881 Epoch: 10 Global Step: 108605 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:24:30,016-Speed 131.37 samples/sec Loss 3.0410 Epoch: 10 Global Step: 108610 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:24:49,626-Speed 130.56 samples/sec Loss 2.9631 Epoch: 10 Global Step: 108615 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:25:09,156-Speed 131.09 samples/sec Loss 3.2767 Epoch: 10 Global Step: 108620 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:25:29,091-Speed 128.43 samples/sec Loss 3.2364 Epoch: 10 Global Step: 108625 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:25:48,529-Speed 131.70 samples/sec Loss 3.0939 Epoch: 10 Global Step: 108630 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:26:08,172-Speed 130.34 samples/sec Loss 2.9868 Epoch: 10 Global Step: 108635 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:26:27,707-Speed 131.05 samples/sec Loss 3.0056 Epoch: 10 Global Step: 108640 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:26:47,340-Speed 130.40 samples/sec Loss 3.3000 Epoch: 10 Global Step: 108645 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:27:06,956-Speed 130.51 samples/sec Loss 3.3348 Epoch: 10 Global Step: 108650 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:27:26,497-Speed 131.01 samples/sec Loss 3.1845 Epoch: 10 Global Step: 108655 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:27:46,101-Speed 130.59 samples/sec Loss 3.4446 Epoch: 10 Global Step: 108660 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:28:05,644-Speed 131.00 samples/sec Loss 3.1192 Epoch: 10 Global Step: 108665 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:28:25,646-Speed 127.99 samples/sec Loss 3.1805 Epoch: 10 Global Step: 108670 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:28:45,094-Speed 131.64 samples/sec Loss 3.0468 Epoch: 10 Global Step: 108675 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:29:04,756-Speed 130.21 samples/sec Loss 2.9202 Epoch: 10 Global Step: 108680 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:29:24,263-Speed 131.24 samples/sec Loss 3.1666 Epoch: 10 Global Step: 108685 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:29:43,801-Speed 131.04 samples/sec Loss 3.1950 Epoch: 10 Global Step: 108690 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:30:03,409-Speed 130.57 samples/sec Loss 3.1211 Epoch: 10 Global Step: 108695 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:30:22,907-Speed 131.30 samples/sec Loss 2.9187 Epoch: 10 Global Step: 108700 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:30:42,515-Speed 130.57 samples/sec Loss 3.0704 Epoch: 10 Global Step: 108705 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:31:02,055-Speed 131.02 samples/sec Loss 3.2182 Epoch: 10 Global Step: 108710 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:31:21,835-Speed 129.43 samples/sec Loss 3.0672 Epoch: 10 Global Step: 108715 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:31:41,326-Speed 131.35 samples/sec Loss 3.2597 Epoch: 10 Global Step: 108720 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:32:01,029-Speed 129.94 samples/sec Loss 3.1907 Epoch: 10 Global Step: 108725 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:32:20,633-Speed 130.60 samples/sec Loss 3.0321 Epoch: 10 Global Step: 108730 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:32:40,175-Speed 131.01 samples/sec Loss 3.3806 Epoch: 10 Global Step: 108735 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:32:59,751-Speed 130.78 samples/sec Loss 3.1110 Epoch: 10 Global Step: 108740 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:33:19,411-Speed 130.22 samples/sec Loss 3.1313 Epoch: 10 Global Step: 108745 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:33:39,006-Speed 130.65 samples/sec Loss 3.3688 Epoch: 10 Global Step: 108750 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:33:58,732-Speed 129.78 samples/sec Loss 3.0901 Epoch: 10 Global Step: 108755 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:34:18,613-Speed 128.78 samples/sec Loss 3.2085 Epoch: 10 Global Step: 108760 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:34:38,038-Speed 131.79 samples/sec Loss 3.0997 Epoch: 10 Global Step: 108765 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:34:57,786-Speed 129.64 samples/sec Loss 3.1562 Epoch: 10 Global Step: 108770 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:35:17,320-Speed 131.06 samples/sec Loss 3.0558 Epoch: 10 Global Step: 108775 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:35:36,875-Speed 130.93 samples/sec Loss 3.1258 Epoch: 10 Global Step: 108780 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:35:56,448-Speed 130.80 samples/sec Loss 3.1600 Epoch: 10 Global Step: 108785 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:36:16,981-Speed 124.68 samples/sec Loss 3.4042 Epoch: 10 Global Step: 108790 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:36:36,609-Speed 130.44 samples/sec Loss 3.2586 Epoch: 10 Global Step: 108795 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:36:56,087-Speed 131.44 samples/sec Loss 2.9510 Epoch: 10 Global Step: 108800 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:37:15,890-Speed 129.28 samples/sec Loss 3.2292 Epoch: 10 Global Step: 108805 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:37:35,473-Speed 130.73 samples/sec Loss 3.0713 Epoch: 10 Global Step: 108810 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:37:55,140-Speed 130.18 samples/sec Loss 3.1810 Epoch: 10 Global Step: 108815 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:38:14,684-Speed 130.99 samples/sec Loss 3.2575 Epoch: 10 Global Step: 108820 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:38:34,488-Speed 129.27 samples/sec Loss 3.2582 Epoch: 10 Global Step: 108825 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:38:54,018-Speed 131.09 samples/sec Loss 2.9476 Epoch: 10 Global Step: 108830 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:39:13,572-Speed 130.93 samples/sec Loss 3.0988 Epoch: 10 Global Step: 108835 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:39:33,175-Speed 130.60 samples/sec Loss 2.9824 Epoch: 10 Global Step: 108840 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:39:52,733-Speed 130.90 samples/sec Loss 3.0920 Epoch: 10 Global Step: 108845 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:40:12,547-Speed 129.21 samples/sec Loss 2.8856 Epoch: 10 Global Step: 108850 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:40:32,209-Speed 130.20 samples/sec Loss 3.2202 Epoch: 10 Global Step: 108855 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:40:51,784-Speed 130.79 samples/sec Loss 2.9770 Epoch: 10 Global Step: 108860 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:41:11,333-Speed 130.96 samples/sec Loss 2.9999 Epoch: 10 Global Step: 108865 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:41:30,935-Speed 130.61 samples/sec Loss 3.2022 Epoch: 10 Global Step: 108870 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:41:50,467-Speed 131.07 samples/sec Loss 3.0603 Epoch: 10 Global Step: 108875 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:42:10,066-Speed 130.63 samples/sec Loss 3.2051 Epoch: 10 Global Step: 108880 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:42:29,639-Speed 130.80 samples/sec Loss 3.1805 Epoch: 10 Global Step: 108885 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:42:49,213-Speed 130.79 samples/sec Loss 3.0350 Epoch: 10 Global Step: 108890 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:43:08,971-Speed 129.58 samples/sec Loss 3.0574 Epoch: 10 Global Step: 108895 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:43:28,817-Speed 129.00 samples/sec Loss 3.1831 Epoch: 10 Global Step: 108900 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:43:48,366-Speed 130.96 samples/sec Loss 3.0091 Epoch: 10 Global Step: 108905 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:44:07,998-Speed 130.41 samples/sec Loss 3.0922 Epoch: 10 Global Step: 108910 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:44:27,536-Speed 131.03 samples/sec Loss 3.1086 Epoch: 10 Global Step: 108915 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:44:47,089-Speed 130.94 samples/sec Loss 3.1818 Epoch: 10 Global Step: 108920 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:45:06,694-Speed 130.59 samples/sec Loss 3.1298 Epoch: 10 Global Step: 108925 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:45:26,248-Speed 130.93 samples/sec Loss 3.1511 Epoch: 10 Global Step: 108930 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:45:45,857-Speed 130.56 samples/sec Loss 3.0627 Epoch: 10 Global Step: 108935 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:46:05,632-Speed 129.46 samples/sec Loss 3.1800 Epoch: 10 Global Step: 108940 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:46:25,199-Speed 130.84 samples/sec Loss 3.3396 Epoch: 10 Global Step: 108945 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:46:44,655-Speed 131.59 samples/sec Loss 3.2829 Epoch: 10 Global Step: 108950 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:47:04,281-Speed 130.44 samples/sec Loss 3.0615 Epoch: 10 Global Step: 108955 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:47:23,924-Speed 130.34 samples/sec Loss 3.1650 Epoch: 10 Global Step: 108960 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:47:43,564-Speed 130.35 samples/sec Loss 3.1947 Epoch: 10 Global Step: 108965 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:48:03,099-Speed 131.05 samples/sec Loss 3.1152 Epoch: 10 Global Step: 108970 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:48:22,642-Speed 131.00 samples/sec Loss 3.0310 Epoch: 10 Global Step: 108975 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:48:42,269-Speed 130.44 samples/sec Loss 3.2089 Epoch: 10 Global Step: 108980 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:49:02,190-Speed 128.52 samples/sec Loss 3.1178 Epoch: 10 Global Step: 108985 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:49:21,627-Speed 131.71 samples/sec Loss 3.1600 Epoch: 10 Global Step: 108990 Fp16 Grad Scale: 8192 Required: 166 hours Training: 2021-06-12 03:49:41,289-Speed 130.21 samples/sec Loss 3.1911 Epoch: 10 Global Step: 108995 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:50:00,877-Speed 130.70 samples/sec Loss 3.1650 Epoch: 10 Global Step: 109000 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:50:20,423-Speed 130.98 samples/sec Loss 3.0353 Epoch: 10 Global Step: 109005 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:50:40,047-Speed 130.46 samples/sec Loss 3.2372 Epoch: 10 Global Step: 109010 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:50:59,609-Speed 130.87 samples/sec Loss 3.3765 Epoch: 10 Global Step: 109015 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:51:19,233-Speed 130.46 samples/sec Loss 3.1676 Epoch: 10 Global Step: 109020 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:51:38,680-Speed 131.65 samples/sec Loss 3.2066 Epoch: 10 Global Step: 109025 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:51:58,555-Speed 128.81 samples/sec Loss 3.1068 Epoch: 10 Global Step: 109030 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:52:18,029-Speed 131.47 samples/sec Loss 3.2764 Epoch: 10 Global Step: 109035 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:52:37,747-Speed 129.84 samples/sec Loss 3.1129 Epoch: 10 Global Step: 109040 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:52:57,179-Speed 131.75 samples/sec Loss 3.0761 Epoch: 10 Global Step: 109045 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:53:16,843-Speed 130.19 samples/sec Loss 3.0286 Epoch: 10 Global Step: 109050 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:53:36,445-Speed 130.61 samples/sec Loss 3.1228 Epoch: 10 Global Step: 109055 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:53:55,898-Speed 131.61 samples/sec Loss 3.2039 Epoch: 10 Global Step: 109060 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:54:15,443-Speed 130.98 samples/sec Loss 3.2365 Epoch: 10 Global Step: 109065 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:54:35,002-Speed 130.89 samples/sec Loss 3.2983 Epoch: 10 Global Step: 109070 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:54:54,957-Speed 128.30 samples/sec Loss 3.0653 Epoch: 10 Global Step: 109075 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:55:14,405-Speed 131.63 samples/sec Loss 3.1641 Epoch: 10 Global Step: 109080 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:55:34,012-Speed 130.58 samples/sec Loss 2.9586 Epoch: 10 Global Step: 109085 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:55:53,548-Speed 131.04 samples/sec Loss 3.1835 Epoch: 10 Global Step: 109090 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:56:13,149-Speed 130.62 samples/sec Loss 3.0671 Epoch: 10 Global Step: 109095 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:56:32,747-Speed 130.63 samples/sec Loss 3.1646 Epoch: 10 Global Step: 109100 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:56:52,190-Speed 131.67 samples/sec Loss 3.0690 Epoch: 10 Global Step: 109105 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:57:11,742-Speed 130.94 samples/sec Loss 3.1430 Epoch: 10 Global Step: 109110 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:57:31,414-Speed 130.14 samples/sec Loss 2.9855 Epoch: 10 Global Step: 109115 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:57:51,370-Speed 128.29 samples/sec Loss 3.3944 Epoch: 10 Global Step: 109120 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:58:10,813-Speed 131.67 samples/sec Loss 3.1300 Epoch: 10 Global Step: 109125 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:58:30,607-Speed 129.34 samples/sec Loss 3.2363 Epoch: 10 Global Step: 109130 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:58:50,155-Speed 130.97 samples/sec Loss 2.9392 Epoch: 10 Global Step: 109135 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:59:09,743-Speed 130.70 samples/sec Loss 3.0829 Epoch: 10 Global Step: 109140 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:59:29,304-Speed 130.88 samples/sec Loss 3.1551 Epoch: 10 Global Step: 109145 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 03:59:48,870-Speed 130.85 samples/sec Loss 3.0953 Epoch: 10 Global Step: 109150 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:00:08,471-Speed 130.61 samples/sec Loss 2.9834 Epoch: 10 Global Step: 109155 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:00:28,010-Speed 131.03 samples/sec Loss 3.1230 Epoch: 10 Global Step: 109160 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:00:47,805-Speed 129.33 samples/sec Loss 3.1765 Epoch: 10 Global Step: 109165 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:01:07,275-Speed 131.49 samples/sec Loss 3.1541 Epoch: 10 Global Step: 109170 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:01:26,928-Speed 130.27 samples/sec Loss 2.9870 Epoch: 10 Global Step: 109175 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:01:46,535-Speed 130.57 samples/sec Loss 3.0911 Epoch: 10 Global Step: 109180 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:02:06,152-Speed 130.51 samples/sec Loss 3.0613 Epoch: 10 Global Step: 109185 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:02:25,587-Speed 131.72 samples/sec Loss 3.1824 Epoch: 10 Global Step: 109190 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:02:45,183-Speed 130.65 samples/sec Loss 3.1565 Epoch: 10 Global Step: 109195 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:03:04,739-Speed 130.91 samples/sec Loss 3.2879 Epoch: 10 Global Step: 109200 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:03:24,348-Speed 130.56 samples/sec Loss 3.0615 Epoch: 10 Global Step: 109205 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:03:44,147-Speed 129.31 samples/sec Loss 3.1647 Epoch: 10 Global Step: 109210 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:04:03,913-Speed 129.52 samples/sec Loss 3.0119 Epoch: 10 Global Step: 109215 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:04:23,566-Speed 130.26 samples/sec Loss 3.1879 Epoch: 10 Global Step: 109220 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:04:43,177-Speed 130.55 samples/sec Loss 3.2872 Epoch: 10 Global Step: 109225 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:05:02,819-Speed 130.34 samples/sec Loss 3.0975 Epoch: 10 Global Step: 109230 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:05:22,254-Speed 131.73 samples/sec Loss 3.0377 Epoch: 10 Global Step: 109235 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:05:41,798-Speed 131.00 samples/sec Loss 3.1322 Epoch: 10 Global Step: 109240 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:06:01,371-Speed 130.80 samples/sec Loss 3.0636 Epoch: 10 Global Step: 109245 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:06:20,996-Speed 130.45 samples/sec Loss 3.1734 Epoch: 10 Global Step: 109250 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:06:40,747-Speed 129.62 samples/sec Loss 3.1433 Epoch: 10 Global Step: 109255 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:07:00,412-Speed 130.18 samples/sec Loss 3.1709 Epoch: 10 Global Step: 109260 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:07:19,951-Speed 131.03 samples/sec Loss 3.3029 Epoch: 10 Global Step: 109265 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:07:39,578-Speed 130.44 samples/sec Loss 3.2572 Epoch: 10 Global Step: 109270 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:07:59,020-Speed 131.68 samples/sec Loss 3.0779 Epoch: 10 Global Step: 109275 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:08:18,577-Speed 130.90 samples/sec Loss 3.0929 Epoch: 10 Global Step: 109280 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:08:38,161-Speed 130.73 samples/sec Loss 3.1297 Epoch: 10 Global Step: 109285 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:08:57,759-Speed 130.64 samples/sec Loss 3.2025 Epoch: 10 Global Step: 109290 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:09:17,358-Speed 130.63 samples/sec Loss 3.1677 Epoch: 10 Global Step: 109295 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:09:37,053-Speed 129.99 samples/sec Loss 3.0068 Epoch: 10 Global Step: 109300 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:09:56,706-Speed 130.27 samples/sec Loss 3.2087 Epoch: 10 Global Step: 109305 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:10:16,438-Speed 129.75 samples/sec Loss 3.0843 Epoch: 10 Global Step: 109310 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:10:36,054-Speed 130.51 samples/sec Loss 3.0848 Epoch: 10 Global Step: 109315 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:10:55,482-Speed 131.77 samples/sec Loss 3.2135 Epoch: 10 Global Step: 109320 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:11:15,099-Speed 130.51 samples/sec Loss 3.0711 Epoch: 10 Global Step: 109325 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:11:34,652-Speed 130.94 samples/sec Loss 3.2158 Epoch: 10 Global Step: 109330 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:11:54,261-Speed 130.56 samples/sec Loss 3.3361 Epoch: 10 Global Step: 109335 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:12:13,843-Speed 130.74 samples/sec Loss 3.2782 Epoch: 10 Global Step: 109340 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:12:33,736-Speed 128.70 samples/sec Loss 3.0208 Epoch: 10 Global Step: 109345 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:12:53,247-Speed 131.21 samples/sec Loss 3.1167 Epoch: 10 Global Step: 109350 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:13:12,873-Speed 130.45 samples/sec Loss 3.1090 Epoch: 10 Global Step: 109355 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:13:32,486-Speed 130.54 samples/sec Loss 2.9896 Epoch: 10 Global Step: 109360 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:13:51,916-Speed 131.76 samples/sec Loss 3.2007 Epoch: 10 Global Step: 109365 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:14:11,610-Speed 130.00 samples/sec Loss 3.1768 Epoch: 10 Global Step: 109370 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:14:31,224-Speed 130.52 samples/sec Loss 3.1207 Epoch: 10 Global Step: 109375 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:14:50,751-Speed 131.11 samples/sec Loss 3.2426 Epoch: 10 Global Step: 109380 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:15:10,286-Speed 131.05 samples/sec Loss 3.0761 Epoch: 10 Global Step: 109385 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:15:30,170-Speed 128.76 samples/sec Loss 2.9317 Epoch: 10 Global Step: 109390 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:15:49,614-Speed 131.66 samples/sec Loss 3.0961 Epoch: 10 Global Step: 109395 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:16:09,420-Speed 129.26 samples/sec Loss 3.0263 Epoch: 10 Global Step: 109400 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:16:28,899-Speed 131.43 samples/sec Loss 3.0923 Epoch: 10 Global Step: 109405 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:16:48,455-Speed 130.92 samples/sec Loss 3.2158 Epoch: 10 Global Step: 109410 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:17:08,045-Speed 130.69 samples/sec Loss 2.9222 Epoch: 10 Global Step: 109415 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:17:27,649-Speed 130.60 samples/sec Loss 2.9065 Epoch: 10 Global Step: 109420 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:17:47,225-Speed 130.78 samples/sec Loss 2.9654 Epoch: 10 Global Step: 109425 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:18:06,685-Speed 131.56 samples/sec Loss 2.9007 Epoch: 10 Global Step: 109430 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:18:26,487-Speed 129.29 samples/sec Loss 3.0399 Epoch: 10 Global Step: 109435 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:18:46,217-Speed 129.76 samples/sec Loss 3.0143 Epoch: 10 Global Step: 109440 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:19:05,661-Speed 131.67 samples/sec Loss 3.1588 Epoch: 10 Global Step: 109445 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:19:25,315-Speed 130.26 samples/sec Loss 3.1588 Epoch: 10 Global Step: 109450 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:19:44,891-Speed 130.78 samples/sec Loss 3.1234 Epoch: 10 Global Step: 109455 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:20:04,507-Speed 130.51 samples/sec Loss 3.2277 Epoch: 10 Global Step: 109460 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:20:24,119-Speed 130.54 samples/sec Loss 3.1097 Epoch: 10 Global Step: 109465 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:20:43,549-Speed 131.76 samples/sec Loss 3.0764 Epoch: 10 Global Step: 109470 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:21:03,100-Speed 130.95 samples/sec Loss 3.0300 Epoch: 10 Global Step: 109475 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:21:22,866-Speed 129.52 samples/sec Loss 3.0285 Epoch: 10 Global Step: 109480 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:21:42,648-Speed 129.42 samples/sec Loss 3.2687 Epoch: 10 Global Step: 109485 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:22:02,092-Speed 131.66 samples/sec Loss 3.1442 Epoch: 10 Global Step: 109490 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:22:21,712-Speed 130.49 samples/sec Loss 3.1239 Epoch: 10 Global Step: 109495 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:22:41,332-Speed 130.48 samples/sec Loss 2.9940 Epoch: 10 Global Step: 109500 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:23:00,942-Speed 130.56 samples/sec Loss 3.1132 Epoch: 10 Global Step: 109505 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:23:20,471-Speed 131.09 samples/sec Loss 3.1873 Epoch: 10 Global Step: 109510 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:23:40,030-Speed 130.89 samples/sec Loss 3.1837 Epoch: 10 Global Step: 109515 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:23:59,618-Speed 130.70 samples/sec Loss 3.4001 Epoch: 10 Global Step: 109520 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:24:19,417-Speed 129.31 samples/sec Loss 3.1943 Epoch: 10 Global Step: 109525 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:24:39,060-Speed 130.33 samples/sec Loss 3.2701 Epoch: 10 Global Step: 109530 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:24:58,629-Speed 130.82 samples/sec Loss 3.0666 Epoch: 10 Global Step: 109535 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:25:18,250-Speed 130.48 samples/sec Loss 2.8210 Epoch: 10 Global Step: 109540 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:25:37,859-Speed 130.56 samples/sec Loss 3.0581 Epoch: 10 Global Step: 109545 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:25:57,462-Speed 130.60 samples/sec Loss 3.0028 Epoch: 10 Global Step: 109550 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:26:17,112-Speed 130.29 samples/sec Loss 3.0928 Epoch: 10 Global Step: 109555 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:26:36,668-Speed 130.91 samples/sec Loss 3.3285 Epoch: 10 Global Step: 109560 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:26:56,236-Speed 130.84 samples/sec Loss 3.4407 Epoch: 10 Global Step: 109565 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:27:16,208-Speed 128.18 samples/sec Loss 2.9687 Epoch: 10 Global Step: 109570 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:27:35,639-Speed 131.75 samples/sec Loss 3.2309 Epoch: 10 Global Step: 109575 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:27:55,344-Speed 129.92 samples/sec Loss 3.2090 Epoch: 10 Global Step: 109580 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:28:14,892-Speed 130.97 samples/sec Loss 3.1464 Epoch: 10 Global Step: 109585 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:28:34,533-Speed 130.34 samples/sec Loss 3.1246 Epoch: 10 Global Step: 109590 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:28:54,093-Speed 130.89 samples/sec Loss 3.1601 Epoch: 10 Global Step: 109595 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:29:13,583-Speed 131.35 samples/sec Loss 2.9382 Epoch: 10 Global Step: 109600 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:30:04,486-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-12 04:30:15,510-Speed 41.34 samples/sec Loss 2.8770 Epoch: 11 Global Step: 109605 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:30:35,525-Speed 127.92 samples/sec Loss 2.3845 Epoch: 11 Global Step: 109610 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:30:55,606-Speed 127.49 samples/sec Loss 2.3907 Epoch: 11 Global Step: 109615 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:31:15,432-Speed 129.13 samples/sec Loss 2.4035 Epoch: 11 Global Step: 109620 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:31:35,981-Speed 124.58 samples/sec Loss 2.2919 Epoch: 11 Global Step: 109625 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:31:55,799-Speed 129.19 samples/sec Loss 2.4929 Epoch: 11 Global Step: 109630 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:32:15,809-Speed 127.93 samples/sec Loss 2.4771 Epoch: 11 Global Step: 109635 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:32:35,735-Speed 128.48 samples/sec Loss 2.5005 Epoch: 11 Global Step: 109640 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:32:55,723-Speed 128.09 samples/sec Loss 2.2734 Epoch: 11 Global Step: 109645 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:33:15,602-Speed 128.78 samples/sec Loss 2.4697 Epoch: 11 Global Step: 109650 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:33:35,403-Speed 129.29 samples/sec Loss 2.3767 Epoch: 11 Global Step: 109655 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:33:55,457-Speed 127.67 samples/sec Loss 2.5404 Epoch: 11 Global Step: 109660 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:34:15,581-Speed 127.22 samples/sec Loss 2.2682 Epoch: 11 Global Step: 109665 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:34:35,827-Speed 126.45 samples/sec Loss 2.5316 Epoch: 11 Global Step: 109670 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:34:55,873-Speed 127.71 samples/sec Loss 2.4373 Epoch: 11 Global Step: 109675 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:35:15,720-Speed 128.99 samples/sec Loss 2.4143 Epoch: 11 Global Step: 109680 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:35:35,642-Speed 128.51 samples/sec Loss 2.4141 Epoch: 11 Global Step: 109685 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:35:55,652-Speed 127.94 samples/sec Loss 2.3055 Epoch: 11 Global Step: 109690 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:36:15,421-Speed 129.50 samples/sec Loss 2.4520 Epoch: 11 Global Step: 109695 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:36:35,825-Speed 125.47 samples/sec Loss 2.5022 Epoch: 11 Global Step: 109700 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:36:55,762-Speed 128.41 samples/sec Loss 2.4913 Epoch: 11 Global Step: 109705 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:37:15,667-Speed 128.61 samples/sec Loss 2.5671 Epoch: 11 Global Step: 109710 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:37:36,167-Speed 124.88 samples/sec Loss 2.4561 Epoch: 11 Global Step: 109715 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:37:56,249-Speed 127.49 samples/sec Loss 2.4076 Epoch: 11 Global Step: 109720 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:38:16,149-Speed 128.65 samples/sec Loss 2.4744 Epoch: 11 Global Step: 109725 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:38:36,199-Speed 127.69 samples/sec Loss 2.3258 Epoch: 11 Global Step: 109730 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:38:55,951-Speed 129.62 samples/sec Loss 2.4629 Epoch: 11 Global Step: 109735 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:39:15,875-Speed 128.50 samples/sec Loss 2.5227 Epoch: 11 Global Step: 109740 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:39:35,908-Speed 127.79 samples/sec Loss 2.4433 Epoch: 11 Global Step: 109745 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:39:55,805-Speed 128.67 samples/sec Loss 2.4846 Epoch: 11 Global Step: 109750 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:40:15,852-Speed 127.71 samples/sec Loss 2.4189 Epoch: 11 Global Step: 109755 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:40:36,301-Speed 125.20 samples/sec Loss 2.4728 Epoch: 11 Global Step: 109760 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:40:56,193-Speed 128.70 samples/sec Loss 2.4313 Epoch: 11 Global Step: 109765 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:41:16,034-Speed 129.03 samples/sec Loss 2.6010 Epoch: 11 Global Step: 109770 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:41:36,046-Speed 127.93 samples/sec Loss 2.3951 Epoch: 11 Global Step: 109775 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:41:55,835-Speed 129.38 samples/sec Loss 2.3618 Epoch: 11 Global Step: 109780 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 04:42:16,056-Speed 126.60 samples/sec Loss 2.3627 Epoch: 11 Global Step: 109785 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:42:35,955-Speed 128.66 samples/sec Loss 2.4614 Epoch: 11 Global Step: 109790 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:42:55,959-Speed 127.98 samples/sec Loss 2.4500 Epoch: 11 Global Step: 109795 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:43:15,792-Speed 129.09 samples/sec Loss 2.2917 Epoch: 11 Global Step: 109800 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:43:36,377-Speed 124.36 samples/sec Loss 2.4172 Epoch: 11 Global Step: 109805 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:43:56,340-Speed 128.24 samples/sec Loss 2.5043 Epoch: 11 Global Step: 109810 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:44:16,274-Speed 128.43 samples/sec Loss 2.3582 Epoch: 11 Global Step: 109815 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:44:36,261-Speed 128.09 samples/sec Loss 2.4293 Epoch: 11 Global Step: 109820 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:44:56,336-Speed 127.53 samples/sec Loss 2.5215 Epoch: 11 Global Step: 109825 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:45:16,380-Speed 127.73 samples/sec Loss 2.5879 Epoch: 11 Global Step: 109830 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:45:36,280-Speed 128.65 samples/sec Loss 2.5490 Epoch: 11 Global Step: 109835 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:45:56,546-Speed 126.33 samples/sec Loss 2.2703 Epoch: 11 Global Step: 109840 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:46:16,351-Speed 129.27 samples/sec Loss 2.5697 Epoch: 11 Global Step: 109845 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:46:36,887-Speed 124.67 samples/sec Loss 2.3468 Epoch: 11 Global Step: 109850 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:46:56,921-Speed 127.78 samples/sec Loss 2.5236 Epoch: 11 Global Step: 109855 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:47:16,649-Speed 129.77 samples/sec Loss 2.6604 Epoch: 11 Global Step: 109860 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:47:36,573-Speed 128.49 samples/sec Loss 2.3148 Epoch: 11 Global Step: 109865 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:47:56,590-Speed 127.90 samples/sec Loss 2.4249 Epoch: 11 Global Step: 109870 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:48:16,523-Speed 128.44 samples/sec Loss 2.4918 Epoch: 11 Global Step: 109875 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:48:36,470-Speed 128.35 samples/sec Loss 2.3336 Epoch: 11 Global Step: 109880 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:48:56,228-Speed 129.57 samples/sec Loss 2.3162 Epoch: 11 Global Step: 109885 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:49:16,231-Speed 127.99 samples/sec Loss 2.3940 Epoch: 11 Global Step: 109890 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:49:36,790-Speed 124.53 samples/sec Loss 2.4570 Epoch: 11 Global Step: 109895 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:49:56,740-Speed 128.33 samples/sec Loss 2.4320 Epoch: 11 Global Step: 109900 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:50:16,509-Speed 129.50 samples/sec Loss 2.3737 Epoch: 11 Global Step: 109905 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:50:36,584-Speed 127.52 samples/sec Loss 2.4189 Epoch: 11 Global Step: 109910 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:50:56,529-Speed 128.37 samples/sec Loss 2.4946 Epoch: 11 Global Step: 109915 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:51:16,480-Speed 128.32 samples/sec Loss 2.5173 Epoch: 11 Global Step: 109920 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:51:36,283-Speed 129.28 samples/sec Loss 2.2888 Epoch: 11 Global Step: 109925 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:51:56,232-Speed 128.33 samples/sec Loss 2.4353 Epoch: 11 Global Step: 109930 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:52:16,251-Speed 127.88 samples/sec Loss 2.4106 Epoch: 11 Global Step: 109935 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:52:36,770-Speed 124.77 samples/sec Loss 2.5702 Epoch: 11 Global Step: 109940 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:52:56,632-Speed 128.90 samples/sec Loss 2.5550 Epoch: 11 Global Step: 109945 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:53:16,532-Speed 128.65 samples/sec Loss 2.5521 Epoch: 11 Global Step: 109950 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:53:36,564-Speed 127.80 samples/sec Loss 2.4437 Epoch: 11 Global Step: 109955 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:53:56,379-Speed 129.21 samples/sec Loss 2.4066 Epoch: 11 Global Step: 109960 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:54:16,298-Speed 128.53 samples/sec Loss 2.4401 Epoch: 11 Global Step: 109965 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:54:36,169-Speed 128.83 samples/sec Loss 2.5507 Epoch: 11 Global Step: 109970 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:54:56,211-Speed 127.74 samples/sec Loss 2.4230 Epoch: 11 Global Step: 109975 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:55:16,250-Speed 127.76 samples/sec Loss 2.4717 Epoch: 11 Global Step: 109980 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:55:36,667-Speed 125.39 samples/sec Loss 2.4810 Epoch: 11 Global Step: 109985 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:55:56,601-Speed 128.43 samples/sec Loss 2.2895 Epoch: 11 Global Step: 109990 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:56:16,581-Speed 128.13 samples/sec Loss 2.4769 Epoch: 11 Global Step: 109995 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 04:56:36,536-Speed 128.30 samples/sec Loss 2.3031 Epoch: 11 Global Step: 110000 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 05:11:23,379-[lfw][110000]XNorm: 22.489162 Training: 2021-06-12 05:11:23,379-[lfw][110000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-06-12 05:11:23,379-[lfw][110000]Accuracy-Highest: 0.99833 Training: 2021-06-12 05:28:40,053-[cfp_fp][110000]XNorm: 22.305307 Training: 2021-06-12 05:28:40,053-[cfp_fp][110000]Accuracy-Flip: 0.99043+-0.00279 Training: 2021-06-12 05:28:40,053-[cfp_fp][110000]Accuracy-Highest: 0.99043 Training: 2021-06-12 05:43:25,854-[agedb_30][110000]XNorm: 23.225446 Training: 2021-06-12 05:43:25,854-[agedb_30][110000]Accuracy-Flip: 0.98200+-0.00600 Training: 2021-06-12 05:43:25,855-[agedb_30][110000]Accuracy-Highest: 0.98200 Training: 2021-06-12 05:43:46,045-Speed 0.90 samples/sec Loss 2.5738 Epoch: 11 Global Step: 110005 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:44:05,815-Speed 129.50 samples/sec Loss 2.5985 Epoch: 11 Global Step: 110010 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:44:25,552-Speed 129.72 samples/sec Loss 2.3209 Epoch: 11 Global Step: 110015 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:44:45,630-Speed 127.51 samples/sec Loss 2.3989 Epoch: 11 Global Step: 110020 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:45:06,104-Speed 125.04 samples/sec Loss 2.5367 Epoch: 11 Global Step: 110025 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:45:25,882-Speed 129.44 samples/sec Loss 2.4595 Epoch: 11 Global Step: 110030 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:45:45,624-Speed 129.68 samples/sec Loss 2.3605 Epoch: 11 Global Step: 110035 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:46:05,352-Speed 129.77 samples/sec Loss 2.5328 Epoch: 11 Global Step: 110040 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:46:25,246-Speed 128.69 samples/sec Loss 2.3624 Epoch: 11 Global Step: 110045 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:46:44,997-Speed 129.63 samples/sec Loss 2.3400 Epoch: 11 Global Step: 110050 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:47:04,747-Speed 129.62 samples/sec Loss 2.6136 Epoch: 11 Global Step: 110055 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:47:24,495-Speed 129.64 samples/sec Loss 2.5443 Epoch: 11 Global Step: 110060 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:47:44,454-Speed 128.27 samples/sec Loss 2.5459 Epoch: 11 Global Step: 110065 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:48:04,244-Speed 129.36 samples/sec Loss 2.5446 Epoch: 11 Global Step: 110070 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:48:24,144-Speed 128.65 samples/sec Loss 2.3790 Epoch: 11 Global Step: 110075 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:48:43,894-Speed 129.63 samples/sec Loss 2.3971 Epoch: 11 Global Step: 110080 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:49:03,647-Speed 129.61 samples/sec Loss 2.5483 Epoch: 11 Global Step: 110085 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:49:23,366-Speed 129.83 samples/sec Loss 2.4444 Epoch: 11 Global Step: 110090 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:49:43,115-Speed 129.63 samples/sec Loss 2.5557 Epoch: 11 Global Step: 110095 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:50:02,870-Speed 129.60 samples/sec Loss 2.4259 Epoch: 11 Global Step: 110100 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:50:22,629-Speed 129.57 samples/sec Loss 2.4855 Epoch: 11 Global Step: 110105 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:50:42,581-Speed 128.31 samples/sec Loss 2.5136 Epoch: 11 Global Step: 110110 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:51:02,661-Speed 127.50 samples/sec Loss 2.5807 Epoch: 11 Global Step: 110115 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:51:22,422-Speed 129.55 samples/sec Loss 2.5027 Epoch: 11 Global Step: 110120 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:51:42,229-Speed 129.26 samples/sec Loss 2.5274 Epoch: 11 Global Step: 110125 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:52:01,977-Speed 129.64 samples/sec Loss 2.5474 Epoch: 11 Global Step: 110130 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:52:21,730-Speed 129.61 samples/sec Loss 2.7861 Epoch: 11 Global Step: 110135 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:52:41,489-Speed 129.57 samples/sec Loss 2.5240 Epoch: 11 Global Step: 110140 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:53:01,234-Speed 129.66 samples/sec Loss 2.4588 Epoch: 11 Global Step: 110145 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:53:21,011-Speed 129.45 samples/sec Loss 2.6527 Epoch: 11 Global Step: 110150 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:53:41,173-Speed 126.98 samples/sec Loss 2.4880 Epoch: 11 Global Step: 110155 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:54:00,771-Speed 130.63 samples/sec Loss 2.5411 Epoch: 11 Global Step: 110160 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:54:20,743-Speed 128.19 samples/sec Loss 2.5344 Epoch: 11 Global Step: 110165 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:54:40,519-Speed 129.46 samples/sec Loss 2.3451 Epoch: 11 Global Step: 110170 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:55:00,294-Speed 129.46 samples/sec Loss 2.3936 Epoch: 11 Global Step: 110175 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:55:20,269-Speed 128.17 samples/sec Loss 2.5245 Epoch: 11 Global Step: 110180 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:55:40,045-Speed 129.46 samples/sec Loss 2.2974 Epoch: 11 Global Step: 110185 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:55:59,782-Speed 129.71 samples/sec Loss 2.3350 Epoch: 11 Global Step: 110190 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:56:19,536-Speed 129.61 samples/sec Loss 2.5071 Epoch: 11 Global Step: 110195 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:56:39,661-Speed 127.21 samples/sec Loss 2.4733 Epoch: 11 Global Step: 110200 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:56:59,435-Speed 129.47 samples/sec Loss 2.4369 Epoch: 11 Global Step: 110205 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:57:19,179-Speed 129.67 samples/sec Loss 2.4937 Epoch: 11 Global Step: 110210 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:57:38,952-Speed 129.48 samples/sec Loss 2.6150 Epoch: 11 Global Step: 110215 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:57:58,709-Speed 129.58 samples/sec Loss 2.4534 Epoch: 11 Global Step: 110220 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:58:18,464-Speed 129.60 samples/sec Loss 2.5039 Epoch: 11 Global Step: 110225 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:58:38,517-Speed 127.67 samples/sec Loss 2.5423 Epoch: 11 Global Step: 110230 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:58:58,285-Speed 129.51 samples/sec Loss 2.4312 Epoch: 11 Global Step: 110235 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:59:18,156-Speed 128.84 samples/sec Loss 2.4534 Epoch: 11 Global Step: 110240 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:59:38,297-Speed 127.12 samples/sec Loss 2.3478 Epoch: 11 Global Step: 110245 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 05:59:57,922-Speed 130.45 samples/sec Loss 2.6273 Epoch: 11 Global Step: 110250 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:00:17,812-Speed 128.71 samples/sec Loss 2.6532 Epoch: 11 Global Step: 110255 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:00:37,605-Speed 129.35 samples/sec Loss 2.4963 Epoch: 11 Global Step: 110260 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:00:57,373-Speed 129.51 samples/sec Loss 2.5300 Epoch: 11 Global Step: 110265 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:01:17,138-Speed 129.53 samples/sec Loss 2.5274 Epoch: 11 Global Step: 110270 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:01:36,902-Speed 129.54 samples/sec Loss 2.3610 Epoch: 11 Global Step: 110275 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:01:56,635-Speed 129.74 samples/sec Loss 2.5595 Epoch: 11 Global Step: 110280 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:02:16,404-Speed 129.50 samples/sec Loss 2.5654 Epoch: 11 Global Step: 110285 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:02:36,674-Speed 126.30 samples/sec Loss 2.7562 Epoch: 11 Global Step: 110290 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:02:56,389-Speed 129.86 samples/sec Loss 2.4565 Epoch: 11 Global Step: 110295 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:03:16,164-Speed 129.46 samples/sec Loss 2.5523 Epoch: 11 Global Step: 110300 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:03:35,919-Speed 129.60 samples/sec Loss 2.5982 Epoch: 11 Global Step: 110305 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:03:55,818-Speed 128.66 samples/sec Loss 2.6779 Epoch: 11 Global Step: 110310 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:04:15,568-Speed 129.63 samples/sec Loss 2.5558 Epoch: 11 Global Step: 110315 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:04:35,317-Speed 129.63 samples/sec Loss 2.5200 Epoch: 11 Global Step: 110320 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:04:55,098-Speed 129.43 samples/sec Loss 2.5416 Epoch: 11 Global Step: 110325 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:05:14,871-Speed 129.48 samples/sec Loss 2.4955 Epoch: 11 Global Step: 110330 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:05:35,007-Speed 127.14 samples/sec Loss 2.3386 Epoch: 11 Global Step: 110335 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:05:54,769-Speed 129.55 samples/sec Loss 2.5554 Epoch: 11 Global Step: 110340 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:06:14,589-Speed 129.16 samples/sec Loss 2.4403 Epoch: 11 Global Step: 110345 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:06:34,359-Speed 129.50 samples/sec Loss 2.4377 Epoch: 11 Global Step: 110350 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:06:54,104-Speed 129.66 samples/sec Loss 2.6180 Epoch: 11 Global Step: 110355 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:07:13,869-Speed 129.53 samples/sec Loss 2.6393 Epoch: 11 Global Step: 110360 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:07:33,634-Speed 129.53 samples/sec Loss 2.5533 Epoch: 11 Global Step: 110365 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:07:54,174-Speed 124.64 samples/sec Loss 2.3125 Epoch: 11 Global Step: 110370 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:08:14,109-Speed 128.43 samples/sec Loss 2.7778 Epoch: 11 Global Step: 110375 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:08:34,121-Speed 127.93 samples/sec Loss 2.6879 Epoch: 11 Global Step: 110380 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:08:53,971-Speed 128.98 samples/sec Loss 2.4143 Epoch: 11 Global Step: 110385 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:09:14,220-Speed 126.43 samples/sec Loss 2.6442 Epoch: 11 Global Step: 110390 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:09:34,126-Speed 128.61 samples/sec Loss 2.5231 Epoch: 11 Global Step: 110395 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:09:53,904-Speed 129.44 samples/sec Loss 2.6573 Epoch: 11 Global Step: 110400 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:10:13,650-Speed 129.66 samples/sec Loss 2.4770 Epoch: 11 Global Step: 110405 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:10:33,557-Speed 128.60 samples/sec Loss 2.6758 Epoch: 11 Global Step: 110410 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:10:53,432-Speed 128.81 samples/sec Loss 2.3266 Epoch: 11 Global Step: 110415 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:11:13,168-Speed 129.72 samples/sec Loss 2.6426 Epoch: 11 Global Step: 110420 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:11:33,274-Speed 127.33 samples/sec Loss 2.5942 Epoch: 11 Global Step: 110425 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:11:52,972-Speed 129.97 samples/sec Loss 2.4465 Epoch: 11 Global Step: 110430 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:12:12,907-Speed 128.42 samples/sec Loss 2.4990 Epoch: 11 Global Step: 110435 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:12:32,693-Speed 129.39 samples/sec Loss 2.5862 Epoch: 11 Global Step: 110440 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:12:52,432-Speed 129.70 samples/sec Loss 2.7067 Epoch: 11 Global Step: 110445 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:13:12,171-Speed 129.70 samples/sec Loss 2.4679 Epoch: 11 Global Step: 110450 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:13:31,941-Speed 129.50 samples/sec Loss 2.6820 Epoch: 11 Global Step: 110455 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:13:51,696-Speed 129.60 samples/sec Loss 2.4750 Epoch: 11 Global Step: 110460 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:14:11,435-Speed 129.70 samples/sec Loss 2.7208 Epoch: 11 Global Step: 110465 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:14:31,592-Speed 127.01 samples/sec Loss 2.5615 Epoch: 11 Global Step: 110470 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:14:51,375-Speed 129.41 samples/sec Loss 2.5777 Epoch: 11 Global Step: 110475 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:15:11,172-Speed 129.32 samples/sec Loss 2.6439 Epoch: 11 Global Step: 110480 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:15:30,955-Speed 129.41 samples/sec Loss 2.7144 Epoch: 11 Global Step: 110485 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:15:50,712-Speed 129.58 samples/sec Loss 2.4000 Epoch: 11 Global Step: 110490 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:16:10,495-Speed 129.41 samples/sec Loss 2.4246 Epoch: 11 Global Step: 110495 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:16:30,269-Speed 129.48 samples/sec Loss 2.4848 Epoch: 11 Global Step: 110500 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:16:50,039-Speed 129.50 samples/sec Loss 2.5265 Epoch: 11 Global Step: 110505 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:17:09,781-Speed 129.68 samples/sec Loss 2.6399 Epoch: 11 Global Step: 110510 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:17:29,935-Speed 127.03 samples/sec Loss 2.5000 Epoch: 11 Global Step: 110515 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:17:49,596-Speed 130.21 samples/sec Loss 2.5996 Epoch: 11 Global Step: 110520 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:18:09,533-Speed 128.41 samples/sec Loss 2.7218 Epoch: 11 Global Step: 110525 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:18:29,310-Speed 129.45 samples/sec Loss 2.6270 Epoch: 11 Global Step: 110530 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:18:49,090-Speed 129.43 samples/sec Loss 2.6532 Epoch: 11 Global Step: 110535 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:19:08,845-Speed 129.60 samples/sec Loss 2.5775 Epoch: 11 Global Step: 110540 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:19:28,614-Speed 129.50 samples/sec Loss 2.5001 Epoch: 11 Global Step: 110545 Fp16 Grad Scale: 8192 Required: 165 hours Training: 2021-06-12 06:19:48,505-Speed 128.71 samples/sec Loss 2.7512 Epoch: 11 Global Step: 110550 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:20:08,270-Speed 129.53 samples/sec Loss 2.5675 Epoch: 11 Global Step: 110555 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:20:28,435-Speed 126.96 samples/sec Loss 2.5175 Epoch: 11 Global Step: 110560 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:20:48,199-Speed 129.54 samples/sec Loss 2.5620 Epoch: 11 Global Step: 110565 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:21:07,993-Speed 129.34 samples/sec Loss 2.6330 Epoch: 11 Global Step: 110570 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:21:27,737-Speed 129.67 samples/sec Loss 2.4750 Epoch: 11 Global Step: 110575 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:21:47,545-Speed 129.25 samples/sec Loss 2.4774 Epoch: 11 Global Step: 110580 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:22:07,289-Speed 129.67 samples/sec Loss 2.6524 Epoch: 11 Global Step: 110585 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:22:27,057-Speed 129.51 samples/sec Loss 2.3287 Epoch: 11 Global Step: 110590 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:22:46,803-Speed 129.65 samples/sec Loss 2.4512 Epoch: 11 Global Step: 110595 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:23:06,560-Speed 129.58 samples/sec Loss 2.7124 Epoch: 11 Global Step: 110600 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:23:26,542-Speed 128.13 samples/sec Loss 2.4956 Epoch: 11 Global Step: 110605 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:23:46,466-Speed 128.49 samples/sec Loss 2.6506 Epoch: 11 Global Step: 110610 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:24:06,233-Speed 129.51 samples/sec Loss 2.5267 Epoch: 11 Global Step: 110615 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:24:26,003-Speed 129.50 samples/sec Loss 2.7330 Epoch: 11 Global Step: 110620 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:24:45,791-Speed 129.38 samples/sec Loss 2.6874 Epoch: 11 Global Step: 110625 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:25:05,751-Speed 128.26 samples/sec Loss 2.4780 Epoch: 11 Global Step: 110630 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:25:25,515-Speed 129.54 samples/sec Loss 2.5967 Epoch: 11 Global Step: 110635 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:25:45,290-Speed 129.46 samples/sec Loss 2.6253 Epoch: 11 Global Step: 110640 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:26:05,017-Speed 129.78 samples/sec Loss 2.5872 Epoch: 11 Global Step: 110645 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:26:25,000-Speed 128.12 samples/sec Loss 2.4846 Epoch: 11 Global Step: 110650 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:26:44,946-Speed 128.35 samples/sec Loss 2.6698 Epoch: 11 Global Step: 110655 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:27:04,721-Speed 129.47 samples/sec Loss 2.3358 Epoch: 11 Global Step: 110660 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:27:24,499-Speed 129.44 samples/sec Loss 2.5728 Epoch: 11 Global Step: 110665 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:27:44,306-Speed 129.25 samples/sec Loss 2.6541 Epoch: 11 Global Step: 110670 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:28:04,302-Speed 128.03 samples/sec Loss 2.4982 Epoch: 11 Global Step: 110675 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:28:24,068-Speed 129.53 samples/sec Loss 2.6276 Epoch: 11 Global Step: 110680 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:28:43,835-Speed 129.52 samples/sec Loss 2.6487 Epoch: 11 Global Step: 110685 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:29:03,627-Speed 129.35 samples/sec Loss 2.4602 Epoch: 11 Global Step: 110690 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:29:23,773-Speed 127.08 samples/sec Loss 2.5527 Epoch: 11 Global Step: 110695 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:29:43,352-Speed 130.76 samples/sec Loss 2.6443 Epoch: 11 Global Step: 110700 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:30:03,543-Speed 126.80 samples/sec Loss 2.7103 Epoch: 11 Global Step: 110705 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:30:24,811-Speed 120.37 samples/sec Loss 2.7359 Epoch: 11 Global Step: 110710 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:30:44,686-Speed 128.81 samples/sec Loss 2.5950 Epoch: 11 Global Step: 110715 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:31:04,375-Speed 130.03 samples/sec Loss 2.6079 Epoch: 11 Global Step: 110720 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:31:24,145-Speed 129.50 samples/sec Loss 2.5173 Epoch: 11 Global Step: 110725 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:31:43,879-Speed 129.73 samples/sec Loss 2.5487 Epoch: 11 Global Step: 110730 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:32:03,653-Speed 129.47 samples/sec Loss 2.4517 Epoch: 11 Global Step: 110735 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:32:23,798-Speed 127.09 samples/sec Loss 2.5695 Epoch: 11 Global Step: 110740 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:32:43,712-Speed 128.56 samples/sec Loss 2.6362 Epoch: 11 Global Step: 110745 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:33:03,502-Speed 129.37 samples/sec Loss 2.8092 Epoch: 11 Global Step: 110750 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:33:23,297-Speed 129.33 samples/sec Loss 2.5800 Epoch: 11 Global Step: 110755 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:33:43,056-Speed 129.57 samples/sec Loss 2.7635 Epoch: 11 Global Step: 110760 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:34:02,779-Speed 129.81 samples/sec Loss 2.6057 Epoch: 11 Global Step: 110765 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:34:22,664-Speed 128.75 samples/sec Loss 2.6449 Epoch: 11 Global Step: 110770 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:34:42,434-Speed 129.50 samples/sec Loss 2.5639 Epoch: 11 Global Step: 110775 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:35:02,212-Speed 129.45 samples/sec Loss 2.5628 Epoch: 11 Global Step: 110780 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:35:22,173-Speed 128.26 samples/sec Loss 2.6377 Epoch: 11 Global Step: 110785 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:35:42,049-Speed 128.80 samples/sec Loss 2.3542 Epoch: 11 Global Step: 110790 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:36:01,830-Speed 129.43 samples/sec Loss 2.6096 Epoch: 11 Global Step: 110795 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:36:21,760-Speed 128.46 samples/sec Loss 2.5591 Epoch: 11 Global Step: 110800 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:36:41,440-Speed 130.09 samples/sec Loss 2.5023 Epoch: 11 Global Step: 110805 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:37:01,210-Speed 129.49 samples/sec Loss 2.5735 Epoch: 11 Global Step: 110810 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:37:20,965-Speed 129.59 samples/sec Loss 2.7397 Epoch: 11 Global Step: 110815 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:37:40,792-Speed 129.13 samples/sec Loss 2.6185 Epoch: 11 Global Step: 110820 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:38:00,552-Speed 129.56 samples/sec Loss 2.6286 Epoch: 11 Global Step: 110825 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:38:20,702-Speed 127.06 samples/sec Loss 2.5509 Epoch: 11 Global Step: 110830 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:38:40,470-Speed 129.51 samples/sec Loss 2.7471 Epoch: 11 Global Step: 110835 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:39:00,240-Speed 129.50 samples/sec Loss 2.6676 Epoch: 11 Global Step: 110840 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:39:19,990-Speed 129.63 samples/sec Loss 2.5324 Epoch: 11 Global Step: 110845 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:39:39,781-Speed 129.36 samples/sec Loss 2.7544 Epoch: 11 Global Step: 110850 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:39:59,553-Speed 129.49 samples/sec Loss 2.7739 Epoch: 11 Global Step: 110855 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:40:19,413-Speed 128.91 samples/sec Loss 2.6179 Epoch: 11 Global Step: 110860 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:40:39,200-Speed 129.39 samples/sec Loss 2.5981 Epoch: 11 Global Step: 110865 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:40:58,976-Speed 129.46 samples/sec Loss 2.6097 Epoch: 11 Global Step: 110870 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:41:18,988-Speed 127.93 samples/sec Loss 2.5574 Epoch: 11 Global Step: 110875 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:41:39,025-Speed 127.77 samples/sec Loss 2.5864 Epoch: 11 Global Step: 110880 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:41:58,824-Speed 129.31 samples/sec Loss 2.7310 Epoch: 11 Global Step: 110885 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:42:18,608-Speed 129.40 samples/sec Loss 2.7535 Epoch: 11 Global Step: 110890 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:42:38,354-Speed 129.65 samples/sec Loss 2.7421 Epoch: 11 Global Step: 110895 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:42:58,140-Speed 129.39 samples/sec Loss 2.4520 Epoch: 11 Global Step: 110900 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:43:17,920-Speed 129.43 samples/sec Loss 2.6746 Epoch: 11 Global Step: 110905 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:43:37,676-Speed 129.59 samples/sec Loss 2.5502 Epoch: 11 Global Step: 110910 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:43:57,387-Speed 129.88 samples/sec Loss 2.5073 Epoch: 11 Global Step: 110915 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:44:17,299-Speed 128.57 samples/sec Loss 2.6146 Epoch: 11 Global Step: 110920 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:44:37,257-Speed 128.28 samples/sec Loss 2.7619 Epoch: 11 Global Step: 110925 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:44:57,101-Speed 129.02 samples/sec Loss 2.7173 Epoch: 11 Global Step: 110930 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:45:16,864-Speed 129.54 samples/sec Loss 2.4541 Epoch: 11 Global Step: 110935 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:45:36,581-Speed 129.85 samples/sec Loss 2.8027 Epoch: 11 Global Step: 110940 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:45:56,382-Speed 129.30 samples/sec Loss 2.5022 Epoch: 11 Global Step: 110945 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:46:16,067-Speed 130.06 samples/sec Loss 2.6198 Epoch: 11 Global Step: 110950 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:46:35,755-Speed 130.03 samples/sec Loss 2.5515 Epoch: 11 Global Step: 110955 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:46:55,485-Speed 129.76 samples/sec Loss 2.6488 Epoch: 11 Global Step: 110960 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:47:15,608-Speed 127.23 samples/sec Loss 2.6945 Epoch: 11 Global Step: 110965 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:47:35,415-Speed 129.25 samples/sec Loss 2.5200 Epoch: 11 Global Step: 110970 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:47:55,122-Speed 129.91 samples/sec Loss 2.6299 Epoch: 11 Global Step: 110975 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:48:14,886-Speed 129.54 samples/sec Loss 2.6508 Epoch: 11 Global Step: 110980 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:48:34,725-Speed 129.05 samples/sec Loss 2.7179 Epoch: 11 Global Step: 110985 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:48:54,452-Speed 129.78 samples/sec Loss 2.6076 Epoch: 11 Global Step: 110990 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:49:14,140-Speed 130.03 samples/sec Loss 2.4646 Epoch: 11 Global Step: 110995 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:49:33,919-Speed 129.44 samples/sec Loss 2.5580 Epoch: 11 Global Step: 111000 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:49:53,678-Speed 129.57 samples/sec Loss 2.4944 Epoch: 11 Global Step: 111005 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:50:14,288-Speed 124.22 samples/sec Loss 2.5956 Epoch: 11 Global Step: 111010 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:50:34,315-Speed 127.84 samples/sec Loss 2.6115 Epoch: 11 Global Step: 111015 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:50:54,102-Speed 129.39 samples/sec Loss 2.5902 Epoch: 11 Global Step: 111020 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:51:13,862-Speed 129.56 samples/sec Loss 2.5764 Epoch: 11 Global Step: 111025 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:51:33,651-Speed 129.37 samples/sec Loss 2.4754 Epoch: 11 Global Step: 111030 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:51:53,420-Speed 129.50 samples/sec Loss 2.3687 Epoch: 11 Global Step: 111035 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:52:13,165-Speed 129.66 samples/sec Loss 2.5209 Epoch: 11 Global Step: 111040 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:52:32,999-Speed 129.08 samples/sec Loss 2.6191 Epoch: 11 Global Step: 111045 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:52:52,793-Speed 129.34 samples/sec Loss 2.5103 Epoch: 11 Global Step: 111050 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:53:12,942-Speed 127.06 samples/sec Loss 2.4739 Epoch: 11 Global Step: 111055 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:53:32,774-Speed 129.09 samples/sec Loss 2.5263 Epoch: 11 Global Step: 111060 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:53:52,548-Speed 129.47 samples/sec Loss 2.4525 Epoch: 11 Global Step: 111065 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:54:12,290-Speed 129.68 samples/sec Loss 2.6185 Epoch: 11 Global Step: 111070 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:54:32,048-Speed 129.58 samples/sec Loss 2.8163 Epoch: 11 Global Step: 111075 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:54:51,777-Speed 129.77 samples/sec Loss 2.5928 Epoch: 11 Global Step: 111080 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:55:11,676-Speed 128.65 samples/sec Loss 2.7628 Epoch: 11 Global Step: 111085 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:55:31,451-Speed 129.47 samples/sec Loss 2.7035 Epoch: 11 Global Step: 111090 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:55:51,183-Speed 129.75 samples/sec Loss 2.7795 Epoch: 11 Global Step: 111095 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:56:11,312-Speed 127.19 samples/sec Loss 2.4723 Epoch: 11 Global Step: 111100 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:56:31,041-Speed 129.77 samples/sec Loss 2.5353 Epoch: 11 Global Step: 111105 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:56:50,802-Speed 129.55 samples/sec Loss 2.7344 Epoch: 11 Global Step: 111110 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:57:10,528-Speed 129.79 samples/sec Loss 2.6462 Epoch: 11 Global Step: 111115 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:57:30,257-Speed 129.77 samples/sec Loss 2.5197 Epoch: 11 Global Step: 111120 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:57:49,998-Speed 129.68 samples/sec Loss 2.6656 Epoch: 11 Global Step: 111125 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:58:09,741-Speed 129.68 samples/sec Loss 2.5550 Epoch: 11 Global Step: 111130 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:58:29,544-Speed 129.28 samples/sec Loss 2.5593 Epoch: 11 Global Step: 111135 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:58:49,386-Speed 129.03 samples/sec Loss 2.6521 Epoch: 11 Global Step: 111140 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:59:09,339-Speed 128.31 samples/sec Loss 2.7097 Epoch: 11 Global Step: 111145 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:59:29,141-Speed 129.28 samples/sec Loss 2.7890 Epoch: 11 Global Step: 111150 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 06:59:48,957-Speed 129.20 samples/sec Loss 2.7304 Epoch: 11 Global Step: 111155 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:00:08,696-Speed 129.70 samples/sec Loss 2.5456 Epoch: 11 Global Step: 111160 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:00:28,461-Speed 129.53 samples/sec Loss 2.5423 Epoch: 11 Global Step: 111165 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:00:48,223-Speed 129.55 samples/sec Loss 2.7475 Epoch: 11 Global Step: 111170 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:01:07,978-Speed 129.60 samples/sec Loss 2.7374 Epoch: 11 Global Step: 111175 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:01:27,727-Speed 129.64 samples/sec Loss 2.6758 Epoch: 11 Global Step: 111180 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:01:47,504-Speed 129.45 samples/sec Loss 2.6873 Epoch: 11 Global Step: 111185 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:02:07,646-Speed 127.11 samples/sec Loss 2.6676 Epoch: 11 Global Step: 111190 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:02:27,681-Speed 127.78 samples/sec Loss 2.7985 Epoch: 11 Global Step: 111195 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:02:47,467-Speed 129.39 samples/sec Loss 2.4857 Epoch: 11 Global Step: 111200 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:03:07,314-Speed 128.99 samples/sec Loss 2.7682 Epoch: 11 Global Step: 111205 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:03:27,091-Speed 129.45 samples/sec Loss 2.6191 Epoch: 11 Global Step: 111210 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:03:46,904-Speed 129.22 samples/sec Loss 2.6676 Epoch: 11 Global Step: 111215 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:04:06,661-Speed 129.58 samples/sec Loss 2.5214 Epoch: 11 Global Step: 111220 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:04:26,508-Speed 128.99 samples/sec Loss 2.5485 Epoch: 11 Global Step: 111225 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:04:46,288-Speed 129.43 samples/sec Loss 2.5056 Epoch: 11 Global Step: 111230 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:05:06,440-Speed 127.04 samples/sec Loss 2.5594 Epoch: 11 Global Step: 111235 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:05:26,245-Speed 129.27 samples/sec Loss 2.6867 Epoch: 11 Global Step: 111240 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:05:46,008-Speed 129.55 samples/sec Loss 2.6705 Epoch: 11 Global Step: 111245 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:06:05,779-Speed 129.49 samples/sec Loss 2.6384 Epoch: 11 Global Step: 111250 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:06:25,563-Speed 129.41 samples/sec Loss 2.6971 Epoch: 11 Global Step: 111255 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:06:45,378-Speed 129.20 samples/sec Loss 2.7694 Epoch: 11 Global Step: 111260 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:07:05,152-Speed 129.48 samples/sec Loss 2.6017 Epoch: 11 Global Step: 111265 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:07:24,904-Speed 129.62 samples/sec Loss 2.9890 Epoch: 11 Global Step: 111270 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:07:44,661-Speed 129.58 samples/sec Loss 2.6411 Epoch: 11 Global Step: 111275 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:08:04,618-Speed 128.29 samples/sec Loss 2.7533 Epoch: 11 Global Step: 111280 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:08:24,574-Speed 128.29 samples/sec Loss 2.5345 Epoch: 11 Global Step: 111285 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:08:44,579-Speed 127.98 samples/sec Loss 2.6322 Epoch: 11 Global Step: 111290 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:09:04,356-Speed 129.45 samples/sec Loss 2.6162 Epoch: 11 Global Step: 111295 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:09:24,145-Speed 129.37 samples/sec Loss 2.5855 Epoch: 11 Global Step: 111300 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:09:43,956-Speed 129.23 samples/sec Loss 2.6613 Epoch: 11 Global Step: 111305 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:10:03,725-Speed 129.50 samples/sec Loss 2.8032 Epoch: 11 Global Step: 111310 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:10:23,476-Speed 129.62 samples/sec Loss 2.5884 Epoch: 11 Global Step: 111315 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:10:43,248-Speed 129.48 samples/sec Loss 2.6372 Epoch: 11 Global Step: 111320 Fp16 Grad Scale: 8192 Required: 164 hours Training: 2021-06-12 07:11:03,330-Speed 127.48 samples/sec Loss 2.8618 Epoch: 11 Global Step: 111325 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:11:23,282-Speed 128.32 samples/sec Loss 2.7294 Epoch: 11 Global Step: 111330 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:11:43,057-Speed 129.46 samples/sec Loss 2.6123 Epoch: 11 Global Step: 111335 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:12:02,846-Speed 129.37 samples/sec Loss 2.5690 Epoch: 11 Global Step: 111340 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:12:23,401-Speed 124.55 samples/sec Loss 2.6822 Epoch: 11 Global Step: 111345 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:12:43,165-Speed 129.54 samples/sec Loss 2.6348 Epoch: 11 Global Step: 111350 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:13:02,933-Speed 129.50 samples/sec Loss 2.5936 Epoch: 11 Global Step: 111355 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:13:22,672-Speed 129.70 samples/sec Loss 2.6950 Epoch: 11 Global Step: 111360 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:13:42,466-Speed 129.34 samples/sec Loss 2.7247 Epoch: 11 Global Step: 111365 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:14:02,581-Speed 127.28 samples/sec Loss 2.6657 Epoch: 11 Global Step: 111370 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:14:22,372-Speed 129.36 samples/sec Loss 2.4895 Epoch: 11 Global Step: 111375 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:14:42,143-Speed 129.49 samples/sec Loss 2.6540 Epoch: 11 Global Step: 111380 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:15:01,990-Speed 129.00 samples/sec Loss 2.4347 Epoch: 11 Global Step: 111385 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:15:22,461-Speed 125.06 samples/sec Loss 2.6339 Epoch: 11 Global Step: 111390 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:15:42,217-Speed 129.59 samples/sec Loss 2.5406 Epoch: 11 Global Step: 111395 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:16:01,978-Speed 129.55 samples/sec Loss 2.5165 Epoch: 11 Global Step: 111400 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:16:21,746-Speed 129.51 samples/sec Loss 2.5418 Epoch: 11 Global Step: 111405 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:16:41,513-Speed 129.52 samples/sec Loss 2.6909 Epoch: 11 Global Step: 111410 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:17:01,470-Speed 128.28 samples/sec Loss 2.7150 Epoch: 11 Global Step: 111415 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:17:21,413-Speed 128.37 samples/sec Loss 2.7680 Epoch: 11 Global Step: 111420 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:17:41,160-Speed 129.64 samples/sec Loss 2.7955 Epoch: 11 Global Step: 111425 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:18:00,904-Speed 129.67 samples/sec Loss 2.6991 Epoch: 11 Global Step: 111430 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:18:20,695-Speed 129.36 samples/sec Loss 2.8004 Epoch: 11 Global Step: 111435 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:18:40,714-Speed 127.89 samples/sec Loss 2.7399 Epoch: 11 Global Step: 111440 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:19:00,459-Speed 129.66 samples/sec Loss 2.8609 Epoch: 11 Global Step: 111445 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:19:20,270-Speed 129.23 samples/sec Loss 2.7025 Epoch: 11 Global Step: 111450 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:19:40,001-Speed 129.76 samples/sec Loss 2.7713 Epoch: 11 Global Step: 111455 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:20:00,071-Speed 127.56 samples/sec Loss 2.7436 Epoch: 11 Global Step: 111460 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:20:19,833-Speed 129.55 samples/sec Loss 2.7783 Epoch: 11 Global Step: 111465 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:20:39,724-Speed 128.71 samples/sec Loss 2.6661 Epoch: 11 Global Step: 111470 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:20:59,462-Speed 129.71 samples/sec Loss 2.6555 Epoch: 11 Global Step: 111475 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:21:19,245-Speed 129.41 samples/sec Loss 2.6104 Epoch: 11 Global Step: 111480 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:21:38,979-Speed 129.74 samples/sec Loss 2.7507 Epoch: 11 Global Step: 111485 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:21:58,732-Speed 129.60 samples/sec Loss 2.6721 Epoch: 11 Global Step: 111490 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:22:19,073-Speed 125.86 samples/sec Loss 2.6387 Epoch: 11 Global Step: 111495 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:22:38,855-Speed 129.42 samples/sec Loss 2.8106 Epoch: 11 Global Step: 111500 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:22:59,181-Speed 125.95 samples/sec Loss 2.5608 Epoch: 11 Global Step: 111505 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:23:18,968-Speed 129.39 samples/sec Loss 2.5759 Epoch: 11 Global Step: 111510 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:23:38,755-Speed 129.38 samples/sec Loss 2.8210 Epoch: 11 Global Step: 111515 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:23:58,515-Speed 129.56 samples/sec Loss 2.5996 Epoch: 11 Global Step: 111520 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:24:18,289-Speed 129.47 samples/sec Loss 2.7747 Epoch: 11 Global Step: 111525 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:24:38,028-Speed 129.70 samples/sec Loss 2.7105 Epoch: 11 Global Step: 111530 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:24:57,801-Speed 129.48 samples/sec Loss 2.7197 Epoch: 11 Global Step: 111535 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:25:17,613-Speed 129.22 samples/sec Loss 2.5964 Epoch: 11 Global Step: 111540 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:25:37,383-Speed 129.50 samples/sec Loss 2.5852 Epoch: 11 Global Step: 111545 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:25:57,310-Speed 128.48 samples/sec Loss 2.5531 Epoch: 11 Global Step: 111550 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:26:17,241-Speed 128.45 samples/sec Loss 2.6368 Epoch: 11 Global Step: 111555 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:26:36,991-Speed 129.62 samples/sec Loss 2.6940 Epoch: 11 Global Step: 111560 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:26:56,769-Speed 129.45 samples/sec Loss 2.6163 Epoch: 11 Global Step: 111565 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:27:16,515-Speed 129.65 samples/sec Loss 2.5272 Epoch: 11 Global Step: 111570 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:27:36,288-Speed 129.48 samples/sec Loss 2.5417 Epoch: 11 Global Step: 111575 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:27:56,053-Speed 129.53 samples/sec Loss 2.8625 Epoch: 11 Global Step: 111580 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:28:15,810-Speed 129.58 samples/sec Loss 2.6499 Epoch: 11 Global Step: 111585 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:28:35,549-Speed 129.71 samples/sec Loss 2.8269 Epoch: 11 Global Step: 111590 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:28:55,684-Speed 127.14 samples/sec Loss 2.7266 Epoch: 11 Global Step: 111595 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:29:15,587-Speed 128.63 samples/sec Loss 2.6729 Epoch: 11 Global Step: 111600 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:29:35,393-Speed 129.27 samples/sec Loss 2.8081 Epoch: 11 Global Step: 111605 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:29:55,125-Speed 129.74 samples/sec Loss 2.6182 Epoch: 11 Global Step: 111610 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:30:14,892-Speed 129.51 samples/sec Loss 2.7970 Epoch: 11 Global Step: 111615 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:30:34,666-Speed 129.47 samples/sec Loss 2.6967 Epoch: 11 Global Step: 111620 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:30:54,416-Speed 129.63 samples/sec Loss 2.6378 Epoch: 11 Global Step: 111625 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:31:14,183-Speed 129.51 samples/sec Loss 2.5739 Epoch: 11 Global Step: 111630 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:31:33,943-Speed 129.56 samples/sec Loss 2.8988 Epoch: 11 Global Step: 111635 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:31:54,064-Speed 127.24 samples/sec Loss 2.6324 Epoch: 11 Global Step: 111640 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:32:13,864-Speed 129.30 samples/sec Loss 2.8793 Epoch: 11 Global Step: 111645 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:32:33,610-Speed 129.65 samples/sec Loss 2.6698 Epoch: 11 Global Step: 111650 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:32:53,343-Speed 129.74 samples/sec Loss 2.7999 Epoch: 11 Global Step: 111655 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:33:13,117-Speed 129.47 samples/sec Loss 2.5949 Epoch: 11 Global Step: 111660 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:33:32,862-Speed 129.66 samples/sec Loss 2.7391 Epoch: 11 Global Step: 111665 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:33:52,618-Speed 129.59 samples/sec Loss 2.5915 Epoch: 11 Global Step: 111670 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:34:12,365-Speed 129.65 samples/sec Loss 2.6960 Epoch: 11 Global Step: 111675 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:34:32,136-Speed 129.49 samples/sec Loss 2.7283 Epoch: 11 Global Step: 111680 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:34:52,121-Speed 128.10 samples/sec Loss 2.7381 Epoch: 11 Global Step: 111685 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:35:12,044-Speed 128.50 samples/sec Loss 2.7817 Epoch: 11 Global Step: 111690 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:35:31,799-Speed 129.60 samples/sec Loss 2.4449 Epoch: 11 Global Step: 111695 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:35:52,333-Speed 124.68 samples/sec Loss 2.7599 Epoch: 11 Global Step: 111700 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:36:12,176-Speed 129.02 samples/sec Loss 2.6683 Epoch: 11 Global Step: 111705 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:36:31,923-Speed 129.65 samples/sec Loss 2.6957 Epoch: 11 Global Step: 111710 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:36:51,669-Speed 129.65 samples/sec Loss 2.6118 Epoch: 11 Global Step: 111715 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:37:11,410-Speed 129.69 samples/sec Loss 2.6736 Epoch: 11 Global Step: 111720 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:37:31,199-Speed 129.37 samples/sec Loss 2.6295 Epoch: 11 Global Step: 111725 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:37:51,341-Speed 127.10 samples/sec Loss 2.7756 Epoch: 11 Global Step: 111730 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:38:11,015-Speed 130.13 samples/sec Loss 2.6422 Epoch: 11 Global Step: 111735 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:38:31,050-Speed 127.78 samples/sec Loss 2.6074 Epoch: 11 Global Step: 111740 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:38:50,814-Speed 129.53 samples/sec Loss 2.6745 Epoch: 11 Global Step: 111745 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:39:10,671-Speed 128.93 samples/sec Loss 2.8466 Epoch: 11 Global Step: 111750 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:39:30,420-Speed 129.63 samples/sec Loss 2.6570 Epoch: 11 Global Step: 111755 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:39:50,152-Speed 129.75 samples/sec Loss 2.7658 Epoch: 11 Global Step: 111760 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:40:10,139-Speed 128.09 samples/sec Loss 2.6721 Epoch: 11 Global Step: 111765 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:40:29,955-Speed 129.20 samples/sec Loss 2.8387 Epoch: 11 Global Step: 111770 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:40:50,154-Speed 126.74 samples/sec Loss 2.7764 Epoch: 11 Global Step: 111775 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:41:09,812-Speed 130.24 samples/sec Loss 2.9032 Epoch: 11 Global Step: 111780 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:41:29,742-Speed 128.45 samples/sec Loss 2.7280 Epoch: 11 Global Step: 111785 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:41:49,475-Speed 129.74 samples/sec Loss 2.7955 Epoch: 11 Global Step: 111790 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:42:09,350-Speed 128.81 samples/sec Loss 2.5696 Epoch: 11 Global Step: 111795 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:42:29,181-Speed 129.10 samples/sec Loss 2.7376 Epoch: 11 Global Step: 111800 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:42:48,904-Speed 129.80 samples/sec Loss 2.9322 Epoch: 11 Global Step: 111805 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:43:08,708-Speed 129.28 samples/sec Loss 2.6658 Epoch: 11 Global Step: 111810 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:43:28,436-Speed 129.77 samples/sec Loss 2.7048 Epoch: 11 Global Step: 111815 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:43:48,505-Speed 127.57 samples/sec Loss 2.7015 Epoch: 11 Global Step: 111820 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:44:08,237-Speed 129.74 samples/sec Loss 2.7246 Epoch: 11 Global Step: 111825 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:44:28,135-Speed 128.66 samples/sec Loss 2.6268 Epoch: 11 Global Step: 111830 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:44:48,427-Speed 126.17 samples/sec Loss 2.6805 Epoch: 11 Global Step: 111835 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:45:08,191-Speed 129.54 samples/sec Loss 2.5884 Epoch: 11 Global Step: 111840 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:45:27,933-Speed 129.68 samples/sec Loss 2.7335 Epoch: 11 Global Step: 111845 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:45:47,697-Speed 129.53 samples/sec Loss 2.7928 Epoch: 11 Global Step: 111850 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:46:07,447-Speed 129.62 samples/sec Loss 2.7821 Epoch: 11 Global Step: 111855 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:46:27,198-Speed 129.63 samples/sec Loss 2.7750 Epoch: 11 Global Step: 111860 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:46:47,310-Speed 127.29 samples/sec Loss 2.5950 Epoch: 11 Global Step: 111865 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:47:06,935-Speed 130.45 samples/sec Loss 2.5886 Epoch: 11 Global Step: 111870 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:47:26,866-Speed 128.45 samples/sec Loss 2.8217 Epoch: 11 Global Step: 111875 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:47:46,616-Speed 129.63 samples/sec Loss 2.7250 Epoch: 11 Global Step: 111880 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:48:06,754-Speed 127.13 samples/sec Loss 2.8381 Epoch: 11 Global Step: 111885 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:48:26,488-Speed 129.73 samples/sec Loss 2.7675 Epoch: 11 Global Step: 111890 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:48:46,224-Speed 129.73 samples/sec Loss 2.9269 Epoch: 11 Global Step: 111895 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:49:06,002-Speed 129.44 samples/sec Loss 2.9995 Epoch: 11 Global Step: 111900 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:49:25,773-Speed 129.49 samples/sec Loss 2.5714 Epoch: 11 Global Step: 111905 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:49:45,790-Speed 127.90 samples/sec Loss 2.9632 Epoch: 11 Global Step: 111910 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:50:05,581-Speed 129.36 samples/sec Loss 2.6783 Epoch: 11 Global Step: 111915 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:50:25,430-Speed 128.98 samples/sec Loss 2.8377 Epoch: 11 Global Step: 111920 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:50:45,160-Speed 129.76 samples/sec Loss 2.6280 Epoch: 11 Global Step: 111925 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:51:04,926-Speed 129.52 samples/sec Loss 2.7813 Epoch: 11 Global Step: 111930 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:51:24,713-Speed 129.39 samples/sec Loss 2.7124 Epoch: 11 Global Step: 111935 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:51:44,473-Speed 129.56 samples/sec Loss 2.5873 Epoch: 11 Global Step: 111940 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:52:04,210-Speed 129.71 samples/sec Loss 2.5349 Epoch: 11 Global Step: 111945 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:52:23,959-Speed 129.63 samples/sec Loss 2.5356 Epoch: 11 Global Step: 111950 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:52:44,101-Speed 127.11 samples/sec Loss 2.7497 Epoch: 11 Global Step: 111955 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:53:03,935-Speed 129.08 samples/sec Loss 2.9081 Epoch: 11 Global Step: 111960 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:53:23,676-Speed 129.69 samples/sec Loss 2.6210 Epoch: 11 Global Step: 111965 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:53:43,429-Speed 129.61 samples/sec Loss 2.7735 Epoch: 11 Global Step: 111970 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:54:03,216-Speed 129.38 samples/sec Loss 2.7379 Epoch: 11 Global Step: 111975 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:54:22,947-Speed 129.75 samples/sec Loss 2.6816 Epoch: 11 Global Step: 111980 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:54:42,842-Speed 128.69 samples/sec Loss 2.8691 Epoch: 11 Global Step: 111985 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:55:02,579-Speed 129.71 samples/sec Loss 2.7200 Epoch: 11 Global Step: 111990 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:55:22,320-Speed 129.69 samples/sec Loss 2.6453 Epoch: 11 Global Step: 111995 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:55:42,309-Speed 128.08 samples/sec Loss 2.6278 Epoch: 11 Global Step: 112000 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:56:02,251-Speed 128.38 samples/sec Loss 2.7657 Epoch: 11 Global Step: 112005 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:56:22,026-Speed 129.47 samples/sec Loss 2.6896 Epoch: 11 Global Step: 112010 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:56:41,763-Speed 129.71 samples/sec Loss 2.7587 Epoch: 11 Global Step: 112015 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:57:01,511-Speed 129.64 samples/sec Loss 3.0256 Epoch: 11 Global Step: 112020 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:57:21,267-Speed 129.59 samples/sec Loss 2.8085 Epoch: 11 Global Step: 112025 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:57:41,026-Speed 129.57 samples/sec Loss 2.7494 Epoch: 11 Global Step: 112030 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:58:00,768-Speed 129.68 samples/sec Loss 2.8945 Epoch: 11 Global Step: 112035 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:58:20,573-Speed 129.27 samples/sec Loss 2.7399 Epoch: 11 Global Step: 112040 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:58:40,705-Speed 127.17 samples/sec Loss 2.7357 Epoch: 11 Global Step: 112045 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:59:00,323-Speed 130.50 samples/sec Loss 2.7702 Epoch: 11 Global Step: 112050 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:59:20,252-Speed 128.46 samples/sec Loss 2.7036 Epoch: 11 Global Step: 112055 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:59:40,018-Speed 129.52 samples/sec Loss 2.6456 Epoch: 11 Global Step: 112060 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 07:59:59,781-Speed 129.54 samples/sec Loss 2.8521 Epoch: 11 Global Step: 112065 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 08:00:19,525-Speed 129.67 samples/sec Loss 2.8651 Epoch: 11 Global Step: 112070 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 08:00:39,387-Speed 128.89 samples/sec Loss 2.9565 Epoch: 11 Global Step: 112075 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 08:00:59,122-Speed 129.73 samples/sec Loss 2.6001 Epoch: 11 Global Step: 112080 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 08:01:18,907-Speed 129.40 samples/sec Loss 2.7980 Epoch: 11 Global Step: 112085 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 08:01:39,125-Speed 126.63 samples/sec Loss 2.6040 Epoch: 11 Global Step: 112090 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 08:01:58,732-Speed 130.57 samples/sec Loss 2.6973 Epoch: 11 Global Step: 112095 Fp16 Grad Scale: 8192 Required: 163 hours Training: 2021-06-12 08:02:18,667-Speed 128.42 samples/sec Loss 2.8018 Epoch: 11 Global Step: 112100 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:02:39,171-Speed 124.86 samples/sec Loss 2.9252 Epoch: 11 Global Step: 112105 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:02:58,936-Speed 129.53 samples/sec Loss 2.7045 Epoch: 11 Global Step: 112110 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:03:18,675-Speed 129.70 samples/sec Loss 2.5306 Epoch: 11 Global Step: 112115 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:03:38,424-Speed 129.63 samples/sec Loss 2.4773 Epoch: 11 Global Step: 112120 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:03:58,179-Speed 129.60 samples/sec Loss 2.5849 Epoch: 11 Global Step: 112125 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:04:18,533-Speed 125.78 samples/sec Loss 2.9110 Epoch: 11 Global Step: 112130 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:04:38,679-Speed 127.08 samples/sec Loss 2.6043 Epoch: 11 Global Step: 112135 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:04:58,433-Speed 129.60 samples/sec Loss 2.7882 Epoch: 11 Global Step: 112140 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:05:18,217-Speed 129.41 samples/sec Loss 2.8113 Epoch: 11 Global Step: 112145 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:05:37,950-Speed 129.74 samples/sec Loss 2.6249 Epoch: 11 Global Step: 112150 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:05:57,707-Speed 129.58 samples/sec Loss 2.7341 Epoch: 11 Global Step: 112155 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:06:17,482-Speed 129.46 samples/sec Loss 2.6494 Epoch: 11 Global Step: 112160 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:06:37,508-Speed 127.84 samples/sec Loss 2.7127 Epoch: 11 Global Step: 112165 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:06:57,276-Speed 129.51 samples/sec Loss 2.6419 Epoch: 11 Global Step: 112170 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:07:17,024-Speed 129.64 samples/sec Loss 2.8192 Epoch: 11 Global Step: 112175 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:07:36,953-Speed 128.47 samples/sec Loss 2.6978 Epoch: 11 Global Step: 112180 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:07:56,869-Speed 128.54 samples/sec Loss 2.8254 Epoch: 11 Global Step: 112185 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:08:16,604-Speed 129.72 samples/sec Loss 2.6470 Epoch: 11 Global Step: 112190 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:08:36,370-Speed 129.52 samples/sec Loss 2.8772 Epoch: 11 Global Step: 112195 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:08:56,102-Speed 129.74 samples/sec Loss 2.6112 Epoch: 11 Global Step: 112200 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:09:16,629-Speed 124.72 samples/sec Loss 2.8994 Epoch: 11 Global Step: 112205 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:09:36,380-Speed 129.63 samples/sec Loss 2.7179 Epoch: 11 Global Step: 112210 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:09:56,127-Speed 129.64 samples/sec Loss 2.7084 Epoch: 11 Global Step: 112215 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:10:15,868-Speed 129.69 samples/sec Loss 2.8681 Epoch: 11 Global Step: 112220 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:10:36,166-Speed 126.13 samples/sec Loss 3.0034 Epoch: 11 Global Step: 112225 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:10:55,924-Speed 129.58 samples/sec Loss 2.6835 Epoch: 11 Global Step: 112230 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:11:15,936-Speed 127.93 samples/sec Loss 2.7127 Epoch: 11 Global Step: 112235 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:11:35,796-Speed 128.91 samples/sec Loss 2.7872 Epoch: 11 Global Step: 112240 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:11:55,541-Speed 129.66 samples/sec Loss 2.8046 Epoch: 11 Global Step: 112245 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:12:15,285-Speed 129.67 samples/sec Loss 2.7735 Epoch: 11 Global Step: 112250 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:12:35,118-Speed 129.08 samples/sec Loss 2.7461 Epoch: 11 Global Step: 112255 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:12:54,833-Speed 129.85 samples/sec Loss 2.7419 Epoch: 11 Global Step: 112260 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:13:14,554-Speed 129.82 samples/sec Loss 2.6708 Epoch: 11 Global Step: 112265 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:13:34,670-Speed 127.27 samples/sec Loss 3.0770 Epoch: 11 Global Step: 112270 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:13:54,384-Speed 129.86 samples/sec Loss 2.7213 Epoch: 11 Global Step: 112275 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:14:14,164-Speed 129.43 samples/sec Loss 2.5723 Epoch: 11 Global Step: 112280 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:14:34,707-Speed 124.62 samples/sec Loss 2.5492 Epoch: 11 Global Step: 112285 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:14:54,441-Speed 129.74 samples/sec Loss 2.8080 Epoch: 11 Global Step: 112290 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:15:14,185-Speed 129.67 samples/sec Loss 2.8192 Epoch: 11 Global Step: 112295 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:15:33,940-Speed 129.60 samples/sec Loss 2.6790 Epoch: 11 Global Step: 112300 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:15:53,688-Speed 129.64 samples/sec Loss 2.6309 Epoch: 11 Global Step: 112305 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:16:13,572-Speed 128.76 samples/sec Loss 2.8943 Epoch: 11 Global Step: 112310 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:16:33,654-Speed 127.48 samples/sec Loss 2.6135 Epoch: 11 Global Step: 112315 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:16:53,439-Speed 129.40 samples/sec Loss 2.7748 Epoch: 11 Global Step: 112320 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:17:13,196-Speed 129.59 samples/sec Loss 2.5089 Epoch: 11 Global Step: 112325 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:17:32,941-Speed 129.66 samples/sec Loss 2.9538 Epoch: 11 Global Step: 112330 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:17:52,699-Speed 129.57 samples/sec Loss 2.9553 Epoch: 11 Global Step: 112335 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:18:13,014-Speed 126.03 samples/sec Loss 2.6701 Epoch: 11 Global Step: 112340 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:18:32,724-Speed 129.89 samples/sec Loss 2.6854 Epoch: 11 Global Step: 112345 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:18:52,499-Speed 129.46 samples/sec Loss 2.7548 Epoch: 11 Global Step: 112350 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:19:12,220-Speed 129.82 samples/sec Loss 2.7013 Epoch: 11 Global Step: 112355 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:19:32,292-Speed 127.55 samples/sec Loss 2.8999 Epoch: 11 Global Step: 112360 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:19:51,871-Speed 130.76 samples/sec Loss 2.8125 Epoch: 11 Global Step: 112365 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:20:11,813-Speed 128.38 samples/sec Loss 2.8407 Epoch: 11 Global Step: 112370 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:20:31,563-Speed 129.63 samples/sec Loss 2.8628 Epoch: 11 Global Step: 112375 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:20:51,302-Speed 129.70 samples/sec Loss 2.8825 Epoch: 11 Global Step: 112380 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:21:11,030-Speed 129.77 samples/sec Loss 2.7130 Epoch: 11 Global Step: 112385 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:21:30,700-Speed 130.16 samples/sec Loss 2.8456 Epoch: 11 Global Step: 112390 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:21:50,419-Speed 129.83 samples/sec Loss 2.9681 Epoch: 11 Global Step: 112395 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:22:10,170-Speed 129.62 samples/sec Loss 2.8289 Epoch: 11 Global Step: 112400 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:22:30,134-Speed 128.24 samples/sec Loss 2.8253 Epoch: 11 Global Step: 112405 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:22:49,905-Speed 129.49 samples/sec Loss 2.8026 Epoch: 11 Global Step: 112410 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:23:09,769-Speed 128.88 samples/sec Loss 2.7447 Epoch: 11 Global Step: 112415 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:23:29,502-Speed 129.74 samples/sec Loss 2.7233 Epoch: 11 Global Step: 112420 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:23:49,224-Speed 129.81 samples/sec Loss 2.6148 Epoch: 11 Global Step: 112425 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:24:08,967-Speed 129.67 samples/sec Loss 2.7264 Epoch: 11 Global Step: 112430 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:24:28,686-Speed 129.83 samples/sec Loss 2.9393 Epoch: 11 Global Step: 112435 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:24:48,405-Speed 129.83 samples/sec Loss 3.0291 Epoch: 11 Global Step: 112440 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:25:08,139-Speed 129.73 samples/sec Loss 2.8259 Epoch: 11 Global Step: 112445 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:25:28,256-Speed 127.26 samples/sec Loss 2.8513 Epoch: 11 Global Step: 112450 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:25:48,005-Speed 129.63 samples/sec Loss 2.7786 Epoch: 11 Global Step: 112455 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:26:07,765-Speed 129.56 samples/sec Loss 2.7688 Epoch: 11 Global Step: 112460 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:26:27,527-Speed 129.55 samples/sec Loss 2.7627 Epoch: 11 Global Step: 112465 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:26:47,257-Speed 129.76 samples/sec Loss 2.8530 Epoch: 11 Global Step: 112470 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:27:07,820-Speed 124.51 samples/sec Loss 2.7895 Epoch: 11 Global Step: 112475 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:27:27,572-Speed 129.61 samples/sec Loss 2.8059 Epoch: 11 Global Step: 112480 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:27:47,335-Speed 129.54 samples/sec Loss 2.9213 Epoch: 11 Global Step: 112485 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:28:07,062-Speed 129.78 samples/sec Loss 2.8746 Epoch: 11 Global Step: 112490 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:28:28,043-Speed 122.02 samples/sec Loss 2.7917 Epoch: 11 Global Step: 112495 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:28:47,998-Speed 128.30 samples/sec Loss 2.7410 Epoch: 11 Global Step: 112500 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:29:07,756-Speed 129.58 samples/sec Loss 2.7712 Epoch: 11 Global Step: 112505 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:29:27,581-Speed 129.14 samples/sec Loss 2.6997 Epoch: 11 Global Step: 112510 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:29:47,300-Speed 129.83 samples/sec Loss 2.7771 Epoch: 11 Global Step: 112515 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:30:07,237-Speed 128.41 samples/sec Loss 2.7263 Epoch: 11 Global Step: 112520 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:30:26,971-Speed 129.73 samples/sec Loss 2.7538 Epoch: 11 Global Step: 112525 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:30:46,686-Speed 129.86 samples/sec Loss 2.8104 Epoch: 11 Global Step: 112530 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:31:06,427-Speed 129.69 samples/sec Loss 2.6841 Epoch: 11 Global Step: 112535 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:31:26,347-Speed 128.52 samples/sec Loss 2.7342 Epoch: 11 Global Step: 112540 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:31:46,401-Speed 127.67 samples/sec Loss 2.7942 Epoch: 11 Global Step: 112545 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:32:06,184-Speed 129.41 samples/sec Loss 2.8027 Epoch: 11 Global Step: 112550 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:32:25,912-Speed 129.77 samples/sec Loss 2.7561 Epoch: 11 Global Step: 112555 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:32:45,652-Speed 129.69 samples/sec Loss 2.9104 Epoch: 11 Global Step: 112560 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:33:05,381-Speed 129.76 samples/sec Loss 2.7495 Epoch: 11 Global Step: 112565 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:33:25,139-Speed 129.58 samples/sec Loss 2.7874 Epoch: 11 Global Step: 112570 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:33:44,895-Speed 129.59 samples/sec Loss 2.7454 Epoch: 11 Global Step: 112575 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:34:04,629-Speed 129.74 samples/sec Loss 2.7568 Epoch: 11 Global Step: 112580 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:34:24,714-Speed 127.47 samples/sec Loss 2.7642 Epoch: 11 Global Step: 112585 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:34:44,419-Speed 129.92 samples/sec Loss 2.7800 Epoch: 11 Global Step: 112590 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:35:04,181-Speed 129.55 samples/sec Loss 2.7983 Epoch: 11 Global Step: 112595 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:35:23,928-Speed 129.65 samples/sec Loss 2.7425 Epoch: 11 Global Step: 112600 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:35:43,770-Speed 129.03 samples/sec Loss 2.7467 Epoch: 11 Global Step: 112605 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:36:03,591-Speed 129.17 samples/sec Loss 3.1086 Epoch: 11 Global Step: 112610 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:36:23,302-Speed 129.88 samples/sec Loss 3.1001 Epoch: 11 Global Step: 112615 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:36:43,047-Speed 129.66 samples/sec Loss 2.8915 Epoch: 11 Global Step: 112620 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:37:02,810-Speed 129.54 samples/sec Loss 2.8817 Epoch: 11 Global Step: 112625 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:37:22,869-Speed 127.63 samples/sec Loss 2.8942 Epoch: 11 Global Step: 112630 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:37:42,643-Speed 129.47 samples/sec Loss 2.8000 Epoch: 11 Global Step: 112635 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:38:02,394-Speed 129.62 samples/sec Loss 2.6915 Epoch: 11 Global Step: 112640 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:38:22,117-Speed 129.80 samples/sec Loss 2.5261 Epoch: 11 Global Step: 112645 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:38:41,867-Speed 129.63 samples/sec Loss 2.7303 Epoch: 11 Global Step: 112650 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:39:01,609-Speed 129.68 samples/sec Loss 2.6000 Epoch: 11 Global Step: 112655 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:39:21,334-Speed 129.79 samples/sec Loss 2.7126 Epoch: 11 Global Step: 112660 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:39:41,073-Speed 129.70 samples/sec Loss 2.8734 Epoch: 11 Global Step: 112665 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:40:00,912-Speed 129.05 samples/sec Loss 2.9242 Epoch: 11 Global Step: 112670 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:40:21,001-Speed 127.44 samples/sec Loss 2.8744 Epoch: 11 Global Step: 112675 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:40:40,728-Speed 129.78 samples/sec Loss 2.5772 Epoch: 11 Global Step: 112680 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:41:00,460-Speed 129.74 samples/sec Loss 2.8842 Epoch: 11 Global Step: 112685 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:41:20,177-Speed 129.85 samples/sec Loss 2.5968 Epoch: 11 Global Step: 112690 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:41:39,918-Speed 129.69 samples/sec Loss 2.7692 Epoch: 11 Global Step: 112695 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:41:59,614-Speed 129.98 samples/sec Loss 2.7827 Epoch: 11 Global Step: 112700 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:42:19,329-Speed 129.86 samples/sec Loss 2.9271 Epoch: 11 Global Step: 112705 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:42:39,047-Speed 129.83 samples/sec Loss 2.9661 Epoch: 11 Global Step: 112710 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:42:58,777-Speed 129.77 samples/sec Loss 2.8241 Epoch: 11 Global Step: 112715 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:43:18,759-Speed 128.12 samples/sec Loss 3.0158 Epoch: 11 Global Step: 112720 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:43:38,589-Speed 129.10 samples/sec Loss 2.7557 Epoch: 11 Global Step: 112725 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:43:58,365-Speed 129.46 samples/sec Loss 2.8431 Epoch: 11 Global Step: 112730 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:44:18,067-Speed 129.94 samples/sec Loss 2.9012 Epoch: 11 Global Step: 112735 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:44:37,816-Speed 129.63 samples/sec Loss 2.7570 Epoch: 11 Global Step: 112740 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:44:57,534-Speed 129.84 samples/sec Loss 2.9987 Epoch: 11 Global Step: 112745 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:45:17,249-Speed 129.86 samples/sec Loss 2.8236 Epoch: 11 Global Step: 112750 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:45:36,964-Speed 129.86 samples/sec Loss 2.8744 Epoch: 11 Global Step: 112755 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:45:57,107-Speed 127.10 samples/sec Loss 2.9176 Epoch: 11 Global Step: 112760 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:46:17,300-Speed 126.78 samples/sec Loss 2.7722 Epoch: 11 Global Step: 112765 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:46:37,013-Speed 129.87 samples/sec Loss 2.7745 Epoch: 11 Global Step: 112770 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:46:56,700-Speed 130.04 samples/sec Loss 2.7998 Epoch: 11 Global Step: 112775 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:47:16,459-Speed 129.57 samples/sec Loss 2.8466 Epoch: 11 Global Step: 112780 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:47:36,156-Speed 129.98 samples/sec Loss 2.8093 Epoch: 11 Global Step: 112785 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:47:55,860-Speed 129.93 samples/sec Loss 2.8040 Epoch: 11 Global Step: 112790 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:48:15,671-Speed 129.22 samples/sec Loss 2.9298 Epoch: 11 Global Step: 112795 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:48:35,242-Speed 130.81 samples/sec Loss 2.9117 Epoch: 11 Global Step: 112800 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:48:55,613-Speed 125.68 samples/sec Loss 2.7540 Epoch: 11 Global Step: 112805 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:49:15,486-Speed 128.82 samples/sec Loss 2.7234 Epoch: 11 Global Step: 112810 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:49:35,288-Speed 129.29 samples/sec Loss 3.0087 Epoch: 11 Global Step: 112815 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:49:54,995-Speed 129.91 samples/sec Loss 2.7554 Epoch: 11 Global Step: 112820 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:50:14,694-Speed 129.96 samples/sec Loss 2.7292 Epoch: 11 Global Step: 112825 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:50:34,363-Speed 130.16 samples/sec Loss 3.1471 Epoch: 11 Global Step: 112830 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:50:54,000-Speed 130.37 samples/sec Loss 2.8368 Epoch: 11 Global Step: 112835 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:51:13,694-Speed 130.00 samples/sec Loss 2.8237 Epoch: 11 Global Step: 112840 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:51:33,439-Speed 129.66 samples/sec Loss 3.0274 Epoch: 11 Global Step: 112845 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:51:53,054-Speed 130.52 samples/sec Loss 2.7587 Epoch: 11 Global Step: 112850 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:52:12,877-Speed 129.15 samples/sec Loss 3.0980 Epoch: 11 Global Step: 112855 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:52:32,609-Speed 129.75 samples/sec Loss 2.9733 Epoch: 11 Global Step: 112860 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:52:52,283-Speed 130.13 samples/sec Loss 2.9264 Epoch: 11 Global Step: 112865 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:53:11,961-Speed 130.10 samples/sec Loss 2.8755 Epoch: 11 Global Step: 112870 Fp16 Grad Scale: 8192 Required: 162 hours Training: 2021-06-12 08:53:31,591-Speed 130.43 samples/sec Loss 2.7654 Epoch: 11 Global Step: 112875 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:53:51,145-Speed 130.93 samples/sec Loss 2.7684 Epoch: 11 Global Step: 112880 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:54:10,848-Speed 129.94 samples/sec Loss 2.8699 Epoch: 11 Global Step: 112885 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:54:30,564-Speed 129.85 samples/sec Loss 2.8011 Epoch: 11 Global Step: 112890 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:54:50,244-Speed 130.09 samples/sec Loss 3.0368 Epoch: 11 Global Step: 112895 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:55:10,099-Speed 128.95 samples/sec Loss 2.8362 Epoch: 11 Global Step: 112900 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:55:29,776-Speed 130.10 samples/sec Loss 2.8708 Epoch: 11 Global Step: 112905 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:55:49,510-Speed 129.73 samples/sec Loss 2.7173 Epoch: 11 Global Step: 112910 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:56:09,188-Speed 130.10 samples/sec Loss 2.8249 Epoch: 11 Global Step: 112915 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:56:28,785-Speed 130.64 samples/sec Loss 2.9187 Epoch: 11 Global Step: 112920 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:56:48,622-Speed 129.06 samples/sec Loss 2.8017 Epoch: 11 Global Step: 112925 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:57:08,302-Speed 130.09 samples/sec Loss 2.8789 Epoch: 11 Global Step: 112930 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:57:27,922-Speed 130.49 samples/sec Loss 2.8284 Epoch: 11 Global Step: 112935 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:57:47,560-Speed 130.36 samples/sec Loss 2.5860 Epoch: 11 Global Step: 112940 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:58:07,487-Speed 128.48 samples/sec Loss 2.8660 Epoch: 11 Global Step: 112945 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:58:27,009-Speed 131.14 samples/sec Loss 2.9204 Epoch: 11 Global Step: 112950 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:58:46,791-Speed 129.42 samples/sec Loss 2.6684 Epoch: 11 Global Step: 112955 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:59:06,504-Speed 129.88 samples/sec Loss 2.8967 Epoch: 11 Global Step: 112960 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:59:26,074-Speed 130.81 samples/sec Loss 3.0428 Epoch: 11 Global Step: 112965 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 08:59:45,797-Speed 129.81 samples/sec Loss 2.9328 Epoch: 11 Global Step: 112970 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:00:05,441-Speed 130.32 samples/sec Loss 2.7000 Epoch: 11 Global Step: 112975 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:00:25,123-Speed 130.08 samples/sec Loss 2.8810 Epoch: 11 Global Step: 112980 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:00:44,764-Speed 130.35 samples/sec Loss 2.8770 Epoch: 11 Global Step: 112985 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:01:04,641-Speed 128.80 samples/sec Loss 2.8556 Epoch: 11 Global Step: 112990 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:01:24,228-Speed 130.71 samples/sec Loss 2.8777 Epoch: 11 Global Step: 112995 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:01:44,044-Speed 129.20 samples/sec Loss 2.9868 Epoch: 11 Global Step: 113000 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:02:03,680-Speed 130.38 samples/sec Loss 2.9342 Epoch: 11 Global Step: 113005 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:02:23,235-Speed 130.92 samples/sec Loss 2.8044 Epoch: 11 Global Step: 113010 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:02:42,955-Speed 129.82 samples/sec Loss 3.1167 Epoch: 11 Global Step: 113015 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:03:02,585-Speed 130.43 samples/sec Loss 2.8446 Epoch: 11 Global Step: 113020 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:03:22,271-Speed 130.05 samples/sec Loss 2.4888 Epoch: 11 Global Step: 113025 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:03:41,901-Speed 130.42 samples/sec Loss 2.7824 Epoch: 11 Global Step: 113030 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:04:01,752-Speed 128.97 samples/sec Loss 2.6822 Epoch: 11 Global Step: 113035 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:04:21,505-Speed 129.61 samples/sec Loss 2.7937 Epoch: 11 Global Step: 113040 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:04:41,273-Speed 129.51 samples/sec Loss 2.8666 Epoch: 11 Global Step: 113045 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:05:00,852-Speed 130.77 samples/sec Loss 2.8520 Epoch: 11 Global Step: 113050 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:05:20,537-Speed 130.05 samples/sec Loss 2.7908 Epoch: 11 Global Step: 113055 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:05:40,209-Speed 130.14 samples/sec Loss 2.9814 Epoch: 11 Global Step: 113060 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:05:59,862-Speed 130.26 samples/sec Loss 2.7508 Epoch: 11 Global Step: 113065 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:06:19,529-Speed 130.18 samples/sec Loss 3.0158 Epoch: 11 Global Step: 113070 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:06:39,133-Speed 130.59 samples/sec Loss 2.6841 Epoch: 11 Global Step: 113075 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:06:59,085-Speed 128.32 samples/sec Loss 2.6689 Epoch: 11 Global Step: 113080 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:07:18,945-Speed 128.91 samples/sec Loss 2.9086 Epoch: 11 Global Step: 113085 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:07:38,496-Speed 130.95 samples/sec Loss 3.0279 Epoch: 11 Global Step: 113090 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:07:58,061-Speed 130.85 samples/sec Loss 2.9202 Epoch: 11 Global Step: 113095 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:08:17,783-Speed 129.81 samples/sec Loss 2.8753 Epoch: 11 Global Step: 113100 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:08:37,446-Speed 130.20 samples/sec Loss 2.8584 Epoch: 11 Global Step: 113105 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:08:57,124-Speed 130.10 samples/sec Loss 2.8072 Epoch: 11 Global Step: 113110 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:09:16,746-Speed 130.47 samples/sec Loss 2.6340 Epoch: 11 Global Step: 113115 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:09:36,413-Speed 130.17 samples/sec Loss 2.8114 Epoch: 11 Global Step: 113120 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:09:56,332-Speed 128.53 samples/sec Loss 2.8961 Epoch: 11 Global Step: 113125 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:10:16,158-Speed 129.13 samples/sec Loss 2.8967 Epoch: 11 Global Step: 113130 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:10:35,766-Speed 130.56 samples/sec Loss 2.6995 Epoch: 11 Global Step: 113135 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:10:55,445-Speed 130.10 samples/sec Loss 2.8557 Epoch: 11 Global Step: 113140 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:11:15,081-Speed 130.38 samples/sec Loss 2.9548 Epoch: 11 Global Step: 113145 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:11:35,031-Speed 128.33 samples/sec Loss 2.8355 Epoch: 11 Global Step: 113150 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:11:54,712-Speed 130.08 samples/sec Loss 2.7115 Epoch: 11 Global Step: 113155 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:12:14,331-Speed 130.49 samples/sec Loss 2.7718 Epoch: 11 Global Step: 113160 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:12:34,116-Speed 129.40 samples/sec Loss 2.8961 Epoch: 11 Global Step: 113165 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:12:53,979-Speed 128.89 samples/sec Loss 2.8150 Epoch: 11 Global Step: 113170 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:13:13,942-Speed 128.24 samples/sec Loss 2.7549 Epoch: 11 Global Step: 113175 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:13:33,595-Speed 130.27 samples/sec Loss 2.8312 Epoch: 11 Global Step: 113180 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:13:53,304-Speed 129.89 samples/sec Loss 2.8499 Epoch: 11 Global Step: 113185 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:14:12,984-Speed 130.09 samples/sec Loss 2.8979 Epoch: 11 Global Step: 113190 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:14:32,637-Speed 130.27 samples/sec Loss 2.9854 Epoch: 11 Global Step: 113195 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:14:52,216-Speed 130.76 samples/sec Loss 2.8174 Epoch: 11 Global Step: 113200 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:15:12,070-Speed 128.95 samples/sec Loss 2.8060 Epoch: 11 Global Step: 113205 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:15:31,804-Speed 129.74 samples/sec Loss 3.0315 Epoch: 11 Global Step: 113210 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:15:51,849-Speed 127.72 samples/sec Loss 2.8353 Epoch: 11 Global Step: 113215 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:16:11,355-Speed 131.25 samples/sec Loss 2.8546 Epoch: 11 Global Step: 113220 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:16:31,400-Speed 127.72 samples/sec Loss 2.8690 Epoch: 11 Global Step: 113225 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:16:51,036-Speed 130.38 samples/sec Loss 2.6953 Epoch: 11 Global Step: 113230 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:17:10,695-Speed 130.23 samples/sec Loss 2.8146 Epoch: 11 Global Step: 113235 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:17:30,371-Speed 130.12 samples/sec Loss 2.8247 Epoch: 11 Global Step: 113240 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:17:49,918-Speed 130.97 samples/sec Loss 2.7672 Epoch: 11 Global Step: 113245 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:18:09,654-Speed 129.72 samples/sec Loss 2.7283 Epoch: 11 Global Step: 113250 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:18:29,335-Speed 130.08 samples/sec Loss 2.7417 Epoch: 11 Global Step: 113255 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:18:49,253-Speed 128.53 samples/sec Loss 2.9699 Epoch: 11 Global Step: 113260 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:19:08,778-Speed 131.12 samples/sec Loss 2.9555 Epoch: 11 Global Step: 113265 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:19:28,639-Speed 128.91 samples/sec Loss 2.9318 Epoch: 11 Global Step: 113270 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:19:48,298-Speed 130.23 samples/sec Loss 2.9727 Epoch: 11 Global Step: 113275 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:20:07,874-Speed 130.78 samples/sec Loss 2.9119 Epoch: 11 Global Step: 113280 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:20:27,537-Speed 130.20 samples/sec Loss 2.7831 Epoch: 11 Global Step: 113285 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:20:47,350-Speed 129.22 samples/sec Loss 2.8207 Epoch: 11 Global Step: 113290 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:21:07,069-Speed 129.83 samples/sec Loss 2.9669 Epoch: 11 Global Step: 113295 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:21:26,762-Speed 130.00 samples/sec Loss 2.9186 Epoch: 11 Global Step: 113300 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:21:46,631-Speed 128.85 samples/sec Loss 2.8205 Epoch: 11 Global Step: 113305 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:22:06,170-Speed 131.03 samples/sec Loss 2.8266 Epoch: 11 Global Step: 113310 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:22:25,950-Speed 129.43 samples/sec Loss 2.8893 Epoch: 11 Global Step: 113315 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:22:45,578-Speed 130.43 samples/sec Loss 3.0618 Epoch: 11 Global Step: 113320 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:23:05,142-Speed 130.86 samples/sec Loss 2.8755 Epoch: 11 Global Step: 113325 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:23:24,946-Speed 129.27 samples/sec Loss 2.9917 Epoch: 11 Global Step: 113330 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:23:44,586-Speed 130.35 samples/sec Loss 2.7966 Epoch: 11 Global Step: 113335 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:24:04,352-Speed 129.53 samples/sec Loss 3.0816 Epoch: 11 Global Step: 113340 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:24:23,941-Speed 130.69 samples/sec Loss 2.7578 Epoch: 11 Global Step: 113345 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:24:43,812-Speed 128.84 samples/sec Loss 2.9824 Epoch: 11 Global Step: 113350 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:25:03,526-Speed 129.86 samples/sec Loss 2.9186 Epoch: 11 Global Step: 113355 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:25:23,344-Speed 129.18 samples/sec Loss 2.9077 Epoch: 11 Global Step: 113360 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:25:42,913-Speed 130.83 samples/sec Loss 2.7652 Epoch: 11 Global Step: 113365 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:26:02,581-Speed 130.17 samples/sec Loss 2.7883 Epoch: 11 Global Step: 113370 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:26:22,116-Speed 131.05 samples/sec Loss 2.8972 Epoch: 11 Global Step: 113375 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:26:41,829-Speed 129.87 samples/sec Loss 2.8378 Epoch: 11 Global Step: 113380 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:27:01,478-Speed 130.30 samples/sec Loss 2.6769 Epoch: 11 Global Step: 113385 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:27:21,033-Speed 130.92 samples/sec Loss 2.7283 Epoch: 11 Global Step: 113390 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:27:40,897-Speed 128.88 samples/sec Loss 3.0106 Epoch: 11 Global Step: 113395 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:28:00,623-Speed 129.78 samples/sec Loss 2.9864 Epoch: 11 Global Step: 113400 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:28:20,317-Speed 130.00 samples/sec Loss 2.7624 Epoch: 11 Global Step: 113405 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:28:40,038-Speed 129.81 samples/sec Loss 2.7077 Epoch: 11 Global Step: 113410 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:28:59,595-Speed 130.91 samples/sec Loss 2.8093 Epoch: 11 Global Step: 113415 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:29:19,198-Speed 130.60 samples/sec Loss 2.7888 Epoch: 11 Global Step: 113420 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:29:38,923-Speed 129.79 samples/sec Loss 2.9523 Epoch: 11 Global Step: 113425 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:29:58,592-Speed 130.16 samples/sec Loss 2.7195 Epoch: 11 Global Step: 113430 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:30:18,282-Speed 130.03 samples/sec Loss 2.7914 Epoch: 11 Global Step: 113435 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:30:38,037-Speed 129.59 samples/sec Loss 2.6606 Epoch: 11 Global Step: 113440 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:30:57,802-Speed 129.53 samples/sec Loss 2.8846 Epoch: 11 Global Step: 113445 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:31:17,411-Speed 130.56 samples/sec Loss 2.6882 Epoch: 11 Global Step: 113450 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:31:37,154-Speed 129.67 samples/sec Loss 2.7433 Epoch: 11 Global Step: 113455 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:31:56,754-Speed 130.62 samples/sec Loss 2.9378 Epoch: 11 Global Step: 113460 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:32:16,448-Speed 130.00 samples/sec Loss 2.9839 Epoch: 11 Global Step: 113465 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:32:36,469-Speed 127.87 samples/sec Loss 2.8478 Epoch: 11 Global Step: 113470 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:32:56,043-Speed 130.80 samples/sec Loss 3.0185 Epoch: 11 Global Step: 113475 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:33:15,698-Speed 130.25 samples/sec Loss 2.9444 Epoch: 11 Global Step: 113480 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:33:35,697-Speed 128.02 samples/sec Loss 2.6747 Epoch: 11 Global Step: 113485 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:33:55,496-Speed 129.30 samples/sec Loss 2.7528 Epoch: 11 Global Step: 113490 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:34:15,165-Speed 130.16 samples/sec Loss 3.0202 Epoch: 11 Global Step: 113495 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:34:34,824-Speed 130.23 samples/sec Loss 3.0056 Epoch: 11 Global Step: 113500 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:34:54,409-Speed 130.72 samples/sec Loss 2.8341 Epoch: 11 Global Step: 113505 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:35:14,078-Speed 130.16 samples/sec Loss 3.0654 Epoch: 11 Global Step: 113510 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:35:33,670-Speed 130.68 samples/sec Loss 2.8206 Epoch: 11 Global Step: 113515 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:35:53,359-Speed 130.02 samples/sec Loss 2.9304 Epoch: 11 Global Step: 113520 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:36:13,158-Speed 129.31 samples/sec Loss 2.9185 Epoch: 11 Global Step: 113525 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:36:33,331-Speed 126.91 samples/sec Loss 3.0447 Epoch: 11 Global Step: 113530 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:36:52,832-Speed 131.28 samples/sec Loss 2.7848 Epoch: 11 Global Step: 113535 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:37:12,556-Speed 129.80 samples/sec Loss 2.8984 Epoch: 11 Global Step: 113540 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:37:32,158-Speed 130.61 samples/sec Loss 2.7833 Epoch: 11 Global Step: 113545 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:37:51,774-Speed 130.51 samples/sec Loss 2.8231 Epoch: 11 Global Step: 113550 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:38:11,443-Speed 130.16 samples/sec Loss 3.0251 Epoch: 11 Global Step: 113555 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:38:31,130-Speed 130.04 samples/sec Loss 2.8453 Epoch: 11 Global Step: 113560 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:38:50,812-Speed 130.07 samples/sec Loss 2.7843 Epoch: 11 Global Step: 113565 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:39:10,358-Speed 130.98 samples/sec Loss 2.9776 Epoch: 11 Global Step: 113570 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:39:30,394-Speed 127.78 samples/sec Loss 2.9705 Epoch: 11 Global Step: 113575 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:39:49,883-Speed 131.36 samples/sec Loss 2.7971 Epoch: 11 Global Step: 113580 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:40:09,713-Speed 129.11 samples/sec Loss 2.8678 Epoch: 11 Global Step: 113585 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:40:29,288-Speed 130.79 samples/sec Loss 2.8193 Epoch: 11 Global Step: 113590 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:40:48,962-Speed 130.13 samples/sec Loss 3.0196 Epoch: 11 Global Step: 113595 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:41:08,611-Speed 130.30 samples/sec Loss 3.0483 Epoch: 11 Global Step: 113600 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:41:28,156-Speed 130.98 samples/sec Loss 2.8737 Epoch: 11 Global Step: 113605 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:41:47,815-Speed 130.23 samples/sec Loss 2.9031 Epoch: 11 Global Step: 113610 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:42:07,365-Speed 130.95 samples/sec Loss 2.8541 Epoch: 11 Global Step: 113615 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:42:27,287-Speed 128.51 samples/sec Loss 2.8615 Epoch: 11 Global Step: 113620 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:42:46,998-Speed 129.88 samples/sec Loss 3.0395 Epoch: 11 Global Step: 113625 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:43:06,951-Speed 128.31 samples/sec Loss 2.8353 Epoch: 11 Global Step: 113630 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:43:26,774-Speed 129.15 samples/sec Loss 3.0010 Epoch: 11 Global Step: 113635 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:43:46,584-Speed 129.23 samples/sec Loss 2.8643 Epoch: 11 Global Step: 113640 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:44:06,229-Speed 130.32 samples/sec Loss 2.8230 Epoch: 11 Global Step: 113645 Fp16 Grad Scale: 8192 Required: 161 hours Training: 2021-06-12 09:44:25,879-Speed 130.29 samples/sec Loss 2.7096 Epoch: 11 Global Step: 113650 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:44:45,429-Speed 130.96 samples/sec Loss 2.8133 Epoch: 11 Global Step: 113655 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:45:05,149-Speed 129.82 samples/sec Loss 2.8416 Epoch: 11 Global Step: 113660 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:45:25,009-Speed 128.91 samples/sec Loss 3.0034 Epoch: 11 Global Step: 113665 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:45:44,810-Speed 129.29 samples/sec Loss 3.1103 Epoch: 11 Global Step: 113670 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:46:04,324-Speed 131.20 samples/sec Loss 2.9250 Epoch: 11 Global Step: 113675 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:46:24,061-Speed 129.71 samples/sec Loss 2.8855 Epoch: 11 Global Step: 113680 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:46:43,741-Speed 130.08 samples/sec Loss 3.0396 Epoch: 11 Global Step: 113685 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:47:03,281-Speed 131.02 samples/sec Loss 2.7421 Epoch: 11 Global Step: 113690 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:47:22,991-Speed 129.89 samples/sec Loss 2.9870 Epoch: 11 Global Step: 113695 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:47:42,710-Speed 129.84 samples/sec Loss 2.9428 Epoch: 11 Global Step: 113700 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:48:02,380-Speed 130.15 samples/sec Loss 2.9094 Epoch: 11 Global Step: 113705 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:48:22,295-Speed 128.55 samples/sec Loss 2.8753 Epoch: 11 Global Step: 113710 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:48:41,987-Speed 130.01 samples/sec Loss 2.8590 Epoch: 11 Global Step: 113715 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:49:01,693-Speed 129.92 samples/sec Loss 2.8408 Epoch: 11 Global Step: 113720 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:49:21,344-Speed 130.28 samples/sec Loss 2.9315 Epoch: 11 Global Step: 113725 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:49:40,987-Speed 130.34 samples/sec Loss 2.5993 Epoch: 11 Global Step: 113730 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:50:00,653-Speed 130.18 samples/sec Loss 2.8469 Epoch: 11 Global Step: 113735 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:50:20,178-Speed 131.12 samples/sec Loss 2.7337 Epoch: 11 Global Step: 113740 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:50:39,893-Speed 129.85 samples/sec Loss 2.7419 Epoch: 11 Global Step: 113745 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:50:59,549-Speed 130.25 samples/sec Loss 2.8126 Epoch: 11 Global Step: 113750 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:51:19,519-Speed 128.20 samples/sec Loss 2.9246 Epoch: 11 Global Step: 113755 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:51:39,227-Speed 129.90 samples/sec Loss 2.6518 Epoch: 11 Global Step: 113760 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:51:58,880-Speed 130.27 samples/sec Loss 2.7730 Epoch: 11 Global Step: 113765 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:52:18,552-Speed 130.14 samples/sec Loss 2.8558 Epoch: 11 Global Step: 113770 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:52:38,171-Speed 130.49 samples/sec Loss 2.8560 Epoch: 11 Global Step: 113775 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:52:57,826-Speed 130.25 samples/sec Loss 2.9820 Epoch: 11 Global Step: 113780 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:53:17,538-Speed 129.88 samples/sec Loss 2.9745 Epoch: 11 Global Step: 113785 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:53:37,197-Speed 130.23 samples/sec Loss 2.9181 Epoch: 11 Global Step: 113790 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:53:56,877-Speed 130.09 samples/sec Loss 2.8414 Epoch: 11 Global Step: 113795 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:54:16,774-Speed 128.67 samples/sec Loss 3.1279 Epoch: 11 Global Step: 113800 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:54:36,300-Speed 131.12 samples/sec Loss 2.8943 Epoch: 11 Global Step: 113805 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:54:56,120-Speed 129.17 samples/sec Loss 2.7584 Epoch: 11 Global Step: 113810 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:55:15,782-Speed 130.20 samples/sec Loss 2.8537 Epoch: 11 Global Step: 113815 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:55:35,443-Speed 130.21 samples/sec Loss 3.0092 Epoch: 11 Global Step: 113820 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:55:55,047-Speed 130.59 samples/sec Loss 2.8775 Epoch: 11 Global Step: 113825 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:56:14,703-Speed 130.25 samples/sec Loss 2.9549 Epoch: 11 Global Step: 113830 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:56:34,436-Speed 129.73 samples/sec Loss 2.9855 Epoch: 11 Global Step: 113835 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:56:54,036-Speed 130.62 samples/sec Loss 3.0488 Epoch: 11 Global Step: 113840 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:57:13,898-Speed 128.90 samples/sec Loss 2.8433 Epoch: 11 Global Step: 113845 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:57:33,766-Speed 128.86 samples/sec Loss 2.9608 Epoch: 11 Global Step: 113850 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:57:53,428-Speed 130.21 samples/sec Loss 2.8150 Epoch: 11 Global Step: 113855 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:58:12,987-Speed 130.89 samples/sec Loss 2.9075 Epoch: 11 Global Step: 113860 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:58:32,638-Speed 130.28 samples/sec Loss 3.0298 Epoch: 11 Global Step: 113865 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:58:52,421-Speed 129.41 samples/sec Loss 2.8312 Epoch: 11 Global Step: 113870 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:59:12,084-Speed 130.21 samples/sec Loss 3.0189 Epoch: 11 Global Step: 113875 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:59:31,730-Speed 130.31 samples/sec Loss 3.0382 Epoch: 11 Global Step: 113880 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 09:59:51,263-Speed 131.07 samples/sec Loss 2.8702 Epoch: 11 Global Step: 113885 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:00:11,129-Speed 128.87 samples/sec Loss 2.8881 Epoch: 11 Global Step: 113890 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:00:31,007-Speed 128.79 samples/sec Loss 2.7664 Epoch: 11 Global Step: 113895 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:00:50,663-Speed 130.25 samples/sec Loss 2.8656 Epoch: 11 Global Step: 113900 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:01:10,312-Speed 130.30 samples/sec Loss 2.9366 Epoch: 11 Global Step: 113905 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:01:29,925-Speed 130.53 samples/sec Loss 2.9779 Epoch: 11 Global Step: 113910 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:01:49,595-Speed 130.15 samples/sec Loss 2.9329 Epoch: 11 Global Step: 113915 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:02:09,229-Speed 130.39 samples/sec Loss 2.9515 Epoch: 11 Global Step: 113920 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:02:28,890-Speed 130.22 samples/sec Loss 3.0045 Epoch: 11 Global Step: 113925 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:02:48,547-Speed 130.24 samples/sec Loss 3.0347 Epoch: 11 Global Step: 113930 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:03:08,639-Speed 127.42 samples/sec Loss 2.8926 Epoch: 11 Global Step: 113935 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:03:28,168-Speed 131.10 samples/sec Loss 3.0000 Epoch: 11 Global Step: 113940 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:03:47,943-Speed 129.46 samples/sec Loss 2.8850 Epoch: 11 Global Step: 113945 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:04:07,574-Speed 130.41 samples/sec Loss 3.0073 Epoch: 11 Global Step: 113950 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:04:27,261-Speed 130.05 samples/sec Loss 2.6908 Epoch: 11 Global Step: 113955 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:04:46,918-Speed 130.24 samples/sec Loss 2.6834 Epoch: 11 Global Step: 113960 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:05:06,564-Speed 130.32 samples/sec Loss 2.7528 Epoch: 11 Global Step: 113965 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:05:26,095-Speed 131.08 samples/sec Loss 3.1677 Epoch: 11 Global Step: 113970 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:05:45,712-Speed 130.51 samples/sec Loss 2.9099 Epoch: 11 Global Step: 113975 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:06:05,828-Speed 127.27 samples/sec Loss 2.9964 Epoch: 11 Global Step: 113980 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:06:25,348-Speed 131.15 samples/sec Loss 2.8257 Epoch: 11 Global Step: 113985 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:06:45,349-Speed 128.00 samples/sec Loss 2.9633 Epoch: 11 Global Step: 113990 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:07:05,033-Speed 130.06 samples/sec Loss 2.9523 Epoch: 11 Global Step: 113995 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:07:24,984-Speed 128.32 samples/sec Loss 2.9600 Epoch: 11 Global Step: 114000 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:07:44,602-Speed 130.50 samples/sec Loss 3.0142 Epoch: 11 Global Step: 114005 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:08:04,166-Speed 130.86 samples/sec Loss 2.8745 Epoch: 11 Global Step: 114010 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:08:23,809-Speed 130.33 samples/sec Loss 2.9032 Epoch: 11 Global Step: 114015 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:08:43,522-Speed 129.88 samples/sec Loss 2.9364 Epoch: 11 Global Step: 114020 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:09:03,561-Speed 127.75 samples/sec Loss 3.0369 Epoch: 11 Global Step: 114025 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:09:23,076-Speed 131.19 samples/sec Loss 2.9650 Epoch: 11 Global Step: 114030 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:09:42,767-Speed 130.02 samples/sec Loss 2.8322 Epoch: 11 Global Step: 114035 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:10:02,603-Speed 129.06 samples/sec Loss 2.9746 Epoch: 11 Global Step: 114040 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:10:22,247-Speed 130.33 samples/sec Loss 2.8437 Epoch: 11 Global Step: 114045 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:10:41,922-Speed 130.13 samples/sec Loss 2.8088 Epoch: 11 Global Step: 114050 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:11:01,454-Speed 131.08 samples/sec Loss 2.7870 Epoch: 11 Global Step: 114055 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:11:21,106-Speed 130.27 samples/sec Loss 2.8507 Epoch: 11 Global Step: 114060 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:11:40,752-Speed 130.31 samples/sec Loss 3.0731 Epoch: 11 Global Step: 114065 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:12:00,606-Speed 128.95 samples/sec Loss 2.9549 Epoch: 11 Global Step: 114070 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:12:20,342-Speed 129.72 samples/sec Loss 3.0684 Epoch: 11 Global Step: 114075 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:12:40,078-Speed 129.72 samples/sec Loss 2.9582 Epoch: 11 Global Step: 114080 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:12:59,750-Speed 130.14 samples/sec Loss 2.8696 Epoch: 11 Global Step: 114085 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:13:19,403-Speed 130.27 samples/sec Loss 2.9760 Epoch: 11 Global Step: 114090 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:13:38,985-Speed 130.74 samples/sec Loss 2.9891 Epoch: 11 Global Step: 114095 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:13:58,671-Speed 130.05 samples/sec Loss 2.9513 Epoch: 11 Global Step: 114100 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:14:18,356-Speed 130.05 samples/sec Loss 2.8850 Epoch: 11 Global Step: 114105 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:14:37,976-Speed 130.48 samples/sec Loss 2.8559 Epoch: 11 Global Step: 114110 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:14:57,813-Speed 129.06 samples/sec Loss 2.9570 Epoch: 11 Global Step: 114115 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:15:17,502-Speed 130.03 samples/sec Loss 3.0231 Epoch: 11 Global Step: 114120 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:15:37,193-Speed 130.02 samples/sec Loss 2.9225 Epoch: 11 Global Step: 114125 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:15:56,851-Speed 130.23 samples/sec Loss 2.8721 Epoch: 11 Global Step: 114130 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:16:16,506-Speed 130.25 samples/sec Loss 3.0803 Epoch: 11 Global Step: 114135 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:16:36,027-Speed 131.15 samples/sec Loss 3.1978 Epoch: 11 Global Step: 114140 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:16:55,622-Speed 130.65 samples/sec Loss 2.8979 Epoch: 11 Global Step: 114145 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:17:15,320-Speed 129.97 samples/sec Loss 2.8314 Epoch: 11 Global Step: 114150 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:17:34,983-Speed 130.20 samples/sec Loss 2.9695 Epoch: 11 Global Step: 114155 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:17:54,967-Speed 128.11 samples/sec Loss 2.9821 Epoch: 11 Global Step: 114160 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:18:14,453-Speed 131.38 samples/sec Loss 3.1160 Epoch: 11 Global Step: 114165 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:18:34,320-Speed 128.87 samples/sec Loss 3.0381 Epoch: 11 Global Step: 114170 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:18:53,983-Speed 130.20 samples/sec Loss 2.8846 Epoch: 11 Global Step: 114175 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:19:13,620-Speed 130.37 samples/sec Loss 2.8140 Epoch: 11 Global Step: 114180 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:19:33,265-Speed 130.32 samples/sec Loss 2.7187 Epoch: 11 Global Step: 114185 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:19:52,924-Speed 130.23 samples/sec Loss 2.9225 Epoch: 11 Global Step: 114190 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:20:12,566-Speed 130.34 samples/sec Loss 2.8960 Epoch: 11 Global Step: 114195 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:20:32,221-Speed 130.26 samples/sec Loss 3.0060 Epoch: 11 Global Step: 114200 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:20:52,186-Speed 128.23 samples/sec Loss 2.8511 Epoch: 11 Global Step: 114205 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:21:11,847-Speed 130.22 samples/sec Loss 2.9355 Epoch: 11 Global Step: 114210 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:21:31,658-Speed 129.23 samples/sec Loss 3.0542 Epoch: 11 Global Step: 114215 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:21:51,295-Speed 130.37 samples/sec Loss 2.8168 Epoch: 11 Global Step: 114220 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:22:10,871-Speed 130.78 samples/sec Loss 2.8113 Epoch: 11 Global Step: 114225 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:22:30,514-Speed 130.33 samples/sec Loss 2.8261 Epoch: 11 Global Step: 114230 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:22:50,210-Speed 129.99 samples/sec Loss 2.8742 Epoch: 11 Global Step: 114235 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:23:09,861-Speed 130.28 samples/sec Loss 2.9775 Epoch: 11 Global Step: 114240 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:23:29,456-Speed 130.66 samples/sec Loss 2.8636 Epoch: 11 Global Step: 114245 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:23:49,382-Speed 128.48 samples/sec Loss 3.0604 Epoch: 11 Global Step: 114250 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:24:08,869-Speed 131.37 samples/sec Loss 2.7544 Epoch: 11 Global Step: 114255 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:24:28,802-Speed 128.44 samples/sec Loss 3.0056 Epoch: 11 Global Step: 114260 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:24:48,392-Speed 130.69 samples/sec Loss 2.7937 Epoch: 11 Global Step: 114265 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:25:07,966-Speed 130.79 samples/sec Loss 2.9532 Epoch: 11 Global Step: 114270 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:25:27,583-Speed 130.51 samples/sec Loss 2.8330 Epoch: 11 Global Step: 114275 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:25:47,233-Speed 130.28 samples/sec Loss 2.9206 Epoch: 11 Global Step: 114280 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:26:06,890-Speed 130.24 samples/sec Loss 2.9638 Epoch: 11 Global Step: 114285 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:26:26,596-Speed 129.92 samples/sec Loss 2.9387 Epoch: 11 Global Step: 114290 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:26:46,567-Speed 128.19 samples/sec Loss 2.9452 Epoch: 11 Global Step: 114295 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:27:06,046-Speed 131.43 samples/sec Loss 2.8874 Epoch: 11 Global Step: 114300 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:27:25,938-Speed 128.70 samples/sec Loss 2.9294 Epoch: 11 Global Step: 114305 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:27:45,519-Speed 130.75 samples/sec Loss 2.9033 Epoch: 11 Global Step: 114310 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:28:05,184-Speed 130.18 samples/sec Loss 2.9268 Epoch: 11 Global Step: 114315 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:28:24,866-Speed 130.08 samples/sec Loss 2.8840 Epoch: 11 Global Step: 114320 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:28:44,526-Speed 130.22 samples/sec Loss 2.9173 Epoch: 11 Global Step: 114325 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:29:04,189-Speed 130.20 samples/sec Loss 2.9695 Epoch: 11 Global Step: 114330 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:29:23,856-Speed 130.18 samples/sec Loss 3.1222 Epoch: 11 Global Step: 114335 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:29:43,865-Speed 127.95 samples/sec Loss 2.9708 Epoch: 11 Global Step: 114340 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:30:03,435-Speed 130.82 samples/sec Loss 2.9420 Epoch: 11 Global Step: 114345 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:30:23,264-Speed 129.11 samples/sec Loss 2.7130 Epoch: 11 Global Step: 114350 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:30:42,918-Speed 130.26 samples/sec Loss 3.1438 Epoch: 11 Global Step: 114355 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:31:02,514-Speed 130.65 samples/sec Loss 3.1421 Epoch: 11 Global Step: 114360 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:31:22,222-Speed 129.90 samples/sec Loss 2.8580 Epoch: 11 Global Step: 114365 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:31:41,896-Speed 130.13 samples/sec Loss 2.9958 Epoch: 11 Global Step: 114370 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:32:01,559-Speed 130.20 samples/sec Loss 2.8671 Epoch: 11 Global Step: 114375 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:32:21,083-Speed 131.13 samples/sec Loss 3.1015 Epoch: 11 Global Step: 114380 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:32:40,959-Speed 128.81 samples/sec Loss 3.0526 Epoch: 11 Global Step: 114385 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:33:00,531-Speed 130.80 samples/sec Loss 3.0023 Epoch: 11 Global Step: 114390 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:33:20,280-Speed 129.63 samples/sec Loss 2.8121 Epoch: 11 Global Step: 114395 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:33:39,880-Speed 130.62 samples/sec Loss 2.9227 Epoch: 11 Global Step: 114400 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:33:59,564-Speed 130.06 samples/sec Loss 3.0798 Epoch: 11 Global Step: 114405 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:34:19,222-Speed 130.23 samples/sec Loss 3.0415 Epoch: 11 Global Step: 114410 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:34:39,360-Speed 127.13 samples/sec Loss 2.8505 Epoch: 11 Global Step: 114415 Fp16 Grad Scale: 8192 Required: 160 hours Training: 2021-06-12 10:34:59,020-Speed 130.22 samples/sec Loss 2.7621 Epoch: 11 Global Step: 114420 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:35:18,608-Speed 130.71 samples/sec Loss 2.9161 Epoch: 11 Global Step: 114425 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:35:38,503-Speed 128.68 samples/sec Loss 2.9224 Epoch: 11 Global Step: 114430 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:35:58,229-Speed 129.79 samples/sec Loss 3.3627 Epoch: 11 Global Step: 114435 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:36:17,965-Speed 129.72 samples/sec Loss 2.9463 Epoch: 11 Global Step: 114440 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:36:37,521-Speed 130.92 samples/sec Loss 2.7628 Epoch: 11 Global Step: 114445 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:36:57,195-Speed 130.12 samples/sec Loss 2.7402 Epoch: 11 Global Step: 114450 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:37:16,773-Speed 130.77 samples/sec Loss 3.1245 Epoch: 11 Global Step: 114455 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:37:36,468-Speed 129.99 samples/sec Loss 3.0506 Epoch: 11 Global Step: 114460 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:37:56,082-Speed 130.52 samples/sec Loss 2.9959 Epoch: 11 Global Step: 114465 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:38:15,701-Speed 130.49 samples/sec Loss 2.9729 Epoch: 11 Global Step: 114470 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:38:35,841-Speed 127.12 samples/sec Loss 2.9109 Epoch: 11 Global Step: 114475 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:38:55,406-Speed 130.85 samples/sec Loss 2.9002 Epoch: 11 Global Step: 114480 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:39:15,212-Speed 129.26 samples/sec Loss 3.0616 Epoch: 11 Global Step: 114485 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:39:34,922-Speed 129.89 samples/sec Loss 3.0533 Epoch: 11 Global Step: 114490 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:39:54,592-Speed 130.16 samples/sec Loss 3.1008 Epoch: 11 Global Step: 114495 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:40:14,249-Speed 130.24 samples/sec Loss 3.1099 Epoch: 11 Global Step: 114500 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:40:33,891-Speed 130.34 samples/sec Loss 3.0581 Epoch: 11 Global Step: 114505 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:40:53,516-Speed 130.45 samples/sec Loss 2.9260 Epoch: 11 Global Step: 114510 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:41:13,137-Speed 130.48 samples/sec Loss 2.7062 Epoch: 11 Global Step: 114515 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:41:33,007-Speed 128.85 samples/sec Loss 3.0584 Epoch: 11 Global Step: 114520 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:41:52,848-Speed 129.04 samples/sec Loss 2.8747 Epoch: 11 Global Step: 114525 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:42:12,531-Speed 130.06 samples/sec Loss 2.9082 Epoch: 11 Global Step: 114530 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:42:32,199-Speed 130.17 samples/sec Loss 2.8706 Epoch: 11 Global Step: 114535 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:42:51,783-Speed 130.72 samples/sec Loss 3.1457 Epoch: 11 Global Step: 114540 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:43:11,407-Speed 130.46 samples/sec Loss 2.9994 Epoch: 11 Global Step: 114545 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:43:31,156-Speed 129.64 samples/sec Loss 3.1510 Epoch: 11 Global Step: 114550 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:43:50,859-Speed 129.93 samples/sec Loss 2.9724 Epoch: 11 Global Step: 114555 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:44:10,495-Speed 130.38 samples/sec Loss 2.8489 Epoch: 11 Global Step: 114560 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:44:30,399-Speed 128.63 samples/sec Loss 3.0560 Epoch: 11 Global Step: 114565 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:44:50,102-Speed 129.93 samples/sec Loss 3.0128 Epoch: 11 Global Step: 114570 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:45:09,789-Speed 130.04 samples/sec Loss 2.9106 Epoch: 11 Global Step: 114575 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:45:29,468-Speed 130.10 samples/sec Loss 3.0276 Epoch: 11 Global Step: 114580 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:45:49,147-Speed 130.09 samples/sec Loss 2.6813 Epoch: 11 Global Step: 114585 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:46:08,783-Speed 130.38 samples/sec Loss 3.0441 Epoch: 11 Global Step: 114590 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:46:28,344-Speed 130.88 samples/sec Loss 2.9673 Epoch: 11 Global Step: 114595 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:46:47,997-Speed 130.27 samples/sec Loss 2.8669 Epoch: 11 Global Step: 114600 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:47:07,701-Speed 129.93 samples/sec Loss 3.1408 Epoch: 11 Global Step: 114605 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:47:27,661-Speed 128.26 samples/sec Loss 3.2887 Epoch: 11 Global Step: 114610 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:47:47,169-Speed 131.24 samples/sec Loss 3.0347 Epoch: 11 Global Step: 114615 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:48:06,883-Speed 129.87 samples/sec Loss 3.0102 Epoch: 11 Global Step: 114620 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:48:26,548-Speed 130.19 samples/sec Loss 2.9198 Epoch: 11 Global Step: 114625 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:48:46,243-Speed 129.99 samples/sec Loss 3.0344 Epoch: 11 Global Step: 114630 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:49:05,908-Speed 130.18 samples/sec Loss 2.7500 Epoch: 11 Global Step: 114635 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:49:25,600-Speed 130.01 samples/sec Loss 2.7428 Epoch: 11 Global Step: 114640 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:49:45,247-Speed 130.31 samples/sec Loss 3.2226 Epoch: 11 Global Step: 114645 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:50:04,919-Speed 130.14 samples/sec Loss 3.0078 Epoch: 11 Global Step: 114650 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:50:24,868-Speed 128.34 samples/sec Loss 2.8949 Epoch: 11 Global Step: 114655 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:50:44,369-Speed 131.28 samples/sec Loss 2.9608 Epoch: 11 Global Step: 114660 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:51:04,250-Speed 128.78 samples/sec Loss 2.8963 Epoch: 11 Global Step: 114665 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:51:23,905-Speed 130.26 samples/sec Loss 3.1595 Epoch: 11 Global Step: 114670 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:51:43,562-Speed 130.24 samples/sec Loss 2.8773 Epoch: 11 Global Step: 114675 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:52:03,216-Speed 130.26 samples/sec Loss 2.9388 Epoch: 11 Global Step: 114680 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:52:22,892-Speed 130.12 samples/sec Loss 2.8107 Epoch: 11 Global Step: 114685 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:52:42,549-Speed 130.25 samples/sec Loss 2.8611 Epoch: 11 Global Step: 114690 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:53:02,107-Speed 130.90 samples/sec Loss 3.0285 Epoch: 11 Global Step: 114695 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:53:21,960-Speed 128.96 samples/sec Loss 3.0702 Epoch: 11 Global Step: 114700 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:53:41,651-Speed 130.01 samples/sec Loss 2.9894 Epoch: 11 Global Step: 114705 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:54:01,385-Speed 129.74 samples/sec Loss 2.9845 Epoch: 11 Global Step: 114710 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:54:20,995-Speed 130.55 samples/sec Loss 2.9534 Epoch: 11 Global Step: 114715 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:54:40,632-Speed 130.37 samples/sec Loss 2.8431 Epoch: 11 Global Step: 114720 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:55:00,332-Speed 129.95 samples/sec Loss 2.9602 Epoch: 11 Global Step: 114725 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:55:19,981-Speed 130.30 samples/sec Loss 2.8546 Epoch: 11 Global Step: 114730 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:55:39,632-Speed 130.28 samples/sec Loss 3.1667 Epoch: 11 Global Step: 114735 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:55:59,219-Speed 130.70 samples/sec Loss 3.1139 Epoch: 11 Global Step: 114740 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:56:19,070-Speed 128.97 samples/sec Loss 2.8931 Epoch: 11 Global Step: 114745 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:56:38,934-Speed 128.88 samples/sec Loss 2.9130 Epoch: 11 Global Step: 114750 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:56:58,570-Speed 130.38 samples/sec Loss 2.9899 Epoch: 11 Global Step: 114755 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:57:18,392-Speed 129.16 samples/sec Loss 2.9066 Epoch: 11 Global Step: 114760 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:57:38,105-Speed 129.87 samples/sec Loss 2.9922 Epoch: 11 Global Step: 114765 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:57:57,821-Speed 129.85 samples/sec Loss 2.7911 Epoch: 11 Global Step: 114770 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:58:17,481-Speed 130.22 samples/sec Loss 3.1712 Epoch: 11 Global Step: 114775 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:58:37,170-Speed 130.03 samples/sec Loss 3.0747 Epoch: 11 Global Step: 114780 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:58:56,754-Speed 130.72 samples/sec Loss 2.8051 Epoch: 11 Global Step: 114785 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:59:16,641-Speed 128.74 samples/sec Loss 2.8612 Epoch: 11 Global Step: 114790 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:59:36,334-Speed 130.00 samples/sec Loss 2.7756 Epoch: 11 Global Step: 114795 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 10:59:55,909-Speed 130.80 samples/sec Loss 2.9213 Epoch: 11 Global Step: 114800 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:00:15,579-Speed 130.15 samples/sec Loss 2.9780 Epoch: 11 Global Step: 114805 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:00:35,280-Speed 129.95 samples/sec Loss 2.9647 Epoch: 11 Global Step: 114810 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:00:54,928-Speed 130.30 samples/sec Loss 2.9773 Epoch: 11 Global Step: 114815 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:01:14,582-Speed 130.26 samples/sec Loss 2.9703 Epoch: 11 Global Step: 114820 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:01:34,404-Speed 129.16 samples/sec Loss 3.0173 Epoch: 11 Global Step: 114825 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:01:54,501-Speed 127.39 samples/sec Loss 3.1757 Epoch: 11 Global Step: 114830 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:02:14,486-Speed 128.10 samples/sec Loss 3.0283 Epoch: 11 Global Step: 114835 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:02:34,173-Speed 130.04 samples/sec Loss 2.8248 Epoch: 11 Global Step: 114840 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:02:53,706-Speed 131.07 samples/sec Loss 3.0477 Epoch: 11 Global Step: 114845 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:03:13,434-Speed 129.77 samples/sec Loss 3.0365 Epoch: 11 Global Step: 114850 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:03:33,103-Speed 130.16 samples/sec Loss 3.0162 Epoch: 11 Global Step: 114855 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:03:52,685-Speed 130.74 samples/sec Loss 3.0498 Epoch: 11 Global Step: 114860 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:04:12,518-Speed 129.08 samples/sec Loss 3.0656 Epoch: 11 Global Step: 114865 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:04:32,171-Speed 130.27 samples/sec Loss 2.9456 Epoch: 11 Global Step: 114870 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:04:51,865-Speed 130.00 samples/sec Loss 3.0561 Epoch: 11 Global Step: 114875 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:05:11,936-Speed 127.55 samples/sec Loss 2.9406 Epoch: 11 Global Step: 114880 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:05:31,454-Speed 131.17 samples/sec Loss 3.0780 Epoch: 11 Global Step: 114885 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:05:51,150-Speed 129.98 samples/sec Loss 2.8249 Epoch: 11 Global Step: 114890 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:06:10,851-Speed 129.95 samples/sec Loss 2.9802 Epoch: 11 Global Step: 114895 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:06:30,379-Speed 131.10 samples/sec Loss 3.0204 Epoch: 11 Global Step: 114900 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:06:50,057-Speed 130.10 samples/sec Loss 2.8630 Epoch: 11 Global Step: 114905 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:07:09,589-Speed 131.07 samples/sec Loss 2.9715 Epoch: 11 Global Step: 114910 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:07:29,272-Speed 130.07 samples/sec Loss 2.8565 Epoch: 11 Global Step: 114915 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:07:48,964-Speed 130.01 samples/sec Loss 2.8015 Epoch: 11 Global Step: 114920 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:08:08,870-Speed 128.61 samples/sec Loss 3.1553 Epoch: 11 Global Step: 114925 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:08:28,348-Speed 131.44 samples/sec Loss 3.1365 Epoch: 11 Global Step: 114930 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:08:48,023-Speed 130.12 samples/sec Loss 2.8252 Epoch: 11 Global Step: 114935 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:09:07,700-Speed 130.11 samples/sec Loss 2.9602 Epoch: 11 Global Step: 114940 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:09:27,416-Speed 129.85 samples/sec Loss 3.0745 Epoch: 11 Global Step: 114945 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:09:47,083-Speed 130.18 samples/sec Loss 2.9752 Epoch: 11 Global Step: 114950 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:10:06,626-Speed 131.00 samples/sec Loss 3.0273 Epoch: 11 Global Step: 114955 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:10:26,348-Speed 129.81 samples/sec Loss 2.9513 Epoch: 11 Global Step: 114960 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:10:45,913-Speed 130.86 samples/sec Loss 3.0646 Epoch: 11 Global Step: 114965 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:11:05,859-Speed 128.35 samples/sec Loss 3.0122 Epoch: 11 Global Step: 114970 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:11:25,376-Speed 131.17 samples/sec Loss 3.0807 Epoch: 11 Global Step: 114975 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:11:45,121-Speed 129.66 samples/sec Loss 3.1081 Epoch: 11 Global Step: 114980 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:12:04,650-Speed 131.10 samples/sec Loss 2.9006 Epoch: 11 Global Step: 114985 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:12:24,301-Speed 130.29 samples/sec Loss 3.0004 Epoch: 11 Global Step: 114990 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:12:43,837-Speed 131.04 samples/sec Loss 2.9348 Epoch: 11 Global Step: 114995 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:13:03,500-Speed 130.20 samples/sec Loss 2.9326 Epoch: 11 Global Step: 115000 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:13:23,196-Speed 129.99 samples/sec Loss 3.0580 Epoch: 11 Global Step: 115005 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:13:42,831-Speed 130.39 samples/sec Loss 3.1768 Epoch: 11 Global Step: 115010 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:14:02,875-Speed 127.73 samples/sec Loss 2.9159 Epoch: 11 Global Step: 115015 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:14:22,444-Speed 130.82 samples/sec Loss 3.1206 Epoch: 11 Global Step: 115020 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:14:42,277-Speed 129.09 samples/sec Loss 3.0169 Epoch: 11 Global Step: 115025 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:15:01,918-Speed 130.35 samples/sec Loss 2.9281 Epoch: 11 Global Step: 115030 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:15:21,592-Speed 130.12 samples/sec Loss 3.0106 Epoch: 11 Global Step: 115035 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:15:41,138-Speed 130.98 samples/sec Loss 2.9797 Epoch: 11 Global Step: 115040 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:16:00,802-Speed 130.19 samples/sec Loss 3.0973 Epoch: 11 Global Step: 115045 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:16:20,452-Speed 130.29 samples/sec Loss 2.9734 Epoch: 11 Global Step: 115050 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:16:40,075-Speed 130.47 samples/sec Loss 3.0005 Epoch: 11 Global Step: 115055 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:16:59,908-Speed 129.08 samples/sec Loss 2.9260 Epoch: 11 Global Step: 115060 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:17:19,644-Speed 129.72 samples/sec Loss 2.8242 Epoch: 11 Global Step: 115065 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:17:39,338-Speed 130.00 samples/sec Loss 2.8291 Epoch: 11 Global Step: 115070 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:17:58,926-Speed 130.70 samples/sec Loss 2.8608 Epoch: 11 Global Step: 115075 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:18:18,573-Speed 130.30 samples/sec Loss 2.9909 Epoch: 11 Global Step: 115080 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:18:38,257-Speed 130.06 samples/sec Loss 3.0120 Epoch: 11 Global Step: 115085 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:18:57,959-Speed 129.95 samples/sec Loss 2.9375 Epoch: 11 Global Step: 115090 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:19:17,615-Speed 130.25 samples/sec Loss 2.9517 Epoch: 11 Global Step: 115095 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:19:37,141-Speed 131.12 samples/sec Loss 2.9083 Epoch: 11 Global Step: 115100 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:19:57,015-Speed 128.82 samples/sec Loss 3.0426 Epoch: 11 Global Step: 115105 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:20:16,865-Speed 128.97 samples/sec Loss 2.8525 Epoch: 11 Global Step: 115110 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:20:36,548-Speed 130.07 samples/sec Loss 3.1212 Epoch: 11 Global Step: 115115 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:20:56,206-Speed 130.23 samples/sec Loss 2.9286 Epoch: 11 Global Step: 115120 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:21:15,748-Speed 131.01 samples/sec Loss 2.8538 Epoch: 11 Global Step: 115125 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:21:35,439-Speed 130.02 samples/sec Loss 2.9788 Epoch: 11 Global Step: 115130 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:21:55,083-Speed 130.32 samples/sec Loss 3.0595 Epoch: 11 Global Step: 115135 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:22:14,734-Speed 130.28 samples/sec Loss 3.1353 Epoch: 11 Global Step: 115140 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:22:34,374-Speed 130.35 samples/sec Loss 3.0562 Epoch: 11 Global Step: 115145 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:22:54,280-Speed 128.61 samples/sec Loss 3.1179 Epoch: 11 Global Step: 115150 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:23:14,110-Speed 129.10 samples/sec Loss 2.9186 Epoch: 11 Global Step: 115155 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:23:33,724-Speed 130.53 samples/sec Loss 3.1611 Epoch: 11 Global Step: 115160 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:23:53,402-Speed 130.10 samples/sec Loss 2.9725 Epoch: 11 Global Step: 115165 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:24:13,056-Speed 130.26 samples/sec Loss 3.0883 Epoch: 11 Global Step: 115170 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:24:32,776-Speed 129.83 samples/sec Loss 2.7512 Epoch: 11 Global Step: 115175 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:24:52,375-Speed 130.62 samples/sec Loss 3.0089 Epoch: 11 Global Step: 115180 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:25:12,038-Speed 130.21 samples/sec Loss 3.0728 Epoch: 11 Global Step: 115185 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:25:31,663-Speed 130.45 samples/sec Loss 3.0687 Epoch: 11 Global Step: 115190 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:25:51,700-Speed 127.77 samples/sec Loss 3.1016 Epoch: 11 Global Step: 115195 Fp16 Grad Scale: 8192 Required: 159 hours Training: 2021-06-12 11:26:11,249-Speed 130.96 samples/sec Loss 2.9922 Epoch: 11 Global Step: 115200 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:26:30,946-Speed 129.98 samples/sec Loss 3.0021 Epoch: 11 Global Step: 115205 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:26:50,602-Speed 130.25 samples/sec Loss 3.0516 Epoch: 11 Global Step: 115210 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:27:10,250-Speed 130.30 samples/sec Loss 3.1040 Epoch: 11 Global Step: 115215 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:27:29,894-Speed 130.33 samples/sec Loss 2.9520 Epoch: 11 Global Step: 115220 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:27:49,500-Speed 130.58 samples/sec Loss 3.0705 Epoch: 11 Global Step: 115225 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:28:09,142-Speed 130.34 samples/sec Loss 3.0737 Epoch: 11 Global Step: 115230 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:28:28,787-Speed 130.33 samples/sec Loss 2.9378 Epoch: 11 Global Step: 115235 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:28:48,875-Speed 127.44 samples/sec Loss 3.0963 Epoch: 11 Global Step: 115240 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:29:08,355-Speed 131.42 samples/sec Loss 3.0131 Epoch: 11 Global Step: 115245 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:29:28,160-Speed 129.27 samples/sec Loss 3.2013 Epoch: 11 Global Step: 115250 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:29:47,830-Speed 130.15 samples/sec Loss 3.1043 Epoch: 11 Global Step: 115255 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:30:07,539-Speed 129.90 samples/sec Loss 2.9844 Epoch: 11 Global Step: 115260 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:30:27,185-Speed 130.32 samples/sec Loss 3.1396 Epoch: 11 Global Step: 115265 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:30:46,850-Speed 130.18 samples/sec Loss 3.0087 Epoch: 11 Global Step: 115270 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:31:06,648-Speed 129.32 samples/sec Loss 3.1515 Epoch: 11 Global Step: 115275 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:31:26,250-Speed 130.61 samples/sec Loss 3.1511 Epoch: 11 Global Step: 115280 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:31:46,306-Speed 127.65 samples/sec Loss 3.1084 Epoch: 11 Global Step: 115285 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:32:05,797-Speed 131.35 samples/sec Loss 3.1106 Epoch: 11 Global Step: 115290 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:32:25,600-Speed 129.28 samples/sec Loss 3.1264 Epoch: 11 Global Step: 115295 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:32:45,286-Speed 130.05 samples/sec Loss 2.9284 Epoch: 11 Global Step: 115300 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:33:04,854-Speed 130.84 samples/sec Loss 2.9329 Epoch: 11 Global Step: 115305 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:33:24,719-Speed 128.88 samples/sec Loss 3.0692 Epoch: 11 Global Step: 115310 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:33:44,303-Speed 130.73 samples/sec Loss 3.1039 Epoch: 11 Global Step: 115315 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:34:03,960-Speed 130.24 samples/sec Loss 3.1874 Epoch: 11 Global Step: 115320 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:34:23,590-Speed 130.42 samples/sec Loss 2.9315 Epoch: 11 Global Step: 115325 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:34:43,546-Speed 128.29 samples/sec Loss 3.1152 Epoch: 11 Global Step: 115330 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:35:03,041-Speed 131.33 samples/sec Loss 3.1584 Epoch: 11 Global Step: 115335 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:35:22,747-Speed 129.92 samples/sec Loss 2.8946 Epoch: 11 Global Step: 115340 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:35:42,447-Speed 129.95 samples/sec Loss 2.9775 Epoch: 11 Global Step: 115345 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:36:02,210-Speed 129.54 samples/sec Loss 3.1042 Epoch: 11 Global Step: 115350 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:36:21,850-Speed 130.36 samples/sec Loss 3.1011 Epoch: 11 Global Step: 115355 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:36:41,515-Speed 130.19 samples/sec Loss 3.0725 Epoch: 11 Global Step: 115360 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:37:01,133-Speed 130.50 samples/sec Loss 2.9443 Epoch: 11 Global Step: 115365 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:37:20,780-Speed 130.31 samples/sec Loss 2.8580 Epoch: 11 Global Step: 115370 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:37:40,619-Speed 129.05 samples/sec Loss 3.0011 Epoch: 11 Global Step: 115375 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:38:00,437-Speed 129.18 samples/sec Loss 2.9084 Epoch: 11 Global Step: 115380 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:38:20,023-Speed 130.71 samples/sec Loss 2.8500 Epoch: 11 Global Step: 115385 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:38:39,686-Speed 130.20 samples/sec Loss 3.1512 Epoch: 11 Global Step: 115390 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:38:59,283-Speed 130.64 samples/sec Loss 3.0535 Epoch: 11 Global Step: 115395 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:39:18,948-Speed 130.19 samples/sec Loss 3.0224 Epoch: 11 Global Step: 115400 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:39:38,613-Speed 130.18 samples/sec Loss 2.9992 Epoch: 11 Global Step: 115405 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:39:58,322-Speed 129.90 samples/sec Loss 3.0269 Epoch: 11 Global Step: 115410 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:40:17,966-Speed 130.33 samples/sec Loss 2.9807 Epoch: 11 Global Step: 115415 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:40:37,853-Speed 128.73 samples/sec Loss 2.9858 Epoch: 11 Global Step: 115420 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:40:57,605-Speed 129.62 samples/sec Loss 3.0324 Epoch: 11 Global Step: 115425 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:41:17,329-Speed 129.80 samples/sec Loss 3.2081 Epoch: 11 Global Step: 115430 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:41:37,006-Speed 130.10 samples/sec Loss 3.0594 Epoch: 11 Global Step: 115435 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:41:56,659-Speed 130.27 samples/sec Loss 3.1487 Epoch: 11 Global Step: 115440 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:42:16,327-Speed 130.17 samples/sec Loss 2.9910 Epoch: 11 Global Step: 115445 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:42:35,947-Speed 130.48 samples/sec Loss 2.9915 Epoch: 11 Global Step: 115450 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:42:55,571-Speed 130.46 samples/sec Loss 2.8744 Epoch: 11 Global Step: 115455 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:43:15,231-Speed 130.22 samples/sec Loss 3.0118 Epoch: 11 Global Step: 115460 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:43:35,240-Speed 127.95 samples/sec Loss 2.8261 Epoch: 11 Global Step: 115465 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:43:54,920-Speed 130.09 samples/sec Loss 3.0162 Epoch: 11 Global Step: 115470 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:44:14,651-Speed 129.75 samples/sec Loss 3.0803 Epoch: 11 Global Step: 115475 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:44:34,285-Speed 130.39 samples/sec Loss 2.9289 Epoch: 11 Global Step: 115480 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:44:54,177-Speed 128.70 samples/sec Loss 3.0062 Epoch: 11 Global Step: 115485 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:45:13,840-Speed 130.20 samples/sec Loss 3.2266 Epoch: 11 Global Step: 115490 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:45:33,625-Speed 129.40 samples/sec Loss 3.1340 Epoch: 11 Global Step: 115495 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:45:53,275-Speed 130.29 samples/sec Loss 2.9938 Epoch: 11 Global Step: 115500 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:46:12,893-Speed 130.50 samples/sec Loss 3.1629 Epoch: 11 Global Step: 115505 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:46:32,901-Speed 127.96 samples/sec Loss 2.9685 Epoch: 11 Global Step: 115510 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:46:52,376-Speed 131.46 samples/sec Loss 2.9567 Epoch: 11 Global Step: 115515 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:47:12,262-Speed 128.74 samples/sec Loss 2.9870 Epoch: 11 Global Step: 115520 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:47:31,912-Speed 130.28 samples/sec Loss 3.2351 Epoch: 11 Global Step: 115525 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:47:51,560-Speed 130.30 samples/sec Loss 3.3041 Epoch: 11 Global Step: 115530 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:48:11,370-Speed 129.24 samples/sec Loss 3.0192 Epoch: 11 Global Step: 115535 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:48:30,969-Speed 130.63 samples/sec Loss 2.9578 Epoch: 11 Global Step: 115540 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:48:50,615-Speed 130.31 samples/sec Loss 3.1115 Epoch: 11 Global Step: 115545 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:49:10,418-Speed 129.28 samples/sec Loss 2.9902 Epoch: 11 Global Step: 115550 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:49:30,424-Speed 127.97 samples/sec Loss 2.9999 Epoch: 11 Global Step: 115555 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:49:50,141-Speed 129.84 samples/sec Loss 3.0061 Epoch: 11 Global Step: 115560 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:50:09,892-Speed 129.62 samples/sec Loss 3.0798 Epoch: 11 Global Step: 115565 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:50:29,572-Speed 130.09 samples/sec Loss 3.0412 Epoch: 11 Global Step: 115570 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:50:49,227-Speed 130.25 samples/sec Loss 3.0992 Epoch: 11 Global Step: 115575 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:51:08,842-Speed 130.53 samples/sec Loss 3.2385 Epoch: 11 Global Step: 115580 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:51:28,368-Speed 131.11 samples/sec Loss 3.0987 Epoch: 11 Global Step: 115585 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:51:48,033-Speed 130.19 samples/sec Loss 2.8585 Epoch: 11 Global Step: 115590 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:52:07,683-Speed 130.28 samples/sec Loss 2.8368 Epoch: 11 Global Step: 115595 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:52:27,638-Speed 128.30 samples/sec Loss 3.0634 Epoch: 11 Global Step: 115600 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:52:47,180-Speed 131.01 samples/sec Loss 3.0342 Epoch: 11 Global Step: 115605 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:53:06,815-Speed 130.38 samples/sec Loss 3.1067 Epoch: 11 Global Step: 115610 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:53:26,644-Speed 129.12 samples/sec Loss 2.9419 Epoch: 11 Global Step: 115615 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:53:46,349-Speed 129.92 samples/sec Loss 2.9711 Epoch: 11 Global Step: 115620 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:54:05,939-Speed 130.69 samples/sec Loss 2.9628 Epoch: 11 Global Step: 115625 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:54:25,582-Speed 130.33 samples/sec Loss 3.0168 Epoch: 11 Global Step: 115630 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:54:45,239-Speed 130.24 samples/sec Loss 3.2003 Epoch: 11 Global Step: 115635 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:55:04,904-Speed 130.19 samples/sec Loss 3.1591 Epoch: 11 Global Step: 115640 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:55:24,780-Speed 128.81 samples/sec Loss 3.1154 Epoch: 11 Global Step: 115645 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:55:44,689-Speed 128.59 samples/sec Loss 3.2741 Epoch: 11 Global Step: 115650 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:56:04,532-Speed 129.02 samples/sec Loss 3.2689 Epoch: 11 Global Step: 115655 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:56:24,264-Speed 129.74 samples/sec Loss 2.9485 Epoch: 11 Global Step: 115660 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:56:43,910-Speed 130.32 samples/sec Loss 3.0335 Epoch: 11 Global Step: 115665 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:57:03,459-Speed 130.96 samples/sec Loss 2.8929 Epoch: 11 Global Step: 115670 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:57:23,582-Speed 127.23 samples/sec Loss 3.1472 Epoch: 11 Global Step: 115675 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:57:43,233-Speed 130.29 samples/sec Loss 3.1459 Epoch: 11 Global Step: 115680 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:58:02,914-Speed 130.08 samples/sec Loss 3.0572 Epoch: 11 Global Step: 115685 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:58:22,791-Speed 128.80 samples/sec Loss 2.8640 Epoch: 11 Global Step: 115690 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:58:42,507-Speed 129.85 samples/sec Loss 3.0516 Epoch: 11 Global Step: 115695 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:59:02,256-Speed 129.63 samples/sec Loss 3.0626 Epoch: 11 Global Step: 115700 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:59:21,974-Speed 129.84 samples/sec Loss 3.2206 Epoch: 11 Global Step: 115705 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 11:59:41,629-Speed 130.25 samples/sec Loss 2.8570 Epoch: 11 Global Step: 115710 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:00:01,149-Speed 131.15 samples/sec Loss 3.1521 Epoch: 11 Global Step: 115715 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:00:20,765-Speed 130.52 samples/sec Loss 3.1624 Epoch: 11 Global Step: 115720 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:00:40,432-Speed 130.17 samples/sec Loss 2.8417 Epoch: 11 Global Step: 115725 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:01:00,023-Speed 130.68 samples/sec Loss 2.9611 Epoch: 11 Global Step: 115730 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:01:19,892-Speed 128.85 samples/sec Loss 2.9530 Epoch: 11 Global Step: 115735 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:01:39,593-Speed 129.95 samples/sec Loss 2.8414 Epoch: 11 Global Step: 115740 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:01:59,278-Speed 130.06 samples/sec Loss 3.1318 Epoch: 11 Global Step: 115745 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:02:18,817-Speed 131.02 samples/sec Loss 3.1067 Epoch: 11 Global Step: 115750 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:02:38,452-Speed 130.39 samples/sec Loss 2.9757 Epoch: 11 Global Step: 115755 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:02:58,002-Speed 130.95 samples/sec Loss 3.1325 Epoch: 11 Global Step: 115760 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:03:17,677-Speed 130.12 samples/sec Loss 3.0502 Epoch: 11 Global Step: 115765 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:03:37,372-Speed 129.99 samples/sec Loss 3.0921 Epoch: 11 Global Step: 115770 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:03:56,900-Speed 131.10 samples/sec Loss 2.9401 Epoch: 11 Global Step: 115775 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:04:16,747-Speed 129.00 samples/sec Loss 3.0317 Epoch: 11 Global Step: 115780 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:04:36,497-Speed 129.63 samples/sec Loss 2.7645 Epoch: 11 Global Step: 115785 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:04:56,218-Speed 129.81 samples/sec Loss 3.4186 Epoch: 11 Global Step: 115790 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:05:15,792-Speed 130.79 samples/sec Loss 3.0156 Epoch: 11 Global Step: 115795 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:05:35,470-Speed 130.10 samples/sec Loss 3.0365 Epoch: 11 Global Step: 115800 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:05:55,118-Speed 130.30 samples/sec Loss 2.8441 Epoch: 11 Global Step: 115805 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:06:14,727-Speed 130.56 samples/sec Loss 2.9455 Epoch: 11 Global Step: 115810 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:06:34,505-Speed 129.45 samples/sec Loss 3.0080 Epoch: 11 Global Step: 115815 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:06:54,099-Speed 130.65 samples/sec Loss 3.1667 Epoch: 11 Global Step: 115820 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:07:13,928-Speed 129.12 samples/sec Loss 3.1487 Epoch: 11 Global Step: 115825 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:07:33,770-Speed 129.02 samples/sec Loss 3.0174 Epoch: 11 Global Step: 115830 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:07:53,334-Speed 130.86 samples/sec Loss 2.9804 Epoch: 11 Global Step: 115835 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:08:12,927-Speed 130.66 samples/sec Loss 3.0588 Epoch: 11 Global Step: 115840 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:08:32,513-Speed 130.71 samples/sec Loss 3.0529 Epoch: 11 Global Step: 115845 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:08:52,176-Speed 130.20 samples/sec Loss 3.2047 Epoch: 11 Global Step: 115850 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:09:11,846-Speed 130.16 samples/sec Loss 2.6884 Epoch: 11 Global Step: 115855 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:09:31,401-Speed 130.92 samples/sec Loss 2.9571 Epoch: 11 Global Step: 115860 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:09:51,035-Speed 130.39 samples/sec Loss 3.0865 Epoch: 11 Global Step: 115865 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:10:11,054-Speed 127.89 samples/sec Loss 2.9431 Epoch: 11 Global Step: 115870 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:10:30,850-Speed 129.33 samples/sec Loss 3.1499 Epoch: 11 Global Step: 115875 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:10:50,375-Speed 131.12 samples/sec Loss 3.2380 Epoch: 11 Global Step: 115880 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:11:10,037-Speed 130.21 samples/sec Loss 3.1603 Epoch: 11 Global Step: 115885 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:11:29,699-Speed 130.21 samples/sec Loss 3.1693 Epoch: 11 Global Step: 115890 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:11:49,396-Speed 129.97 samples/sec Loss 2.9787 Epoch: 11 Global Step: 115895 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:12:08,911-Speed 131.19 samples/sec Loss 3.0566 Epoch: 11 Global Step: 115900 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:12:28,499-Speed 130.70 samples/sec Loss 3.0968 Epoch: 11 Global Step: 115905 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:12:48,150-Speed 130.28 samples/sec Loss 3.2661 Epoch: 11 Global Step: 115910 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:13:08,174-Speed 127.85 samples/sec Loss 2.8741 Epoch: 11 Global Step: 115915 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:13:27,697-Speed 131.13 samples/sec Loss 3.0737 Epoch: 11 Global Step: 115920 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:13:47,313-Speed 130.52 samples/sec Loss 3.0152 Epoch: 11 Global Step: 115925 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:14:06,956-Speed 130.33 samples/sec Loss 3.1592 Epoch: 11 Global Step: 115930 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:14:26,608-Speed 130.27 samples/sec Loss 3.0583 Epoch: 11 Global Step: 115935 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:14:46,306-Speed 129.97 samples/sec Loss 3.0721 Epoch: 11 Global Step: 115940 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:15:05,840-Speed 131.06 samples/sec Loss 3.1377 Epoch: 11 Global Step: 115945 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:15:25,489-Speed 130.29 samples/sec Loss 3.0946 Epoch: 11 Global Step: 115950 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:15:45,106-Speed 130.50 samples/sec Loss 2.9768 Epoch: 11 Global Step: 115955 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:16:04,998-Speed 128.70 samples/sec Loss 3.0711 Epoch: 11 Global Step: 115960 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:16:24,633-Speed 130.39 samples/sec Loss 3.0542 Epoch: 11 Global Step: 115965 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:16:44,372-Speed 129.70 samples/sec Loss 3.0512 Epoch: 11 Global Step: 115970 Fp16 Grad Scale: 8192 Required: 158 hours Training: 2021-06-12 12:17:04,033-Speed 130.22 samples/sec Loss 2.9820 Epoch: 11 Global Step: 115975 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:17:23,694-Speed 130.21 samples/sec Loss 3.0786 Epoch: 11 Global Step: 115980 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:17:43,436-Speed 129.68 samples/sec Loss 3.0636 Epoch: 11 Global Step: 115985 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:18:03,010-Speed 130.80 samples/sec Loss 3.2505 Epoch: 11 Global Step: 115990 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:18:22,640-Speed 130.42 samples/sec Loss 3.0962 Epoch: 11 Global Step: 115995 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:18:42,287-Speed 130.31 samples/sec Loss 2.8842 Epoch: 11 Global Step: 116000 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:19:02,193-Speed 128.61 samples/sec Loss 2.9330 Epoch: 11 Global Step: 116005 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:19:21,773-Speed 130.75 samples/sec Loss 3.0246 Epoch: 11 Global Step: 116010 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:19:41,466-Speed 130.01 samples/sec Loss 3.2636 Epoch: 11 Global Step: 116015 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:20:01,090-Speed 130.46 samples/sec Loss 3.1541 Epoch: 11 Global Step: 116020 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:20:20,793-Speed 129.93 samples/sec Loss 3.2272 Epoch: 11 Global Step: 116025 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:20:40,309-Speed 131.19 samples/sec Loss 3.1374 Epoch: 11 Global Step: 116030 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:20:59,877-Speed 130.83 samples/sec Loss 3.1155 Epoch: 11 Global Step: 116035 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:21:19,513-Speed 130.38 samples/sec Loss 2.8484 Epoch: 11 Global Step: 116040 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:21:39,180-Speed 130.18 samples/sec Loss 2.9840 Epoch: 11 Global Step: 116045 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:21:59,305-Speed 127.21 samples/sec Loss 3.0995 Epoch: 11 Global Step: 116050 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:22:19,028-Speed 129.81 samples/sec Loss 2.9926 Epoch: 11 Global Step: 116055 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:22:38,660-Speed 130.41 samples/sec Loss 3.0903 Epoch: 11 Global Step: 116060 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:22:58,318-Speed 130.23 samples/sec Loss 3.0493 Epoch: 11 Global Step: 116065 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:23:17,981-Speed 130.20 samples/sec Loss 2.9641 Epoch: 11 Global Step: 116070 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:23:37,620-Speed 130.36 samples/sec Loss 3.2665 Epoch: 11 Global Step: 116075 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:23:57,287-Speed 130.17 samples/sec Loss 2.9384 Epoch: 11 Global Step: 116080 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:24:16,910-Speed 130.47 samples/sec Loss 3.2203 Epoch: 11 Global Step: 116085 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:24:36,616-Speed 129.92 samples/sec Loss 3.0495 Epoch: 11 Global Step: 116090 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:24:56,379-Speed 129.54 samples/sec Loss 3.1118 Epoch: 11 Global Step: 116095 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:25:16,143-Speed 129.53 samples/sec Loss 3.1772 Epoch: 11 Global Step: 116100 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:25:35,779-Speed 130.38 samples/sec Loss 2.8474 Epoch: 11 Global Step: 116105 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:25:55,489-Speed 129.89 samples/sec Loss 3.4343 Epoch: 11 Global Step: 116110 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:26:15,011-Speed 131.14 samples/sec Loss 3.0116 Epoch: 11 Global Step: 116115 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:26:34,605-Speed 130.66 samples/sec Loss 3.1180 Epoch: 11 Global Step: 116120 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:26:54,275-Speed 130.15 samples/sec Loss 3.2498 Epoch: 11 Global Step: 116125 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:27:13,954-Speed 130.10 samples/sec Loss 3.0608 Epoch: 11 Global Step: 116130 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:27:33,597-Speed 130.33 samples/sec Loss 3.1168 Epoch: 11 Global Step: 116135 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:27:53,377-Speed 129.43 samples/sec Loss 3.1495 Epoch: 11 Global Step: 116140 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:28:13,111-Speed 129.74 samples/sec Loss 3.1843 Epoch: 11 Global Step: 116145 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:28:32,797-Speed 130.04 samples/sec Loss 3.1517 Epoch: 11 Global Step: 116150 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:28:52,464-Speed 130.18 samples/sec Loss 3.0233 Epoch: 11 Global Step: 116155 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:29:12,033-Speed 130.83 samples/sec Loss 3.0119 Epoch: 11 Global Step: 116160 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:29:31,691-Speed 130.24 samples/sec Loss 3.1956 Epoch: 11 Global Step: 116165 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:29:51,492-Speed 129.29 samples/sec Loss 3.1172 Epoch: 11 Global Step: 116170 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:30:11,207-Speed 129.86 samples/sec Loss 2.9975 Epoch: 11 Global Step: 116175 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:30:30,867-Speed 130.22 samples/sec Loss 2.9866 Epoch: 11 Global Step: 116180 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:30:50,849-Speed 128.13 samples/sec Loss 2.8638 Epoch: 11 Global Step: 116185 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:31:10,580-Speed 129.75 samples/sec Loss 2.9434 Epoch: 11 Global Step: 116190 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:31:30,337-Speed 129.58 samples/sec Loss 2.9712 Epoch: 11 Global Step: 116195 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:31:50,008-Speed 130.15 samples/sec Loss 3.2390 Epoch: 11 Global Step: 116200 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:32:09,610-Speed 130.60 samples/sec Loss 2.9685 Epoch: 11 Global Step: 116205 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:32:29,254-Speed 130.33 samples/sec Loss 3.0738 Epoch: 11 Global Step: 116210 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:32:48,891-Speed 130.37 samples/sec Loss 2.9256 Epoch: 11 Global Step: 116215 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:33:08,513-Speed 130.48 samples/sec Loss 3.1163 Epoch: 11 Global Step: 116220 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:33:28,137-Speed 130.46 samples/sec Loss 3.2480 Epoch: 11 Global Step: 116225 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:33:48,162-Speed 127.85 samples/sec Loss 3.0864 Epoch: 11 Global Step: 116230 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:34:07,644-Speed 131.41 samples/sec Loss 2.9473 Epoch: 11 Global Step: 116235 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:34:27,454-Speed 129.23 samples/sec Loss 3.1402 Epoch: 11 Global Step: 116240 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:34:47,712-Speed 126.38 samples/sec Loss 3.0340 Epoch: 11 Global Step: 116245 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:35:07,336-Speed 130.46 samples/sec Loss 3.0605 Epoch: 11 Global Step: 116250 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:35:26,972-Speed 130.38 samples/sec Loss 3.1584 Epoch: 11 Global Step: 116255 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:35:46,630-Speed 130.24 samples/sec Loss 3.2507 Epoch: 11 Global Step: 116260 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:36:06,236-Speed 130.58 samples/sec Loss 2.9438 Epoch: 11 Global Step: 116265 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:36:25,924-Speed 130.03 samples/sec Loss 3.3058 Epoch: 11 Global Step: 116270 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:36:45,821-Speed 128.67 samples/sec Loss 2.9632 Epoch: 11 Global Step: 116275 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:37:05,312-Speed 131.36 samples/sec Loss 2.9684 Epoch: 11 Global Step: 116280 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:37:25,055-Speed 129.67 samples/sec Loss 2.9907 Epoch: 11 Global Step: 116285 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:37:44,717-Speed 130.21 samples/sec Loss 2.9330 Epoch: 11 Global Step: 116290 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:38:04,248-Speed 131.08 samples/sec Loss 3.1408 Epoch: 11 Global Step: 116295 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:38:23,905-Speed 130.24 samples/sec Loss 3.1358 Epoch: 11 Global Step: 116300 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:38:43,532-Speed 130.44 samples/sec Loss 3.1979 Epoch: 11 Global Step: 116305 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:39:03,215-Speed 130.07 samples/sec Loss 3.1630 Epoch: 11 Global Step: 116310 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:39:23,084-Speed 128.86 samples/sec Loss 3.0726 Epoch: 11 Global Step: 116315 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:39:42,971-Speed 128.73 samples/sec Loss 2.9942 Epoch: 11 Global Step: 116320 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:40:02,552-Speed 130.75 samples/sec Loss 3.1371 Epoch: 11 Global Step: 116325 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:40:22,463-Speed 128.58 samples/sec Loss 2.9281 Epoch: 11 Global Step: 116330 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:40:42,039-Speed 130.78 samples/sec Loss 3.2563 Epoch: 11 Global Step: 116335 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:41:01,689-Speed 130.29 samples/sec Loss 2.9632 Epoch: 11 Global Step: 116340 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:41:21,413-Speed 129.80 samples/sec Loss 3.0714 Epoch: 11 Global Step: 116345 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:41:41,018-Speed 130.58 samples/sec Loss 2.9647 Epoch: 11 Global Step: 116350 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:42:00,667-Speed 130.30 samples/sec Loss 3.0398 Epoch: 11 Global Step: 116355 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:42:20,230-Speed 130.87 samples/sec Loss 2.9854 Epoch: 11 Global Step: 116360 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:42:40,209-Speed 128.14 samples/sec Loss 3.0167 Epoch: 11 Global Step: 116365 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:42:59,700-Speed 131.35 samples/sec Loss 3.0651 Epoch: 11 Global Step: 116370 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:43:19,563-Speed 128.89 samples/sec Loss 3.2888 Epoch: 11 Global Step: 116375 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:43:39,120-Speed 130.91 samples/sec Loss 3.1341 Epoch: 11 Global Step: 116380 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:43:58,741-Speed 130.48 samples/sec Loss 3.1163 Epoch: 11 Global Step: 116385 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:44:18,363-Speed 130.47 samples/sec Loss 3.0504 Epoch: 11 Global Step: 116390 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:44:38,306-Speed 128.37 samples/sec Loss 3.1684 Epoch: 11 Global Step: 116395 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:44:57,937-Speed 130.42 samples/sec Loss 3.0969 Epoch: 11 Global Step: 116400 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:45:17,776-Speed 129.05 samples/sec Loss 3.2028 Epoch: 11 Global Step: 116405 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:45:37,678-Speed 128.64 samples/sec Loss 3.0806 Epoch: 11 Global Step: 116410 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:45:57,422-Speed 129.67 samples/sec Loss 3.0595 Epoch: 11 Global Step: 116415 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:46:17,022-Speed 130.62 samples/sec Loss 3.1681 Epoch: 11 Global Step: 116420 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:46:36,820-Speed 129.32 samples/sec Loss 3.0884 Epoch: 11 Global Step: 116425 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:46:56,530-Speed 129.89 samples/sec Loss 3.0278 Epoch: 11 Global Step: 116430 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:47:16,125-Speed 130.65 samples/sec Loss 2.9941 Epoch: 11 Global Step: 116435 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:47:35,781-Speed 130.25 samples/sec Loss 3.0226 Epoch: 11 Global Step: 116440 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:47:55,386-Speed 130.59 samples/sec Loss 2.9679 Epoch: 11 Global Step: 116445 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:48:15,236-Speed 128.97 samples/sec Loss 3.2086 Epoch: 11 Global Step: 116450 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:48:35,070-Speed 129.08 samples/sec Loss 3.1487 Epoch: 11 Global Step: 116455 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:48:54,905-Speed 129.07 samples/sec Loss 3.1205 Epoch: 11 Global Step: 116460 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:49:14,445-Speed 131.02 samples/sec Loss 3.1164 Epoch: 11 Global Step: 116465 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:49:34,149-Speed 129.93 samples/sec Loss 3.0799 Epoch: 11 Global Step: 116470 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:49:53,789-Speed 130.35 samples/sec Loss 3.1088 Epoch: 11 Global Step: 116475 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:50:13,434-Speed 130.32 samples/sec Loss 3.1900 Epoch: 11 Global Step: 116480 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:50:33,130-Speed 129.98 samples/sec Loss 3.1768 Epoch: 11 Global Step: 116485 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:50:52,684-Speed 130.93 samples/sec Loss 3.1936 Epoch: 11 Global Step: 116490 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:51:12,360-Speed 130.12 samples/sec Loss 3.1348 Epoch: 11 Global Step: 116495 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:51:32,204-Speed 129.01 samples/sec Loss 2.9152 Epoch: 11 Global Step: 116500 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:51:52,013-Speed 129.25 samples/sec Loss 2.9788 Epoch: 11 Global Step: 116505 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:52:11,600-Speed 130.70 samples/sec Loss 3.0854 Epoch: 11 Global Step: 116510 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:52:31,253-Speed 130.27 samples/sec Loss 2.9171 Epoch: 11 Global Step: 116515 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:52:50,917-Speed 130.20 samples/sec Loss 3.2557 Epoch: 11 Global Step: 116520 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:53:10,480-Speed 130.86 samples/sec Loss 3.1188 Epoch: 11 Global Step: 116525 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:53:30,128-Speed 130.30 samples/sec Loss 3.0897 Epoch: 11 Global Step: 116530 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:53:49,746-Speed 130.50 samples/sec Loss 3.1832 Epoch: 11 Global Step: 116535 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:54:09,402-Speed 130.25 samples/sec Loss 2.9626 Epoch: 11 Global Step: 116540 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:54:29,228-Speed 129.13 samples/sec Loss 3.1274 Epoch: 11 Global Step: 116545 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:54:48,965-Speed 129.71 samples/sec Loss 2.9041 Epoch: 11 Global Step: 116550 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:55:08,651-Speed 130.05 samples/sec Loss 2.8873 Epoch: 11 Global Step: 116555 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:55:28,300-Speed 130.29 samples/sec Loss 3.1362 Epoch: 11 Global Step: 116560 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:55:47,860-Speed 130.88 samples/sec Loss 3.3558 Epoch: 11 Global Step: 116565 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:56:07,570-Speed 129.89 samples/sec Loss 3.1188 Epoch: 11 Global Step: 116570 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:56:27,221-Speed 130.28 samples/sec Loss 3.1424 Epoch: 11 Global Step: 116575 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:56:46,936-Speed 129.86 samples/sec Loss 3.1709 Epoch: 11 Global Step: 116580 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:57:06,600-Speed 130.20 samples/sec Loss 3.0104 Epoch: 11 Global Step: 116585 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:57:26,459-Speed 128.91 samples/sec Loss 2.9993 Epoch: 11 Global Step: 116590 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:57:46,079-Speed 130.49 samples/sec Loss 3.0274 Epoch: 11 Global Step: 116595 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:58:05,779-Speed 129.95 samples/sec Loss 3.0007 Epoch: 11 Global Step: 116600 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:58:25,428-Speed 130.30 samples/sec Loss 3.1273 Epoch: 11 Global Step: 116605 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:58:45,049-Speed 130.48 samples/sec Loss 3.1387 Epoch: 11 Global Step: 116610 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:59:04,631-Speed 130.74 samples/sec Loss 3.0748 Epoch: 11 Global Step: 116615 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:59:24,248-Speed 130.50 samples/sec Loss 3.1559 Epoch: 11 Global Step: 116620 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 12:59:43,913-Speed 130.19 samples/sec Loss 3.0511 Epoch: 11 Global Step: 116625 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:00:03,684-Speed 129.49 samples/sec Loss 2.9595 Epoch: 11 Global Step: 116630 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:00:23,532-Speed 128.99 samples/sec Loss 3.1005 Epoch: 11 Global Step: 116635 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:00:43,224-Speed 130.01 samples/sec Loss 3.3604 Epoch: 11 Global Step: 116640 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:01:02,913-Speed 130.03 samples/sec Loss 3.1507 Epoch: 11 Global Step: 116645 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:01:22,568-Speed 130.26 samples/sec Loss 3.1871 Epoch: 11 Global Step: 116650 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:01:42,108-Speed 131.02 samples/sec Loss 3.1052 Epoch: 11 Global Step: 116655 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:02:01,690-Speed 130.74 samples/sec Loss 2.9401 Epoch: 11 Global Step: 116660 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:02:21,393-Speed 129.94 samples/sec Loss 2.9417 Epoch: 11 Global Step: 116665 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:02:41,046-Speed 130.27 samples/sec Loss 2.9834 Epoch: 11 Global Step: 116670 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:03:00,702-Speed 130.25 samples/sec Loss 3.3249 Epoch: 11 Global Step: 116675 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:03:20,467-Speed 129.53 samples/sec Loss 3.0366 Epoch: 11 Global Step: 116680 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:03:40,134-Speed 130.17 samples/sec Loss 2.9187 Epoch: 11 Global Step: 116685 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:03:59,781-Speed 130.31 samples/sec Loss 3.1491 Epoch: 11 Global Step: 116690 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:04:19,424-Speed 130.33 samples/sec Loss 3.1243 Epoch: 11 Global Step: 116695 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:04:39,002-Speed 130.77 samples/sec Loss 3.1845 Epoch: 11 Global Step: 116700 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:04:58,682-Speed 130.09 samples/sec Loss 3.0206 Epoch: 11 Global Step: 116705 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:05:18,331-Speed 130.30 samples/sec Loss 3.2251 Epoch: 11 Global Step: 116710 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:05:37,864-Speed 131.07 samples/sec Loss 2.9258 Epoch: 11 Global Step: 116715 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:05:57,787-Speed 128.50 samples/sec Loss 3.1937 Epoch: 11 Global Step: 116720 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:06:17,594-Speed 129.26 samples/sec Loss 3.1141 Epoch: 11 Global Step: 116725 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:06:37,278-Speed 130.06 samples/sec Loss 3.1375 Epoch: 11 Global Step: 116730 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:06:56,947-Speed 130.16 samples/sec Loss 3.0373 Epoch: 11 Global Step: 116735 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:07:16,617-Speed 130.15 samples/sec Loss 3.0770 Epoch: 11 Global Step: 116740 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:07:36,191-Speed 130.79 samples/sec Loss 3.0990 Epoch: 11 Global Step: 116745 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:07:55,881-Speed 130.02 samples/sec Loss 3.1284 Epoch: 11 Global Step: 116750 Fp16 Grad Scale: 8192 Required: 157 hours Training: 2021-06-12 13:08:15,551-Speed 130.15 samples/sec Loss 3.1492 Epoch: 11 Global Step: 116755 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:08:35,098-Speed 130.98 samples/sec Loss 2.9825 Epoch: 11 Global Step: 116760 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:08:54,719-Speed 130.48 samples/sec Loss 3.1807 Epoch: 11 Global Step: 116765 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:09:14,664-Speed 128.36 samples/sec Loss 3.1100 Epoch: 11 Global Step: 116770 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:09:34,231-Speed 130.84 samples/sec Loss 3.1431 Epoch: 11 Global Step: 116775 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:09:53,883-Speed 130.27 samples/sec Loss 3.0364 Epoch: 11 Global Step: 116780 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:10:13,505-Speed 130.47 samples/sec Loss 3.1469 Epoch: 11 Global Step: 116785 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:10:33,025-Speed 131.16 samples/sec Loss 3.2100 Epoch: 11 Global Step: 116790 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:10:52,720-Speed 129.99 samples/sec Loss 3.0295 Epoch: 11 Global Step: 116795 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:11:12,255-Speed 131.05 samples/sec Loss 3.0287 Epoch: 11 Global Step: 116800 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:11:31,909-Speed 130.26 samples/sec Loss 3.0172 Epoch: 11 Global Step: 116805 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:11:51,480-Speed 130.81 samples/sec Loss 3.0362 Epoch: 11 Global Step: 116810 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:12:11,399-Speed 128.52 samples/sec Loss 3.2181 Epoch: 11 Global Step: 116815 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:12:30,968-Speed 130.83 samples/sec Loss 3.1670 Epoch: 11 Global Step: 116820 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:12:50,590-Speed 130.47 samples/sec Loss 2.9403 Epoch: 11 Global Step: 116825 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:13:10,199-Speed 130.56 samples/sec Loss 3.0303 Epoch: 11 Global Step: 116830 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:13:29,854-Speed 130.25 samples/sec Loss 3.2093 Epoch: 11 Global Step: 116835 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:13:49,519-Speed 130.19 samples/sec Loss 2.9336 Epoch: 11 Global Step: 116840 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:14:09,164-Speed 130.32 samples/sec Loss 3.0864 Epoch: 11 Global Step: 116845 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:14:28,743-Speed 130.76 samples/sec Loss 3.1374 Epoch: 11 Global Step: 116850 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:14:48,347-Speed 130.59 samples/sec Loss 3.0329 Epoch: 11 Global Step: 116855 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:15:08,262-Speed 128.55 samples/sec Loss 3.0184 Epoch: 11 Global Step: 116860 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:15:28,034-Speed 129.49 samples/sec Loss 3.1013 Epoch: 11 Global Step: 116865 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:15:47,606-Speed 130.80 samples/sec Loss 3.1670 Epoch: 11 Global Step: 116870 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:16:07,240-Speed 130.39 samples/sec Loss 3.0500 Epoch: 11 Global Step: 116875 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:16:26,915-Speed 130.13 samples/sec Loss 3.0255 Epoch: 11 Global Step: 116880 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:16:46,506-Speed 130.68 samples/sec Loss 3.0990 Epoch: 11 Global Step: 116885 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:17:06,159-Speed 130.26 samples/sec Loss 3.2476 Epoch: 11 Global Step: 116890 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:17:25,773-Speed 130.53 samples/sec Loss 3.2257 Epoch: 11 Global Step: 116895 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:17:45,470-Speed 129.98 samples/sec Loss 3.0911 Epoch: 11 Global Step: 116900 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:18:06,025-Speed 124.55 samples/sec Loss 3.2863 Epoch: 11 Global Step: 116905 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:18:25,766-Speed 129.69 samples/sec Loss 3.3272 Epoch: 11 Global Step: 116910 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:18:45,357-Speed 130.68 samples/sec Loss 3.1039 Epoch: 11 Global Step: 116915 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:19:05,069-Speed 129.88 samples/sec Loss 3.0724 Epoch: 11 Global Step: 116920 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:19:24,666-Speed 130.64 samples/sec Loss 3.2894 Epoch: 11 Global Step: 116925 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:19:44,307-Speed 130.34 samples/sec Loss 3.0751 Epoch: 11 Global Step: 116930 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:20:03,828-Speed 131.15 samples/sec Loss 3.2318 Epoch: 11 Global Step: 116935 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:20:23,446-Speed 130.50 samples/sec Loss 3.1874 Epoch: 11 Global Step: 116940 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:20:43,112-Speed 130.18 samples/sec Loss 3.1742 Epoch: 11 Global Step: 116945 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:21:03,009-Speed 128.67 samples/sec Loss 3.1519 Epoch: 11 Global Step: 116950 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:21:22,491-Speed 131.41 samples/sec Loss 3.1635 Epoch: 11 Global Step: 116955 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:21:42,122-Speed 130.41 samples/sec Loss 3.1824 Epoch: 11 Global Step: 116960 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:22:01,772-Speed 130.29 samples/sec Loss 3.2697 Epoch: 11 Global Step: 116965 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:22:21,350-Speed 130.77 samples/sec Loss 3.1754 Epoch: 11 Global Step: 116970 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:22:41,022-Speed 130.14 samples/sec Loss 2.9468 Epoch: 11 Global Step: 116975 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:23:00,554-Speed 131.08 samples/sec Loss 3.0022 Epoch: 11 Global Step: 116980 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:23:20,263-Speed 129.90 samples/sec Loss 3.0205 Epoch: 11 Global Step: 116985 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:23:39,898-Speed 130.39 samples/sec Loss 3.1953 Epoch: 11 Global Step: 116990 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:23:59,802-Speed 128.62 samples/sec Loss 3.2275 Epoch: 11 Global Step: 116995 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:24:19,288-Speed 131.38 samples/sec Loss 3.1044 Epoch: 11 Global Step: 117000 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:24:39,132-Speed 129.02 samples/sec Loss 3.0845 Epoch: 11 Global Step: 117005 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:24:58,777-Speed 130.32 samples/sec Loss 2.9528 Epoch: 11 Global Step: 117010 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:25:18,302-Speed 131.13 samples/sec Loss 3.3133 Epoch: 11 Global Step: 117015 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:25:37,955-Speed 130.26 samples/sec Loss 3.2322 Epoch: 11 Global Step: 117020 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:25:57,668-Speed 129.87 samples/sec Loss 3.2339 Epoch: 11 Global Step: 117025 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:26:17,398-Speed 129.76 samples/sec Loss 3.2821 Epoch: 11 Global Step: 117030 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:26:37,049-Speed 130.28 samples/sec Loss 2.9581 Epoch: 11 Global Step: 117035 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:26:56,945-Speed 128.67 samples/sec Loss 3.1143 Epoch: 11 Global Step: 117040 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:27:16,440-Speed 131.33 samples/sec Loss 3.0446 Epoch: 11 Global Step: 117045 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:27:36,304-Speed 128.88 samples/sec Loss 3.1985 Epoch: 11 Global Step: 117050 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:27:55,921-Speed 130.51 samples/sec Loss 3.1491 Epoch: 11 Global Step: 117055 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:28:15,547-Speed 130.45 samples/sec Loss 3.2264 Epoch: 11 Global Step: 117060 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:28:35,300-Speed 129.61 samples/sec Loss 3.0635 Epoch: 11 Global Step: 117065 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:28:55,001-Speed 129.94 samples/sec Loss 3.0097 Epoch: 11 Global Step: 117070 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:29:14,647-Speed 130.32 samples/sec Loss 3.0321 Epoch: 11 Global Step: 117075 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:29:34,185-Speed 131.04 samples/sec Loss 3.2693 Epoch: 11 Global Step: 117080 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:29:54,088-Speed 128.63 samples/sec Loss 3.1580 Epoch: 11 Global Step: 117085 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:30:13,723-Speed 130.38 samples/sec Loss 3.1494 Epoch: 11 Global Step: 117090 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:30:33,543-Speed 129.17 samples/sec Loss 3.1121 Epoch: 11 Global Step: 117095 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:30:53,075-Speed 131.08 samples/sec Loss 2.9143 Epoch: 11 Global Step: 117100 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:31:12,595-Speed 131.16 samples/sec Loss 3.0545 Epoch: 11 Global Step: 117105 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:31:32,276-Speed 130.08 samples/sec Loss 3.0717 Epoch: 11 Global Step: 117110 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:31:51,920-Speed 130.33 samples/sec Loss 3.1419 Epoch: 11 Global Step: 117115 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:32:11,570-Speed 130.28 samples/sec Loss 2.9529 Epoch: 11 Global Step: 117120 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:32:31,110-Speed 131.02 samples/sec Loss 3.0516 Epoch: 11 Global Step: 117125 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:32:51,006-Speed 128.67 samples/sec Loss 3.1138 Epoch: 11 Global Step: 117130 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:33:10,696-Speed 130.02 samples/sec Loss 3.0745 Epoch: 11 Global Step: 117135 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:33:30,704-Speed 127.96 samples/sec Loss 3.1208 Epoch: 11 Global Step: 117140 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:33:50,233-Speed 131.09 samples/sec Loss 3.1313 Epoch: 11 Global Step: 117145 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:34:09,881-Speed 130.31 samples/sec Loss 3.1784 Epoch: 11 Global Step: 117150 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:34:29,581-Speed 129.96 samples/sec Loss 3.2804 Epoch: 11 Global Step: 117155 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:34:49,514-Speed 128.44 samples/sec Loss 3.0983 Epoch: 11 Global Step: 117160 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:35:09,126-Speed 130.54 samples/sec Loss 3.2084 Epoch: 11 Global Step: 117165 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:35:28,653-Speed 131.10 samples/sec Loss 3.0829 Epoch: 11 Global Step: 117170 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:35:48,657-Speed 127.98 samples/sec Loss 3.2269 Epoch: 11 Global Step: 117175 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:36:08,438-Speed 129.42 samples/sec Loss 3.2557 Epoch: 11 Global Step: 117180 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:36:28,033-Speed 130.65 samples/sec Loss 3.1116 Epoch: 11 Global Step: 117185 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:36:47,551-Speed 131.16 samples/sec Loss 3.1661 Epoch: 11 Global Step: 117190 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:37:07,212-Speed 130.22 samples/sec Loss 3.1251 Epoch: 11 Global Step: 117195 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:37:26,866-Speed 130.26 samples/sec Loss 3.1641 Epoch: 11 Global Step: 117200 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:37:46,542-Speed 130.12 samples/sec Loss 3.1901 Epoch: 11 Global Step: 117205 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:38:06,165-Speed 130.47 samples/sec Loss 3.1994 Epoch: 11 Global Step: 117210 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:38:25,700-Speed 131.05 samples/sec Loss 3.2312 Epoch: 11 Global Step: 117215 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:38:45,588-Speed 128.73 samples/sec Loss 3.0853 Epoch: 11 Global Step: 117220 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:39:05,288-Speed 129.96 samples/sec Loss 3.1550 Epoch: 11 Global Step: 117225 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:39:24,950-Speed 130.21 samples/sec Loss 2.9817 Epoch: 11 Global Step: 117230 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:39:44,473-Speed 131.13 samples/sec Loss 3.0927 Epoch: 11 Global Step: 117235 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:40:04,184-Speed 129.88 samples/sec Loss 3.0319 Epoch: 11 Global Step: 117240 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:40:24,070-Speed 128.74 samples/sec Loss 3.0359 Epoch: 11 Global Step: 117245 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:40:43,740-Speed 130.16 samples/sec Loss 3.0051 Epoch: 11 Global Step: 117250 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:41:03,269-Speed 131.10 samples/sec Loss 3.1232 Epoch: 11 Global Step: 117255 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:41:22,922-Speed 130.27 samples/sec Loss 3.1565 Epoch: 11 Global Step: 117260 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:41:42,833-Speed 128.58 samples/sec Loss 3.0779 Epoch: 11 Global Step: 117265 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:42:02,611-Speed 129.44 samples/sec Loss 3.2149 Epoch: 11 Global Step: 117270 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:42:22,115-Speed 131.26 samples/sec Loss 2.9427 Epoch: 11 Global Step: 117275 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:42:41,868-Speed 129.61 samples/sec Loss 3.1558 Epoch: 11 Global Step: 117280 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:43:01,528-Speed 130.22 samples/sec Loss 3.3470 Epoch: 11 Global Step: 117285 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:43:21,182-Speed 130.27 samples/sec Loss 3.1396 Epoch: 11 Global Step: 117290 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:43:40,832-Speed 130.29 samples/sec Loss 3.2636 Epoch: 11 Global Step: 117295 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:44:00,487-Speed 130.25 samples/sec Loss 3.2072 Epoch: 11 Global Step: 117300 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:44:20,237-Speed 129.64 samples/sec Loss 3.2198 Epoch: 11 Global Step: 117305 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:44:40,096-Speed 128.92 samples/sec Loss 3.1396 Epoch: 11 Global Step: 117310 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:44:59,850-Speed 129.60 samples/sec Loss 3.1498 Epoch: 11 Global Step: 117315 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:45:19,426-Speed 130.78 samples/sec Loss 2.9470 Epoch: 11 Global Step: 117320 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:45:39,096-Speed 130.15 samples/sec Loss 3.1188 Epoch: 11 Global Step: 117325 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:45:58,735-Speed 130.36 samples/sec Loss 3.0030 Epoch: 11 Global Step: 117330 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:46:18,274-Speed 131.03 samples/sec Loss 3.0689 Epoch: 11 Global Step: 117335 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:46:37,880-Speed 130.58 samples/sec Loss 3.1033 Epoch: 11 Global Step: 117340 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:46:57,573-Speed 130.00 samples/sec Loss 3.1408 Epoch: 11 Global Step: 117345 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:47:17,239-Speed 130.18 samples/sec Loss 3.1411 Epoch: 11 Global Step: 117350 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:47:37,341-Speed 127.36 samples/sec Loss 3.1480 Epoch: 11 Global Step: 117355 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:47:56,824-Speed 131.40 samples/sec Loss 2.9693 Epoch: 11 Global Step: 117360 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:48:16,630-Speed 129.26 samples/sec Loss 3.1589 Epoch: 11 Global Step: 117365 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:48:36,285-Speed 130.25 samples/sec Loss 3.0402 Epoch: 11 Global Step: 117370 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:48:56,125-Speed 129.04 samples/sec Loss 3.2805 Epoch: 11 Global Step: 117375 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:49:15,758-Speed 130.40 samples/sec Loss 3.0141 Epoch: 11 Global Step: 117380 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:49:35,360-Speed 130.61 samples/sec Loss 3.3430 Epoch: 11 Global Step: 117385 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:49:55,070-Speed 129.89 samples/sec Loss 3.3561 Epoch: 11 Global Step: 117390 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:50:14,719-Speed 130.29 samples/sec Loss 3.2057 Epoch: 11 Global Step: 117395 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:50:34,673-Speed 128.30 samples/sec Loss 3.0276 Epoch: 11 Global Step: 117400 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:50:54,148-Speed 131.46 samples/sec Loss 3.2764 Epoch: 11 Global Step: 117405 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:51:14,131-Speed 128.12 samples/sec Loss 3.0670 Epoch: 11 Global Step: 117410 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:51:33,778-Speed 130.30 samples/sec Loss 3.3146 Epoch: 11 Global Step: 117415 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:51:53,348-Speed 130.82 samples/sec Loss 3.2810 Epoch: 11 Global Step: 117420 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:52:12,954-Speed 130.58 samples/sec Loss 3.1306 Epoch: 11 Global Step: 117425 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:52:32,607-Speed 130.27 samples/sec Loss 3.0097 Epoch: 11 Global Step: 117430 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:52:52,248-Speed 130.35 samples/sec Loss 3.0267 Epoch: 11 Global Step: 117435 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:53:11,899-Speed 130.28 samples/sec Loss 3.2581 Epoch: 11 Global Step: 117440 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:53:32,170-Speed 126.30 samples/sec Loss 3.1969 Epoch: 11 Global Step: 117445 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:53:51,678-Speed 131.23 samples/sec Loss 3.1418 Epoch: 11 Global Step: 117450 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:54:11,541-Speed 128.89 samples/sec Loss 3.2541 Epoch: 11 Global Step: 117455 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:54:31,186-Speed 130.32 samples/sec Loss 3.0485 Epoch: 11 Global Step: 117460 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:54:50,854-Speed 130.17 samples/sec Loss 3.2509 Epoch: 11 Global Step: 117465 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:55:10,503-Speed 130.29 samples/sec Loss 3.1912 Epoch: 11 Global Step: 117470 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:55:30,207-Speed 129.93 samples/sec Loss 3.1476 Epoch: 11 Global Step: 117475 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:55:49,867-Speed 130.22 samples/sec Loss 3.1311 Epoch: 11 Global Step: 117480 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:56:09,520-Speed 130.27 samples/sec Loss 3.1643 Epoch: 11 Global Step: 117485 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:56:29,365-Speed 129.01 samples/sec Loss 3.2197 Epoch: 11 Global Step: 117490 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:56:48,936-Speed 130.81 samples/sec Loss 3.1818 Epoch: 11 Global Step: 117495 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:57:08,803-Speed 128.86 samples/sec Loss 3.1110 Epoch: 11 Global Step: 117500 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:57:28,375-Speed 130.81 samples/sec Loss 2.9091 Epoch: 11 Global Step: 117505 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:57:48,034-Speed 130.23 samples/sec Loss 3.0321 Epoch: 11 Global Step: 117510 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:58:07,586-Speed 130.94 samples/sec Loss 3.1869 Epoch: 11 Global Step: 117515 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:58:27,285-Speed 129.96 samples/sec Loss 3.0313 Epoch: 11 Global Step: 117520 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:58:46,932-Speed 130.31 samples/sec Loss 3.2444 Epoch: 11 Global Step: 117525 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:59:06,483-Speed 130.94 samples/sec Loss 3.0385 Epoch: 11 Global Step: 117530 Fp16 Grad Scale: 8192 Required: 156 hours Training: 2021-06-12 13:59:26,313-Speed 129.11 samples/sec Loss 3.0972 Epoch: 11 Global Step: 117535 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 13:59:46,124-Speed 129.23 samples/sec Loss 3.1599 Epoch: 11 Global Step: 117540 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:00:05,763-Speed 130.36 samples/sec Loss 2.9243 Epoch: 11 Global Step: 117545 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:00:25,428-Speed 130.19 samples/sec Loss 3.0265 Epoch: 11 Global Step: 117550 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:00:45,032-Speed 130.59 samples/sec Loss 3.1013 Epoch: 11 Global Step: 117555 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:01:04,729-Speed 129.98 samples/sec Loss 3.1503 Epoch: 11 Global Step: 117560 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:01:24,384-Speed 130.25 samples/sec Loss 3.2060 Epoch: 11 Global Step: 117565 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:01:44,030-Speed 130.31 samples/sec Loss 3.2685 Epoch: 11 Global Step: 117570 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:02:03,673-Speed 130.33 samples/sec Loss 3.0341 Epoch: 11 Global Step: 117575 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:02:23,481-Speed 129.25 samples/sec Loss 3.3373 Epoch: 11 Global Step: 117580 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:02:43,287-Speed 129.26 samples/sec Loss 3.2828 Epoch: 11 Global Step: 117585 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:03:02,861-Speed 130.79 samples/sec Loss 3.1339 Epoch: 11 Global Step: 117590 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:03:22,626-Speed 129.53 samples/sec Loss 3.1788 Epoch: 11 Global Step: 117595 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:03:42,250-Speed 130.46 samples/sec Loss 2.9934 Epoch: 11 Global Step: 117600 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:04:01,862-Speed 130.54 samples/sec Loss 3.1427 Epoch: 11 Global Step: 117605 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:04:21,510-Speed 130.30 samples/sec Loss 3.1347 Epoch: 11 Global Step: 117610 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:04:41,415-Speed 128.62 samples/sec Loss 2.9894 Epoch: 11 Global Step: 117615 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:05:01,043-Speed 130.44 samples/sec Loss 3.2318 Epoch: 11 Global Step: 117620 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:05:20,966-Speed 128.50 samples/sec Loss 3.1207 Epoch: 11 Global Step: 117625 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:05:40,915-Speed 128.33 samples/sec Loss 3.1661 Epoch: 11 Global Step: 117630 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:06:00,723-Speed 129.25 samples/sec Loss 3.2630 Epoch: 11 Global Step: 117635 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:06:20,366-Speed 130.33 samples/sec Loss 3.1955 Epoch: 11 Global Step: 117640 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:06:40,067-Speed 129.95 samples/sec Loss 3.1961 Epoch: 11 Global Step: 117645 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:06:59,717-Speed 130.29 samples/sec Loss 3.2760 Epoch: 11 Global Step: 117650 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:07:19,374-Speed 130.25 samples/sec Loss 3.1661 Epoch: 11 Global Step: 117655 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:07:39,017-Speed 130.33 samples/sec Loss 3.0882 Epoch: 11 Global Step: 117660 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:07:58,665-Speed 130.30 samples/sec Loss 3.1117 Epoch: 11 Global Step: 117665 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:08:18,736-Speed 127.55 samples/sec Loss 3.0863 Epoch: 11 Global Step: 117670 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:08:38,229-Speed 131.34 samples/sec Loss 3.2260 Epoch: 11 Global Step: 117675 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:08:58,042-Speed 129.22 samples/sec Loss 3.1690 Epoch: 11 Global Step: 117680 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:09:17,740-Speed 129.97 samples/sec Loss 2.9728 Epoch: 11 Global Step: 117685 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:09:37,396-Speed 130.25 samples/sec Loss 3.2042 Epoch: 11 Global Step: 117690 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:09:57,078-Speed 130.08 samples/sec Loss 3.1259 Epoch: 11 Global Step: 117695 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:10:16,724-Speed 130.31 samples/sec Loss 3.3070 Epoch: 11 Global Step: 117700 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:10:36,367-Speed 130.33 samples/sec Loss 3.1603 Epoch: 11 Global Step: 117705 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:10:56,054-Speed 130.04 samples/sec Loss 3.1710 Epoch: 11 Global Step: 117710 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:11:16,039-Speed 128.10 samples/sec Loss 3.1609 Epoch: 11 Global Step: 117715 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:11:35,524-Speed 131.40 samples/sec Loss 3.0540 Epoch: 11 Global Step: 117720 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:11:55,283-Speed 129.57 samples/sec Loss 3.0682 Epoch: 11 Global Step: 117725 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:12:15,009-Speed 129.80 samples/sec Loss 2.9127 Epoch: 11 Global Step: 117730 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:12:34,629-Speed 130.48 samples/sec Loss 3.1769 Epoch: 11 Global Step: 117735 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:12:54,301-Speed 130.14 samples/sec Loss 3.0653 Epoch: 11 Global Step: 117740 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:13:13,958-Speed 130.24 samples/sec Loss 3.1622 Epoch: 11 Global Step: 117745 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:13:33,646-Speed 130.04 samples/sec Loss 3.1416 Epoch: 11 Global Step: 117750 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:13:53,188-Speed 131.01 samples/sec Loss 3.2504 Epoch: 11 Global Step: 117755 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:14:13,167-Speed 128.14 samples/sec Loss 3.0851 Epoch: 11 Global Step: 117760 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:14:32,810-Speed 130.33 samples/sec Loss 3.2073 Epoch: 11 Global Step: 117765 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:14:52,721-Speed 128.58 samples/sec Loss 3.3155 Epoch: 11 Global Step: 117770 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:15:12,352-Speed 130.41 samples/sec Loss 3.2082 Epoch: 11 Global Step: 117775 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:15:32,020-Speed 130.17 samples/sec Loss 3.1046 Epoch: 11 Global Step: 117780 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:15:51,667-Speed 130.31 samples/sec Loss 3.0974 Epoch: 11 Global Step: 117785 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:16:11,185-Speed 131.16 samples/sec Loss 3.0584 Epoch: 11 Global Step: 117790 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:16:30,880-Speed 129.99 samples/sec Loss 3.1152 Epoch: 11 Global Step: 117795 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:16:50,500-Speed 130.48 samples/sec Loss 3.0875 Epoch: 11 Global Step: 117800 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:17:10,478-Speed 128.15 samples/sec Loss 3.1889 Epoch: 11 Global Step: 117805 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:17:30,262-Speed 129.40 samples/sec Loss 3.0630 Epoch: 11 Global Step: 117810 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:17:49,947-Speed 130.06 samples/sec Loss 3.3286 Epoch: 11 Global Step: 117815 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:18:09,602-Speed 130.25 samples/sec Loss 3.0339 Epoch: 11 Global Step: 117820 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:18:29,274-Speed 130.15 samples/sec Loss 3.2797 Epoch: 11 Global Step: 117825 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:18:48,975-Speed 129.95 samples/sec Loss 3.2494 Epoch: 11 Global Step: 117830 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:19:08,622-Speed 130.31 samples/sec Loss 3.2916 Epoch: 11 Global Step: 117835 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:19:28,304-Speed 130.07 samples/sec Loss 3.1999 Epoch: 11 Global Step: 117840 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:19:48,120-Speed 129.20 samples/sec Loss 3.1596 Epoch: 11 Global Step: 117845 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:20:07,978-Speed 128.92 samples/sec Loss 3.1354 Epoch: 11 Global Step: 117850 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:20:27,700-Speed 129.81 samples/sec Loss 3.1048 Epoch: 11 Global Step: 117855 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:20:47,308-Speed 130.57 samples/sec Loss 3.2492 Epoch: 11 Global Step: 117860 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:21:07,046-Speed 129.70 samples/sec Loss 3.0892 Epoch: 11 Global Step: 117865 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:21:26,676-Speed 130.42 samples/sec Loss 3.1599 Epoch: 11 Global Step: 117870 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:21:46,193-Speed 131.17 samples/sec Loss 3.0185 Epoch: 11 Global Step: 117875 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:22:05,842-Speed 130.29 samples/sec Loss 3.2823 Epoch: 11 Global Step: 117880 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:22:25,467-Speed 130.46 samples/sec Loss 3.1224 Epoch: 11 Global Step: 117885 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:22:45,155-Speed 130.04 samples/sec Loss 3.2032 Epoch: 11 Global Step: 117890 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:23:04,983-Speed 129.12 samples/sec Loss 3.1158 Epoch: 11 Global Step: 117895 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:23:24,830-Speed 128.99 samples/sec Loss 3.2055 Epoch: 11 Global Step: 117900 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:23:44,496-Speed 130.18 samples/sec Loss 3.0900 Epoch: 11 Global Step: 117905 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:24:04,159-Speed 130.20 samples/sec Loss 3.3137 Epoch: 11 Global Step: 117910 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:24:23,758-Speed 130.62 samples/sec Loss 3.2989 Epoch: 11 Global Step: 117915 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:24:43,292-Speed 131.06 samples/sec Loss 3.3085 Epoch: 11 Global Step: 117920 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:25:02,973-Speed 130.08 samples/sec Loss 3.3621 Epoch: 11 Global Step: 117925 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:25:22,654-Speed 130.09 samples/sec Loss 3.2957 Epoch: 11 Global Step: 117930 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:25:42,339-Speed 130.06 samples/sec Loss 3.1555 Epoch: 11 Global Step: 117935 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:26:02,223-Speed 128.75 samples/sec Loss 3.4189 Epoch: 11 Global Step: 117940 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:26:21,811-Speed 130.70 samples/sec Loss 2.9745 Epoch: 11 Global Step: 117945 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:26:41,594-Speed 129.41 samples/sec Loss 3.1185 Epoch: 11 Global Step: 117950 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:27:01,239-Speed 130.32 samples/sec Loss 3.0413 Epoch: 11 Global Step: 117955 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:27:20,839-Speed 130.62 samples/sec Loss 3.2546 Epoch: 11 Global Step: 117960 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:27:40,492-Speed 130.27 samples/sec Loss 3.1912 Epoch: 11 Global Step: 117965 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:28:00,354-Speed 128.89 samples/sec Loss 3.1299 Epoch: 11 Global Step: 117970 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:28:19,976-Speed 130.47 samples/sec Loss 3.0986 Epoch: 11 Global Step: 117975 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:28:39,746-Speed 129.50 samples/sec Loss 3.2060 Epoch: 11 Global Step: 117980 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:28:59,719-Speed 128.18 samples/sec Loss 3.2444 Epoch: 11 Global Step: 117985 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:29:19,228-Speed 131.23 samples/sec Loss 3.2024 Epoch: 11 Global Step: 117990 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:29:39,140-Speed 128.57 samples/sec Loss 3.1247 Epoch: 11 Global Step: 117995 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:29:58,754-Speed 130.52 samples/sec Loss 3.1965 Epoch: 11 Global Step: 118000 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:30:18,395-Speed 130.35 samples/sec Loss 3.0085 Epoch: 11 Global Step: 118005 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:30:38,085-Speed 130.02 samples/sec Loss 3.2631 Epoch: 11 Global Step: 118010 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:30:57,739-Speed 130.27 samples/sec Loss 3.2180 Epoch: 11 Global Step: 118015 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:31:17,408-Speed 130.16 samples/sec Loss 3.3351 Epoch: 11 Global Step: 118020 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:31:36,962-Speed 130.93 samples/sec Loss 3.1410 Epoch: 11 Global Step: 118025 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:31:56,868-Speed 128.61 samples/sec Loss 3.5208 Epoch: 11 Global Step: 118030 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:32:16,387-Speed 131.16 samples/sec Loss 3.2852 Epoch: 11 Global Step: 118035 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:32:36,200-Speed 129.22 samples/sec Loss 3.0173 Epoch: 11 Global Step: 118040 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:32:55,765-Speed 130.85 samples/sec Loss 3.1993 Epoch: 11 Global Step: 118045 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:33:15,283-Speed 131.17 samples/sec Loss 3.0842 Epoch: 11 Global Step: 118050 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:33:35,002-Speed 129.84 samples/sec Loss 3.1771 Epoch: 11 Global Step: 118055 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:33:54,640-Speed 130.36 samples/sec Loss 3.0971 Epoch: 11 Global Step: 118060 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:34:14,291-Speed 130.28 samples/sec Loss 3.0306 Epoch: 11 Global Step: 118065 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:34:33,896-Speed 130.59 samples/sec Loss 3.3626 Epoch: 11 Global Step: 118070 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:34:53,784-Speed 128.72 samples/sec Loss 3.3046 Epoch: 11 Global Step: 118075 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:35:13,408-Speed 130.46 samples/sec Loss 3.0430 Epoch: 11 Global Step: 118080 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:35:33,389-Speed 128.12 samples/sec Loss 3.3094 Epoch: 11 Global Step: 118085 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:35:52,991-Speed 130.61 samples/sec Loss 3.0858 Epoch: 11 Global Step: 118090 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:36:12,588-Speed 130.64 samples/sec Loss 3.2756 Epoch: 11 Global Step: 118095 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:36:32,245-Speed 130.24 samples/sec Loss 3.2229 Epoch: 11 Global Step: 118100 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:36:51,898-Speed 130.27 samples/sec Loss 3.1037 Epoch: 11 Global Step: 118105 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:37:11,664-Speed 129.53 samples/sec Loss 3.2313 Epoch: 11 Global Step: 118110 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:37:31,264-Speed 130.62 samples/sec Loss 3.1497 Epoch: 11 Global Step: 118115 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:37:51,232-Speed 128.21 samples/sec Loss 3.0356 Epoch: 11 Global Step: 118120 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:38:10,865-Speed 130.40 samples/sec Loss 3.1807 Epoch: 11 Global Step: 118125 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:38:30,642-Speed 129.45 samples/sec Loss 3.1558 Epoch: 11 Global Step: 118130 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:38:50,237-Speed 130.65 samples/sec Loss 3.3026 Epoch: 11 Global Step: 118135 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:39:09,972-Speed 129.73 samples/sec Loss 3.1812 Epoch: 11 Global Step: 118140 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:39:29,642-Speed 130.15 samples/sec Loss 3.2289 Epoch: 11 Global Step: 118145 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:39:49,150-Speed 131.24 samples/sec Loss 3.0581 Epoch: 11 Global Step: 118150 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:40:08,806-Speed 130.25 samples/sec Loss 3.1816 Epoch: 11 Global Step: 118155 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:40:28,437-Speed 130.41 samples/sec Loss 2.9514 Epoch: 11 Global Step: 118160 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:40:48,349-Speed 128.58 samples/sec Loss 3.1929 Epoch: 11 Global Step: 118165 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:41:08,175-Speed 129.13 samples/sec Loss 3.1161 Epoch: 11 Global Step: 118170 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:41:27,801-Speed 130.45 samples/sec Loss 3.1575 Epoch: 11 Global Step: 118175 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:41:47,415-Speed 130.53 samples/sec Loss 3.1141 Epoch: 11 Global Step: 118180 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:42:07,123-Speed 129.90 samples/sec Loss 3.2752 Epoch: 11 Global Step: 118185 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:42:26,789-Speed 130.18 samples/sec Loss 3.0432 Epoch: 11 Global Step: 118190 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:42:46,328-Speed 131.02 samples/sec Loss 3.1475 Epoch: 11 Global Step: 118195 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:43:05,970-Speed 130.34 samples/sec Loss 3.1982 Epoch: 11 Global Step: 118200 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:43:25,608-Speed 130.37 samples/sec Loss 3.1348 Epoch: 11 Global Step: 118205 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:43:45,525-Speed 128.54 samples/sec Loss 3.1080 Epoch: 11 Global Step: 118210 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:44:05,150-Speed 130.45 samples/sec Loss 3.2716 Epoch: 11 Global Step: 118215 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:44:24,753-Speed 130.61 samples/sec Loss 3.1212 Epoch: 11 Global Step: 118220 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:44:44,369-Speed 130.51 samples/sec Loss 3.2548 Epoch: 11 Global Step: 118225 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:45:04,012-Speed 130.34 samples/sec Loss 3.0009 Epoch: 11 Global Step: 118230 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:45:23,620-Speed 130.56 samples/sec Loss 3.1656 Epoch: 11 Global Step: 118235 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:45:43,173-Speed 130.94 samples/sec Loss 3.2197 Epoch: 11 Global Step: 118240 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:46:02,829-Speed 130.25 samples/sec Loss 3.1734 Epoch: 11 Global Step: 118245 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:46:22,501-Speed 130.14 samples/sec Loss 3.1004 Epoch: 11 Global Step: 118250 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:46:42,355-Speed 128.95 samples/sec Loss 3.3543 Epoch: 11 Global Step: 118255 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:47:01,916-Speed 130.88 samples/sec Loss 3.2600 Epoch: 11 Global Step: 118260 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:47:21,527-Speed 130.55 samples/sec Loss 3.1219 Epoch: 11 Global Step: 118265 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:47:41,269-Speed 129.68 samples/sec Loss 3.0822 Epoch: 11 Global Step: 118270 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:48:00,875-Speed 130.58 samples/sec Loss 3.1437 Epoch: 11 Global Step: 118275 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:48:20,476-Speed 130.61 samples/sec Loss 3.4065 Epoch: 11 Global Step: 118280 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:48:40,028-Speed 130.94 samples/sec Loss 2.9876 Epoch: 11 Global Step: 118285 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:48:59,743-Speed 129.86 samples/sec Loss 3.2456 Epoch: 11 Global Step: 118290 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:49:19,444-Speed 129.95 samples/sec Loss 3.2239 Epoch: 11 Global Step: 118295 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:49:39,300-Speed 128.93 samples/sec Loss 3.2433 Epoch: 11 Global Step: 118300 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:49:58,856-Speed 130.91 samples/sec Loss 3.1431 Epoch: 11 Global Step: 118305 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:50:18,553-Speed 129.98 samples/sec Loss 3.1923 Epoch: 11 Global Step: 118310 Fp16 Grad Scale: 8192 Required: 155 hours Training: 2021-06-12 14:50:38,217-Speed 130.20 samples/sec Loss 3.2346 Epoch: 11 Global Step: 118315 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:50:57,874-Speed 130.25 samples/sec Loss 3.0881 Epoch: 11 Global Step: 118320 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:51:17,452-Speed 130.76 samples/sec Loss 3.2368 Epoch: 11 Global Step: 118325 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:51:37,306-Speed 128.95 samples/sec Loss 3.2747 Epoch: 11 Global Step: 118330 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:51:57,014-Speed 129.90 samples/sec Loss 3.3906 Epoch: 11 Global Step: 118335 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:52:16,655-Speed 130.34 samples/sec Loss 3.2415 Epoch: 11 Global Step: 118340 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:52:36,491-Speed 129.07 samples/sec Loss 3.2425 Epoch: 11 Global Step: 118345 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:52:56,265-Speed 129.47 samples/sec Loss 3.2757 Epoch: 11 Global Step: 118350 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:53:15,896-Speed 130.42 samples/sec Loss 3.2460 Epoch: 11 Global Step: 118355 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:53:35,549-Speed 130.26 samples/sec Loss 3.2836 Epoch: 11 Global Step: 118360 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:53:55,135-Speed 130.72 samples/sec Loss 3.1072 Epoch: 11 Global Step: 118365 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:54:14,709-Speed 130.79 samples/sec Loss 3.2236 Epoch: 11 Global Step: 118370 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:54:34,358-Speed 130.29 samples/sec Loss 3.4503 Epoch: 11 Global Step: 118375 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:54:54,080-Speed 129.81 samples/sec Loss 3.2386 Epoch: 11 Global Step: 118380 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:55:13,785-Speed 129.92 samples/sec Loss 3.2635 Epoch: 11 Global Step: 118385 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:55:33,600-Speed 129.21 samples/sec Loss 3.2954 Epoch: 11 Global Step: 118390 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:55:53,228-Speed 130.43 samples/sec Loss 3.1161 Epoch: 11 Global Step: 118395 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:56:12,883-Speed 130.26 samples/sec Loss 3.3376 Epoch: 11 Global Step: 118400 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:56:32,550-Speed 130.18 samples/sec Loss 3.3500 Epoch: 11 Global Step: 118405 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:56:52,280-Speed 129.76 samples/sec Loss 3.2097 Epoch: 11 Global Step: 118410 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:57:11,906-Speed 130.44 samples/sec Loss 3.0065 Epoch: 11 Global Step: 118415 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:57:31,552-Speed 130.32 samples/sec Loss 3.1253 Epoch: 11 Global Step: 118420 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:57:51,226-Speed 130.13 samples/sec Loss 3.1829 Epoch: 11 Global Step: 118425 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:58:10,864-Speed 130.37 samples/sec Loss 3.3294 Epoch: 11 Global Step: 118430 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:58:30,822-Speed 128.28 samples/sec Loss 3.3664 Epoch: 11 Global Step: 118435 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:58:50,314-Speed 131.34 samples/sec Loss 3.1838 Epoch: 11 Global Step: 118440 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:59:10,140-Speed 129.13 samples/sec Loss 3.0725 Epoch: 11 Global Step: 118445 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:59:29,751-Speed 130.55 samples/sec Loss 3.0724 Epoch: 11 Global Step: 118450 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 14:59:49,471-Speed 129.83 samples/sec Loss 3.1747 Epoch: 11 Global Step: 118455 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:00:09,110-Speed 130.36 samples/sec Loss 3.2202 Epoch: 11 Global Step: 118460 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:00:28,806-Speed 129.99 samples/sec Loss 3.1827 Epoch: 11 Global Step: 118465 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:00:48,466-Speed 130.21 samples/sec Loss 3.3866 Epoch: 11 Global Step: 118470 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:01:08,089-Speed 130.47 samples/sec Loss 3.0961 Epoch: 11 Global Step: 118475 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:01:28,470-Speed 125.62 samples/sec Loss 3.1991 Epoch: 11 Global Step: 118480 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:01:48,059-Speed 130.69 samples/sec Loss 3.0498 Epoch: 11 Global Step: 118485 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:02:08,156-Speed 127.39 samples/sec Loss 3.2514 Epoch: 11 Global Step: 118490 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:02:27,749-Speed 130.66 samples/sec Loss 3.0789 Epoch: 11 Global Step: 118495 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:02:47,333-Speed 130.73 samples/sec Loss 3.4484 Epoch: 11 Global Step: 118500 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:03:06,996-Speed 130.21 samples/sec Loss 3.2252 Epoch: 11 Global Step: 118505 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:03:26,688-Speed 130.00 samples/sec Loss 3.1607 Epoch: 11 Global Step: 118510 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:03:46,349-Speed 130.22 samples/sec Loss 3.2705 Epoch: 11 Global Step: 118515 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:04:06,030-Speed 130.08 samples/sec Loss 3.2278 Epoch: 11 Global Step: 118520 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:04:25,864-Speed 129.08 samples/sec Loss 3.3255 Epoch: 11 Global Step: 118525 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:04:45,566-Speed 129.94 samples/sec Loss 2.9611 Epoch: 11 Global Step: 118530 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:05:05,318-Speed 129.61 samples/sec Loss 3.1317 Epoch: 11 Global Step: 118535 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:05:25,146-Speed 129.12 samples/sec Loss 3.2527 Epoch: 11 Global Step: 118540 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:05:44,793-Speed 130.30 samples/sec Loss 3.4364 Epoch: 11 Global Step: 118545 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:06:04,497-Speed 129.93 samples/sec Loss 2.9864 Epoch: 11 Global Step: 118550 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:06:24,100-Speed 130.60 samples/sec Loss 3.4206 Epoch: 11 Global Step: 118555 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:06:43,736-Speed 130.38 samples/sec Loss 3.2302 Epoch: 11 Global Step: 118560 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:07:03,255-Speed 131.16 samples/sec Loss 3.2969 Epoch: 11 Global Step: 118565 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:07:23,084-Speed 129.12 samples/sec Loss 3.3474 Epoch: 11 Global Step: 118570 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:07:42,939-Speed 128.94 samples/sec Loss 3.1831 Epoch: 11 Global Step: 118575 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:08:02,453-Speed 131.20 samples/sec Loss 3.4219 Epoch: 11 Global Step: 118580 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:08:21,968-Speed 131.19 samples/sec Loss 3.0938 Epoch: 11 Global Step: 118585 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:08:41,611-Speed 130.33 samples/sec Loss 3.1951 Epoch: 11 Global Step: 118590 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:09:01,313-Speed 129.94 samples/sec Loss 3.4167 Epoch: 11 Global Step: 118595 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:09:21,036-Speed 129.81 samples/sec Loss 3.1107 Epoch: 11 Global Step: 118600 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:09:40,572-Speed 131.05 samples/sec Loss 3.2082 Epoch: 11 Global Step: 118605 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:10:00,095-Speed 131.14 samples/sec Loss 3.2576 Epoch: 11 Global Step: 118610 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:10:19,958-Speed 128.89 samples/sec Loss 3.2041 Epoch: 11 Global Step: 118615 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:10:39,807-Speed 128.98 samples/sec Loss 3.3721 Epoch: 11 Global Step: 118620 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:10:59,334-Speed 131.11 samples/sec Loss 3.3041 Epoch: 11 Global Step: 118625 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:11:19,050-Speed 129.85 samples/sec Loss 3.2254 Epoch: 11 Global Step: 118630 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:11:38,706-Speed 130.25 samples/sec Loss 3.1291 Epoch: 11 Global Step: 118635 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:11:58,369-Speed 130.20 samples/sec Loss 3.1699 Epoch: 11 Global Step: 118640 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:12:17,994-Speed 130.45 samples/sec Loss 3.1710 Epoch: 11 Global Step: 118645 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:12:37,522-Speed 131.10 samples/sec Loss 3.2362 Epoch: 11 Global Step: 118650 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:12:57,167-Speed 130.32 samples/sec Loss 3.1051 Epoch: 11 Global Step: 118655 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:13:17,111-Speed 128.37 samples/sec Loss 3.1553 Epoch: 11 Global Step: 118660 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:13:36,892-Speed 129.43 samples/sec Loss 3.3317 Epoch: 11 Global Step: 118665 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:13:56,431-Speed 131.03 samples/sec Loss 3.2574 Epoch: 11 Global Step: 118670 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:14:16,095-Speed 130.19 samples/sec Loss 3.0449 Epoch: 11 Global Step: 118675 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:14:35,782-Speed 130.04 samples/sec Loss 3.2347 Epoch: 11 Global Step: 118680 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:14:55,443-Speed 130.21 samples/sec Loss 3.2704 Epoch: 11 Global Step: 118685 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:15:14,970-Speed 131.11 samples/sec Loss 3.3327 Epoch: 11 Global Step: 118690 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:15:34,539-Speed 130.83 samples/sec Loss 3.2666 Epoch: 11 Global Step: 118695 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:15:54,209-Speed 130.15 samples/sec Loss 3.2689 Epoch: 11 Global Step: 118700 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:16:14,212-Speed 127.99 samples/sec Loss 3.3758 Epoch: 11 Global Step: 118705 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:16:33,784-Speed 130.80 samples/sec Loss 3.2896 Epoch: 11 Global Step: 118710 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:16:53,422-Speed 130.37 samples/sec Loss 3.4008 Epoch: 11 Global Step: 118715 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:17:13,354-Speed 128.45 samples/sec Loss 3.1770 Epoch: 11 Global Step: 118720 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:17:33,001-Speed 130.30 samples/sec Loss 3.5265 Epoch: 11 Global Step: 118725 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:17:52,649-Speed 130.31 samples/sec Loss 3.2150 Epoch: 11 Global Step: 118730 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:18:12,181-Speed 131.07 samples/sec Loss 3.2673 Epoch: 11 Global Step: 118735 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:18:31,841-Speed 130.22 samples/sec Loss 3.3582 Epoch: 11 Global Step: 118740 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:18:51,538-Speed 129.98 samples/sec Loss 3.0027 Epoch: 11 Global Step: 118745 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:19:11,558-Speed 127.88 samples/sec Loss 3.0027 Epoch: 11 Global Step: 118750 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:19:31,053-Speed 131.32 samples/sec Loss 3.2300 Epoch: 11 Global Step: 118755 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:19:50,924-Speed 128.84 samples/sec Loss 3.0768 Epoch: 11 Global Step: 118760 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:20:10,637-Speed 129.87 samples/sec Loss 3.3472 Epoch: 11 Global Step: 118765 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:20:30,244-Speed 130.57 samples/sec Loss 3.2673 Epoch: 11 Global Step: 118770 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:20:49,870-Speed 130.45 samples/sec Loss 3.1078 Epoch: 11 Global Step: 118775 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:21:09,582-Speed 129.88 samples/sec Loss 3.2624 Epoch: 11 Global Step: 118780 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:21:29,248-Speed 130.18 samples/sec Loss 3.2389 Epoch: 11 Global Step: 118785 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:21:48,913-Speed 130.19 samples/sec Loss 3.0483 Epoch: 11 Global Step: 118790 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:22:08,797-Speed 128.76 samples/sec Loss 3.2625 Epoch: 11 Global Step: 118795 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:22:28,312-Speed 131.19 samples/sec Loss 3.1746 Epoch: 11 Global Step: 118800 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:22:48,062-Speed 129.63 samples/sec Loss 3.2878 Epoch: 11 Global Step: 118805 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:23:07,693-Speed 130.41 samples/sec Loss 3.1493 Epoch: 11 Global Step: 118810 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:23:27,343-Speed 130.29 samples/sec Loss 3.3667 Epoch: 11 Global Step: 118815 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:23:46,855-Speed 131.21 samples/sec Loss 3.3060 Epoch: 11 Global Step: 118820 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:24:06,528-Speed 130.14 samples/sec Loss 3.2369 Epoch: 11 Global Step: 118825 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:24:26,367-Speed 129.05 samples/sec Loss 3.0873 Epoch: 11 Global Step: 118830 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:24:46,010-Speed 130.33 samples/sec Loss 3.3052 Epoch: 11 Global Step: 118835 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:25:05,847-Speed 129.06 samples/sec Loss 3.2708 Epoch: 11 Global Step: 118840 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:25:25,394-Speed 130.97 samples/sec Loss 3.0682 Epoch: 11 Global Step: 118845 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:25:45,234-Speed 129.04 samples/sec Loss 3.0977 Epoch: 11 Global Step: 118850 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:26:04,874-Speed 130.35 samples/sec Loss 3.2586 Epoch: 11 Global Step: 118855 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:26:24,532-Speed 130.23 samples/sec Loss 3.4244 Epoch: 11 Global Step: 118860 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:26:44,180-Speed 130.31 samples/sec Loss 3.1634 Epoch: 11 Global Step: 118865 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:27:03,836-Speed 130.25 samples/sec Loss 3.0874 Epoch: 11 Global Step: 118870 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:27:23,550-Speed 129.86 samples/sec Loss 3.1114 Epoch: 11 Global Step: 118875 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:27:43,207-Speed 130.24 samples/sec Loss 3.2568 Epoch: 11 Global Step: 118880 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:28:03,056-Speed 128.98 samples/sec Loss 3.1305 Epoch: 11 Global Step: 118885 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:28:22,777-Speed 129.82 samples/sec Loss 3.2038 Epoch: 11 Global Step: 118890 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:28:42,546-Speed 129.50 samples/sec Loss 3.1392 Epoch: 11 Global Step: 118895 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:29:02,092-Speed 130.98 samples/sec Loss 3.0814 Epoch: 11 Global Step: 118900 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:29:21,716-Speed 130.46 samples/sec Loss 3.2800 Epoch: 11 Global Step: 118905 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:29:41,395-Speed 130.10 samples/sec Loss 3.2073 Epoch: 11 Global Step: 118910 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:30:01,152-Speed 129.58 samples/sec Loss 3.1130 Epoch: 11 Global Step: 118915 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:30:20,805-Speed 130.27 samples/sec Loss 3.2303 Epoch: 11 Global Step: 118920 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:30:40,354-Speed 130.96 samples/sec Loss 3.2639 Epoch: 11 Global Step: 118925 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:31:00,170-Speed 129.20 samples/sec Loss 3.1684 Epoch: 11 Global Step: 118930 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:31:19,874-Speed 129.93 samples/sec Loss 3.0316 Epoch: 11 Global Step: 118935 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:31:40,254-Speed 125.63 samples/sec Loss 3.3524 Epoch: 11 Global Step: 118940 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:31:59,899-Speed 130.32 samples/sec Loss 3.2420 Epoch: 11 Global Step: 118945 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:32:19,549-Speed 130.29 samples/sec Loss 3.3171 Epoch: 11 Global Step: 118950 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:32:39,172-Speed 130.46 samples/sec Loss 3.2363 Epoch: 11 Global Step: 118955 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:32:58,817-Speed 130.32 samples/sec Loss 3.3275 Epoch: 11 Global Step: 118960 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:33:18,418-Speed 130.62 samples/sec Loss 3.1561 Epoch: 11 Global Step: 118965 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:33:38,216-Speed 129.31 samples/sec Loss 3.1507 Epoch: 11 Global Step: 118970 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:33:58,072-Speed 128.94 samples/sec Loss 3.0549 Epoch: 11 Global Step: 118975 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:34:17,955-Speed 128.76 samples/sec Loss 3.2053 Epoch: 11 Global Step: 118980 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:34:37,546-Speed 130.68 samples/sec Loss 3.1829 Epoch: 11 Global Step: 118985 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:34:57,199-Speed 130.27 samples/sec Loss 3.3487 Epoch: 11 Global Step: 118990 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:35:16,841-Speed 130.34 samples/sec Loss 3.3558 Epoch: 11 Global Step: 118995 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:35:36,525-Speed 130.06 samples/sec Loss 3.2640 Epoch: 11 Global Step: 119000 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:35:56,118-Speed 130.67 samples/sec Loss 3.2111 Epoch: 11 Global Step: 119005 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:36:15,720-Speed 130.61 samples/sec Loss 3.1113 Epoch: 11 Global Step: 119010 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:36:35,348-Speed 130.43 samples/sec Loss 3.2314 Epoch: 11 Global Step: 119015 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:36:55,361-Speed 127.93 samples/sec Loss 3.2140 Epoch: 11 Global Step: 119020 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:37:14,854-Speed 131.33 samples/sec Loss 3.2517 Epoch: 11 Global Step: 119025 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:37:34,529-Speed 130.12 samples/sec Loss 3.2165 Epoch: 11 Global Step: 119030 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:37:54,218-Speed 130.02 samples/sec Loss 3.0410 Epoch: 11 Global Step: 119035 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:38:13,809-Speed 130.68 samples/sec Loss 3.1859 Epoch: 11 Global Step: 119040 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:38:33,452-Speed 130.33 samples/sec Loss 3.1971 Epoch: 11 Global Step: 119045 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:38:52,996-Speed 130.99 samples/sec Loss 3.2166 Epoch: 11 Global Step: 119050 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:39:12,638-Speed 130.34 samples/sec Loss 3.2681 Epoch: 11 Global Step: 119055 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:39:32,285-Speed 130.31 samples/sec Loss 3.1467 Epoch: 11 Global Step: 119060 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:39:52,321-Speed 127.78 samples/sec Loss 3.2670 Epoch: 11 Global Step: 119065 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:40:11,821-Speed 131.28 samples/sec Loss 3.2202 Epoch: 11 Global Step: 119070 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:40:31,470-Speed 130.30 samples/sec Loss 3.3401 Epoch: 11 Global Step: 119075 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:40:51,281-Speed 129.23 samples/sec Loss 3.1975 Epoch: 11 Global Step: 119080 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:41:11,004-Speed 129.81 samples/sec Loss 3.0984 Epoch: 11 Global Step: 119085 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:41:30,649-Speed 130.32 samples/sec Loss 3.1899 Epoch: 11 Global Step: 119090 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:41:50,286-Speed 130.37 samples/sec Loss 3.1271 Epoch: 11 Global Step: 119095 Fp16 Grad Scale: 8192 Required: 154 hours Training: 2021-06-12 15:42:09,910-Speed 130.46 samples/sec Loss 3.2508 Epoch: 11 Global Step: 119100 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:42:29,520-Speed 130.56 samples/sec Loss 3.2115 Epoch: 11 Global Step: 119105 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:42:49,551-Speed 127.81 samples/sec Loss 3.3257 Epoch: 11 Global Step: 119110 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:43:09,038-Speed 131.37 samples/sec Loss 3.3171 Epoch: 11 Global Step: 119115 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:43:28,764-Speed 129.78 samples/sec Loss 3.3404 Epoch: 11 Global Step: 119120 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:43:48,471-Speed 129.92 samples/sec Loss 3.2247 Epoch: 11 Global Step: 119125 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:44:08,138-Speed 130.17 samples/sec Loss 3.3258 Epoch: 11 Global Step: 119130 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:44:27,776-Speed 130.37 samples/sec Loss 3.1351 Epoch: 11 Global Step: 119135 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:44:47,357-Speed 130.74 samples/sec Loss 3.1313 Epoch: 11 Global Step: 119140 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:45:06,927-Speed 130.83 samples/sec Loss 3.1750 Epoch: 11 Global Step: 119145 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:45:26,533-Speed 130.58 samples/sec Loss 3.4718 Epoch: 11 Global Step: 119150 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:45:46,599-Speed 127.58 samples/sec Loss 3.1845 Epoch: 11 Global Step: 119155 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:46:06,206-Speed 130.57 samples/sec Loss 3.0574 Epoch: 11 Global Step: 119160 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:46:26,399-Speed 126.78 samples/sec Loss 3.2927 Epoch: 11 Global Step: 119165 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:46:46,032-Speed 130.40 samples/sec Loss 3.2381 Epoch: 11 Global Step: 119170 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:47:05,652-Speed 130.48 samples/sec Loss 3.3803 Epoch: 11 Global Step: 119175 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:47:25,312-Speed 130.22 samples/sec Loss 3.2748 Epoch: 11 Global Step: 119180 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:47:44,937-Speed 130.45 samples/sec Loss 3.1005 Epoch: 11 Global Step: 119185 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:48:04,615-Speed 130.11 samples/sec Loss 3.1728 Epoch: 11 Global Step: 119190 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:48:24,268-Speed 130.26 samples/sec Loss 3.1953 Epoch: 11 Global Step: 119195 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:48:44,189-Speed 128.51 samples/sec Loss 3.2658 Epoch: 11 Global Step: 119200 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:49:03,670-Speed 131.42 samples/sec Loss 3.1363 Epoch: 11 Global Step: 119205 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:49:23,415-Speed 129.67 samples/sec Loss 3.0651 Epoch: 11 Global Step: 119210 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:49:43,066-Speed 130.29 samples/sec Loss 3.2369 Epoch: 11 Global Step: 119215 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:50:02,676-Speed 130.55 samples/sec Loss 3.1001 Epoch: 11 Global Step: 119220 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:50:22,316-Speed 130.35 samples/sec Loss 3.2050 Epoch: 11 Global Step: 119225 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:50:42,005-Speed 130.03 samples/sec Loss 3.1521 Epoch: 11 Global Step: 119230 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:51:01,651-Speed 130.31 samples/sec Loss 3.3102 Epoch: 11 Global Step: 119235 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:51:21,296-Speed 130.32 samples/sec Loss 3.1484 Epoch: 11 Global Step: 119240 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:51:41,151-Speed 128.95 samples/sec Loss 3.3192 Epoch: 11 Global Step: 119245 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:52:00,830-Speed 130.09 samples/sec Loss 3.2297 Epoch: 11 Global Step: 119250 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:52:20,605-Speed 129.46 samples/sec Loss 3.2074 Epoch: 11 Global Step: 119255 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:52:40,253-Speed 130.30 samples/sec Loss 3.3972 Epoch: 11 Global Step: 119260 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:52:59,906-Speed 130.27 samples/sec Loss 3.0866 Epoch: 11 Global Step: 119265 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:53:19,531-Speed 130.45 samples/sec Loss 3.2982 Epoch: 11 Global Step: 119270 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:53:39,155-Speed 130.46 samples/sec Loss 3.3567 Epoch: 11 Global Step: 119275 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:53:59,246-Speed 127.43 samples/sec Loss 3.3429 Epoch: 11 Global Step: 119280 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:54:18,803-Speed 130.91 samples/sec Loss 3.3173 Epoch: 11 Global Step: 119285 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:54:38,650-Speed 129.00 samples/sec Loss 3.0477 Epoch: 11 Global Step: 119290 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:54:58,502-Speed 128.96 samples/sec Loss 3.2835 Epoch: 11 Global Step: 119295 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:55:18,237-Speed 129.72 samples/sec Loss 3.3184 Epoch: 11 Global Step: 119300 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:55:37,779-Speed 131.01 samples/sec Loss 3.2026 Epoch: 11 Global Step: 119305 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:55:57,458-Speed 130.09 samples/sec Loss 3.4266 Epoch: 11 Global Step: 119310 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:56:17,144-Speed 130.05 samples/sec Loss 3.1970 Epoch: 11 Global Step: 119315 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:56:36,891-Speed 129.64 samples/sec Loss 3.3731 Epoch: 11 Global Step: 119320 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:56:56,551-Speed 130.22 samples/sec Loss 3.2360 Epoch: 11 Global Step: 119325 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:57:16,219-Speed 130.17 samples/sec Loss 3.1856 Epoch: 11 Global Step: 119330 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:57:36,083-Speed 128.88 samples/sec Loss 3.3609 Epoch: 11 Global Step: 119335 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:57:55,874-Speed 129.36 samples/sec Loss 3.2233 Epoch: 11 Global Step: 119340 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:58:15,494-Speed 130.49 samples/sec Loss 3.2052 Epoch: 11 Global Step: 119345 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:58:35,152-Speed 130.24 samples/sec Loss 3.1385 Epoch: 11 Global Step: 119350 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:58:54,781-Speed 130.43 samples/sec Loss 3.2613 Epoch: 11 Global Step: 119355 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:59:14,350-Speed 130.83 samples/sec Loss 3.3096 Epoch: 11 Global Step: 119360 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:59:33,986-Speed 130.38 samples/sec Loss 3.4268 Epoch: 11 Global Step: 119365 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 15:59:53,612-Speed 130.45 samples/sec Loss 3.1883 Epoch: 11 Global Step: 119370 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:00:13,251-Speed 130.36 samples/sec Loss 3.3256 Epoch: 11 Global Step: 119375 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:00:33,164-Speed 128.56 samples/sec Loss 3.2551 Epoch: 11 Global Step: 119380 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:00:52,863-Speed 129.97 samples/sec Loss 3.3171 Epoch: 11 Global Step: 119385 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:01:12,460-Speed 130.64 samples/sec Loss 3.3757 Epoch: 11 Global Step: 119390 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:01:32,093-Speed 130.40 samples/sec Loss 3.2474 Epoch: 11 Global Step: 119395 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:01:51,777-Speed 130.06 samples/sec Loss 3.2971 Epoch: 11 Global Step: 119400 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:02:11,444-Speed 130.17 samples/sec Loss 3.2635 Epoch: 11 Global Step: 119405 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:02:31,123-Speed 130.10 samples/sec Loss 3.2194 Epoch: 11 Global Step: 119410 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:02:50,703-Speed 130.76 samples/sec Loss 3.2044 Epoch: 11 Global Step: 119415 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:03:10,359-Speed 130.25 samples/sec Loss 3.2079 Epoch: 11 Global Step: 119420 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:03:30,380-Speed 127.87 samples/sec Loss 3.3202 Epoch: 11 Global Step: 119425 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:03:49,883-Speed 131.27 samples/sec Loss 3.1570 Epoch: 11 Global Step: 119430 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:04:09,569-Speed 130.05 samples/sec Loss 3.4232 Epoch: 11 Global Step: 119435 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:04:29,328-Speed 129.57 samples/sec Loss 3.1049 Epoch: 11 Global Step: 119440 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:04:48,887-Speed 130.90 samples/sec Loss 3.2148 Epoch: 11 Global Step: 119445 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:05:08,518-Speed 130.41 samples/sec Loss 3.4400 Epoch: 11 Global Step: 119450 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:05:28,077-Speed 130.90 samples/sec Loss 3.1241 Epoch: 11 Global Step: 119455 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:05:47,725-Speed 130.30 samples/sec Loss 3.3641 Epoch: 11 Global Step: 119460 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:06:07,367-Speed 130.34 samples/sec Loss 3.4095 Epoch: 11 Global Step: 119465 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:06:27,378-Speed 127.94 samples/sec Loss 3.2081 Epoch: 11 Global Step: 119470 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:06:46,900-Speed 131.14 samples/sec Loss 3.3150 Epoch: 11 Global Step: 119475 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:07:06,609-Speed 129.90 samples/sec Loss 3.2495 Epoch: 11 Global Step: 119480 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:07:26,298-Speed 130.03 samples/sec Loss 3.0046 Epoch: 11 Global Step: 119485 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:07:45,913-Speed 130.52 samples/sec Loss 3.3552 Epoch: 11 Global Step: 119490 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:08:05,553-Speed 130.35 samples/sec Loss 3.4429 Epoch: 11 Global Step: 119495 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:08:25,163-Speed 130.55 samples/sec Loss 3.2576 Epoch: 11 Global Step: 119500 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:08:44,780-Speed 130.51 samples/sec Loss 3.0267 Epoch: 11 Global Step: 119505 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:09:04,440-Speed 130.22 samples/sec Loss 3.2224 Epoch: 11 Global Step: 119510 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:09:24,401-Speed 128.26 samples/sec Loss 3.3257 Epoch: 11 Global Step: 119515 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:09:43,895-Speed 131.33 samples/sec Loss 3.1197 Epoch: 11 Global Step: 119520 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:10:03,560-Speed 130.19 samples/sec Loss 3.0837 Epoch: 11 Global Step: 119525 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:10:23,199-Speed 130.36 samples/sec Loss 3.0776 Epoch: 11 Global Step: 119530 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:10:42,748-Speed 130.96 samples/sec Loss 3.2943 Epoch: 11 Global Step: 119535 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:11:02,373-Speed 130.45 samples/sec Loss 3.3277 Epoch: 11 Global Step: 119540 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:11:22,088-Speed 129.86 samples/sec Loss 3.2911 Epoch: 11 Global Step: 119545 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:11:41,777-Speed 130.03 samples/sec Loss 3.2374 Epoch: 11 Global Step: 119550 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:12:01,355-Speed 130.76 samples/sec Loss 3.1365 Epoch: 11 Global Step: 119555 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:12:21,263-Speed 128.60 samples/sec Loss 3.1482 Epoch: 11 Global Step: 119560 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:12:40,697-Speed 131.73 samples/sec Loss 3.1237 Epoch: 11 Global Step: 119565 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:13:27,665-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-12 16:13:43,147-Speed 40.99 samples/sec Loss 3.0645 Epoch: 12 Global Step: 119570 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:14:02,969-Speed 129.16 samples/sec Loss 2.5503 Epoch: 12 Global Step: 119575 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:14:22,653-Speed 130.06 samples/sec Loss 2.4437 Epoch: 12 Global Step: 119580 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:14:42,441-Speed 129.38 samples/sec Loss 2.5538 Epoch: 12 Global Step: 119585 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:15:02,052-Speed 130.55 samples/sec Loss 2.3923 Epoch: 12 Global Step: 119590 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:15:21,769-Speed 129.84 samples/sec Loss 2.7076 Epoch: 12 Global Step: 119595 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:15:41,560-Speed 129.36 samples/sec Loss 2.4344 Epoch: 12 Global Step: 119600 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:16:01,333-Speed 129.47 samples/sec Loss 2.7163 Epoch: 12 Global Step: 119605 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:16:21,044-Speed 129.89 samples/sec Loss 2.5090 Epoch: 12 Global Step: 119610 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:16:40,877-Speed 129.08 samples/sec Loss 2.4236 Epoch: 12 Global Step: 119615 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:17:00,783-Speed 128.61 samples/sec Loss 2.5991 Epoch: 12 Global Step: 119620 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:17:20,400-Speed 130.51 samples/sec Loss 2.5325 Epoch: 12 Global Step: 119625 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:17:40,118-Speed 129.84 samples/sec Loss 2.4227 Epoch: 12 Global Step: 119630 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:17:59,698-Speed 130.75 samples/sec Loss 2.6697 Epoch: 12 Global Step: 119635 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:18:19,318-Speed 130.49 samples/sec Loss 2.4794 Epoch: 12 Global Step: 119640 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:18:39,107-Speed 129.37 samples/sec Loss 2.6240 Epoch: 12 Global Step: 119645 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:18:58,996-Speed 128.72 samples/sec Loss 2.3580 Epoch: 12 Global Step: 119650 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:19:18,434-Speed 131.71 samples/sec Loss 2.5324 Epoch: 12 Global Step: 119655 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:19:38,447-Speed 127.92 samples/sec Loss 2.5534 Epoch: 12 Global Step: 119660 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:19:58,317-Speed 128.84 samples/sec Loss 2.5721 Epoch: 12 Global Step: 119665 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:20:17,914-Speed 130.64 samples/sec Loss 2.4357 Epoch: 12 Global Step: 119670 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:20:37,815-Speed 128.64 samples/sec Loss 2.6018 Epoch: 12 Global Step: 119675 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:20:57,341-Speed 131.11 samples/sec Loss 2.4216 Epoch: 12 Global Step: 119680 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:21:17,171-Speed 129.11 samples/sec Loss 2.5826 Epoch: 12 Global Step: 119685 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:21:36,926-Speed 129.59 samples/sec Loss 2.6371 Epoch: 12 Global Step: 119690 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:21:56,754-Speed 129.12 samples/sec Loss 2.5248 Epoch: 12 Global Step: 119695 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:22:16,185-Speed 131.75 samples/sec Loss 2.4993 Epoch: 12 Global Step: 119700 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:22:36,106-Speed 128.51 samples/sec Loss 2.4638 Epoch: 12 Global Step: 119705 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:22:55,818-Speed 129.88 samples/sec Loss 2.6049 Epoch: 12 Global Step: 119710 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:23:15,302-Speed 131.39 samples/sec Loss 2.5838 Epoch: 12 Global Step: 119715 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:23:35,305-Speed 127.99 samples/sec Loss 2.5027 Epoch: 12 Global Step: 119720 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:23:54,766-Speed 131.55 samples/sec Loss 2.4962 Epoch: 12 Global Step: 119725 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:24:14,712-Speed 128.36 samples/sec Loss 2.8359 Epoch: 12 Global Step: 119730 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:24:34,496-Speed 129.40 samples/sec Loss 2.6323 Epoch: 12 Global Step: 119735 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:24:54,362-Speed 128.87 samples/sec Loss 2.4063 Epoch: 12 Global Step: 119740 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:25:13,801-Speed 131.70 samples/sec Loss 2.3986 Epoch: 12 Global Step: 119745 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:25:33,716-Speed 128.55 samples/sec Loss 2.6833 Epoch: 12 Global Step: 119750 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:25:53,422-Speed 129.92 samples/sec Loss 2.5225 Epoch: 12 Global Step: 119755 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:26:13,046-Speed 130.46 samples/sec Loss 2.6596 Epoch: 12 Global Step: 119760 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:26:32,746-Speed 129.96 samples/sec Loss 2.6017 Epoch: 12 Global Step: 119765 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:26:52,312-Speed 130.85 samples/sec Loss 2.4814 Epoch: 12 Global Step: 119770 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:27:12,245-Speed 128.44 samples/sec Loss 2.5984 Epoch: 12 Global Step: 119775 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:27:32,048-Speed 129.28 samples/sec Loss 2.5281 Epoch: 12 Global Step: 119780 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:27:51,737-Speed 130.03 samples/sec Loss 2.4828 Epoch: 12 Global Step: 119785 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:28:11,310-Speed 130.80 samples/sec Loss 2.5605 Epoch: 12 Global Step: 119790 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:28:31,234-Speed 128.50 samples/sec Loss 2.6697 Epoch: 12 Global Step: 119795 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:28:50,980-Speed 129.66 samples/sec Loss 2.6502 Epoch: 12 Global Step: 119800 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:29:10,591-Speed 130.54 samples/sec Loss 2.6887 Epoch: 12 Global Step: 119805 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:29:30,582-Speed 128.07 samples/sec Loss 2.5078 Epoch: 12 Global Step: 119810 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:29:50,227-Speed 130.32 samples/sec Loss 2.6220 Epoch: 12 Global Step: 119815 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:30:10,005-Speed 129.45 samples/sec Loss 2.5918 Epoch: 12 Global Step: 119820 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:30:29,877-Speed 128.83 samples/sec Loss 2.5786 Epoch: 12 Global Step: 119825 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:30:49,615-Speed 129.71 samples/sec Loss 2.6567 Epoch: 12 Global Step: 119830 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:31:09,343-Speed 129.77 samples/sec Loss 2.5719 Epoch: 12 Global Step: 119835 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:31:29,237-Speed 128.69 samples/sec Loss 2.6763 Epoch: 12 Global Step: 119840 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:31:48,907-Speed 130.16 samples/sec Loss 2.5183 Epoch: 12 Global Step: 119845 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:32:08,522-Speed 130.52 samples/sec Loss 2.6373 Epoch: 12 Global Step: 119850 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:32:28,268-Speed 129.65 samples/sec Loss 2.5999 Epoch: 12 Global Step: 119855 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:32:47,884-Speed 130.51 samples/sec Loss 2.3556 Epoch: 12 Global Step: 119860 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:33:07,593-Speed 129.90 samples/sec Loss 2.6452 Epoch: 12 Global Step: 119865 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:33:27,505-Speed 128.57 samples/sec Loss 2.5473 Epoch: 12 Global Step: 119870 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:33:47,239-Speed 129.73 samples/sec Loss 2.5175 Epoch: 12 Global Step: 119875 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:34:06,953-Speed 129.87 samples/sec Loss 2.5379 Epoch: 12 Global Step: 119880 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:34:26,760-Speed 129.25 samples/sec Loss 2.4990 Epoch: 12 Global Step: 119885 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:34:46,501-Speed 129.69 samples/sec Loss 2.5368 Epoch: 12 Global Step: 119890 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 16:35:06,045-Speed 130.99 samples/sec Loss 2.5601 Epoch: 12 Global Step: 119895 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:35:25,818-Speed 129.48 samples/sec Loss 2.7455 Epoch: 12 Global Step: 119900 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:35:45,396-Speed 130.76 samples/sec Loss 2.6185 Epoch: 12 Global Step: 119905 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:36:05,025-Speed 130.43 samples/sec Loss 2.5641 Epoch: 12 Global Step: 119910 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:36:24,831-Speed 129.26 samples/sec Loss 2.4688 Epoch: 12 Global Step: 119915 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:36:44,622-Speed 129.36 samples/sec Loss 2.6172 Epoch: 12 Global Step: 119920 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:37:04,338-Speed 129.85 samples/sec Loss 2.6668 Epoch: 12 Global Step: 119925 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:37:24,065-Speed 129.78 samples/sec Loss 2.6211 Epoch: 12 Global Step: 119930 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:37:43,974-Speed 128.59 samples/sec Loss 2.6952 Epoch: 12 Global Step: 119935 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:38:03,518-Speed 131.00 samples/sec Loss 2.4514 Epoch: 12 Global Step: 119940 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:38:23,345-Speed 129.12 samples/sec Loss 2.4898 Epoch: 12 Global Step: 119945 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:38:42,926-Speed 130.75 samples/sec Loss 2.4547 Epoch: 12 Global Step: 119950 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:39:02,650-Speed 129.79 samples/sec Loss 2.5547 Epoch: 12 Global Step: 119955 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:39:22,782-Speed 127.17 samples/sec Loss 2.6749 Epoch: 12 Global Step: 119960 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:39:42,499-Speed 129.85 samples/sec Loss 2.6835 Epoch: 12 Global Step: 119965 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:40:02,176-Speed 130.11 samples/sec Loss 2.5811 Epoch: 12 Global Step: 119970 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:40:21,869-Speed 130.00 samples/sec Loss 2.7055 Epoch: 12 Global Step: 119975 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:40:41,790-Speed 128.51 samples/sec Loss 2.6992 Epoch: 12 Global Step: 119980 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:41:01,304-Speed 131.20 samples/sec Loss 2.4075 Epoch: 12 Global Step: 119985 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:41:21,161-Speed 128.93 samples/sec Loss 2.6550 Epoch: 12 Global Step: 119990 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:41:40,684-Speed 131.14 samples/sec Loss 2.4370 Epoch: 12 Global Step: 119995 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:42:00,374-Speed 130.02 samples/sec Loss 2.6461 Epoch: 12 Global Step: 120000 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 16:56:51,139-[lfw][120000]XNorm: 22.420792 Training: 2021-06-12 16:56:51,139-[lfw][120000]Accuracy-Flip: 0.99833+-0.00236 Training: 2021-06-12 16:56:51,139-[lfw][120000]Accuracy-Highest: 0.99833 Training: 2021-06-12 17:14:06,349-[cfp_fp][120000]XNorm: 22.145785 Training: 2021-06-12 17:14:06,349-[cfp_fp][120000]Accuracy-Flip: 0.98971+-0.00403 Training: 2021-06-12 17:14:06,349-[cfp_fp][120000]Accuracy-Highest: 0.99043 Training: 2021-06-12 17:28:55,550-[agedb_30][120000]XNorm: 22.789384 Training: 2021-06-12 17:28:55,551-[agedb_30][120000]Accuracy-Flip: 0.98150+-0.00736 Training: 2021-06-12 17:28:55,551-[agedb_30][120000]Accuracy-Highest: 0.98200 Training: 2021-06-12 17:29:15,210-Speed 0.90 samples/sec Loss 2.7350 Epoch: 12 Global Step: 120005 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:29:34,983-Speed 129.48 samples/sec Loss 2.6493 Epoch: 12 Global Step: 120010 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:29:54,574-Speed 130.68 samples/sec Loss 2.4888 Epoch: 12 Global Step: 120015 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:30:14,022-Speed 131.64 samples/sec Loss 2.5829 Epoch: 12 Global Step: 120020 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:30:33,632-Speed 130.56 samples/sec Loss 2.6457 Epoch: 12 Global Step: 120025 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:30:53,065-Speed 131.74 samples/sec Loss 2.7673 Epoch: 12 Global Step: 120030 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:31:12,582-Speed 131.17 samples/sec Loss 2.8524 Epoch: 12 Global Step: 120035 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:31:32,165-Speed 130.73 samples/sec Loss 2.6187 Epoch: 12 Global Step: 120040 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:31:51,668-Speed 131.27 samples/sec Loss 2.6685 Epoch: 12 Global Step: 120045 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:32:11,332-Speed 130.19 samples/sec Loss 2.5549 Epoch: 12 Global Step: 120050 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:32:31,268-Speed 128.42 samples/sec Loss 2.7704 Epoch: 12 Global Step: 120055 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:32:50,809-Speed 131.01 samples/sec Loss 2.5962 Epoch: 12 Global Step: 120060 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:33:10,360-Speed 130.95 samples/sec Loss 2.7269 Epoch: 12 Global Step: 120065 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:33:31,209-Speed 122.79 samples/sec Loss 2.6334 Epoch: 12 Global Step: 120070 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:33:50,747-Speed 131.04 samples/sec Loss 2.8298 Epoch: 12 Global Step: 120075 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:34:10,469-Speed 129.81 samples/sec Loss 2.5711 Epoch: 12 Global Step: 120080 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:34:29,916-Speed 131.65 samples/sec Loss 2.5932 Epoch: 12 Global Step: 120085 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:34:49,521-Speed 130.59 samples/sec Loss 2.6797 Epoch: 12 Global Step: 120090 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:35:09,049-Speed 131.10 samples/sec Loss 2.7137 Epoch: 12 Global Step: 120095 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:35:29,110-Speed 127.62 samples/sec Loss 2.6304 Epoch: 12 Global Step: 120100 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:35:48,595-Speed 131.39 samples/sec Loss 2.6651 Epoch: 12 Global Step: 120105 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:36:08,248-Speed 130.26 samples/sec Loss 2.6654 Epoch: 12 Global Step: 120110 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:36:27,703-Speed 131.60 samples/sec Loss 2.6307 Epoch: 12 Global Step: 120115 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:36:47,336-Speed 130.40 samples/sec Loss 2.7584 Epoch: 12 Global Step: 120120 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:37:06,935-Speed 130.63 samples/sec Loss 2.5551 Epoch: 12 Global Step: 120125 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:37:26,486-Speed 130.95 samples/sec Loss 2.5404 Epoch: 12 Global Step: 120130 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:37:46,095-Speed 130.56 samples/sec Loss 2.4656 Epoch: 12 Global Step: 120135 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:38:05,538-Speed 131.68 samples/sec Loss 2.6775 Epoch: 12 Global Step: 120140 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:38:25,473-Speed 128.42 samples/sec Loss 2.6049 Epoch: 12 Global Step: 120145 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:38:44,911-Speed 131.71 samples/sec Loss 2.8633 Epoch: 12 Global Step: 120150 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:39:04,660-Speed 129.63 samples/sec Loss 2.6236 Epoch: 12 Global Step: 120155 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:39:24,200-Speed 131.02 samples/sec Loss 2.6684 Epoch: 12 Global Step: 120160 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:39:43,766-Speed 130.85 samples/sec Loss 2.7068 Epoch: 12 Global Step: 120165 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:40:03,537-Speed 129.48 samples/sec Loss 2.6687 Epoch: 12 Global Step: 120170 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:40:23,037-Speed 131.29 samples/sec Loss 2.7421 Epoch: 12 Global Step: 120175 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:40:42,642-Speed 130.58 samples/sec Loss 2.7121 Epoch: 12 Global Step: 120180 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:41:02,234-Speed 130.68 samples/sec Loss 2.6471 Epoch: 12 Global Step: 120185 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:41:22,103-Speed 128.85 samples/sec Loss 2.8107 Epoch: 12 Global Step: 120190 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:41:41,529-Speed 131.79 samples/sec Loss 2.7363 Epoch: 12 Global Step: 120195 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:42:01,131-Speed 130.61 samples/sec Loss 2.6284 Epoch: 12 Global Step: 120200 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:42:20,735-Speed 130.59 samples/sec Loss 2.5642 Epoch: 12 Global Step: 120205 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:42:40,171-Speed 131.72 samples/sec Loss 2.6966 Epoch: 12 Global Step: 120210 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:42:59,786-Speed 130.52 samples/sec Loss 2.8256 Epoch: 12 Global Step: 120215 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:43:19,337-Speed 130.95 samples/sec Loss 2.7570 Epoch: 12 Global Step: 120220 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:43:38,940-Speed 130.60 samples/sec Loss 2.7098 Epoch: 12 Global Step: 120225 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:43:58,668-Speed 129.77 samples/sec Loss 2.5857 Epoch: 12 Global Step: 120230 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:44:18,532-Speed 128.88 samples/sec Loss 2.7175 Epoch: 12 Global Step: 120235 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:44:37,972-Speed 131.69 samples/sec Loss 2.9260 Epoch: 12 Global Step: 120240 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:44:57,700-Speed 129.77 samples/sec Loss 2.4882 Epoch: 12 Global Step: 120245 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:45:17,213-Speed 131.21 samples/sec Loss 2.6334 Epoch: 12 Global Step: 120250 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:45:36,821-Speed 130.57 samples/sec Loss 2.6868 Epoch: 12 Global Step: 120255 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:45:56,263-Speed 131.68 samples/sec Loss 2.7457 Epoch: 12 Global Step: 120260 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:46:15,791-Speed 131.10 samples/sec Loss 2.7740 Epoch: 12 Global Step: 120265 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:46:35,398-Speed 130.58 samples/sec Loss 2.6405 Epoch: 12 Global Step: 120270 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:46:54,870-Speed 131.48 samples/sec Loss 2.6126 Epoch: 12 Global Step: 120275 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:47:14,672-Speed 129.29 samples/sec Loss 2.6504 Epoch: 12 Global Step: 120280 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:47:34,426-Speed 129.60 samples/sec Loss 2.7481 Epoch: 12 Global Step: 120285 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:47:53,976-Speed 130.95 samples/sec Loss 2.9179 Epoch: 12 Global Step: 120290 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:48:13,460-Speed 131.40 samples/sec Loss 2.7092 Epoch: 12 Global Step: 120295 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:48:33,030-Speed 130.82 samples/sec Loss 2.6104 Epoch: 12 Global Step: 120300 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:48:52,571-Speed 131.02 samples/sec Loss 2.6743 Epoch: 12 Global Step: 120305 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:49:12,166-Speed 130.65 samples/sec Loss 2.7466 Epoch: 12 Global Step: 120310 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:49:31,718-Speed 130.94 samples/sec Loss 2.5583 Epoch: 12 Global Step: 120315 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:49:51,281-Speed 130.87 samples/sec Loss 2.6755 Epoch: 12 Global Step: 120320 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:50:11,089-Speed 129.25 samples/sec Loss 2.6461 Epoch: 12 Global Step: 120325 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:50:30,696-Speed 130.58 samples/sec Loss 2.6474 Epoch: 12 Global Step: 120330 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:50:50,254-Speed 130.90 samples/sec Loss 2.7127 Epoch: 12 Global Step: 120335 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:51:09,802-Speed 130.97 samples/sec Loss 2.7335 Epoch: 12 Global Step: 120340 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:51:29,375-Speed 130.80 samples/sec Loss 2.7754 Epoch: 12 Global Step: 120345 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:51:48,811-Speed 131.72 samples/sec Loss 2.9440 Epoch: 12 Global Step: 120350 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:52:08,438-Speed 130.44 samples/sec Loss 2.7565 Epoch: 12 Global Step: 120355 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:52:28,038-Speed 130.62 samples/sec Loss 2.7253 Epoch: 12 Global Step: 120360 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:52:47,579-Speed 131.01 samples/sec Loss 2.8134 Epoch: 12 Global Step: 120365 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:53:07,427-Speed 128.99 samples/sec Loss 2.7576 Epoch: 12 Global Step: 120370 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:53:27,057-Speed 130.42 samples/sec Loss 2.7475 Epoch: 12 Global Step: 120375 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:53:46,675-Speed 130.50 samples/sec Loss 2.7947 Epoch: 12 Global Step: 120380 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:54:06,283-Speed 130.56 samples/sec Loss 2.7382 Epoch: 12 Global Step: 120385 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:54:25,830-Speed 130.97 samples/sec Loss 2.9135 Epoch: 12 Global Step: 120390 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:54:45,381-Speed 130.95 samples/sec Loss 2.7505 Epoch: 12 Global Step: 120395 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:55:05,009-Speed 130.44 samples/sec Loss 2.7125 Epoch: 12 Global Step: 120400 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:55:24,570-Speed 130.88 samples/sec Loss 2.5672 Epoch: 12 Global Step: 120405 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:55:44,179-Speed 130.56 samples/sec Loss 2.7898 Epoch: 12 Global Step: 120410 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:56:03,990-Speed 129.22 samples/sec Loss 2.7372 Epoch: 12 Global Step: 120415 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:56:23,435-Speed 131.66 samples/sec Loss 2.9184 Epoch: 12 Global Step: 120420 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:56:43,135-Speed 129.96 samples/sec Loss 2.9293 Epoch: 12 Global Step: 120425 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:57:02,794-Speed 130.23 samples/sec Loss 2.9318 Epoch: 12 Global Step: 120430 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:57:22,245-Speed 131.62 samples/sec Loss 2.8207 Epoch: 12 Global Step: 120435 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:57:41,935-Speed 130.02 samples/sec Loss 2.9309 Epoch: 12 Global Step: 120440 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:58:01,507-Speed 130.80 samples/sec Loss 2.7308 Epoch: 12 Global Step: 120445 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:58:21,129-Speed 130.48 samples/sec Loss 2.5406 Epoch: 12 Global Step: 120450 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:58:40,648-Speed 131.16 samples/sec Loss 2.7493 Epoch: 12 Global Step: 120455 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:59:00,469-Speed 129.16 samples/sec Loss 2.6566 Epoch: 12 Global Step: 120460 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:59:19,900-Speed 131.75 samples/sec Loss 2.7104 Epoch: 12 Global Step: 120465 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:59:39,580-Speed 130.09 samples/sec Loss 2.7492 Epoch: 12 Global Step: 120470 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 17:59:59,056-Speed 131.45 samples/sec Loss 2.6491 Epoch: 12 Global Step: 120475 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:00:18,494-Speed 131.71 samples/sec Loss 2.5122 Epoch: 12 Global Step: 120480 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:00:38,230-Speed 129.72 samples/sec Loss 2.7890 Epoch: 12 Global Step: 120485 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:00:57,775-Speed 130.99 samples/sec Loss 2.6698 Epoch: 12 Global Step: 120490 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:01:18,159-Speed 125.60 samples/sec Loss 2.6764 Epoch: 12 Global Step: 120495 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:01:37,606-Speed 131.65 samples/sec Loss 2.8001 Epoch: 12 Global Step: 120500 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:01:57,522-Speed 128.54 samples/sec Loss 2.6108 Epoch: 12 Global Step: 120505 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:02:17,125-Speed 130.60 samples/sec Loss 2.7800 Epoch: 12 Global Step: 120510 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:02:36,846-Speed 129.82 samples/sec Loss 2.6105 Epoch: 12 Global Step: 120515 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:02:56,281-Speed 131.73 samples/sec Loss 2.7039 Epoch: 12 Global Step: 120520 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:03:15,836-Speed 130.92 samples/sec Loss 2.7746 Epoch: 12 Global Step: 120525 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:03:35,442-Speed 130.58 samples/sec Loss 2.8018 Epoch: 12 Global Step: 120530 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:03:54,908-Speed 131.52 samples/sec Loss 2.7376 Epoch: 12 Global Step: 120535 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:04:14,504-Speed 130.64 samples/sec Loss 2.7647 Epoch: 12 Global Step: 120540 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:04:33,955-Speed 131.62 samples/sec Loss 2.7736 Epoch: 12 Global Step: 120545 Fp16 Grad Scale: 8192 Required: 153 hours Training: 2021-06-12 18:04:53,764-Speed 129.24 samples/sec Loss 2.6863 Epoch: 12 Global Step: 120550 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:05:13,328-Speed 130.86 samples/sec Loss 2.7011 Epoch: 12 Global Step: 120555 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:05:32,978-Speed 130.29 samples/sec Loss 2.7834 Epoch: 12 Global Step: 120560 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:05:52,523-Speed 130.98 samples/sec Loss 2.4990 Epoch: 12 Global Step: 120565 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:06:12,156-Speed 130.40 samples/sec Loss 2.8342 Epoch: 12 Global Step: 120570 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:06:31,695-Speed 131.02 samples/sec Loss 2.7678 Epoch: 12 Global Step: 120575 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:06:51,328-Speed 130.40 samples/sec Loss 2.7175 Epoch: 12 Global Step: 120580 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:07:10,773-Speed 131.66 samples/sec Loss 2.6053 Epoch: 12 Global Step: 120585 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:07:30,317-Speed 130.99 samples/sec Loss 2.7229 Epoch: 12 Global Step: 120590 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:07:50,140-Speed 129.15 samples/sec Loss 2.7333 Epoch: 12 Global Step: 120595 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:08:09,854-Speed 129.86 samples/sec Loss 2.5406 Epoch: 12 Global Step: 120600 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:08:29,327-Speed 131.47 samples/sec Loss 2.8345 Epoch: 12 Global Step: 120605 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:08:48,798-Speed 131.49 samples/sec Loss 2.9160 Epoch: 12 Global Step: 120610 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:09:08,421-Speed 130.47 samples/sec Loss 2.6864 Epoch: 12 Global Step: 120615 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:09:27,983-Speed 130.87 samples/sec Loss 2.9182 Epoch: 12 Global Step: 120620 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:09:47,598-Speed 130.52 samples/sec Loss 2.8003 Epoch: 12 Global Step: 120625 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:10:07,132-Speed 131.06 samples/sec Loss 2.8525 Epoch: 12 Global Step: 120630 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:10:26,615-Speed 131.40 samples/sec Loss 2.6908 Epoch: 12 Global Step: 120635 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:10:46,434-Speed 129.18 samples/sec Loss 2.7653 Epoch: 12 Global Step: 120640 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:11:06,206-Speed 129.49 samples/sec Loss 2.5765 Epoch: 12 Global Step: 120645 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:11:25,693-Speed 131.38 samples/sec Loss 2.6315 Epoch: 12 Global Step: 120650 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:11:45,332-Speed 130.36 samples/sec Loss 2.8233 Epoch: 12 Global Step: 120655 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:12:04,871-Speed 131.03 samples/sec Loss 2.9647 Epoch: 12 Global Step: 120660 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:12:24,471-Speed 130.62 samples/sec Loss 2.9830 Epoch: 12 Global Step: 120665 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:12:44,079-Speed 130.57 samples/sec Loss 2.9581 Epoch: 12 Global Step: 120670 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:13:03,550-Speed 131.48 samples/sec Loss 2.7062 Epoch: 12 Global Step: 120675 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:13:23,103-Speed 130.93 samples/sec Loss 2.8646 Epoch: 12 Global Step: 120680 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:13:42,853-Speed 129.63 samples/sec Loss 2.7031 Epoch: 12 Global Step: 120685 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:14:02,579-Speed 129.79 samples/sec Loss 2.9511 Epoch: 12 Global Step: 120690 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:14:22,025-Speed 131.66 samples/sec Loss 2.7494 Epoch: 12 Global Step: 120695 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:14:41,696-Speed 130.15 samples/sec Loss 2.8281 Epoch: 12 Global Step: 120700 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:15:01,242-Speed 130.98 samples/sec Loss 2.9210 Epoch: 12 Global Step: 120705 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:15:20,914-Speed 130.15 samples/sec Loss 2.7308 Epoch: 12 Global Step: 120710 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:15:40,497-Speed 130.73 samples/sec Loss 2.8228 Epoch: 12 Global Step: 120715 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:16:00,023-Speed 131.11 samples/sec Loss 2.7397 Epoch: 12 Global Step: 120720 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:16:19,626-Speed 130.60 samples/sec Loss 2.9380 Epoch: 12 Global Step: 120725 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:16:39,301-Speed 130.12 samples/sec Loss 2.6532 Epoch: 12 Global Step: 120730 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:16:59,135-Speed 129.08 samples/sec Loss 2.8180 Epoch: 12 Global Step: 120735 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:17:18,747-Speed 130.54 samples/sec Loss 2.7149 Epoch: 12 Global Step: 120740 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:17:38,357-Speed 130.55 samples/sec Loss 2.8296 Epoch: 12 Global Step: 120745 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:17:57,814-Speed 131.58 samples/sec Loss 2.8294 Epoch: 12 Global Step: 120750 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:18:17,336-Speed 131.14 samples/sec Loss 2.9448 Epoch: 12 Global Step: 120755 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:18:36,926-Speed 130.68 samples/sec Loss 2.7906 Epoch: 12 Global Step: 120760 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:18:56,470-Speed 131.00 samples/sec Loss 2.7740 Epoch: 12 Global Step: 120765 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:19:16,173-Speed 129.94 samples/sec Loss 2.7493 Epoch: 12 Global Step: 120770 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:19:35,931-Speed 129.58 samples/sec Loss 2.8443 Epoch: 12 Global Step: 120775 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:19:55,648-Speed 129.84 samples/sec Loss 2.8370 Epoch: 12 Global Step: 120780 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:20:15,181-Speed 131.07 samples/sec Loss 2.7136 Epoch: 12 Global Step: 120785 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:20:34,807-Speed 130.45 samples/sec Loss 2.8878 Epoch: 12 Global Step: 120790 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:20:54,359-Speed 130.94 samples/sec Loss 2.6945 Epoch: 12 Global Step: 120795 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:21:13,912-Speed 130.93 samples/sec Loss 2.6500 Epoch: 12 Global Step: 120800 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:21:33,496-Speed 130.73 samples/sec Loss 2.9338 Epoch: 12 Global Step: 120805 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:21:53,088-Speed 130.67 samples/sec Loss 2.6471 Epoch: 12 Global Step: 120810 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:22:12,606-Speed 131.17 samples/sec Loss 2.7825 Epoch: 12 Global Step: 120815 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:22:32,580-Speed 128.17 samples/sec Loss 2.8905 Epoch: 12 Global Step: 120820 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:22:52,171-Speed 130.68 samples/sec Loss 2.7621 Epoch: 12 Global Step: 120825 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:23:11,761-Speed 130.68 samples/sec Loss 2.6790 Epoch: 12 Global Step: 120830 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:23:31,357-Speed 130.65 samples/sec Loss 2.8683 Epoch: 12 Global Step: 120835 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:23:50,897-Speed 131.02 samples/sec Loss 2.9964 Epoch: 12 Global Step: 120840 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:24:10,399-Speed 131.28 samples/sec Loss 2.6183 Epoch: 12 Global Step: 120845 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:24:29,933-Speed 131.06 samples/sec Loss 2.7153 Epoch: 12 Global Step: 120850 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:24:49,552-Speed 130.50 samples/sec Loss 2.7643 Epoch: 12 Global Step: 120855 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:25:09,139-Speed 130.71 samples/sec Loss 2.8042 Epoch: 12 Global Step: 120860 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:25:29,172-Speed 127.79 samples/sec Loss 2.6988 Epoch: 12 Global Step: 120865 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:25:48,597-Speed 131.80 samples/sec Loss 2.7711 Epoch: 12 Global Step: 120870 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:26:08,316-Speed 129.83 samples/sec Loss 2.7949 Epoch: 12 Global Step: 120875 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:26:27,924-Speed 130.57 samples/sec Loss 2.8581 Epoch: 12 Global Step: 120880 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:26:47,398-Speed 131.46 samples/sec Loss 2.9139 Epoch: 12 Global Step: 120885 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:27:06,978-Speed 130.75 samples/sec Loss 2.9385 Epoch: 12 Global Step: 120890 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:27:27,075-Speed 127.39 samples/sec Loss 2.8204 Epoch: 12 Global Step: 120895 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:27:46,689-Speed 130.53 samples/sec Loss 3.0020 Epoch: 12 Global Step: 120900 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:28:06,240-Speed 130.95 samples/sec Loss 2.7294 Epoch: 12 Global Step: 120905 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:28:26,056-Speed 129.20 samples/sec Loss 2.6901 Epoch: 12 Global Step: 120910 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:28:45,574-Speed 131.17 samples/sec Loss 2.8572 Epoch: 12 Global Step: 120915 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:29:05,316-Speed 129.68 samples/sec Loss 2.9344 Epoch: 12 Global Step: 120920 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:29:24,856-Speed 131.02 samples/sec Loss 2.6637 Epoch: 12 Global Step: 120925 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:29:44,460-Speed 130.59 samples/sec Loss 2.8656 Epoch: 12 Global Step: 120930 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:30:04,161-Speed 129.95 samples/sec Loss 2.6852 Epoch: 12 Global Step: 120935 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:30:23,814-Speed 130.27 samples/sec Loss 2.8668 Epoch: 12 Global Step: 120940 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:30:43,403-Speed 130.69 samples/sec Loss 2.7067 Epoch: 12 Global Step: 120945 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:31:02,879-Speed 131.45 samples/sec Loss 2.7461 Epoch: 12 Global Step: 120950 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:31:22,752-Speed 128.83 samples/sec Loss 3.0670 Epoch: 12 Global Step: 120955 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:31:42,365-Speed 130.53 samples/sec Loss 2.8582 Epoch: 12 Global Step: 120960 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:32:02,013-Speed 130.30 samples/sec Loss 2.6480 Epoch: 12 Global Step: 120965 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:32:21,452-Speed 131.70 samples/sec Loss 2.9479 Epoch: 12 Global Step: 120970 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:32:41,070-Speed 130.50 samples/sec Loss 2.7280 Epoch: 12 Global Step: 120975 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:33:00,634-Speed 130.86 samples/sec Loss 2.7647 Epoch: 12 Global Step: 120980 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:33:20,255-Speed 130.48 samples/sec Loss 2.7450 Epoch: 12 Global Step: 120985 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:33:39,820-Speed 130.85 samples/sec Loss 2.6582 Epoch: 12 Global Step: 120990 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:33:59,264-Speed 131.67 samples/sec Loss 3.0382 Epoch: 12 Global Step: 120995 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:34:19,071-Speed 129.26 samples/sec Loss 2.8316 Epoch: 12 Global Step: 121000 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:34:38,659-Speed 130.70 samples/sec Loss 2.8927 Epoch: 12 Global Step: 121005 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:34:58,376-Speed 129.85 samples/sec Loss 2.8710 Epoch: 12 Global Step: 121010 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:35:17,812-Speed 131.72 samples/sec Loss 2.8989 Epoch: 12 Global Step: 121015 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:35:37,427-Speed 130.52 samples/sec Loss 2.8571 Epoch: 12 Global Step: 121020 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:35:56,985-Speed 130.90 samples/sec Loss 2.8444 Epoch: 12 Global Step: 121025 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:36:16,596-Speed 130.55 samples/sec Loss 2.7763 Epoch: 12 Global Step: 121030 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:36:36,022-Speed 131.79 samples/sec Loss 2.8957 Epoch: 12 Global Step: 121035 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:36:55,477-Speed 131.59 samples/sec Loss 2.7503 Epoch: 12 Global Step: 121040 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:37:15,269-Speed 129.36 samples/sec Loss 2.8622 Epoch: 12 Global Step: 121045 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:37:34,968-Speed 129.96 samples/sec Loss 2.9220 Epoch: 12 Global Step: 121050 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:37:54,510-Speed 131.01 samples/sec Loss 2.8133 Epoch: 12 Global Step: 121055 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:38:14,150-Speed 130.36 samples/sec Loss 2.8729 Epoch: 12 Global Step: 121060 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:38:33,741-Speed 130.68 samples/sec Loss 2.7969 Epoch: 12 Global Step: 121065 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:38:53,355-Speed 130.53 samples/sec Loss 2.7990 Epoch: 12 Global Step: 121070 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:39:12,956-Speed 130.62 samples/sec Loss 2.8209 Epoch: 12 Global Step: 121075 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:39:32,410-Speed 131.60 samples/sec Loss 2.8708 Epoch: 12 Global Step: 121080 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:39:52,022-Speed 130.54 samples/sec Loss 2.8117 Epoch: 12 Global Step: 121085 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:40:11,812-Speed 129.36 samples/sec Loss 2.8299 Epoch: 12 Global Step: 121090 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:40:31,529-Speed 129.84 samples/sec Loss 2.9095 Epoch: 12 Global Step: 121095 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:40:50,986-Speed 131.58 samples/sec Loss 2.8868 Epoch: 12 Global Step: 121100 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:41:10,609-Speed 130.47 samples/sec Loss 2.8120 Epoch: 12 Global Step: 121105 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:41:30,167-Speed 130.90 samples/sec Loss 2.9147 Epoch: 12 Global Step: 121110 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:41:49,853-Speed 130.05 samples/sec Loss 2.8936 Epoch: 12 Global Step: 121115 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:42:09,301-Speed 131.64 samples/sec Loss 2.9340 Epoch: 12 Global Step: 121120 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:42:28,791-Speed 131.36 samples/sec Loss 2.9555 Epoch: 12 Global Step: 121125 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:42:48,366-Speed 130.79 samples/sec Loss 2.9899 Epoch: 12 Global Step: 121130 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:43:08,294-Speed 128.47 samples/sec Loss 2.8249 Epoch: 12 Global Step: 121135 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:43:27,726-Speed 131.75 samples/sec Loss 2.9607 Epoch: 12 Global Step: 121140 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:43:47,198-Speed 131.48 samples/sec Loss 2.8149 Epoch: 12 Global Step: 121145 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:44:06,781-Speed 130.73 samples/sec Loss 2.9508 Epoch: 12 Global Step: 121150 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:44:26,410-Speed 130.43 samples/sec Loss 2.9841 Epoch: 12 Global Step: 121155 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:44:46,061-Speed 130.28 samples/sec Loss 2.9720 Epoch: 12 Global Step: 121160 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:45:05,504-Speed 131.67 samples/sec Loss 3.0458 Epoch: 12 Global Step: 121165 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:45:25,071-Speed 130.84 samples/sec Loss 3.0358 Epoch: 12 Global Step: 121170 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:45:44,642-Speed 130.82 samples/sec Loss 2.8831 Epoch: 12 Global Step: 121175 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:46:04,631-Speed 128.07 samples/sec Loss 2.9091 Epoch: 12 Global Step: 121180 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:46:24,060-Speed 131.77 samples/sec Loss 2.9802 Epoch: 12 Global Step: 121185 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:46:43,610-Speed 130.96 samples/sec Loss 2.8295 Epoch: 12 Global Step: 121190 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:47:03,229-Speed 130.49 samples/sec Loss 2.7196 Epoch: 12 Global Step: 121195 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:47:22,867-Speed 130.37 samples/sec Loss 2.7399 Epoch: 12 Global Step: 121200 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:47:42,472-Speed 130.58 samples/sec Loss 2.7894 Epoch: 12 Global Step: 121205 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:48:01,925-Speed 131.61 samples/sec Loss 2.9071 Epoch: 12 Global Step: 121210 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:48:21,595-Speed 130.15 samples/sec Loss 2.7317 Epoch: 12 Global Step: 121215 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:48:41,169-Speed 130.80 samples/sec Loss 2.9496 Epoch: 12 Global Step: 121220 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:49:01,108-Speed 128.40 samples/sec Loss 2.7948 Epoch: 12 Global Step: 121225 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:49:20,730-Speed 130.47 samples/sec Loss 2.9066 Epoch: 12 Global Step: 121230 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:49:40,443-Speed 129.87 samples/sec Loss 2.8325 Epoch: 12 Global Step: 121235 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:50:00,035-Speed 130.68 samples/sec Loss 2.9332 Epoch: 12 Global Step: 121240 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:50:19,605-Speed 130.82 samples/sec Loss 2.9718 Epoch: 12 Global Step: 121245 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:50:39,194-Speed 130.69 samples/sec Loss 2.8569 Epoch: 12 Global Step: 121250 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:50:58,733-Speed 131.03 samples/sec Loss 2.8334 Epoch: 12 Global Step: 121255 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:51:18,340-Speed 130.58 samples/sec Loss 2.9925 Epoch: 12 Global Step: 121260 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:51:37,882-Speed 131.00 samples/sec Loss 2.7942 Epoch: 12 Global Step: 121265 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:51:57,693-Speed 129.23 samples/sec Loss 2.9617 Epoch: 12 Global Step: 121270 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:52:17,141-Speed 131.64 samples/sec Loss 2.7682 Epoch: 12 Global Step: 121275 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:52:36,816-Speed 130.12 samples/sec Loss 2.8728 Epoch: 12 Global Step: 121280 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:52:56,358-Speed 131.01 samples/sec Loss 3.1758 Epoch: 12 Global Step: 121285 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:53:15,871-Speed 131.20 samples/sec Loss 2.8457 Epoch: 12 Global Step: 121290 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:53:35,415-Speed 130.99 samples/sec Loss 2.9408 Epoch: 12 Global Step: 121295 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:53:54,955-Speed 131.03 samples/sec Loss 2.9763 Epoch: 12 Global Step: 121300 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:54:14,522-Speed 130.84 samples/sec Loss 2.8461 Epoch: 12 Global Step: 121305 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:54:34,118-Speed 130.65 samples/sec Loss 3.0845 Epoch: 12 Global Step: 121310 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:54:53,915-Speed 129.32 samples/sec Loss 3.0669 Epoch: 12 Global Step: 121315 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:55:13,507-Speed 130.67 samples/sec Loss 2.8124 Epoch: 12 Global Step: 121320 Fp16 Grad Scale: 8192 Required: 152 hours Training: 2021-06-12 18:55:33,315-Speed 129.25 samples/sec Loss 3.0040 Epoch: 12 Global Step: 121325 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:55:52,921-Speed 130.58 samples/sec Loss 2.8050 Epoch: 12 Global Step: 121330 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:56:12,569-Speed 130.30 samples/sec Loss 2.9066 Epoch: 12 Global Step: 121335 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:56:32,110-Speed 131.02 samples/sec Loss 2.8274 Epoch: 12 Global Step: 121340 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:56:51,542-Speed 131.75 samples/sec Loss 2.8128 Epoch: 12 Global Step: 121345 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:57:11,150-Speed 130.57 samples/sec Loss 2.9525 Epoch: 12 Global Step: 121350 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:57:30,780-Speed 130.42 samples/sec Loss 2.9242 Epoch: 12 Global Step: 121355 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:57:50,590-Speed 129.23 samples/sec Loss 2.9165 Epoch: 12 Global Step: 121360 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:58:10,114-Speed 131.13 samples/sec Loss 2.7492 Epoch: 12 Global Step: 121365 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:58:29,806-Speed 130.01 samples/sec Loss 3.0045 Epoch: 12 Global Step: 121370 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:58:49,446-Speed 130.36 samples/sec Loss 2.7909 Epoch: 12 Global Step: 121375 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:59:09,163-Speed 129.84 samples/sec Loss 2.8370 Epoch: 12 Global Step: 121380 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:59:28,703-Speed 131.02 samples/sec Loss 3.0876 Epoch: 12 Global Step: 121385 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 18:59:48,266-Speed 130.86 samples/sec Loss 2.7607 Epoch: 12 Global Step: 121390 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:00:08,018-Speed 129.62 samples/sec Loss 2.9385 Epoch: 12 Global Step: 121395 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:00:27,638-Speed 130.49 samples/sec Loss 2.9302 Epoch: 12 Global Step: 121400 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:00:47,474-Speed 129.06 samples/sec Loss 2.9710 Epoch: 12 Global Step: 121405 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:01:06,969-Speed 131.33 samples/sec Loss 2.8441 Epoch: 12 Global Step: 121410 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:01:26,634-Speed 130.19 samples/sec Loss 3.2480 Epoch: 12 Global Step: 121415 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:01:46,181-Speed 130.97 samples/sec Loss 2.9674 Epoch: 12 Global Step: 121420 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:02:05,815-Speed 130.40 samples/sec Loss 3.0318 Epoch: 12 Global Step: 121425 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:02:25,350-Speed 131.06 samples/sec Loss 2.9210 Epoch: 12 Global Step: 121430 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:02:44,876-Speed 131.12 samples/sec Loss 2.9579 Epoch: 12 Global Step: 121435 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:03:04,491-Speed 130.52 samples/sec Loss 3.0512 Epoch: 12 Global Step: 121440 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:03:24,039-Speed 130.97 samples/sec Loss 2.7704 Epoch: 12 Global Step: 121445 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:03:43,841-Speed 129.29 samples/sec Loss 2.7951 Epoch: 12 Global Step: 121450 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:04:03,595-Speed 129.60 samples/sec Loss 2.8030 Epoch: 12 Global Step: 121455 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:04:23,209-Speed 130.52 samples/sec Loss 2.9385 Epoch: 12 Global Step: 121460 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:04:42,673-Speed 131.54 samples/sec Loss 2.8332 Epoch: 12 Global Step: 121465 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:05:02,301-Speed 130.43 samples/sec Loss 3.0052 Epoch: 12 Global Step: 121470 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:05:21,849-Speed 130.97 samples/sec Loss 3.0081 Epoch: 12 Global Step: 121475 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:05:41,457-Speed 130.56 samples/sec Loss 3.0608 Epoch: 12 Global Step: 121480 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:06:01,070-Speed 130.54 samples/sec Loss 2.8170 Epoch: 12 Global Step: 121485 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:06:20,559-Speed 131.36 samples/sec Loss 2.9602 Epoch: 12 Global Step: 121490 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:06:40,330-Speed 129.49 samples/sec Loss 2.9831 Epoch: 12 Global Step: 121495 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:07:00,078-Speed 129.64 samples/sec Loss 2.8228 Epoch: 12 Global Step: 121500 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:07:19,643-Speed 130.85 samples/sec Loss 2.9372 Epoch: 12 Global Step: 121505 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:07:39,149-Speed 131.25 samples/sec Loss 2.8693 Epoch: 12 Global Step: 121510 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:07:58,692-Speed 131.00 samples/sec Loss 2.9307 Epoch: 12 Global Step: 121515 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:08:18,252-Speed 130.89 samples/sec Loss 2.9580 Epoch: 12 Global Step: 121520 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:08:37,901-Speed 130.29 samples/sec Loss 2.9045 Epoch: 12 Global Step: 121525 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:08:57,597-Speed 129.98 samples/sec Loss 2.9021 Epoch: 12 Global Step: 121530 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:09:17,222-Speed 130.46 samples/sec Loss 3.0454 Epoch: 12 Global Step: 121535 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:09:37,001-Speed 129.44 samples/sec Loss 2.7153 Epoch: 12 Global Step: 121540 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:09:56,593-Speed 130.67 samples/sec Loss 2.9143 Epoch: 12 Global Step: 121545 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:10:16,066-Speed 131.47 samples/sec Loss 3.0459 Epoch: 12 Global Step: 121550 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:10:35,719-Speed 130.27 samples/sec Loss 2.8157 Epoch: 12 Global Step: 121555 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:10:55,261-Speed 131.01 samples/sec Loss 2.9282 Epoch: 12 Global Step: 121560 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:11:14,887-Speed 130.45 samples/sec Loss 2.7588 Epoch: 12 Global Step: 121565 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:11:34,502-Speed 130.52 samples/sec Loss 2.9137 Epoch: 12 Global Step: 121570 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:11:54,051-Speed 130.96 samples/sec Loss 2.8726 Epoch: 12 Global Step: 121575 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:12:13,573-Speed 131.14 samples/sec Loss 3.0867 Epoch: 12 Global Step: 121580 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:12:33,552-Speed 128.14 samples/sec Loss 2.9504 Epoch: 12 Global Step: 121585 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:12:53,003-Speed 131.62 samples/sec Loss 3.0829 Epoch: 12 Global Step: 121590 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:13:12,823-Speed 129.17 samples/sec Loss 2.9887 Epoch: 12 Global Step: 121595 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:13:32,404-Speed 130.75 samples/sec Loss 3.0679 Epoch: 12 Global Step: 121600 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:13:52,030-Speed 130.45 samples/sec Loss 2.9338 Epoch: 12 Global Step: 121605 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:14:11,647-Speed 130.51 samples/sec Loss 3.0119 Epoch: 12 Global Step: 121610 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:14:31,088-Speed 131.69 samples/sec Loss 2.6949 Epoch: 12 Global Step: 121615 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:14:50,635-Speed 130.97 samples/sec Loss 2.8553 Epoch: 12 Global Step: 121620 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:15:10,221-Speed 130.72 samples/sec Loss 3.0920 Epoch: 12 Global Step: 121625 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:15:30,118-Speed 128.67 samples/sec Loss 3.0651 Epoch: 12 Global Step: 121630 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:15:49,556-Speed 131.71 samples/sec Loss 2.9992 Epoch: 12 Global Step: 121635 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:16:09,165-Speed 130.56 samples/sec Loss 2.9919 Epoch: 12 Global Step: 121640 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:16:28,716-Speed 130.95 samples/sec Loss 2.9877 Epoch: 12 Global Step: 121645 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:16:48,198-Speed 131.41 samples/sec Loss 3.0591 Epoch: 12 Global Step: 121650 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:17:07,941-Speed 129.68 samples/sec Loss 2.6981 Epoch: 12 Global Step: 121655 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:17:27,383-Speed 131.68 samples/sec Loss 2.9546 Epoch: 12 Global Step: 121660 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:17:46,998-Speed 130.52 samples/sec Loss 2.7088 Epoch: 12 Global Step: 121665 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:18:06,559-Speed 130.88 samples/sec Loss 2.8467 Epoch: 12 Global Step: 121670 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:18:26,371-Speed 129.22 samples/sec Loss 2.9945 Epoch: 12 Global Step: 121675 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:18:46,021-Speed 130.29 samples/sec Loss 3.0048 Epoch: 12 Global Step: 121680 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:19:05,707-Speed 130.05 samples/sec Loss 3.0784 Epoch: 12 Global Step: 121685 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:19:25,274-Speed 130.84 samples/sec Loss 3.0668 Epoch: 12 Global Step: 121690 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:19:44,893-Speed 130.49 samples/sec Loss 3.0196 Epoch: 12 Global Step: 121695 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:20:04,476-Speed 130.74 samples/sec Loss 2.9212 Epoch: 12 Global Step: 121700 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:20:24,028-Speed 130.94 samples/sec Loss 3.0305 Epoch: 12 Global Step: 121705 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:20:43,605-Speed 130.77 samples/sec Loss 2.9934 Epoch: 12 Global Step: 121710 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:21:03,123-Speed 131.17 samples/sec Loss 3.2021 Epoch: 12 Global Step: 121715 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:21:22,929-Speed 129.26 samples/sec Loss 3.0049 Epoch: 12 Global Step: 121720 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:21:42,503-Speed 130.80 samples/sec Loss 3.0047 Epoch: 12 Global Step: 121725 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:22:02,168-Speed 130.19 samples/sec Loss 2.9456 Epoch: 12 Global Step: 121730 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:22:21,709-Speed 131.01 samples/sec Loss 2.9376 Epoch: 12 Global Step: 121735 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:22:41,326-Speed 130.51 samples/sec Loss 2.9374 Epoch: 12 Global Step: 121740 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:23:00,856-Speed 131.09 samples/sec Loss 2.9187 Epoch: 12 Global Step: 121745 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:23:20,417-Speed 130.88 samples/sec Loss 2.8684 Epoch: 12 Global Step: 121750 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:23:40,040-Speed 130.47 samples/sec Loss 3.1991 Epoch: 12 Global Step: 121755 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:23:59,583-Speed 131.00 samples/sec Loss 3.0824 Epoch: 12 Global Step: 121760 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:24:19,625-Speed 127.73 samples/sec Loss 2.8014 Epoch: 12 Global Step: 121765 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:24:39,314-Speed 130.03 samples/sec Loss 3.2152 Epoch: 12 Global Step: 121770 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:24:58,971-Speed 130.24 samples/sec Loss 3.1541 Epoch: 12 Global Step: 121775 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:25:18,422-Speed 131.62 samples/sec Loss 2.9915 Epoch: 12 Global Step: 121780 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:25:38,037-Speed 130.52 samples/sec Loss 2.9139 Epoch: 12 Global Step: 121785 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:25:57,646-Speed 130.56 samples/sec Loss 3.2046 Epoch: 12 Global Step: 121790 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:26:17,199-Speed 130.94 samples/sec Loss 2.9935 Epoch: 12 Global Step: 121795 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:26:36,803-Speed 130.59 samples/sec Loss 2.8203 Epoch: 12 Global Step: 121800 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:26:56,242-Speed 131.71 samples/sec Loss 2.8975 Epoch: 12 Global Step: 121805 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:27:16,049-Speed 129.25 samples/sec Loss 2.9525 Epoch: 12 Global Step: 121810 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:27:35,620-Speed 130.81 samples/sec Loss 2.9909 Epoch: 12 Global Step: 121815 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:27:55,286-Speed 130.18 samples/sec Loss 2.8925 Epoch: 12 Global Step: 121820 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:28:14,792-Speed 131.25 samples/sec Loss 2.8516 Epoch: 12 Global Step: 121825 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:28:34,379-Speed 130.71 samples/sec Loss 3.1053 Epoch: 12 Global Step: 121830 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:28:53,916-Speed 131.04 samples/sec Loss 2.8638 Epoch: 12 Global Step: 121835 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:29:13,513-Speed 130.64 samples/sec Loss 2.8337 Epoch: 12 Global Step: 121840 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:29:33,127-Speed 130.52 samples/sec Loss 2.9843 Epoch: 12 Global Step: 121845 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:29:52,676-Speed 130.96 samples/sec Loss 3.1134 Epoch: 12 Global Step: 121850 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:30:12,490-Speed 129.21 samples/sec Loss 2.9560 Epoch: 12 Global Step: 121855 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:30:32,131-Speed 130.35 samples/sec Loss 2.9934 Epoch: 12 Global Step: 121860 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:30:51,636-Speed 131.25 samples/sec Loss 3.1730 Epoch: 12 Global Step: 121865 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:31:11,260-Speed 130.47 samples/sec Loss 2.8051 Epoch: 12 Global Step: 121870 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:31:30,852-Speed 130.67 samples/sec Loss 2.8902 Epoch: 12 Global Step: 121875 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:31:50,398-Speed 130.98 samples/sec Loss 3.1275 Epoch: 12 Global Step: 121880 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:32:10,005-Speed 130.57 samples/sec Loss 2.8749 Epoch: 12 Global Step: 121885 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:32:29,478-Speed 131.48 samples/sec Loss 2.8969 Epoch: 12 Global Step: 121890 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:32:49,097-Speed 130.50 samples/sec Loss 2.9757 Epoch: 12 Global Step: 121895 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:33:08,872-Speed 129.46 samples/sec Loss 3.0842 Epoch: 12 Global Step: 121900 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:33:28,527-Speed 130.25 samples/sec Loss 2.7664 Epoch: 12 Global Step: 121905 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:33:48,072-Speed 130.99 samples/sec Loss 2.9468 Epoch: 12 Global Step: 121910 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:34:07,671-Speed 130.63 samples/sec Loss 3.1250 Epoch: 12 Global Step: 121915 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:34:27,227-Speed 130.91 samples/sec Loss 3.1228 Epoch: 12 Global Step: 121920 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:34:46,899-Speed 130.14 samples/sec Loss 3.0485 Epoch: 12 Global Step: 121925 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:35:06,448-Speed 130.96 samples/sec Loss 3.0024 Epoch: 12 Global Step: 121930 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:35:26,083-Speed 130.39 samples/sec Loss 3.1107 Epoch: 12 Global Step: 121935 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:35:45,619-Speed 131.05 samples/sec Loss 3.1066 Epoch: 12 Global Step: 121940 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:36:05,548-Speed 128.46 samples/sec Loss 3.1130 Epoch: 12 Global Step: 121945 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:36:24,975-Speed 131.78 samples/sec Loss 2.8953 Epoch: 12 Global Step: 121950 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:36:44,880-Speed 128.62 samples/sec Loss 3.1825 Epoch: 12 Global Step: 121955 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:37:04,414-Speed 131.06 samples/sec Loss 2.9079 Epoch: 12 Global Step: 121960 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:37:23,977-Speed 130.87 samples/sec Loss 3.2003 Epoch: 12 Global Step: 121965 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:37:43,580-Speed 130.60 samples/sec Loss 2.9868 Epoch: 12 Global Step: 121970 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:38:03,120-Speed 131.02 samples/sec Loss 3.1301 Epoch: 12 Global Step: 121975 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:38:22,721-Speed 130.61 samples/sec Loss 2.9304 Epoch: 12 Global Step: 121980 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:38:42,177-Speed 131.59 samples/sec Loss 2.9544 Epoch: 12 Global Step: 121985 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:39:01,971-Speed 129.34 samples/sec Loss 3.2034 Epoch: 12 Global Step: 121990 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:39:21,547-Speed 130.78 samples/sec Loss 3.0966 Epoch: 12 Global Step: 121995 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:39:41,178-Speed 130.41 samples/sec Loss 2.9164 Epoch: 12 Global Step: 122000 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:40:01,619-Speed 125.24 samples/sec Loss 3.2254 Epoch: 12 Global Step: 122005 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:40:21,237-Speed 130.50 samples/sec Loss 2.9482 Epoch: 12 Global Step: 122010 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:40:40,795-Speed 130.90 samples/sec Loss 3.0272 Epoch: 12 Global Step: 122015 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:41:00,394-Speed 130.62 samples/sec Loss 3.0654 Epoch: 12 Global Step: 122020 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:41:19,966-Speed 130.81 samples/sec Loss 2.9170 Epoch: 12 Global Step: 122025 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:41:39,533-Speed 130.84 samples/sec Loss 3.0658 Epoch: 12 Global Step: 122030 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:41:59,358-Speed 129.13 samples/sec Loss 2.9603 Epoch: 12 Global Step: 122035 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:42:18,927-Speed 130.83 samples/sec Loss 2.9505 Epoch: 12 Global Step: 122040 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:42:38,650-Speed 129.80 samples/sec Loss 2.9395 Epoch: 12 Global Step: 122045 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:42:58,201-Speed 130.94 samples/sec Loss 2.9195 Epoch: 12 Global Step: 122050 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:43:17,809-Speed 130.57 samples/sec Loss 3.1628 Epoch: 12 Global Step: 122055 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:43:37,280-Speed 131.48 samples/sec Loss 3.0533 Epoch: 12 Global Step: 122060 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:43:56,899-Speed 130.49 samples/sec Loss 2.9103 Epoch: 12 Global Step: 122065 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:44:16,516-Speed 130.51 samples/sec Loss 2.9685 Epoch: 12 Global Step: 122070 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:44:36,074-Speed 130.90 samples/sec Loss 3.0939 Epoch: 12 Global Step: 122075 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:44:55,882-Speed 129.25 samples/sec Loss 3.0774 Epoch: 12 Global Step: 122080 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:45:15,366-Speed 131.39 samples/sec Loss 3.0446 Epoch: 12 Global Step: 122085 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:45:35,047-Speed 130.09 samples/sec Loss 3.0375 Epoch: 12 Global Step: 122090 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:45:54,525-Speed 131.44 samples/sec Loss 2.9785 Epoch: 12 Global Step: 122095 Fp16 Grad Scale: 8192 Required: 151 hours Training: 2021-06-12 19:46:14,172-Speed 130.31 samples/sec Loss 3.0212 Epoch: 12 Global Step: 122100 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:46:33,760-Speed 130.70 samples/sec Loss 3.1369 Epoch: 12 Global Step: 122105 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:46:53,309-Speed 130.96 samples/sec Loss 3.0728 Epoch: 12 Global Step: 122110 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:47:12,881-Speed 130.81 samples/sec Loss 3.0960 Epoch: 12 Global Step: 122115 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:47:32,528-Speed 130.31 samples/sec Loss 3.1121 Epoch: 12 Global Step: 122120 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:47:52,348-Speed 129.17 samples/sec Loss 3.0428 Epoch: 12 Global Step: 122125 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:48:12,113-Speed 129.53 samples/sec Loss 3.1105 Epoch: 12 Global Step: 122130 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:48:31,623-Speed 131.22 samples/sec Loss 2.9164 Epoch: 12 Global Step: 122135 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:48:51,345-Speed 129.81 samples/sec Loss 3.1066 Epoch: 12 Global Step: 122140 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:49:10,960-Speed 130.52 samples/sec Loss 3.0008 Epoch: 12 Global Step: 122145 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:49:30,561-Speed 130.61 samples/sec Loss 2.8745 Epoch: 12 Global Step: 122150 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:49:50,176-Speed 130.52 samples/sec Loss 3.2833 Epoch: 12 Global Step: 122155 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:50:09,684-Speed 131.24 samples/sec Loss 2.8293 Epoch: 12 Global Step: 122160 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:50:29,227-Speed 131.00 samples/sec Loss 2.9596 Epoch: 12 Global Step: 122165 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:50:49,007-Speed 129.44 samples/sec Loss 3.2210 Epoch: 12 Global Step: 122170 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:51:08,730-Speed 129.80 samples/sec Loss 3.0474 Epoch: 12 Global Step: 122175 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:51:28,228-Speed 131.30 samples/sec Loss 3.1234 Epoch: 12 Global Step: 122180 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:51:47,787-Speed 130.90 samples/sec Loss 3.0004 Epoch: 12 Global Step: 122185 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:52:07,325-Speed 131.04 samples/sec Loss 2.9627 Epoch: 12 Global Step: 122190 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:52:26,965-Speed 130.35 samples/sec Loss 3.2219 Epoch: 12 Global Step: 122195 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:52:46,557-Speed 130.68 samples/sec Loss 2.9000 Epoch: 12 Global Step: 122200 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:53:06,075-Speed 131.17 samples/sec Loss 2.7730 Epoch: 12 Global Step: 122205 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:53:25,654-Speed 130.76 samples/sec Loss 2.8482 Epoch: 12 Global Step: 122210 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:53:45,464-Speed 129.23 samples/sec Loss 3.0180 Epoch: 12 Global Step: 122215 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:54:05,181-Speed 129.84 samples/sec Loss 3.1338 Epoch: 12 Global Step: 122220 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:54:24,730-Speed 130.96 samples/sec Loss 2.9079 Epoch: 12 Global Step: 122225 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:54:44,339-Speed 130.56 samples/sec Loss 3.0561 Epoch: 12 Global Step: 122230 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:55:04,014-Speed 130.12 samples/sec Loss 3.0733 Epoch: 12 Global Step: 122235 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:55:23,575-Speed 130.88 samples/sec Loss 3.1476 Epoch: 12 Global Step: 122240 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:55:43,234-Speed 130.23 samples/sec Loss 3.2331 Epoch: 12 Global Step: 122245 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:56:02,811-Speed 130.77 samples/sec Loss 3.0160 Epoch: 12 Global Step: 122250 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:56:22,407-Speed 130.65 samples/sec Loss 2.8962 Epoch: 12 Global Step: 122255 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:56:42,216-Speed 129.24 samples/sec Loss 2.9316 Epoch: 12 Global Step: 122260 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:57:01,847-Speed 130.41 samples/sec Loss 3.1018 Epoch: 12 Global Step: 122265 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:57:21,263-Speed 131.86 samples/sec Loss 3.0426 Epoch: 12 Global Step: 122270 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:57:40,856-Speed 130.66 samples/sec Loss 2.9782 Epoch: 12 Global Step: 122275 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:58:00,320-Speed 131.54 samples/sec Loss 2.9489 Epoch: 12 Global Step: 122280 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:58:19,879-Speed 130.89 samples/sec Loss 3.1438 Epoch: 12 Global Step: 122285 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:58:39,460-Speed 130.75 samples/sec Loss 3.0169 Epoch: 12 Global Step: 122290 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:58:58,905-Speed 131.67 samples/sec Loss 3.1742 Epoch: 12 Global Step: 122295 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:59:18,524-Speed 130.49 samples/sec Loss 2.9620 Epoch: 12 Global Step: 122300 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:59:38,393-Speed 128.85 samples/sec Loss 3.0016 Epoch: 12 Global Step: 122305 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 19:59:58,086-Speed 130.01 samples/sec Loss 3.0396 Epoch: 12 Global Step: 122310 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:00:17,536-Speed 131.62 samples/sec Loss 3.1050 Epoch: 12 Global Step: 122315 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:00:37,165-Speed 130.43 samples/sec Loss 3.0732 Epoch: 12 Global Step: 122320 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:00:56,704-Speed 131.02 samples/sec Loss 2.9005 Epoch: 12 Global Step: 122325 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:01:16,374-Speed 130.16 samples/sec Loss 3.0893 Epoch: 12 Global Step: 122330 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:01:35,957-Speed 130.73 samples/sec Loss 2.9642 Epoch: 12 Global Step: 122335 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:01:55,563-Speed 130.58 samples/sec Loss 3.0843 Epoch: 12 Global Step: 122340 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:02:15,200-Speed 130.37 samples/sec Loss 3.1069 Epoch: 12 Global Step: 122345 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:02:34,985-Speed 129.40 samples/sec Loss 2.8612 Epoch: 12 Global Step: 122350 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:02:54,629-Speed 130.32 samples/sec Loss 3.0744 Epoch: 12 Global Step: 122355 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:03:14,243-Speed 130.52 samples/sec Loss 3.0787 Epoch: 12 Global Step: 122360 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:03:33,848-Speed 130.59 samples/sec Loss 3.0843 Epoch: 12 Global Step: 122365 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:03:53,414-Speed 130.85 samples/sec Loss 2.9560 Epoch: 12 Global Step: 122370 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:04:13,071-Speed 130.24 samples/sec Loss 3.0563 Epoch: 12 Global Step: 122375 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:04:32,556-Speed 131.40 samples/sec Loss 3.2179 Epoch: 12 Global Step: 122380 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:04:52,146-Speed 130.68 samples/sec Loss 3.2191 Epoch: 12 Global Step: 122385 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:05:11,724-Speed 130.77 samples/sec Loss 2.9297 Epoch: 12 Global Step: 122390 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:05:31,681-Speed 128.28 samples/sec Loss 2.8994 Epoch: 12 Global Step: 122395 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:05:51,116-Speed 131.72 samples/sec Loss 2.9926 Epoch: 12 Global Step: 122400 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:06:10,784-Speed 130.17 samples/sec Loss 3.0491 Epoch: 12 Global Step: 122405 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:06:30,326-Speed 131.01 samples/sec Loss 2.9776 Epoch: 12 Global Step: 122410 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:06:49,876-Speed 130.95 samples/sec Loss 3.0657 Epoch: 12 Global Step: 122415 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:07:09,474-Speed 130.63 samples/sec Loss 3.0706 Epoch: 12 Global Step: 122420 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:07:29,077-Speed 130.60 samples/sec Loss 2.9508 Epoch: 12 Global Step: 122425 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:07:48,696-Speed 130.50 samples/sec Loss 3.1364 Epoch: 12 Global Step: 122430 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:08:08,301-Speed 130.58 samples/sec Loss 3.2133 Epoch: 12 Global Step: 122435 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:08:28,115-Speed 129.21 samples/sec Loss 2.9994 Epoch: 12 Global Step: 122440 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:08:47,569-Speed 131.60 samples/sec Loss 3.1214 Epoch: 12 Global Step: 122445 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:09:07,383-Speed 129.21 samples/sec Loss 3.1765 Epoch: 12 Global Step: 122450 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:09:26,982-Speed 130.62 samples/sec Loss 3.1578 Epoch: 12 Global Step: 122455 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:09:46,867-Speed 128.74 samples/sec Loss 3.2623 Epoch: 12 Global Step: 122460 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:10:06,292-Speed 131.80 samples/sec Loss 2.9587 Epoch: 12 Global Step: 122465 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:10:25,929-Speed 130.37 samples/sec Loss 3.0661 Epoch: 12 Global Step: 122470 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:10:45,496-Speed 130.84 samples/sec Loss 3.2362 Epoch: 12 Global Step: 122475 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:11:05,039-Speed 131.00 samples/sec Loss 3.0477 Epoch: 12 Global Step: 122480 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:11:24,856-Speed 129.19 samples/sec Loss 3.0711 Epoch: 12 Global Step: 122485 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:11:44,345-Speed 131.37 samples/sec Loss 3.1194 Epoch: 12 Global Step: 122490 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:12:04,060-Speed 129.86 samples/sec Loss 3.1677 Epoch: 12 Global Step: 122495 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:12:23,593-Speed 131.06 samples/sec Loss 3.1079 Epoch: 12 Global Step: 122500 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:12:43,247-Speed 130.27 samples/sec Loss 2.9803 Epoch: 12 Global Step: 122505 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:13:02,824-Speed 130.77 samples/sec Loss 2.8724 Epoch: 12 Global Step: 122510 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:13:22,457-Speed 130.40 samples/sec Loss 2.9133 Epoch: 12 Global Step: 122515 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:13:41,690-Speed 133.12 samples/sec Loss 3.1548 Epoch: 12 Global Step: 122520 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:14:01,216-Speed 131.11 samples/sec Loss 3.3446 Epoch: 12 Global Step: 122525 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:14:21,079-Speed 128.89 samples/sec Loss 3.0661 Epoch: 12 Global Step: 122530 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:14:40,717-Speed 130.36 samples/sec Loss 3.0675 Epoch: 12 Global Step: 122535 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:15:00,346-Speed 130.43 samples/sec Loss 3.1532 Epoch: 12 Global Step: 122540 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:15:20,406-Speed 127.62 samples/sec Loss 3.1353 Epoch: 12 Global Step: 122545 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:15:39,682-Speed 132.82 samples/sec Loss 3.3362 Epoch: 12 Global Step: 122550 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:15:59,380-Speed 129.97 samples/sec Loss 3.1575 Epoch: 12 Global Step: 122555 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:16:19,386-Speed 127.97 samples/sec Loss 2.9993 Epoch: 12 Global Step: 122560 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:16:39,015-Speed 130.42 samples/sec Loss 2.9871 Epoch: 12 Global Step: 122565 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:16:58,980-Speed 128.23 samples/sec Loss 3.1596 Epoch: 12 Global Step: 122570 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:17:18,558-Speed 130.77 samples/sec Loss 3.1933 Epoch: 12 Global Step: 122575 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:17:38,531-Speed 128.18 samples/sec Loss 3.3312 Epoch: 12 Global Step: 122580 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:17:58,153-Speed 130.48 samples/sec Loss 3.0311 Epoch: 12 Global Step: 122585 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:18:17,936-Speed 129.41 samples/sec Loss 3.2100 Epoch: 12 Global Step: 122590 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:18:37,503-Speed 130.83 samples/sec Loss 3.1430 Epoch: 12 Global Step: 122595 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:18:57,599-Speed 127.39 samples/sec Loss 3.0217 Epoch: 12 Global Step: 122600 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:19:17,319-Speed 129.83 samples/sec Loss 3.1090 Epoch: 12 Global Step: 122605 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:19:36,785-Speed 131.51 samples/sec Loss 3.0613 Epoch: 12 Global Step: 122610 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:19:56,694-Speed 128.60 samples/sec Loss 2.9618 Epoch: 12 Global Step: 122615 Fp16 Grad Scale: 4096 Required: 150 hours Training: 2021-06-12 20:20:16,328-Speed 130.40 samples/sec Loss 3.0130 Epoch: 12 Global Step: 122620 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:20:36,247-Speed 128.53 samples/sec Loss 2.8873 Epoch: 12 Global Step: 122625 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:20:55,802-Speed 130.92 samples/sec Loss 3.2126 Epoch: 12 Global Step: 122630 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:21:15,726-Speed 128.49 samples/sec Loss 2.9220 Epoch: 12 Global Step: 122635 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:21:35,310-Speed 130.73 samples/sec Loss 3.0763 Epoch: 12 Global Step: 122640 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:21:55,480-Speed 126.93 samples/sec Loss 2.9587 Epoch: 12 Global Step: 122645 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:22:15,212-Speed 129.75 samples/sec Loss 3.0728 Epoch: 12 Global Step: 122650 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:22:34,824-Speed 130.54 samples/sec Loss 3.0029 Epoch: 12 Global Step: 122655 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:22:54,547-Speed 129.80 samples/sec Loss 3.0770 Epoch: 12 Global Step: 122660 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:23:14,169-Speed 130.48 samples/sec Loss 3.2460 Epoch: 12 Global Step: 122665 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:23:34,043-Speed 128.82 samples/sec Loss 3.2477 Epoch: 12 Global Step: 122670 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:23:53,495-Speed 131.62 samples/sec Loss 3.2479 Epoch: 12 Global Step: 122675 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:24:13,408-Speed 128.57 samples/sec Loss 3.2138 Epoch: 12 Global Step: 122680 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:24:33,139-Speed 129.75 samples/sec Loss 3.0772 Epoch: 12 Global Step: 122685 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:24:53,244-Speed 127.34 samples/sec Loss 3.0616 Epoch: 12 Global Step: 122690 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:25:13,027-Speed 129.41 samples/sec Loss 3.2470 Epoch: 12 Global Step: 122695 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:25:32,597-Speed 130.82 samples/sec Loss 3.0563 Epoch: 12 Global Step: 122700 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:25:52,374-Speed 129.45 samples/sec Loss 3.0937 Epoch: 12 Global Step: 122705 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:26:11,969-Speed 130.66 samples/sec Loss 3.0992 Epoch: 12 Global Step: 122710 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:26:31,688-Speed 129.83 samples/sec Loss 3.2243 Epoch: 12 Global Step: 122715 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:26:51,157-Speed 131.50 samples/sec Loss 3.0181 Epoch: 12 Global Step: 122720 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:27:11,036-Speed 128.79 samples/sec Loss 2.8104 Epoch: 12 Global Step: 122725 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:27:30,674-Speed 130.36 samples/sec Loss 3.0873 Epoch: 12 Global Step: 122730 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:27:50,757-Speed 127.48 samples/sec Loss 3.1124 Epoch: 12 Global Step: 122735 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:28:10,479-Speed 129.81 samples/sec Loss 3.0932 Epoch: 12 Global Step: 122740 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:28:30,060-Speed 130.74 samples/sec Loss 3.4293 Epoch: 12 Global Step: 122745 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:28:49,857-Speed 129.31 samples/sec Loss 3.0015 Epoch: 12 Global Step: 122750 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:29:09,435-Speed 130.77 samples/sec Loss 3.2407 Epoch: 12 Global Step: 122755 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:29:29,053-Speed 130.50 samples/sec Loss 3.2387 Epoch: 12 Global Step: 122760 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:29:48,651-Speed 130.63 samples/sec Loss 3.0994 Epoch: 12 Global Step: 122765 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:30:08,387-Speed 129.72 samples/sec Loss 3.1139 Epoch: 12 Global Step: 122770 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:30:27,935-Speed 130.96 samples/sec Loss 3.0928 Epoch: 12 Global Step: 122775 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:30:48,048-Speed 127.29 samples/sec Loss 3.0829 Epoch: 12 Global Step: 122780 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:31:07,916-Speed 128.85 samples/sec Loss 2.9443 Epoch: 12 Global Step: 122785 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:31:27,528-Speed 130.54 samples/sec Loss 2.9384 Epoch: 12 Global Step: 122790 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:31:47,413-Speed 128.74 samples/sec Loss 3.1447 Epoch: 12 Global Step: 122795 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:32:07,367-Speed 128.30 samples/sec Loss 3.1861 Epoch: 12 Global Step: 122800 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:32:27,053-Speed 130.05 samples/sec Loss 3.0719 Epoch: 12 Global Step: 122805 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:32:46,615-Speed 130.88 samples/sec Loss 2.9453 Epoch: 12 Global Step: 122810 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:33:06,365-Speed 129.63 samples/sec Loss 3.1573 Epoch: 12 Global Step: 122815 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:33:25,991-Speed 130.44 samples/sec Loss 3.1531 Epoch: 12 Global Step: 122820 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:33:45,834-Speed 129.02 samples/sec Loss 3.2628 Epoch: 12 Global Step: 122825 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:34:05,732-Speed 128.66 samples/sec Loss 3.0093 Epoch: 12 Global Step: 122830 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:34:25,311-Speed 130.76 samples/sec Loss 3.0765 Epoch: 12 Global Step: 122835 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:34:45,225-Speed 128.56 samples/sec Loss 3.2356 Epoch: 12 Global Step: 122840 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:35:04,733-Speed 131.24 samples/sec Loss 3.1109 Epoch: 12 Global Step: 122845 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:35:24,545-Speed 129.22 samples/sec Loss 3.0386 Epoch: 12 Global Step: 122850 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:35:44,160-Speed 130.52 samples/sec Loss 3.0608 Epoch: 12 Global Step: 122855 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:36:03,909-Speed 129.63 samples/sec Loss 3.0747 Epoch: 12 Global Step: 122860 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:36:23,495-Speed 130.72 samples/sec Loss 3.2368 Epoch: 12 Global Step: 122865 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:36:43,462-Speed 128.21 samples/sec Loss 3.1519 Epoch: 12 Global Step: 122870 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:37:03,243-Speed 129.42 samples/sec Loss 3.1748 Epoch: 12 Global Step: 122875 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:37:22,792-Speed 130.96 samples/sec Loss 3.2050 Epoch: 12 Global Step: 122880 Fp16 Grad Scale: 8192 Required: 150 hours Training: 2021-06-12 20:37:42,549-Speed 129.58 samples/sec Loss 3.1792 Epoch: 12 Global Step: 122885 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:38:01,986-Speed 131.72 samples/sec Loss 3.2509 Epoch: 12 Global Step: 122890 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:38:21,896-Speed 128.58 samples/sec Loss 3.2243 Epoch: 12 Global Step: 122895 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:38:41,512-Speed 130.52 samples/sec Loss 3.1059 Epoch: 12 Global Step: 122900 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:39:01,414-Speed 128.64 samples/sec Loss 3.1754 Epoch: 12 Global Step: 122905 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:39:20,853-Speed 131.70 samples/sec Loss 3.0142 Epoch: 12 Global Step: 122910 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:39:40,930-Speed 127.51 samples/sec Loss 3.0240 Epoch: 12 Global Step: 122915 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:40:00,679-Speed 129.63 samples/sec Loss 3.0347 Epoch: 12 Global Step: 122920 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:40:20,226-Speed 130.97 samples/sec Loss 3.1409 Epoch: 12 Global Step: 122925 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:40:39,928-Speed 129.94 samples/sec Loss 3.2005 Epoch: 12 Global Step: 122930 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:40:59,475-Speed 130.98 samples/sec Loss 3.0688 Epoch: 12 Global Step: 122935 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:41:19,422-Speed 128.34 samples/sec Loss 3.3012 Epoch: 12 Global Step: 122940 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:41:38,987-Speed 130.85 samples/sec Loss 2.9636 Epoch: 12 Global Step: 122945 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:41:58,794-Speed 129.26 samples/sec Loss 3.1252 Epoch: 12 Global Step: 122950 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:42:18,348-Speed 130.93 samples/sec Loss 3.3171 Epoch: 12 Global Step: 122955 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:42:38,486-Speed 127.12 samples/sec Loss 3.2406 Epoch: 12 Global Step: 122960 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:42:58,219-Speed 129.74 samples/sec Loss 3.1282 Epoch: 12 Global Step: 122965 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:43:17,839-Speed 130.49 samples/sec Loss 3.0901 Epoch: 12 Global Step: 122970 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:43:37,469-Speed 130.42 samples/sec Loss 3.1307 Epoch: 12 Global Step: 122975 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:43:57,013-Speed 130.99 samples/sec Loss 3.1400 Epoch: 12 Global Step: 122980 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:44:16,873-Speed 128.91 samples/sec Loss 3.2536 Epoch: 12 Global Step: 122985 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:44:36,459-Speed 130.71 samples/sec Loss 3.2468 Epoch: 12 Global Step: 122990 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:44:56,232-Speed 129.48 samples/sec Loss 3.1255 Epoch: 12 Global Step: 122995 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:45:15,812-Speed 130.75 samples/sec Loss 3.0973 Epoch: 12 Global Step: 123000 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:45:35,907-Speed 127.40 samples/sec Loss 3.1304 Epoch: 12 Global Step: 123005 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:45:55,896-Speed 128.08 samples/sec Loss 3.1167 Epoch: 12 Global Step: 123010 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:46:15,397-Speed 131.28 samples/sec Loss 3.1093 Epoch: 12 Global Step: 123015 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:46:35,134-Speed 129.71 samples/sec Loss 2.9844 Epoch: 12 Global Step: 123020 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:46:54,728-Speed 130.66 samples/sec Loss 3.0995 Epoch: 12 Global Step: 123025 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:47:14,523-Speed 129.33 samples/sec Loss 3.1595 Epoch: 12 Global Step: 123030 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:47:34,124-Speed 130.61 samples/sec Loss 3.0968 Epoch: 12 Global Step: 123035 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:47:53,757-Speed 130.40 samples/sec Loss 3.0505 Epoch: 12 Global Step: 123040 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:48:13,304-Speed 130.97 samples/sec Loss 3.0309 Epoch: 12 Global Step: 123045 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:48:33,350-Speed 127.71 samples/sec Loss 3.1447 Epoch: 12 Global Step: 123050 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:48:53,339-Speed 128.08 samples/sec Loss 3.1678 Epoch: 12 Global Step: 123055 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:49:12,880-Speed 131.01 samples/sec Loss 3.2448 Epoch: 12 Global Step: 123060 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:49:32,791-Speed 128.58 samples/sec Loss 3.3335 Epoch: 12 Global Step: 123065 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:49:52,315-Speed 131.13 samples/sec Loss 3.1931 Epoch: 12 Global Step: 123070 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:50:12,176-Speed 128.90 samples/sec Loss 2.9518 Epoch: 12 Global Step: 123075 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:50:31,691-Speed 131.19 samples/sec Loss 3.0834 Epoch: 12 Global Step: 123080 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:50:51,423-Speed 129.75 samples/sec Loss 3.3776 Epoch: 12 Global Step: 123085 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:51:11,257-Speed 129.07 samples/sec Loss 3.2259 Epoch: 12 Global Step: 123090 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:51:31,358-Speed 127.37 samples/sec Loss 3.1362 Epoch: 12 Global Step: 123095 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:51:51,510-Speed 127.04 samples/sec Loss 3.3410 Epoch: 12 Global Step: 123100 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:52:11,074-Speed 130.86 samples/sec Loss 3.2162 Epoch: 12 Global Step: 123105 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:52:30,984-Speed 128.58 samples/sec Loss 3.1316 Epoch: 12 Global Step: 123110 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:52:50,608-Speed 130.46 samples/sec Loss 2.9452 Epoch: 12 Global Step: 123115 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:53:10,469-Speed 128.91 samples/sec Loss 3.0211 Epoch: 12 Global Step: 123120 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:53:30,033-Speed 130.86 samples/sec Loss 3.0802 Epoch: 12 Global Step: 123125 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:53:49,928-Speed 128.69 samples/sec Loss 3.2641 Epoch: 12 Global Step: 123130 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:54:09,480-Speed 130.94 samples/sec Loss 3.2475 Epoch: 12 Global Step: 123135 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:54:29,573-Speed 127.42 samples/sec Loss 3.0366 Epoch: 12 Global Step: 123140 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:54:49,229-Speed 130.25 samples/sec Loss 3.0919 Epoch: 12 Global Step: 123145 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:55:08,836-Speed 130.57 samples/sec Loss 2.9553 Epoch: 12 Global Step: 123150 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:55:28,673-Speed 129.05 samples/sec Loss 3.1662 Epoch: 12 Global Step: 123155 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:55:48,208-Speed 131.06 samples/sec Loss 3.0339 Epoch: 12 Global Step: 123160 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:56:08,184-Speed 128.16 samples/sec Loss 3.1216 Epoch: 12 Global Step: 123165 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:56:27,790-Speed 130.59 samples/sec Loss 3.2169 Epoch: 12 Global Step: 123170 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:56:47,627-Speed 129.06 samples/sec Loss 3.1276 Epoch: 12 Global Step: 123175 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:57:07,250-Speed 130.47 samples/sec Loss 3.1228 Epoch: 12 Global Step: 123180 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:57:27,317-Speed 127.58 samples/sec Loss 2.9229 Epoch: 12 Global Step: 123185 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:57:47,199-Speed 128.77 samples/sec Loss 3.0662 Epoch: 12 Global Step: 123190 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:58:06,776-Speed 130.77 samples/sec Loss 3.0096 Epoch: 12 Global Step: 123195 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:58:26,694-Speed 128.53 samples/sec Loss 3.0443 Epoch: 12 Global Step: 123200 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:58:46,329-Speed 130.39 samples/sec Loss 3.2099 Epoch: 12 Global Step: 123205 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:59:06,059-Speed 129.75 samples/sec Loss 3.2491 Epoch: 12 Global Step: 123210 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:59:25,646-Speed 130.71 samples/sec Loss 3.2828 Epoch: 12 Global Step: 123215 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 20:59:45,502-Speed 128.93 samples/sec Loss 3.0542 Epoch: 12 Global Step: 123220 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:00:05,056-Speed 130.93 samples/sec Loss 2.9648 Epoch: 12 Global Step: 123225 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:00:25,061-Speed 127.98 samples/sec Loss 3.2065 Epoch: 12 Global Step: 123230 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:00:44,976-Speed 128.55 samples/sec Loss 2.9839 Epoch: 12 Global Step: 123235 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:01:04,606-Speed 130.42 samples/sec Loss 3.0509 Epoch: 12 Global Step: 123240 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:01:24,353-Speed 129.65 samples/sec Loss 3.3813 Epoch: 12 Global Step: 123245 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:01:43,916-Speed 130.87 samples/sec Loss 3.2514 Epoch: 12 Global Step: 123250 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:02:03,673-Speed 129.58 samples/sec Loss 3.1222 Epoch: 12 Global Step: 123255 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:02:23,293-Speed 130.49 samples/sec Loss 3.2236 Epoch: 12 Global Step: 123260 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:02:43,091-Speed 129.31 samples/sec Loss 2.9948 Epoch: 12 Global Step: 123265 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:03:02,692-Speed 130.62 samples/sec Loss 3.1094 Epoch: 12 Global Step: 123270 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:03:22,717-Speed 127.85 samples/sec Loss 3.0938 Epoch: 12 Global Step: 123275 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:03:42,692-Speed 128.17 samples/sec Loss 3.2734 Epoch: 12 Global Step: 123280 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:04:02,149-Speed 131.58 samples/sec Loss 3.2293 Epoch: 12 Global Step: 123285 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:04:22,055-Speed 128.61 samples/sec Loss 3.0737 Epoch: 12 Global Step: 123290 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:04:41,612-Speed 130.91 samples/sec Loss 3.2900 Epoch: 12 Global Step: 123295 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:05:01,497-Speed 128.75 samples/sec Loss 3.0531 Epoch: 12 Global Step: 123300 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:05:21,105-Speed 130.57 samples/sec Loss 3.1049 Epoch: 12 Global Step: 123305 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:05:40,868-Speed 129.54 samples/sec Loss 3.1446 Epoch: 12 Global Step: 123310 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:06:00,461-Speed 130.66 samples/sec Loss 3.0507 Epoch: 12 Global Step: 123315 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:06:20,575-Speed 127.28 samples/sec Loss 3.1718 Epoch: 12 Global Step: 123320 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:06:40,375-Speed 129.30 samples/sec Loss 3.1990 Epoch: 12 Global Step: 123325 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:06:59,992-Speed 130.50 samples/sec Loss 3.2445 Epoch: 12 Global Step: 123330 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:07:19,875-Speed 128.77 samples/sec Loss 3.3296 Epoch: 12 Global Step: 123335 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:07:39,447-Speed 130.80 samples/sec Loss 3.1720 Epoch: 12 Global Step: 123340 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:07:59,237-Speed 129.37 samples/sec Loss 3.0668 Epoch: 12 Global Step: 123345 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:08:18,828-Speed 130.68 samples/sec Loss 3.0924 Epoch: 12 Global Step: 123350 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:08:38,724-Speed 128.67 samples/sec Loss 3.0601 Epoch: 12 Global Step: 123355 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:08:58,400-Speed 130.12 samples/sec Loss 3.2073 Epoch: 12 Global Step: 123360 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:09:18,394-Speed 128.04 samples/sec Loss 3.0848 Epoch: 12 Global Step: 123365 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:09:38,535-Speed 127.11 samples/sec Loss 3.1377 Epoch: 12 Global Step: 123370 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:09:57,996-Speed 131.55 samples/sec Loss 2.9549 Epoch: 12 Global Step: 123375 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:10:17,731-Speed 129.73 samples/sec Loss 3.2262 Epoch: 12 Global Step: 123380 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:10:37,367-Speed 130.38 samples/sec Loss 3.0180 Epoch: 12 Global Step: 123385 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:10:57,166-Speed 129.30 samples/sec Loss 3.2364 Epoch: 12 Global Step: 123390 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:11:16,780-Speed 130.53 samples/sec Loss 3.0330 Epoch: 12 Global Step: 123395 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:11:36,431-Speed 130.29 samples/sec Loss 2.9332 Epoch: 12 Global Step: 123400 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:11:56,056-Speed 130.45 samples/sec Loss 3.0706 Epoch: 12 Global Step: 123405 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:12:16,155-Speed 127.37 samples/sec Loss 3.2149 Epoch: 12 Global Step: 123410 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:12:36,062-Speed 128.60 samples/sec Loss 3.3036 Epoch: 12 Global Step: 123415 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:12:55,615-Speed 130.93 samples/sec Loss 3.0444 Epoch: 12 Global Step: 123420 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:13:15,335-Speed 129.83 samples/sec Loss 3.2657 Epoch: 12 Global Step: 123425 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:13:35,184-Speed 128.98 samples/sec Loss 3.2399 Epoch: 12 Global Step: 123430 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:13:54,946-Speed 129.55 samples/sec Loss 3.2388 Epoch: 12 Global Step: 123435 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:14:14,551-Speed 130.59 samples/sec Loss 3.2554 Epoch: 12 Global Step: 123440 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:14:34,278-Speed 129.78 samples/sec Loss 3.0867 Epoch: 12 Global Step: 123445 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:14:53,831-Speed 130.93 samples/sec Loss 2.9922 Epoch: 12 Global Step: 123450 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:15:13,920-Speed 127.44 samples/sec Loss 3.1228 Epoch: 12 Global Step: 123455 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:15:33,845-Speed 128.49 samples/sec Loss 3.2226 Epoch: 12 Global Step: 123460 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:15:53,387-Speed 131.00 samples/sec Loss 3.2840 Epoch: 12 Global Step: 123465 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:16:13,196-Speed 129.24 samples/sec Loss 3.1800 Epoch: 12 Global Step: 123470 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:16:32,800-Speed 130.59 samples/sec Loss 3.1463 Epoch: 12 Global Step: 123475 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:16:52,445-Speed 130.32 samples/sec Loss 3.1025 Epoch: 12 Global Step: 123480 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:17:12,051-Speed 130.58 samples/sec Loss 3.1882 Epoch: 12 Global Step: 123485 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:17:31,823-Speed 129.49 samples/sec Loss 3.1158 Epoch: 12 Global Step: 123490 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:17:51,415-Speed 130.68 samples/sec Loss 3.1782 Epoch: 12 Global Step: 123495 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:18:11,447-Speed 127.80 samples/sec Loss 3.0868 Epoch: 12 Global Step: 123500 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:18:31,268-Speed 129.17 samples/sec Loss 3.0227 Epoch: 12 Global Step: 123505 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:18:50,854-Speed 130.71 samples/sec Loss 3.2126 Epoch: 12 Global Step: 123510 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:19:10,831-Speed 128.16 samples/sec Loss 3.1192 Epoch: 12 Global Step: 123515 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:19:30,358-Speed 131.11 samples/sec Loss 3.3960 Epoch: 12 Global Step: 123520 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:19:50,385-Speed 127.84 samples/sec Loss 3.1706 Epoch: 12 Global Step: 123525 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:20:09,996-Speed 130.55 samples/sec Loss 3.2135 Epoch: 12 Global Step: 123530 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:20:29,745-Speed 129.63 samples/sec Loss 3.0862 Epoch: 12 Global Step: 123535 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:20:49,586-Speed 129.03 samples/sec Loss 3.2732 Epoch: 12 Global Step: 123540 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:21:09,481-Speed 128.68 samples/sec Loss 3.1572 Epoch: 12 Global Step: 123545 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:21:29,457-Speed 128.16 samples/sec Loss 3.2277 Epoch: 12 Global Step: 123550 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:21:48,899-Speed 131.68 samples/sec Loss 3.2746 Epoch: 12 Global Step: 123555 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:22:08,875-Speed 128.16 samples/sec Loss 3.0611 Epoch: 12 Global Step: 123560 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:22:28,315-Speed 131.70 samples/sec Loss 3.0138 Epoch: 12 Global Step: 123565 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:22:48,176-Speed 128.90 samples/sec Loss 3.3320 Epoch: 12 Global Step: 123570 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:23:07,743-Speed 130.84 samples/sec Loss 3.0762 Epoch: 12 Global Step: 123575 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:23:27,452-Speed 129.89 samples/sec Loss 3.1469 Epoch: 12 Global Step: 123580 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:23:47,083-Speed 130.41 samples/sec Loss 3.2514 Epoch: 12 Global Step: 123585 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:24:07,088-Speed 127.97 samples/sec Loss 3.0711 Epoch: 12 Global Step: 123590 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:24:26,849-Speed 129.56 samples/sec Loss 3.1918 Epoch: 12 Global Step: 123595 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:24:46,378-Speed 131.10 samples/sec Loss 3.2796 Epoch: 12 Global Step: 123600 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:25:06,286-Speed 128.60 samples/sec Loss 3.0558 Epoch: 12 Global Step: 123605 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:25:25,856-Speed 130.82 samples/sec Loss 3.1026 Epoch: 12 Global Step: 123610 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:25:45,768-Speed 128.58 samples/sec Loss 3.1925 Epoch: 12 Global Step: 123615 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:26:05,506-Speed 129.71 samples/sec Loss 3.0732 Epoch: 12 Global Step: 123620 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:26:25,463-Speed 128.28 samples/sec Loss 3.2189 Epoch: 12 Global Step: 123625 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:26:44,994-Speed 131.08 samples/sec Loss 3.1052 Epoch: 12 Global Step: 123630 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:27:05,102-Speed 127.32 samples/sec Loss 3.0834 Epoch: 12 Global Step: 123635 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:27:24,797-Speed 129.99 samples/sec Loss 3.0270 Epoch: 12 Global Step: 123640 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:27:44,409-Speed 130.54 samples/sec Loss 3.1928 Epoch: 12 Global Step: 123645 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:28:04,016-Speed 130.58 samples/sec Loss 3.2064 Epoch: 12 Global Step: 123650 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:28:23,556-Speed 131.02 samples/sec Loss 3.1386 Epoch: 12 Global Step: 123655 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:28:43,436-Speed 128.78 samples/sec Loss 3.1730 Epoch: 12 Global Step: 123660 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:29:03,065-Speed 130.42 samples/sec Loss 3.1489 Epoch: 12 Global Step: 123665 Fp16 Grad Scale: 8192 Required: 149 hours Training: 2021-06-12 21:29:22,952-Speed 128.74 samples/sec Loss 2.9256 Epoch: 12 Global Step: 123670 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:29:42,431-Speed 131.42 samples/sec Loss 3.2342 Epoch: 12 Global Step: 123675 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:30:02,521-Speed 127.43 samples/sec Loss 3.1867 Epoch: 12 Global Step: 123680 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:30:22,260-Speed 129.70 samples/sec Loss 3.0741 Epoch: 12 Global Step: 123685 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:30:41,873-Speed 130.53 samples/sec Loss 3.3220 Epoch: 12 Global Step: 123690 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:31:01,606-Speed 129.74 samples/sec Loss 3.1842 Epoch: 12 Global Step: 123695 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:31:21,207-Speed 130.61 samples/sec Loss 3.1446 Epoch: 12 Global Step: 123700 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:31:40,825-Speed 130.50 samples/sec Loss 3.2001 Epoch: 12 Global Step: 123705 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:32:00,447-Speed 130.48 samples/sec Loss 3.2069 Epoch: 12 Global Step: 123710 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:32:20,299-Speed 128.96 samples/sec Loss 3.1832 Epoch: 12 Global Step: 123715 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:32:39,893-Speed 130.66 samples/sec Loss 3.3445 Epoch: 12 Global Step: 123720 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:32:59,960-Speed 127.58 samples/sec Loss 3.0564 Epoch: 12 Global Step: 123725 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:33:20,008-Speed 127.70 samples/sec Loss 3.2226 Epoch: 12 Global Step: 123730 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:33:39,476-Speed 131.51 samples/sec Loss 3.2136 Epoch: 12 Global Step: 123735 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:33:59,198-Speed 129.81 samples/sec Loss 3.2690 Epoch: 12 Global Step: 123740 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:34:18,801-Speed 130.60 samples/sec Loss 3.1646 Epoch: 12 Global Step: 123745 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:34:38,511-Speed 129.89 samples/sec Loss 3.0692 Epoch: 12 Global Step: 123750 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:34:58,103-Speed 130.67 samples/sec Loss 3.0987 Epoch: 12 Global Step: 123755 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:35:17,808-Speed 129.93 samples/sec Loss 3.1264 Epoch: 12 Global Step: 123760 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:35:37,370-Speed 130.87 samples/sec Loss 3.1510 Epoch: 12 Global Step: 123765 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:35:57,433-Speed 127.60 samples/sec Loss 3.3169 Epoch: 12 Global Step: 123770 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:36:17,494-Speed 127.62 samples/sec Loss 3.1930 Epoch: 12 Global Step: 123775 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:36:37,055-Speed 130.88 samples/sec Loss 3.2862 Epoch: 12 Global Step: 123780 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:36:56,925-Speed 128.85 samples/sec Loss 3.0410 Epoch: 12 Global Step: 123785 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:37:16,490-Speed 130.85 samples/sec Loss 3.3486 Epoch: 12 Global Step: 123790 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:37:36,314-Speed 129.14 samples/sec Loss 3.2007 Epoch: 12 Global Step: 123795 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:37:55,876-Speed 130.88 samples/sec Loss 3.1770 Epoch: 12 Global Step: 123800 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:38:15,621-Speed 129.66 samples/sec Loss 3.1474 Epoch: 12 Global Step: 123805 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:38:35,247-Speed 130.45 samples/sec Loss 3.3918 Epoch: 12 Global Step: 123810 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:38:55,126-Speed 128.79 samples/sec Loss 3.0583 Epoch: 12 Global Step: 123815 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:39:15,035-Speed 128.59 samples/sec Loss 3.1670 Epoch: 12 Global Step: 123820 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:39:34,539-Speed 131.26 samples/sec Loss 3.0394 Epoch: 12 Global Step: 123825 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:39:54,448-Speed 128.59 samples/sec Loss 3.0176 Epoch: 12 Global Step: 123830 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:40:13,912-Speed 131.54 samples/sec Loss 3.4589 Epoch: 12 Global Step: 123835 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:40:33,742-Speed 129.10 samples/sec Loss 3.1884 Epoch: 12 Global Step: 123840 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:40:53,308-Speed 130.85 samples/sec Loss 3.1410 Epoch: 12 Global Step: 123845 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:41:12,938-Speed 130.42 samples/sec Loss 3.2337 Epoch: 12 Global Step: 123850 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:41:32,536-Speed 130.64 samples/sec Loss 3.4530 Epoch: 12 Global Step: 123855 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:41:52,404-Speed 128.86 samples/sec Loss 3.1071 Epoch: 12 Global Step: 123860 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:42:12,383-Speed 128.14 samples/sec Loss 3.2333 Epoch: 12 Global Step: 123865 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:42:31,914-Speed 131.08 samples/sec Loss 3.0711 Epoch: 12 Global Step: 123870 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:42:51,712-Speed 129.31 samples/sec Loss 3.2620 Epoch: 12 Global Step: 123875 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:43:11,303-Speed 130.68 samples/sec Loss 3.1069 Epoch: 12 Global Step: 123880 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:43:31,209-Speed 128.61 samples/sec Loss 3.2119 Epoch: 12 Global Step: 123885 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:43:50,746-Speed 131.04 samples/sec Loss 3.0658 Epoch: 12 Global Step: 123890 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:44:10,506-Speed 129.56 samples/sec Loss 3.3612 Epoch: 12 Global Step: 123895 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:44:30,090-Speed 130.72 samples/sec Loss 3.2724 Epoch: 12 Global Step: 123900 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:44:50,248-Speed 127.01 samples/sec Loss 3.3250 Epoch: 12 Global Step: 123905 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:45:09,978-Speed 129.76 samples/sec Loss 3.0884 Epoch: 12 Global Step: 123910 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:45:29,571-Speed 130.66 samples/sec Loss 2.9406 Epoch: 12 Global Step: 123915 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:45:49,433-Speed 128.90 samples/sec Loss 3.3089 Epoch: 12 Global Step: 123920 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:46:09,049-Speed 130.52 samples/sec Loss 3.0699 Epoch: 12 Global Step: 123925 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:46:28,944-Speed 128.68 samples/sec Loss 3.0957 Epoch: 12 Global Step: 123930 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:46:48,527-Speed 130.73 samples/sec Loss 3.0030 Epoch: 12 Global Step: 123935 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:47:08,421-Speed 128.69 samples/sec Loss 3.4122 Epoch: 12 Global Step: 123940 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:47:27,978-Speed 130.91 samples/sec Loss 3.1475 Epoch: 12 Global Step: 123945 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:47:48,076-Speed 127.38 samples/sec Loss 3.2251 Epoch: 12 Global Step: 123950 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:48:07,806-Speed 129.76 samples/sec Loss 3.0469 Epoch: 12 Global Step: 123955 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:48:27,407-Speed 130.61 samples/sec Loss 3.1761 Epoch: 12 Global Step: 123960 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:48:47,129-Speed 129.82 samples/sec Loss 3.2387 Epoch: 12 Global Step: 123965 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:49:06,693-Speed 130.86 samples/sec Loss 3.2274 Epoch: 12 Global Step: 123970 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:49:27,212-Speed 124.77 samples/sec Loss 3.2796 Epoch: 12 Global Step: 123975 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:49:46,663-Speed 131.62 samples/sec Loss 3.2815 Epoch: 12 Global Step: 123980 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:50:06,506-Speed 129.03 samples/sec Loss 3.2889 Epoch: 12 Global Step: 123985 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:50:26,120-Speed 130.53 samples/sec Loss 3.0215 Epoch: 12 Global Step: 123990 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:50:46,185-Speed 127.59 samples/sec Loss 3.3181 Epoch: 12 Global Step: 123995 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:51:05,956-Speed 129.49 samples/sec Loss 3.2218 Epoch: 12 Global Step: 124000 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:51:25,527-Speed 130.82 samples/sec Loss 3.3253 Epoch: 12 Global Step: 124005 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:51:45,322-Speed 129.33 samples/sec Loss 3.2186 Epoch: 12 Global Step: 124010 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:52:04,952-Speed 130.42 samples/sec Loss 3.0892 Epoch: 12 Global Step: 124015 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:52:24,634-Speed 130.08 samples/sec Loss 3.3179 Epoch: 12 Global Step: 124020 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:52:44,208-Speed 130.79 samples/sec Loss 3.1870 Epoch: 12 Global Step: 124025 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:53:04,077-Speed 128.85 samples/sec Loss 3.3122 Epoch: 12 Global Step: 124030 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:53:23,691-Speed 130.52 samples/sec Loss 3.2662 Epoch: 12 Global Step: 124035 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:53:43,696-Speed 127.98 samples/sec Loss 3.0151 Epoch: 12 Global Step: 124040 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:54:03,620-Speed 128.49 samples/sec Loss 3.1833 Epoch: 12 Global Step: 124045 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:54:23,158-Speed 131.03 samples/sec Loss 3.1220 Epoch: 12 Global Step: 124050 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:54:42,906-Speed 129.64 samples/sec Loss 3.1193 Epoch: 12 Global Step: 124055 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:55:02,491-Speed 130.72 samples/sec Loss 3.0486 Epoch: 12 Global Step: 124060 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:55:22,110-Speed 130.49 samples/sec Loss 3.3140 Epoch: 12 Global Step: 124065 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:55:41,743-Speed 130.40 samples/sec Loss 3.1939 Epoch: 12 Global Step: 124070 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:56:01,490-Speed 129.64 samples/sec Loss 3.2531 Epoch: 12 Global Step: 124075 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:56:21,111-Speed 130.48 samples/sec Loss 3.1965 Epoch: 12 Global Step: 124080 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:56:41,051-Speed 128.39 samples/sec Loss 3.3309 Epoch: 12 Global Step: 124085 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:57:00,923-Speed 128.83 samples/sec Loss 3.2360 Epoch: 12 Global Step: 124090 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:57:20,539-Speed 130.51 samples/sec Loss 3.2035 Epoch: 12 Global Step: 124095 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:57:40,398-Speed 128.91 samples/sec Loss 3.2027 Epoch: 12 Global Step: 124100 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:57:59,935-Speed 131.05 samples/sec Loss 3.2177 Epoch: 12 Global Step: 124105 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:58:19,736-Speed 129.29 samples/sec Loss 3.3813 Epoch: 12 Global Step: 124110 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:58:39,288-Speed 130.94 samples/sec Loss 3.1909 Epoch: 12 Global Step: 124115 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:58:59,187-Speed 128.66 samples/sec Loss 3.2430 Epoch: 12 Global Step: 124120 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:59:18,733-Speed 130.98 samples/sec Loss 3.2415 Epoch: 12 Global Step: 124125 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:59:38,816-Speed 127.47 samples/sec Loss 3.2200 Epoch: 12 Global Step: 124130 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 21:59:58,510-Speed 130.00 samples/sec Loss 3.2058 Epoch: 12 Global Step: 124135 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:00:18,101-Speed 130.68 samples/sec Loss 3.2018 Epoch: 12 Global Step: 124140 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:00:37,747-Speed 130.31 samples/sec Loss 3.2515 Epoch: 12 Global Step: 124145 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:00:57,224-Speed 131.45 samples/sec Loss 3.3135 Epoch: 12 Global Step: 124150 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:01:17,125-Speed 128.64 samples/sec Loss 3.2008 Epoch: 12 Global Step: 124155 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:01:36,675-Speed 130.95 samples/sec Loss 3.2819 Epoch: 12 Global Step: 124160 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:01:56,573-Speed 128.67 samples/sec Loss 3.0976 Epoch: 12 Global Step: 124165 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:02:16,140-Speed 130.84 samples/sec Loss 3.1062 Epoch: 12 Global Step: 124170 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:02:36,206-Speed 127.58 samples/sec Loss 3.3635 Epoch: 12 Global Step: 124175 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:02:56,222-Speed 127.90 samples/sec Loss 3.0298 Epoch: 12 Global Step: 124180 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:03:15,798-Speed 130.78 samples/sec Loss 3.2115 Epoch: 12 Global Step: 124185 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:03:35,603-Speed 129.26 samples/sec Loss 3.1802 Epoch: 12 Global Step: 124190 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:03:55,214-Speed 130.54 samples/sec Loss 3.1382 Epoch: 12 Global Step: 124195 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:04:14,851-Speed 130.38 samples/sec Loss 3.1493 Epoch: 12 Global Step: 124200 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:04:34,489-Speed 130.37 samples/sec Loss 3.1465 Epoch: 12 Global Step: 124205 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:04:54,285-Speed 129.33 samples/sec Loss 3.2628 Epoch: 12 Global Step: 124210 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:05:13,830-Speed 130.99 samples/sec Loss 3.2128 Epoch: 12 Global Step: 124215 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:05:34,077-Speed 126.45 samples/sec Loss 3.2842 Epoch: 12 Global Step: 124220 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:05:54,187-Speed 127.30 samples/sec Loss 3.3527 Epoch: 12 Global Step: 124225 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:06:13,725-Speed 131.03 samples/sec Loss 3.4145 Epoch: 12 Global Step: 124230 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:06:33,589-Speed 128.88 samples/sec Loss 3.3747 Epoch: 12 Global Step: 124235 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:06:53,145-Speed 130.92 samples/sec Loss 3.0877 Epoch: 12 Global Step: 124240 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:07:12,946-Speed 129.29 samples/sec Loss 3.3792 Epoch: 12 Global Step: 124245 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:07:33,139-Speed 126.78 samples/sec Loss 3.1582 Epoch: 12 Global Step: 124250 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:07:52,848-Speed 129.90 samples/sec Loss 3.4071 Epoch: 12 Global Step: 124255 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:08:12,477-Speed 130.43 samples/sec Loss 3.3278 Epoch: 12 Global Step: 124260 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:08:32,561-Speed 127.47 samples/sec Loss 3.2493 Epoch: 12 Global Step: 124265 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:08:52,886-Speed 125.96 samples/sec Loss 3.0893 Epoch: 12 Global Step: 124270 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:09:12,975-Speed 127.44 samples/sec Loss 3.3007 Epoch: 12 Global Step: 124275 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:09:32,804-Speed 129.11 samples/sec Loss 3.3143 Epoch: 12 Global Step: 124280 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:09:52,343-Speed 131.03 samples/sec Loss 3.0368 Epoch: 12 Global Step: 124285 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:10:12,331-Speed 128.09 samples/sec Loss 3.2796 Epoch: 12 Global Step: 124290 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:10:31,921-Speed 130.68 samples/sec Loss 3.3653 Epoch: 12 Global Step: 124295 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:10:51,848-Speed 128.48 samples/sec Loss 3.1066 Epoch: 12 Global Step: 124300 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:11:11,436-Speed 130.70 samples/sec Loss 3.2746 Epoch: 12 Global Step: 124305 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:11:31,633-Speed 126.76 samples/sec Loss 3.3303 Epoch: 12 Global Step: 124310 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:11:51,342-Speed 129.89 samples/sec Loss 2.9297 Epoch: 12 Global Step: 124315 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:12:10,898-Speed 130.91 samples/sec Loss 3.2002 Epoch: 12 Global Step: 124320 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:12:30,705-Speed 129.26 samples/sec Loss 3.1978 Epoch: 12 Global Step: 124325 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:12:50,307-Speed 130.60 samples/sec Loss 3.4068 Epoch: 12 Global Step: 124330 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:13:10,201-Speed 128.69 samples/sec Loss 3.1104 Epoch: 12 Global Step: 124335 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:13:29,753-Speed 130.94 samples/sec Loss 3.2015 Epoch: 12 Global Step: 124340 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:13:49,541-Speed 129.37 samples/sec Loss 3.3478 Epoch: 12 Global Step: 124345 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:14:09,123-Speed 130.74 samples/sec Loss 3.1206 Epoch: 12 Global Step: 124350 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:14:29,312-Speed 126.81 samples/sec Loss 3.2836 Epoch: 12 Global Step: 124355 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:14:49,079-Speed 129.51 samples/sec Loss 3.4484 Epoch: 12 Global Step: 124360 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:15:08,681-Speed 130.60 samples/sec Loss 3.1588 Epoch: 12 Global Step: 124365 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:15:28,412-Speed 129.76 samples/sec Loss 3.5509 Epoch: 12 Global Step: 124370 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:15:48,028-Speed 130.51 samples/sec Loss 3.2065 Epoch: 12 Global Step: 124375 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:16:07,781-Speed 129.61 samples/sec Loss 3.3308 Epoch: 12 Global Step: 124380 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:16:27,322-Speed 131.01 samples/sec Loss 3.3013 Epoch: 12 Global Step: 124385 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:16:47,154-Speed 129.09 samples/sec Loss 3.2614 Epoch: 12 Global Step: 124390 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:17:06,717-Speed 130.86 samples/sec Loss 3.3840 Epoch: 12 Global Step: 124395 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:17:26,808-Speed 127.42 samples/sec Loss 3.2486 Epoch: 12 Global Step: 124400 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:17:46,563-Speed 129.60 samples/sec Loss 3.3150 Epoch: 12 Global Step: 124405 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:18:06,137-Speed 130.79 samples/sec Loss 3.3831 Epoch: 12 Global Step: 124410 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:18:25,925-Speed 129.38 samples/sec Loss 3.3435 Epoch: 12 Global Step: 124415 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:18:45,552-Speed 130.44 samples/sec Loss 3.1919 Epoch: 12 Global Step: 124420 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:19:05,271-Speed 129.83 samples/sec Loss 3.1348 Epoch: 12 Global Step: 124425 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:19:24,886-Speed 130.52 samples/sec Loss 3.2987 Epoch: 12 Global Step: 124430 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:19:44,620-Speed 129.74 samples/sec Loss 3.1178 Epoch: 12 Global Step: 124435 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:20:04,249-Speed 130.42 samples/sec Loss 3.2088 Epoch: 12 Global Step: 124440 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:20:24,211-Speed 128.25 samples/sec Loss 3.3887 Epoch: 12 Global Step: 124445 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:20:44,188-Speed 128.15 samples/sec Loss 3.2693 Epoch: 12 Global Step: 124450 Fp16 Grad Scale: 8192 Required: 148 hours Training: 2021-06-12 22:21:03,717-Speed 131.09 samples/sec Loss 3.4626 Epoch: 12 Global Step: 124455 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:21:23,627-Speed 128.59 samples/sec Loss 3.2626 Epoch: 12 Global Step: 124460 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:21:43,248-Speed 130.48 samples/sec Loss 3.2704 Epoch: 12 Global Step: 124465 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:22:03,010-Speed 129.55 samples/sec Loss 3.1254 Epoch: 12 Global Step: 124470 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:22:22,576-Speed 130.85 samples/sec Loss 3.4389 Epoch: 12 Global Step: 124475 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:22:42,240-Speed 130.19 samples/sec Loss 3.3776 Epoch: 12 Global Step: 124480 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:23:01,970-Speed 129.75 samples/sec Loss 3.4408 Epoch: 12 Global Step: 124485 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:23:21,829-Speed 128.92 samples/sec Loss 3.3538 Epoch: 12 Global Step: 124490 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:23:41,713-Speed 128.75 samples/sec Loss 3.1992 Epoch: 12 Global Step: 124495 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:24:01,337-Speed 130.46 samples/sec Loss 3.3246 Epoch: 12 Global Step: 124500 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:24:21,211-Speed 128.81 samples/sec Loss 3.3234 Epoch: 12 Global Step: 124505 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:24:40,759-Speed 130.97 samples/sec Loss 3.4805 Epoch: 12 Global Step: 124510 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:25:00,584-Speed 129.14 samples/sec Loss 3.2448 Epoch: 12 Global Step: 124515 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:25:20,233-Speed 130.29 samples/sec Loss 3.2011 Epoch: 12 Global Step: 124520 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:25:40,065-Speed 129.10 samples/sec Loss 3.4609 Epoch: 12 Global Step: 124525 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:25:59,633-Speed 130.84 samples/sec Loss 3.3776 Epoch: 12 Global Step: 124530 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:26:19,712-Speed 127.50 samples/sec Loss 3.2770 Epoch: 12 Global Step: 124535 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:26:39,495-Speed 129.40 samples/sec Loss 3.2081 Epoch: 12 Global Step: 124540 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:26:59,096-Speed 130.61 samples/sec Loss 3.0613 Epoch: 12 Global Step: 124545 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:27:18,820-Speed 129.80 samples/sec Loss 3.2159 Epoch: 12 Global Step: 124550 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:27:38,362-Speed 131.01 samples/sec Loss 3.4018 Epoch: 12 Global Step: 124555 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:27:58,258-Speed 128.68 samples/sec Loss 3.1788 Epoch: 12 Global Step: 124560 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:28:17,820-Speed 130.87 samples/sec Loss 3.2464 Epoch: 12 Global Step: 124565 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:28:37,687-Speed 128.86 samples/sec Loss 3.2635 Epoch: 12 Global Step: 124570 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:28:57,216-Speed 131.10 samples/sec Loss 3.3135 Epoch: 12 Global Step: 124575 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:29:17,471-Speed 126.39 samples/sec Loss 3.1971 Epoch: 12 Global Step: 124580 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:29:37,260-Speed 129.38 samples/sec Loss 3.1844 Epoch: 12 Global Step: 124585 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:29:56,793-Speed 131.07 samples/sec Loss 3.2317 Epoch: 12 Global Step: 124590 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:30:16,593-Speed 129.30 samples/sec Loss 3.3782 Epoch: 12 Global Step: 124595 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:30:36,214-Speed 130.48 samples/sec Loss 3.2167 Epoch: 12 Global Step: 124600 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:30:56,063-Speed 128.99 samples/sec Loss 3.0857 Epoch: 12 Global Step: 124605 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:31:15,676-Speed 130.53 samples/sec Loss 3.2262 Epoch: 12 Global Step: 124610 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:31:35,500-Speed 129.14 samples/sec Loss 3.2880 Epoch: 12 Global Step: 124615 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:31:55,063-Speed 130.87 samples/sec Loss 3.2344 Epoch: 12 Global Step: 124620 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:32:15,159-Speed 127.40 samples/sec Loss 3.3718 Epoch: 12 Global Step: 124625 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:32:34,980-Speed 129.16 samples/sec Loss 3.2092 Epoch: 12 Global Step: 124630 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:32:54,551-Speed 130.81 samples/sec Loss 3.2834 Epoch: 12 Global Step: 124635 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:33:14,162-Speed 130.55 samples/sec Loss 3.4297 Epoch: 12 Global Step: 124640 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:33:33,776-Speed 130.53 samples/sec Loss 3.3872 Epoch: 12 Global Step: 124645 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:33:53,564-Speed 129.38 samples/sec Loss 3.2443 Epoch: 12 Global Step: 124650 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:34:13,049-Speed 131.39 samples/sec Loss 3.2143 Epoch: 12 Global Step: 124655 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:34:32,812-Speed 129.55 samples/sec Loss 3.2795 Epoch: 12 Global Step: 124660 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:34:52,374-Speed 130.87 samples/sec Loss 3.1995 Epoch: 12 Global Step: 124665 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:35:12,481-Speed 127.33 samples/sec Loss 3.2597 Epoch: 12 Global Step: 124670 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:35:32,592-Speed 127.30 samples/sec Loss 3.4126 Epoch: 12 Global Step: 124675 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:35:52,043-Speed 131.62 samples/sec Loss 3.1557 Epoch: 12 Global Step: 124680 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:36:11,869-Speed 129.13 samples/sec Loss 3.4208 Epoch: 12 Global Step: 124685 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:36:31,458-Speed 130.69 samples/sec Loss 3.4031 Epoch: 12 Global Step: 124690 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:36:51,252-Speed 129.34 samples/sec Loss 3.4020 Epoch: 12 Global Step: 124695 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:37:10,911-Speed 130.23 samples/sec Loss 3.4098 Epoch: 12 Global Step: 124700 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:37:30,535-Speed 130.46 samples/sec Loss 3.2780 Epoch: 12 Global Step: 124705 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:37:50,152-Speed 130.51 samples/sec Loss 3.2428 Epoch: 12 Global Step: 124710 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:38:10,107-Speed 128.29 samples/sec Loss 3.6024 Epoch: 12 Global Step: 124715 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:38:30,024-Speed 128.54 samples/sec Loss 3.2819 Epoch: 12 Global Step: 124720 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:38:49,735-Speed 129.88 samples/sec Loss 3.2053 Epoch: 12 Global Step: 124725 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:39:09,662-Speed 128.47 samples/sec Loss 3.2850 Epoch: 12 Global Step: 124730 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:39:29,247-Speed 130.72 samples/sec Loss 3.1616 Epoch: 12 Global Step: 124735 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:39:49,124-Speed 128.80 samples/sec Loss 3.1859 Epoch: 12 Global Step: 124740 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:40:08,582-Speed 131.57 samples/sec Loss 3.1049 Epoch: 12 Global Step: 124745 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:40:28,469-Speed 128.73 samples/sec Loss 3.2182 Epoch: 12 Global Step: 124750 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:40:48,037-Speed 130.84 samples/sec Loss 3.2438 Epoch: 12 Global Step: 124755 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:41:08,182-Speed 127.08 samples/sec Loss 3.3445 Epoch: 12 Global Step: 124760 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:41:27,793-Speed 130.54 samples/sec Loss 3.4023 Epoch: 12 Global Step: 124765 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:41:47,411-Speed 130.50 samples/sec Loss 3.3101 Epoch: 12 Global Step: 124770 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:42:07,080-Speed 130.16 samples/sec Loss 3.4798 Epoch: 12 Global Step: 124775 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:42:26,710-Speed 130.42 samples/sec Loss 3.3826 Epoch: 12 Global Step: 124780 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:42:46,553-Speed 129.02 samples/sec Loss 3.2029 Epoch: 12 Global Step: 124785 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:43:06,123-Speed 130.81 samples/sec Loss 3.3006 Epoch: 12 Global Step: 124790 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:43:26,023-Speed 128.65 samples/sec Loss 3.1999 Epoch: 12 Global Step: 124795 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:43:45,644-Speed 130.48 samples/sec Loss 3.3856 Epoch: 12 Global Step: 124800 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:44:05,695-Speed 127.69 samples/sec Loss 3.2342 Epoch: 12 Global Step: 124805 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:44:25,431-Speed 129.72 samples/sec Loss 3.2934 Epoch: 12 Global Step: 124810 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:44:45,040-Speed 130.55 samples/sec Loss 3.2399 Epoch: 12 Global Step: 124815 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:45:04,846-Speed 129.26 samples/sec Loss 3.3991 Epoch: 12 Global Step: 124820 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:45:24,617-Speed 129.49 samples/sec Loss 3.3304 Epoch: 12 Global Step: 124825 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:45:44,415-Speed 129.31 samples/sec Loss 3.2541 Epoch: 12 Global Step: 124830 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:46:04,398-Speed 128.12 samples/sec Loss 3.1766 Epoch: 12 Global Step: 124835 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:46:24,234-Speed 129.06 samples/sec Loss 3.1292 Epoch: 12 Global Step: 124840 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:46:43,767-Speed 131.07 samples/sec Loss 3.1131 Epoch: 12 Global Step: 124845 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:47:03,851-Speed 127.47 samples/sec Loss 3.4009 Epoch: 12 Global Step: 124850 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:47:23,834-Speed 128.12 samples/sec Loss 3.1825 Epoch: 12 Global Step: 124855 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:47:43,441-Speed 130.57 samples/sec Loss 3.4040 Epoch: 12 Global Step: 124860 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:48:03,105-Speed 130.20 samples/sec Loss 3.2368 Epoch: 12 Global Step: 124865 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:48:22,691-Speed 130.71 samples/sec Loss 3.3760 Epoch: 12 Global Step: 124870 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:48:42,408-Speed 129.84 samples/sec Loss 3.1616 Epoch: 12 Global Step: 124875 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:49:01,967-Speed 130.89 samples/sec Loss 3.2452 Epoch: 12 Global Step: 124880 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:49:21,820-Speed 128.95 samples/sec Loss 3.2554 Epoch: 12 Global Step: 124885 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:49:41,320-Speed 131.29 samples/sec Loss 3.3434 Epoch: 12 Global Step: 124890 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:50:01,247-Speed 128.48 samples/sec Loss 3.1797 Epoch: 12 Global Step: 124895 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:50:21,328-Speed 127.49 samples/sec Loss 3.1516 Epoch: 12 Global Step: 124900 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:50:40,927-Speed 130.62 samples/sec Loss 3.2350 Epoch: 12 Global Step: 124905 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:51:00,730-Speed 129.28 samples/sec Loss 3.2875 Epoch: 12 Global Step: 124910 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:51:20,275-Speed 130.99 samples/sec Loss 3.1214 Epoch: 12 Global Step: 124915 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:51:40,020-Speed 129.66 samples/sec Loss 3.2899 Epoch: 12 Global Step: 124920 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:51:59,639-Speed 130.49 samples/sec Loss 3.0623 Epoch: 12 Global Step: 124925 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:52:19,445-Speed 129.26 samples/sec Loss 3.2784 Epoch: 12 Global Step: 124930 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:52:39,001-Speed 130.91 samples/sec Loss 3.1235 Epoch: 12 Global Step: 124935 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:52:59,108-Speed 127.32 samples/sec Loss 3.2152 Epoch: 12 Global Step: 124940 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:53:19,072-Speed 128.24 samples/sec Loss 3.3975 Epoch: 12 Global Step: 124945 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:53:38,713-Speed 130.34 samples/sec Loss 3.2182 Epoch: 12 Global Step: 124950 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:53:58,554-Speed 129.03 samples/sec Loss 2.9743 Epoch: 12 Global Step: 124955 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:54:18,282-Speed 129.77 samples/sec Loss 3.1550 Epoch: 12 Global Step: 124960 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:54:38,035-Speed 129.61 samples/sec Loss 3.2535 Epoch: 12 Global Step: 124965 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:54:57,606-Speed 130.81 samples/sec Loss 3.3242 Epoch: 12 Global Step: 124970 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:55:17,353-Speed 129.65 samples/sec Loss 3.2684 Epoch: 12 Global Step: 124975 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:55:36,967-Speed 130.53 samples/sec Loss 3.2528 Epoch: 12 Global Step: 124980 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:55:56,892-Speed 128.49 samples/sec Loss 3.3663 Epoch: 12 Global Step: 124985 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:56:16,788-Speed 128.68 samples/sec Loss 3.2360 Epoch: 12 Global Step: 124990 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:56:36,328-Speed 131.02 samples/sec Loss 3.3156 Epoch: 12 Global Step: 124995 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:56:56,273-Speed 128.36 samples/sec Loss 3.3584 Epoch: 12 Global Step: 125000 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:57:15,741-Speed 131.51 samples/sec Loss 3.3744 Epoch: 12 Global Step: 125005 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:57:35,583-Speed 129.03 samples/sec Loss 3.4266 Epoch: 12 Global Step: 125010 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:57:55,193-Speed 130.55 samples/sec Loss 3.5080 Epoch: 12 Global Step: 125015 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:58:14,924-Speed 129.76 samples/sec Loss 3.4028 Epoch: 12 Global Step: 125020 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:58:34,552-Speed 130.43 samples/sec Loss 3.1751 Epoch: 12 Global Step: 125025 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:58:54,600-Speed 127.70 samples/sec Loss 3.2171 Epoch: 12 Global Step: 125030 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:59:14,396-Speed 129.33 samples/sec Loss 3.2219 Epoch: 12 Global Step: 125035 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:59:33,931-Speed 131.05 samples/sec Loss 3.2705 Epoch: 12 Global Step: 125040 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 22:59:53,840-Speed 128.59 samples/sec Loss 3.3142 Epoch: 12 Global Step: 125045 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:00:13,401-Speed 130.88 samples/sec Loss 3.2827 Epoch: 12 Global Step: 125050 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:00:33,192-Speed 129.36 samples/sec Loss 3.1678 Epoch: 12 Global Step: 125055 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:00:52,807-Speed 130.51 samples/sec Loss 3.4178 Epoch: 12 Global Step: 125060 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:01:12,588-Speed 129.43 samples/sec Loss 3.2065 Epoch: 12 Global Step: 125065 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:01:32,295-Speed 129.91 samples/sec Loss 3.4415 Epoch: 12 Global Step: 125070 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:01:52,403-Speed 127.32 samples/sec Loss 3.3842 Epoch: 12 Global Step: 125075 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:02:12,120-Speed 129.86 samples/sec Loss 3.2743 Epoch: 12 Global Step: 125080 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:02:31,674-Speed 130.93 samples/sec Loss 3.1000 Epoch: 12 Global Step: 125085 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:02:51,495-Speed 129.16 samples/sec Loss 3.2089 Epoch: 12 Global Step: 125090 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:03:10,952-Speed 131.58 samples/sec Loss 3.2644 Epoch: 12 Global Step: 125095 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:03:30,923-Speed 128.19 samples/sec Loss 3.3375 Epoch: 12 Global Step: 125100 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:03:50,470-Speed 130.98 samples/sec Loss 3.2340 Epoch: 12 Global Step: 125105 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:04:10,368-Speed 128.66 samples/sec Loss 3.1687 Epoch: 12 Global Step: 125110 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:04:29,927-Speed 130.89 samples/sec Loss 3.4341 Epoch: 12 Global Step: 125115 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:04:50,064-Speed 127.14 samples/sec Loss 3.2074 Epoch: 12 Global Step: 125120 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:05:09,730-Speed 130.18 samples/sec Loss 3.3746 Epoch: 12 Global Step: 125125 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:05:29,344-Speed 130.53 samples/sec Loss 3.3255 Epoch: 12 Global Step: 125130 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:05:49,125-Speed 129.42 samples/sec Loss 3.5717 Epoch: 12 Global Step: 125135 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:06:08,763-Speed 130.37 samples/sec Loss 3.3180 Epoch: 12 Global Step: 125140 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:06:28,552-Speed 129.37 samples/sec Loss 3.2652 Epoch: 12 Global Step: 125145 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:06:48,117-Speed 130.85 samples/sec Loss 3.3765 Epoch: 12 Global Step: 125150 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:07:07,872-Speed 129.60 samples/sec Loss 3.3173 Epoch: 12 Global Step: 125155 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:07:27,423-Speed 130.95 samples/sec Loss 3.0916 Epoch: 12 Global Step: 125160 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:07:47,504-Speed 127.49 samples/sec Loss 3.1784 Epoch: 12 Global Step: 125165 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:08:07,438-Speed 128.43 samples/sec Loss 3.3207 Epoch: 12 Global Step: 125170 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:08:27,001-Speed 130.87 samples/sec Loss 3.1961 Epoch: 12 Global Step: 125175 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:08:46,637-Speed 130.38 samples/sec Loss 3.0297 Epoch: 12 Global Step: 125180 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:09:06,257-Speed 130.49 samples/sec Loss 3.3853 Epoch: 12 Global Step: 125185 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:09:25,953-Speed 129.98 samples/sec Loss 3.3299 Epoch: 12 Global Step: 125190 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:09:45,500-Speed 130.97 samples/sec Loss 3.4669 Epoch: 12 Global Step: 125195 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:10:05,360-Speed 128.91 samples/sec Loss 3.4320 Epoch: 12 Global Step: 125200 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:10:24,925-Speed 130.86 samples/sec Loss 3.3318 Epoch: 12 Global Step: 125205 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:10:45,079-Speed 127.03 samples/sec Loss 3.2945 Epoch: 12 Global Step: 125210 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:11:04,922-Speed 129.02 samples/sec Loss 3.1141 Epoch: 12 Global Step: 125215 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:11:24,486-Speed 130.86 samples/sec Loss 3.2562 Epoch: 12 Global Step: 125220 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:11:44,218-Speed 129.75 samples/sec Loss 3.1590 Epoch: 12 Global Step: 125225 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:12:03,852-Speed 130.39 samples/sec Loss 3.3892 Epoch: 12 Global Step: 125230 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:12:23,613-Speed 129.56 samples/sec Loss 3.3451 Epoch: 12 Global Step: 125235 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:12:43,145-Speed 131.08 samples/sec Loss 3.1932 Epoch: 12 Global Step: 125240 Fp16 Grad Scale: 8192 Required: 147 hours Training: 2021-06-12 23:13:02,994-Speed 128.98 samples/sec Loss 3.2208 Epoch: 12 Global Step: 125245 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:13:22,494-Speed 131.29 samples/sec Loss 3.2500 Epoch: 12 Global Step: 125250 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:13:42,603-Speed 127.31 samples/sec Loss 3.3179 Epoch: 12 Global Step: 125255 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:14:02,529-Speed 128.48 samples/sec Loss 3.1934 Epoch: 12 Global Step: 125260 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:14:22,089-Speed 130.88 samples/sec Loss 3.3730 Epoch: 12 Global Step: 125265 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:14:41,958-Speed 128.85 samples/sec Loss 3.6122 Epoch: 12 Global Step: 125270 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:15:01,502-Speed 130.99 samples/sec Loss 3.4827 Epoch: 12 Global Step: 125275 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:15:21,326-Speed 129.14 samples/sec Loss 3.5742 Epoch: 12 Global Step: 125280 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:15:40,927-Speed 130.62 samples/sec Loss 3.3404 Epoch: 12 Global Step: 125285 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:16:00,595-Speed 130.16 samples/sec Loss 3.1698 Epoch: 12 Global Step: 125290 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:16:20,211-Speed 130.52 samples/sec Loss 3.1462 Epoch: 12 Global Step: 125295 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:16:40,282-Speed 127.56 samples/sec Loss 3.2692 Epoch: 12 Global Step: 125300 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:17:00,192-Speed 128.59 samples/sec Loss 3.3778 Epoch: 12 Global Step: 125305 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:17:19,653-Speed 131.54 samples/sec Loss 3.3347 Epoch: 12 Global Step: 125310 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:17:39,660-Speed 127.96 samples/sec Loss 3.1566 Epoch: 12 Global Step: 125315 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:17:59,198-Speed 131.04 samples/sec Loss 3.3639 Epoch: 12 Global Step: 125320 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:18:18,859-Speed 130.22 samples/sec Loss 3.3413 Epoch: 12 Global Step: 125325 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:18:38,457-Speed 130.63 samples/sec Loss 3.1524 Epoch: 12 Global Step: 125330 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:18:58,126-Speed 130.16 samples/sec Loss 3.4183 Epoch: 12 Global Step: 125335 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:19:17,740-Speed 130.53 samples/sec Loss 3.2583 Epoch: 12 Global Step: 125340 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:19:37,970-Speed 126.55 samples/sec Loss 3.2523 Epoch: 12 Global Step: 125345 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:19:57,704-Speed 129.73 samples/sec Loss 3.4195 Epoch: 12 Global Step: 125350 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:20:17,300-Speed 130.64 samples/sec Loss 3.1649 Epoch: 12 Global Step: 125355 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:20:37,029-Speed 129.76 samples/sec Loss 3.3046 Epoch: 12 Global Step: 125360 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:20:56,517-Speed 131.37 samples/sec Loss 3.3732 Epoch: 12 Global Step: 125365 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:21:16,678-Speed 126.98 samples/sec Loss 3.2067 Epoch: 12 Global Step: 125370 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:21:36,234-Speed 130.91 samples/sec Loss 3.3109 Epoch: 12 Global Step: 125375 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:21:56,191-Speed 128.28 samples/sec Loss 3.2212 Epoch: 12 Global Step: 125380 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:22:15,755-Speed 130.86 samples/sec Loss 3.1661 Epoch: 12 Global Step: 125385 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:22:35,735-Speed 128.13 samples/sec Loss 3.2868 Epoch: 12 Global Step: 125390 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:22:55,505-Speed 129.50 samples/sec Loss 3.0951 Epoch: 12 Global Step: 125395 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:23:15,118-Speed 130.53 samples/sec Loss 3.2113 Epoch: 12 Global Step: 125400 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:23:34,785-Speed 130.17 samples/sec Loss 3.3394 Epoch: 12 Global Step: 125405 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:23:54,330-Speed 130.99 samples/sec Loss 3.4305 Epoch: 12 Global Step: 125410 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:24:14,159-Speed 129.11 samples/sec Loss 3.1628 Epoch: 12 Global Step: 125415 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:24:33,699-Speed 131.03 samples/sec Loss 3.2152 Epoch: 12 Global Step: 125420 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:24:53,583-Speed 128.75 samples/sec Loss 3.1808 Epoch: 12 Global Step: 125425 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:25:13,048-Speed 131.52 samples/sec Loss 3.4526 Epoch: 12 Global Step: 125430 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:25:33,026-Speed 128.14 samples/sec Loss 3.2975 Epoch: 12 Global Step: 125435 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:25:52,906-Speed 128.78 samples/sec Loss 3.2026 Epoch: 12 Global Step: 125440 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:26:12,486-Speed 130.75 samples/sec Loss 3.3262 Epoch: 12 Global Step: 125445 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:26:32,106-Speed 130.49 samples/sec Loss 3.3064 Epoch: 12 Global Step: 125450 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:26:51,645-Speed 131.02 samples/sec Loss 3.3662 Epoch: 12 Global Step: 125455 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:27:11,450-Speed 129.27 samples/sec Loss 3.6691 Epoch: 12 Global Step: 125460 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:27:31,001-Speed 130.94 samples/sec Loss 3.2565 Epoch: 12 Global Step: 125465 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:27:50,945-Speed 128.37 samples/sec Loss 3.3400 Epoch: 12 Global Step: 125470 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:28:10,646-Speed 129.95 samples/sec Loss 3.3274 Epoch: 12 Global Step: 125475 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:28:30,728-Speed 127.48 samples/sec Loss 3.2857 Epoch: 12 Global Step: 125480 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:28:50,658-Speed 128.46 samples/sec Loss 3.3380 Epoch: 12 Global Step: 125485 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:29:10,231-Speed 130.80 samples/sec Loss 3.1559 Epoch: 12 Global Step: 125490 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:29:29,850-Speed 130.49 samples/sec Loss 3.2134 Epoch: 12 Global Step: 125495 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:29:49,488-Speed 130.37 samples/sec Loss 3.4647 Epoch: 12 Global Step: 125500 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:30:09,242-Speed 129.60 samples/sec Loss 3.4730 Epoch: 12 Global Step: 125505 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:30:28,907-Speed 130.19 samples/sec Loss 3.3174 Epoch: 12 Global Step: 125510 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:30:48,684-Speed 129.45 samples/sec Loss 3.3109 Epoch: 12 Global Step: 125515 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:31:08,160-Speed 131.45 samples/sec Loss 3.3689 Epoch: 12 Global Step: 125520 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:31:28,269-Speed 127.31 samples/sec Loss 3.5872 Epoch: 12 Global Step: 125525 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:31:48,463-Speed 126.78 samples/sec Loss 3.2030 Epoch: 12 Global Step: 125530 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:32:07,931-Speed 131.50 samples/sec Loss 3.3559 Epoch: 12 Global Step: 125535 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:32:27,647-Speed 129.85 samples/sec Loss 3.3340 Epoch: 12 Global Step: 125540 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:32:47,284-Speed 130.37 samples/sec Loss 3.2837 Epoch: 12 Global Step: 125545 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:33:07,087-Speed 129.28 samples/sec Loss 3.5509 Epoch: 12 Global Step: 125550 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:33:26,677-Speed 130.69 samples/sec Loss 3.1351 Epoch: 12 Global Step: 125555 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:33:46,340-Speed 130.20 samples/sec Loss 3.1602 Epoch: 12 Global Step: 125560 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:34:05,925-Speed 130.72 samples/sec Loss 3.2456 Epoch: 12 Global Step: 125565 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:34:25,965-Speed 127.75 samples/sec Loss 3.4305 Epoch: 12 Global Step: 125570 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:34:45,978-Speed 127.92 samples/sec Loss 3.4202 Epoch: 12 Global Step: 125575 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:35:05,432-Speed 131.60 samples/sec Loss 3.4575 Epoch: 12 Global Step: 125580 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:35:25,455-Speed 127.86 samples/sec Loss 3.3952 Epoch: 12 Global Step: 125585 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:35:45,026-Speed 130.81 samples/sec Loss 3.3861 Epoch: 12 Global Step: 125590 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:36:04,873-Speed 129.00 samples/sec Loss 3.4003 Epoch: 12 Global Step: 125595 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:36:24,400-Speed 131.10 samples/sec Loss 3.1164 Epoch: 12 Global Step: 125600 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:36:44,133-Speed 129.74 samples/sec Loss 3.4347 Epoch: 12 Global Step: 125605 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:37:03,756-Speed 130.47 samples/sec Loss 3.3491 Epoch: 12 Global Step: 125610 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:37:24,044-Speed 126.18 samples/sec Loss 3.5580 Epoch: 12 Global Step: 125615 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:37:43,652-Speed 130.57 samples/sec Loss 3.4988 Epoch: 12 Global Step: 125620 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:38:03,292-Speed 130.35 samples/sec Loss 3.3575 Epoch: 12 Global Step: 125625 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:38:23,192-Speed 128.65 samples/sec Loss 3.4955 Epoch: 12 Global Step: 125630 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:38:42,917-Speed 129.79 samples/sec Loss 3.1904 Epoch: 12 Global Step: 125635 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:39:02,813-Speed 128.67 samples/sec Loss 3.0807 Epoch: 12 Global Step: 125640 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:39:22,344-Speed 131.09 samples/sec Loss 3.5447 Epoch: 12 Global Step: 125645 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:39:42,050-Speed 129.92 samples/sec Loss 3.3380 Epoch: 12 Global Step: 125650 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:40:01,633-Speed 130.74 samples/sec Loss 3.3531 Epoch: 12 Global Step: 125655 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:40:21,773-Speed 127.11 samples/sec Loss 3.3133 Epoch: 12 Global Step: 125660 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:40:41,468-Speed 129.99 samples/sec Loss 3.1899 Epoch: 12 Global Step: 125665 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:41:01,075-Speed 130.57 samples/sec Loss 3.2960 Epoch: 12 Global Step: 125670 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:41:20,701-Speed 130.44 samples/sec Loss 3.3772 Epoch: 12 Global Step: 125675 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:41:40,329-Speed 130.43 samples/sec Loss 3.4284 Epoch: 12 Global Step: 125680 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:42:00,094-Speed 129.53 samples/sec Loss 3.3329 Epoch: 12 Global Step: 125685 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:42:19,636-Speed 131.01 samples/sec Loss 3.4256 Epoch: 12 Global Step: 125690 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:42:39,563-Speed 128.47 samples/sec Loss 3.3694 Epoch: 12 Global Step: 125695 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:42:59,115-Speed 130.94 samples/sec Loss 3.3733 Epoch: 12 Global Step: 125700 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:43:19,247-Speed 127.16 samples/sec Loss 3.2203 Epoch: 12 Global Step: 125705 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:43:38,939-Speed 130.01 samples/sec Loss 3.4000 Epoch: 12 Global Step: 125710 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:43:58,555-Speed 130.51 samples/sec Loss 3.2589 Epoch: 12 Global Step: 125715 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:44:18,517-Speed 128.25 samples/sec Loss 3.4545 Epoch: 12 Global Step: 125720 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:44:38,129-Speed 130.54 samples/sec Loss 3.3089 Epoch: 12 Global Step: 125725 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:44:57,844-Speed 129.85 samples/sec Loss 3.2860 Epoch: 12 Global Step: 125730 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:45:17,385-Speed 131.02 samples/sec Loss 3.3957 Epoch: 12 Global Step: 125735 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:45:37,225-Speed 129.04 samples/sec Loss 3.2594 Epoch: 12 Global Step: 125740 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:45:56,868-Speed 130.34 samples/sec Loss 3.3156 Epoch: 12 Global Step: 125745 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:46:16,880-Speed 127.93 samples/sec Loss 3.3922 Epoch: 12 Global Step: 125750 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:46:36,693-Speed 129.21 samples/sec Loss 3.1999 Epoch: 12 Global Step: 125755 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:46:56,264-Speed 130.81 samples/sec Loss 3.2748 Epoch: 12 Global Step: 125760 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:47:16,043-Speed 129.44 samples/sec Loss 3.2501 Epoch: 12 Global Step: 125765 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:47:35,749-Speed 129.91 samples/sec Loss 3.2789 Epoch: 12 Global Step: 125770 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:47:55,430-Speed 130.08 samples/sec Loss 3.5818 Epoch: 12 Global Step: 125775 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:48:14,978-Speed 130.97 samples/sec Loss 3.4954 Epoch: 12 Global Step: 125780 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:48:34,853-Speed 128.81 samples/sec Loss 3.2427 Epoch: 12 Global Step: 125785 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:48:54,404-Speed 130.95 samples/sec Loss 3.2793 Epoch: 12 Global Step: 125790 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:49:14,472-Speed 127.57 samples/sec Loss 3.2304 Epoch: 12 Global Step: 125795 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:49:34,378-Speed 128.61 samples/sec Loss 3.3953 Epoch: 12 Global Step: 125800 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:49:53,989-Speed 130.55 samples/sec Loss 3.4116 Epoch: 12 Global Step: 125805 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:50:13,693-Speed 129.93 samples/sec Loss 3.2225 Epoch: 12 Global Step: 125810 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:50:33,306-Speed 130.53 samples/sec Loss 3.3443 Epoch: 12 Global Step: 125815 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:50:52,906-Speed 130.61 samples/sec Loss 3.3571 Epoch: 12 Global Step: 125820 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:51:12,523-Speed 130.51 samples/sec Loss 3.4167 Epoch: 12 Global Step: 125825 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:51:32,411-Speed 128.73 samples/sec Loss 3.4739 Epoch: 12 Global Step: 125830 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:51:51,878-Speed 131.51 samples/sec Loss 3.3281 Epoch: 12 Global Step: 125835 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:52:11,878-Speed 128.01 samples/sec Loss 3.3515 Epoch: 12 Global Step: 125840 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:52:31,836-Speed 128.28 samples/sec Loss 3.4233 Epoch: 12 Global Step: 125845 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:52:51,491-Speed 130.25 samples/sec Loss 3.2102 Epoch: 12 Global Step: 125850 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:53:11,385-Speed 128.69 samples/sec Loss 3.1989 Epoch: 12 Global Step: 125855 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:53:30,834-Speed 131.63 samples/sec Loss 3.2537 Epoch: 12 Global Step: 125860 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:53:50,502-Speed 130.17 samples/sec Loss 3.4153 Epoch: 12 Global Step: 125865 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:54:10,150-Speed 130.30 samples/sec Loss 3.3143 Epoch: 12 Global Step: 125870 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:54:29,949-Speed 129.31 samples/sec Loss 3.5414 Epoch: 12 Global Step: 125875 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:54:49,563-Speed 130.53 samples/sec Loss 3.2218 Epoch: 12 Global Step: 125880 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:55:09,553-Speed 128.07 samples/sec Loss 3.0276 Epoch: 12 Global Step: 125885 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:55:29,289-Speed 129.72 samples/sec Loss 3.4670 Epoch: 12 Global Step: 125890 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:55:48,735-Speed 131.65 samples/sec Loss 3.2915 Epoch: 12 Global Step: 125895 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:56:08,605-Speed 128.84 samples/sec Loss 3.3892 Epoch: 12 Global Step: 125900 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:56:28,036-Speed 131.75 samples/sec Loss 3.1119 Epoch: 12 Global Step: 125905 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:56:47,938-Speed 128.64 samples/sec Loss 3.3274 Epoch: 12 Global Step: 125910 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:57:07,519-Speed 130.74 samples/sec Loss 3.3030 Epoch: 12 Global Step: 125915 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:57:27,194-Speed 130.12 samples/sec Loss 3.2768 Epoch: 12 Global Step: 125920 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:57:46,873-Speed 130.09 samples/sec Loss 3.3669 Epoch: 12 Global Step: 125925 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:58:06,746-Speed 128.82 samples/sec Loss 3.3262 Epoch: 12 Global Step: 125930 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:58:26,476-Speed 129.76 samples/sec Loss 3.2898 Epoch: 12 Global Step: 125935 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:58:46,014-Speed 131.03 samples/sec Loss 3.2707 Epoch: 12 Global Step: 125940 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:59:05,794-Speed 129.43 samples/sec Loss 3.2210 Epoch: 12 Global Step: 125945 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:59:25,250-Speed 131.59 samples/sec Loss 3.4027 Epoch: 12 Global Step: 125950 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-12 23:59:45,195-Speed 128.36 samples/sec Loss 3.2353 Epoch: 12 Global Step: 125955 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:00:04,718-Speed 131.13 samples/sec Loss 3.2389 Epoch: 12 Global Step: 125960 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:00:24,567-Speed 128.98 samples/sec Loss 3.1805 Epoch: 12 Global Step: 125965 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:00:44,184-Speed 130.50 samples/sec Loss 3.3721 Epoch: 12 Global Step: 125970 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:01:04,290-Speed 127.33 samples/sec Loss 3.0936 Epoch: 12 Global Step: 125975 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:01:24,021-Speed 129.75 samples/sec Loss 3.3432 Epoch: 12 Global Step: 125980 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:01:43,496-Speed 131.45 samples/sec Loss 3.1972 Epoch: 12 Global Step: 125985 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:02:03,261-Speed 129.53 samples/sec Loss 3.4638 Epoch: 12 Global Step: 125990 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:02:22,828-Speed 130.84 samples/sec Loss 3.1782 Epoch: 12 Global Step: 125995 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:02:42,754-Speed 128.48 samples/sec Loss 3.4138 Epoch: 12 Global Step: 126000 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:03:02,297-Speed 131.00 samples/sec Loss 3.1874 Epoch: 12 Global Step: 126005 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:03:22,093-Speed 129.33 samples/sec Loss 3.3032 Epoch: 12 Global Step: 126010 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:03:41,550-Speed 131.58 samples/sec Loss 3.1694 Epoch: 12 Global Step: 126015 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:04:01,670-Speed 127.24 samples/sec Loss 3.2507 Epoch: 12 Global Step: 126020 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:04:21,292-Speed 130.48 samples/sec Loss 3.2086 Epoch: 12 Global Step: 126025 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:04:40,900-Speed 130.56 samples/sec Loss 3.2050 Epoch: 12 Global Step: 126030 Fp16 Grad Scale: 8192 Required: 146 hours Training: 2021-06-13 00:05:00,613-Speed 129.88 samples/sec Loss 3.2415 Epoch: 12 Global Step: 126035 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:05:20,263-Speed 130.29 samples/sec Loss 3.4083 Epoch: 12 Global Step: 126040 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:05:40,011-Speed 129.64 samples/sec Loss 3.1698 Epoch: 12 Global Step: 126045 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:05:59,449-Speed 131.71 samples/sec Loss 3.3815 Epoch: 12 Global Step: 126050 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:06:19,243-Speed 129.34 samples/sec Loss 3.2379 Epoch: 12 Global Step: 126055 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:06:38,816-Speed 130.80 samples/sec Loss 3.4077 Epoch: 12 Global Step: 126060 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:06:58,979-Speed 126.98 samples/sec Loss 3.4859 Epoch: 12 Global Step: 126065 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:07:18,744-Speed 129.53 samples/sec Loss 3.3228 Epoch: 12 Global Step: 126070 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:07:38,316-Speed 130.81 samples/sec Loss 3.3135 Epoch: 12 Global Step: 126075 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:07:58,041-Speed 129.79 samples/sec Loss 3.3491 Epoch: 12 Global Step: 126080 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:08:17,746-Speed 129.92 samples/sec Loss 3.3009 Epoch: 12 Global Step: 126085 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:08:37,343-Speed 130.64 samples/sec Loss 3.3534 Epoch: 12 Global Step: 126090 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:08:56,937-Speed 130.66 samples/sec Loss 3.5404 Epoch: 12 Global Step: 126095 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:09:16,688-Speed 129.62 samples/sec Loss 3.3246 Epoch: 12 Global Step: 126100 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:09:36,262-Speed 130.79 samples/sec Loss 3.2987 Epoch: 12 Global Step: 126105 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:09:56,327-Speed 127.60 samples/sec Loss 3.5225 Epoch: 12 Global Step: 126110 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:10:16,109-Speed 129.42 samples/sec Loss 3.2920 Epoch: 12 Global Step: 126115 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:10:35,718-Speed 130.56 samples/sec Loss 3.2927 Epoch: 12 Global Step: 126120 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:10:55,451-Speed 129.74 samples/sec Loss 3.2177 Epoch: 12 Global Step: 126125 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:11:15,076-Speed 130.46 samples/sec Loss 3.3665 Epoch: 12 Global Step: 126130 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:11:34,679-Speed 130.59 samples/sec Loss 3.1646 Epoch: 12 Global Step: 126135 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:11:54,155-Speed 131.45 samples/sec Loss 3.4894 Epoch: 12 Global Step: 126140 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:12:14,013-Speed 128.93 samples/sec Loss 3.4749 Epoch: 12 Global Step: 126145 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:12:33,637-Speed 130.46 samples/sec Loss 3.3200 Epoch: 12 Global Step: 126150 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:12:53,604-Speed 128.21 samples/sec Loss 3.5422 Epoch: 12 Global Step: 126155 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:13:13,318-Speed 129.86 samples/sec Loss 3.4321 Epoch: 12 Global Step: 126160 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:13:32,887-Speed 130.82 samples/sec Loss 3.3441 Epoch: 12 Global Step: 126165 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:13:52,678-Speed 129.36 samples/sec Loss 3.3063 Epoch: 12 Global Step: 126170 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:14:12,487-Speed 129.24 samples/sec Loss 3.2383 Epoch: 12 Global Step: 126175 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:14:32,101-Speed 130.53 samples/sec Loss 3.5213 Epoch: 12 Global Step: 126180 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:14:51,714-Speed 130.54 samples/sec Loss 3.3235 Epoch: 12 Global Step: 126185 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:15:11,459-Speed 129.66 samples/sec Loss 3.3999 Epoch: 12 Global Step: 126190 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:15:31,088-Speed 130.43 samples/sec Loss 3.2938 Epoch: 12 Global Step: 126195 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:15:51,032-Speed 128.37 samples/sec Loss 3.2156 Epoch: 12 Global Step: 126200 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:16:10,826-Speed 129.34 samples/sec Loss 3.3365 Epoch: 12 Global Step: 126205 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:16:30,377-Speed 130.94 samples/sec Loss 3.1756 Epoch: 12 Global Step: 126210 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:16:50,361-Speed 128.11 samples/sec Loss 3.2566 Epoch: 12 Global Step: 126215 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:17:09,820-Speed 131.57 samples/sec Loss 3.3370 Epoch: 12 Global Step: 126220 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:17:29,481-Speed 130.22 samples/sec Loss 3.3309 Epoch: 12 Global Step: 126225 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:17:49,058-Speed 130.77 samples/sec Loss 3.3810 Epoch: 12 Global Step: 126230 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:18:08,800-Speed 129.68 samples/sec Loss 3.3040 Epoch: 12 Global Step: 126235 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:18:28,518-Speed 129.84 samples/sec Loss 3.4229 Epoch: 12 Global Step: 126240 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:18:48,347-Speed 129.11 samples/sec Loss 3.3169 Epoch: 12 Global Step: 126245 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:19:08,334-Speed 128.09 samples/sec Loss 3.2839 Epoch: 12 Global Step: 126250 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:19:27,954-Speed 130.49 samples/sec Loss 3.5601 Epoch: 12 Global Step: 126255 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:19:47,759-Speed 129.27 samples/sec Loss 3.2616 Epoch: 12 Global Step: 126260 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:20:07,197-Speed 131.71 samples/sec Loss 3.1799 Epoch: 12 Global Step: 126265 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:20:26,845-Speed 130.30 samples/sec Loss 3.3045 Epoch: 12 Global Step: 126270 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:20:46,420-Speed 130.78 samples/sec Loss 3.3152 Epoch: 12 Global Step: 126275 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:21:06,221-Speed 129.30 samples/sec Loss 3.4031 Epoch: 12 Global Step: 126280 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:21:25,852-Speed 130.41 samples/sec Loss 3.4022 Epoch: 12 Global Step: 126285 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:21:45,676-Speed 129.14 samples/sec Loss 3.2774 Epoch: 12 Global Step: 126290 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:22:05,631-Speed 128.30 samples/sec Loss 3.1375 Epoch: 12 Global Step: 126295 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:22:25,262-Speed 130.41 samples/sec Loss 3.2838 Epoch: 12 Global Step: 126300 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:22:45,067-Speed 129.27 samples/sec Loss 3.3779 Epoch: 12 Global Step: 126305 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:23:04,517-Speed 131.63 samples/sec Loss 3.2640 Epoch: 12 Global Step: 126310 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:23:24,319-Speed 129.28 samples/sec Loss 3.3007 Epoch: 12 Global Step: 126315 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:23:43,880-Speed 130.88 samples/sec Loss 3.3772 Epoch: 12 Global Step: 126320 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:24:03,708-Speed 129.12 samples/sec Loss 3.2498 Epoch: 12 Global Step: 126325 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:24:23,315-Speed 130.58 samples/sec Loss 3.4120 Epoch: 12 Global Step: 126330 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:24:43,281-Speed 128.22 samples/sec Loss 3.3720 Epoch: 12 Global Step: 126335 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:25:02,928-Speed 130.31 samples/sec Loss 3.3467 Epoch: 12 Global Step: 126340 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:25:22,553-Speed 130.45 samples/sec Loss 3.5079 Epoch: 12 Global Step: 126345 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:25:42,165-Speed 130.54 samples/sec Loss 3.4243 Epoch: 12 Global Step: 126350 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:26:02,131-Speed 128.22 samples/sec Loss 3.3000 Epoch: 12 Global Step: 126355 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:26:21,989-Speed 128.92 samples/sec Loss 3.2800 Epoch: 12 Global Step: 126360 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:26:41,681-Speed 130.02 samples/sec Loss 3.1737 Epoch: 12 Global Step: 126365 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:27:01,469-Speed 129.38 samples/sec Loss 3.3384 Epoch: 12 Global Step: 126370 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:27:21,043-Speed 130.79 samples/sec Loss 3.3495 Epoch: 12 Global Step: 126375 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:27:41,234-Speed 126.79 samples/sec Loss 3.3560 Epoch: 12 Global Step: 126380 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:28:01,017-Speed 129.41 samples/sec Loss 3.3922 Epoch: 12 Global Step: 126385 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:28:20,630-Speed 130.53 samples/sec Loss 3.2969 Epoch: 12 Global Step: 126390 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:28:40,244-Speed 130.53 samples/sec Loss 3.3393 Epoch: 12 Global Step: 126395 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:28:59,802-Speed 130.90 samples/sec Loss 3.0453 Epoch: 12 Global Step: 126400 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:29:19,601-Speed 129.31 samples/sec Loss 3.1455 Epoch: 12 Global Step: 126405 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:29:39,210-Speed 130.56 samples/sec Loss 3.2717 Epoch: 12 Global Step: 126410 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:29:59,000-Speed 129.36 samples/sec Loss 3.4024 Epoch: 12 Global Step: 126415 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:30:18,554-Speed 130.93 samples/sec Loss 3.2625 Epoch: 12 Global Step: 126420 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:30:38,640-Speed 127.46 samples/sec Loss 3.2230 Epoch: 12 Global Step: 126425 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:30:58,467-Speed 129.13 samples/sec Loss 3.2463 Epoch: 12 Global Step: 126430 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:31:18,099-Speed 130.41 samples/sec Loss 3.2929 Epoch: 12 Global Step: 126435 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:31:37,830-Speed 129.75 samples/sec Loss 3.3318 Epoch: 12 Global Step: 126440 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:31:57,442-Speed 130.54 samples/sec Loss 3.1165 Epoch: 12 Global Step: 126445 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:32:17,071-Speed 130.42 samples/sec Loss 3.5387 Epoch: 12 Global Step: 126450 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:32:36,847-Speed 129.46 samples/sec Loss 3.2365 Epoch: 12 Global Step: 126455 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:32:56,710-Speed 128.89 samples/sec Loss 3.1993 Epoch: 12 Global Step: 126460 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:33:16,287-Speed 130.78 samples/sec Loss 3.3473 Epoch: 12 Global Step: 126465 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:33:36,444-Speed 127.01 samples/sec Loss 3.4223 Epoch: 12 Global Step: 126470 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:33:56,321-Speed 128.80 samples/sec Loss 3.5323 Epoch: 12 Global Step: 126475 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:34:15,862-Speed 131.01 samples/sec Loss 3.3329 Epoch: 12 Global Step: 126480 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:34:35,651-Speed 129.37 samples/sec Loss 3.4622 Epoch: 12 Global Step: 126485 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:34:55,267-Speed 130.51 samples/sec Loss 3.3783 Epoch: 12 Global Step: 126490 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:35:15,038-Speed 129.49 samples/sec Loss 3.3156 Epoch: 12 Global Step: 126495 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:35:34,569-Speed 131.08 samples/sec Loss 3.3876 Epoch: 12 Global Step: 126500 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:35:54,376-Speed 129.26 samples/sec Loss 3.3714 Epoch: 12 Global Step: 126505 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:36:13,958-Speed 130.73 samples/sec Loss 3.3332 Epoch: 12 Global Step: 126510 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:36:33,925-Speed 128.22 samples/sec Loss 3.4816 Epoch: 12 Global Step: 126515 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:36:53,982-Speed 127.65 samples/sec Loss 3.2490 Epoch: 12 Global Step: 126520 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:37:13,520-Speed 131.03 samples/sec Loss 3.3328 Epoch: 12 Global Step: 126525 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:37:33,280-Speed 129.56 samples/sec Loss 3.3152 Epoch: 12 Global Step: 126530 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:37:52,869-Speed 130.70 samples/sec Loss 3.2511 Epoch: 12 Global Step: 126535 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:38:12,499-Speed 130.42 samples/sec Loss 3.4079 Epoch: 12 Global Step: 126540 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:38:32,101-Speed 130.61 samples/sec Loss 3.3362 Epoch: 12 Global Step: 126545 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:38:51,837-Speed 129.72 samples/sec Loss 3.2817 Epoch: 12 Global Step: 126550 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:39:11,362-Speed 131.12 samples/sec Loss 3.3361 Epoch: 12 Global Step: 126555 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:39:31,368-Speed 127.97 samples/sec Loss 3.2540 Epoch: 12 Global Step: 126560 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:39:51,276-Speed 128.60 samples/sec Loss 3.1866 Epoch: 12 Global Step: 126565 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:40:11,066-Speed 129.36 samples/sec Loss 3.4302 Epoch: 12 Global Step: 126570 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:40:30,869-Speed 129.28 samples/sec Loss 3.3889 Epoch: 12 Global Step: 126575 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:40:50,621-Speed 129.61 samples/sec Loss 3.3833 Epoch: 12 Global Step: 126580 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:41:10,464-Speed 129.02 samples/sec Loss 3.2544 Epoch: 12 Global Step: 126585 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:41:30,002-Speed 131.03 samples/sec Loss 3.2860 Epoch: 12 Global Step: 126590 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:41:50,026-Speed 127.86 samples/sec Loss 3.3739 Epoch: 12 Global Step: 126595 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:42:09,641-Speed 130.52 samples/sec Loss 3.4277 Epoch: 12 Global Step: 126600 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:42:29,764-Speed 127.23 samples/sec Loss 3.2275 Epoch: 12 Global Step: 126605 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:42:49,821-Speed 127.64 samples/sec Loss 3.3449 Epoch: 12 Global Step: 126610 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:43:09,405-Speed 130.72 samples/sec Loss 3.3298 Epoch: 12 Global Step: 126615 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:43:29,188-Speed 129.41 samples/sec Loss 3.1951 Epoch: 12 Global Step: 126620 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:43:48,802-Speed 130.53 samples/sec Loss 3.3161 Epoch: 12 Global Step: 126625 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:44:08,711-Speed 128.59 samples/sec Loss 3.4279 Epoch: 12 Global Step: 126630 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:44:28,297-Speed 130.72 samples/sec Loss 3.4343 Epoch: 12 Global Step: 126635 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:44:48,202-Speed 128.62 samples/sec Loss 3.4079 Epoch: 12 Global Step: 126640 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:45:07,681-Speed 131.43 samples/sec Loss 3.6788 Epoch: 12 Global Step: 126645 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:45:27,843-Speed 126.98 samples/sec Loss 3.5278 Epoch: 12 Global Step: 126650 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:45:47,504-Speed 130.21 samples/sec Loss 3.1371 Epoch: 12 Global Step: 126655 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:46:07,111-Speed 130.57 samples/sec Loss 3.4180 Epoch: 12 Global Step: 126660 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:46:26,800-Speed 130.03 samples/sec Loss 3.1986 Epoch: 12 Global Step: 126665 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:46:46,346-Speed 130.98 samples/sec Loss 3.1812 Epoch: 12 Global Step: 126670 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:47:06,223-Speed 128.81 samples/sec Loss 3.6470 Epoch: 12 Global Step: 126675 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:47:25,768-Speed 130.99 samples/sec Loss 3.4412 Epoch: 12 Global Step: 126680 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:47:45,632-Speed 128.88 samples/sec Loss 3.4045 Epoch: 12 Global Step: 126685 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:48:05,252-Speed 130.49 samples/sec Loss 3.3532 Epoch: 12 Global Step: 126690 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:48:25,222-Speed 128.20 samples/sec Loss 3.3874 Epoch: 12 Global Step: 126695 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:48:45,192-Speed 128.20 samples/sec Loss 3.2326 Epoch: 12 Global Step: 126700 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:49:04,841-Speed 130.29 samples/sec Loss 3.3257 Epoch: 12 Global Step: 126705 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:49:24,646-Speed 129.27 samples/sec Loss 3.3255 Epoch: 12 Global Step: 126710 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:49:44,252-Speed 130.58 samples/sec Loss 3.2370 Epoch: 12 Global Step: 126715 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:50:03,959-Speed 129.91 samples/sec Loss 3.4989 Epoch: 12 Global Step: 126720 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:50:23,536-Speed 130.77 samples/sec Loss 3.4061 Epoch: 12 Global Step: 126725 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:50:43,450-Speed 128.56 samples/sec Loss 3.5840 Epoch: 12 Global Step: 126730 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:51:03,061-Speed 130.54 samples/sec Loss 3.2268 Epoch: 12 Global Step: 126735 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:51:23,128-Speed 127.58 samples/sec Loss 3.3515 Epoch: 12 Global Step: 126740 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:51:43,008-Speed 128.77 samples/sec Loss 3.4307 Epoch: 12 Global Step: 126745 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:52:02,614-Speed 130.58 samples/sec Loss 3.2556 Epoch: 12 Global Step: 126750 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:52:22,233-Speed 130.49 samples/sec Loss 3.3675 Epoch: 12 Global Step: 126755 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:52:41,858-Speed 130.45 samples/sec Loss 3.3959 Epoch: 12 Global Step: 126760 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:53:01,517-Speed 130.23 samples/sec Loss 3.3953 Epoch: 12 Global Step: 126765 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:53:21,242-Speed 129.79 samples/sec Loss 3.4144 Epoch: 12 Global Step: 126770 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:53:41,289-Speed 127.71 samples/sec Loss 3.2006 Epoch: 12 Global Step: 126775 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:54:00,847-Speed 130.90 samples/sec Loss 3.3612 Epoch: 12 Global Step: 126780 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:54:20,935-Speed 127.45 samples/sec Loss 3.3225 Epoch: 12 Global Step: 126785 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:54:40,856-Speed 128.51 samples/sec Loss 3.3948 Epoch: 12 Global Step: 126790 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:55:00,445-Speed 130.69 samples/sec Loss 3.2982 Epoch: 12 Global Step: 126795 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:55:20,432-Speed 128.09 samples/sec Loss 3.3547 Epoch: 12 Global Step: 126800 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:55:40,167-Speed 129.72 samples/sec Loss 3.4300 Epoch: 12 Global Step: 126805 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:55:59,882-Speed 129.86 samples/sec Loss 3.2961 Epoch: 12 Global Step: 126810 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:56:19,498-Speed 130.52 samples/sec Loss 3.4652 Epoch: 12 Global Step: 126815 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:56:39,128-Speed 130.41 samples/sec Loss 3.2747 Epoch: 12 Global Step: 126820 Fp16 Grad Scale: 8192 Required: 145 hours Training: 2021-06-13 00:56:58,700-Speed 130.81 samples/sec Loss 3.3492 Epoch: 12 Global Step: 126825 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 00:57:18,715-Speed 127.91 samples/sec Loss 3.4864 Epoch: 12 Global Step: 126830 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 00:57:38,697-Speed 128.12 samples/sec Loss 3.4389 Epoch: 12 Global Step: 126835 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 00:57:58,155-Speed 131.57 samples/sec Loss 3.3694 Epoch: 12 Global Step: 126840 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 00:58:18,106-Speed 128.32 samples/sec Loss 3.3199 Epoch: 12 Global Step: 126845 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 00:58:37,662-Speed 130.91 samples/sec Loss 3.1398 Epoch: 12 Global Step: 126850 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 00:58:57,626-Speed 128.24 samples/sec Loss 3.3521 Epoch: 12 Global Step: 126855 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 00:59:17,316-Speed 130.02 samples/sec Loss 3.4790 Epoch: 12 Global Step: 126860 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 00:59:36,979-Speed 130.20 samples/sec Loss 3.3968 Epoch: 12 Global Step: 126865 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 00:59:56,606-Speed 130.44 samples/sec Loss 3.2515 Epoch: 12 Global Step: 126870 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:00:16,592-Speed 128.10 samples/sec Loss 3.3495 Epoch: 12 Global Step: 126875 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:00:36,335-Speed 129.67 samples/sec Loss 3.5246 Epoch: 12 Global Step: 126880 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:00:55,941-Speed 130.58 samples/sec Loss 3.3066 Epoch: 12 Global Step: 126885 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:01:15,697-Speed 129.59 samples/sec Loss 3.4358 Epoch: 12 Global Step: 126890 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:01:35,166-Speed 131.50 samples/sec Loss 3.3591 Epoch: 12 Global Step: 126895 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:01:55,090-Speed 128.50 samples/sec Loss 3.4677 Epoch: 12 Global Step: 126900 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:02:14,556-Speed 131.52 samples/sec Loss 3.4757 Epoch: 12 Global Step: 126905 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:02:34,480-Speed 128.49 samples/sec Loss 3.5501 Epoch: 12 Global Step: 126910 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:02:54,035-Speed 130.92 samples/sec Loss 3.3549 Epoch: 12 Global Step: 126915 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:03:14,125-Speed 127.43 samples/sec Loss 3.4855 Epoch: 12 Global Step: 126920 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:03:33,947-Speed 129.16 samples/sec Loss 3.4596 Epoch: 12 Global Step: 126925 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:03:53,522-Speed 130.78 samples/sec Loss 3.3068 Epoch: 12 Global Step: 126930 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:04:13,148-Speed 130.44 samples/sec Loss 3.2168 Epoch: 12 Global Step: 126935 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:04:32,761-Speed 130.53 samples/sec Loss 3.3225 Epoch: 12 Global Step: 126940 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:04:52,543-Speed 129.42 samples/sec Loss 3.2470 Epoch: 12 Global Step: 126945 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:05:12,115-Speed 130.80 samples/sec Loss 3.3200 Epoch: 12 Global Step: 126950 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:05:31,917-Speed 129.29 samples/sec Loss 3.3617 Epoch: 12 Global Step: 126955 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:05:51,407-Speed 131.36 samples/sec Loss 3.3268 Epoch: 12 Global Step: 126960 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:06:11,510-Speed 127.35 samples/sec Loss 3.1516 Epoch: 12 Global Step: 126965 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:06:31,540-Speed 127.81 samples/sec Loss 3.2877 Epoch: 12 Global Step: 126970 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:06:51,070-Speed 131.08 samples/sec Loss 3.1292 Epoch: 12 Global Step: 126975 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:07:10,797-Speed 129.78 samples/sec Loss 3.3984 Epoch: 12 Global Step: 126980 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:07:30,400-Speed 130.60 samples/sec Loss 3.4356 Epoch: 12 Global Step: 126985 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:07:50,083-Speed 130.07 samples/sec Loss 3.3808 Epoch: 12 Global Step: 126990 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:08:09,678-Speed 130.65 samples/sec Loss 3.3478 Epoch: 12 Global Step: 126995 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:08:29,401-Speed 129.81 samples/sec Loss 3.5280 Epoch: 12 Global Step: 127000 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:08:49,034-Speed 130.40 samples/sec Loss 3.1472 Epoch: 12 Global Step: 127005 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:09:08,958-Speed 128.49 samples/sec Loss 3.2080 Epoch: 12 Global Step: 127010 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:09:28,886-Speed 128.47 samples/sec Loss 3.4406 Epoch: 12 Global Step: 127015 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:09:48,429-Speed 131.00 samples/sec Loss 3.5062 Epoch: 12 Global Step: 127020 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:10:08,329-Speed 128.65 samples/sec Loss 3.3651 Epoch: 12 Global Step: 127025 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:10:27,940-Speed 130.55 samples/sec Loss 3.2058 Epoch: 12 Global Step: 127030 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:10:47,790-Speed 128.98 samples/sec Loss 3.5697 Epoch: 12 Global Step: 127035 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:11:07,366-Speed 130.78 samples/sec Loss 3.2819 Epoch: 12 Global Step: 127040 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:11:27,189-Speed 129.15 samples/sec Loss 3.2255 Epoch: 12 Global Step: 127045 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:11:46,813-Speed 130.46 samples/sec Loss 3.3823 Epoch: 12 Global Step: 127050 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:12:06,824-Speed 127.94 samples/sec Loss 3.4097 Epoch: 12 Global Step: 127055 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:12:26,750-Speed 128.48 samples/sec Loss 3.3814 Epoch: 12 Global Step: 127060 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:12:46,347-Speed 130.64 samples/sec Loss 3.3320 Epoch: 12 Global Step: 127065 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:13:06,221-Speed 128.82 samples/sec Loss 3.2874 Epoch: 12 Global Step: 127070 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:13:25,787-Speed 130.85 samples/sec Loss 3.5702 Epoch: 12 Global Step: 127075 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:13:46,110-Speed 125.97 samples/sec Loss 3.2974 Epoch: 12 Global Step: 127080 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:14:05,664-Speed 130.93 samples/sec Loss 3.2354 Epoch: 12 Global Step: 127085 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:14:25,432-Speed 129.51 samples/sec Loss 3.4359 Epoch: 12 Global Step: 127090 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:14:45,034-Speed 130.60 samples/sec Loss 3.3374 Epoch: 12 Global Step: 127095 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:15:05,129-Speed 127.40 samples/sec Loss 3.5116 Epoch: 12 Global Step: 127100 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:15:24,837-Speed 129.90 samples/sec Loss 3.1848 Epoch: 12 Global Step: 127105 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:15:44,803-Speed 128.22 samples/sec Loss 3.2677 Epoch: 12 Global Step: 127110 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:16:05,026-Speed 126.60 samples/sec Loss 3.3065 Epoch: 12 Global Step: 127115 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:16:24,563-Speed 131.04 samples/sec Loss 3.2192 Epoch: 12 Global Step: 127120 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:16:44,455-Speed 128.70 samples/sec Loss 3.3595 Epoch: 12 Global Step: 127125 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:17:03,993-Speed 131.03 samples/sec Loss 3.3294 Epoch: 12 Global Step: 127130 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:17:23,807-Speed 129.21 samples/sec Loss 3.2765 Epoch: 12 Global Step: 127135 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:17:43,447-Speed 130.35 samples/sec Loss 3.5567 Epoch: 12 Global Step: 127140 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:18:03,562-Speed 127.28 samples/sec Loss 3.3625 Epoch: 12 Global Step: 127145 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:18:23,302-Speed 129.69 samples/sec Loss 3.3803 Epoch: 12 Global Step: 127150 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:18:42,928-Speed 130.44 samples/sec Loss 3.4527 Epoch: 12 Global Step: 127155 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:19:02,646-Speed 129.84 samples/sec Loss 3.1777 Epoch: 12 Global Step: 127160 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:19:22,174-Speed 131.10 samples/sec Loss 3.3745 Epoch: 12 Global Step: 127165 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:19:41,934-Speed 129.56 samples/sec Loss 3.3504 Epoch: 12 Global Step: 127170 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:20:01,505-Speed 130.82 samples/sec Loss 3.2472 Epoch: 12 Global Step: 127175 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:20:21,494-Speed 128.07 samples/sec Loss 3.2916 Epoch: 12 Global Step: 127180 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:20:41,039-Speed 130.99 samples/sec Loss 3.4577 Epoch: 12 Global Step: 127185 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:21:01,114-Speed 127.53 samples/sec Loss 3.4541 Epoch: 12 Global Step: 127190 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:21:20,874-Speed 129.56 samples/sec Loss 3.4112 Epoch: 12 Global Step: 127195 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:21:40,491-Speed 130.50 samples/sec Loss 3.1047 Epoch: 12 Global Step: 127200 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:22:00,189-Speed 129.98 samples/sec Loss 3.3725 Epoch: 12 Global Step: 127205 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:22:19,811-Speed 130.47 samples/sec Loss 3.4942 Epoch: 12 Global Step: 127210 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:22:39,510-Speed 129.97 samples/sec Loss 3.3229 Epoch: 12 Global Step: 127215 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:22:59,103-Speed 130.66 samples/sec Loss 3.2324 Epoch: 12 Global Step: 127220 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:23:18,901-Speed 129.31 samples/sec Loss 3.1967 Epoch: 12 Global Step: 127225 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:23:38,475-Speed 130.80 samples/sec Loss 3.5683 Epoch: 12 Global Step: 127230 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:23:58,460-Speed 128.10 samples/sec Loss 3.5360 Epoch: 12 Global Step: 127235 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:24:18,376-Speed 128.55 samples/sec Loss 3.5583 Epoch: 12 Global Step: 127240 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:24:37,988-Speed 130.54 samples/sec Loss 3.3860 Epoch: 12 Global Step: 127245 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:24:57,808-Speed 129.17 samples/sec Loss 3.4510 Epoch: 12 Global Step: 127250 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:25:17,530-Speed 129.81 samples/sec Loss 3.4791 Epoch: 12 Global Step: 127255 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:25:37,249-Speed 129.83 samples/sec Loss 3.3388 Epoch: 12 Global Step: 127260 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:25:56,873-Speed 130.46 samples/sec Loss 3.4794 Epoch: 12 Global Step: 127265 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:26:16,681-Speed 129.25 samples/sec Loss 3.4364 Epoch: 12 Global Step: 127270 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:26:36,225-Speed 130.99 samples/sec Loss 3.3939 Epoch: 12 Global Step: 127275 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:26:56,292-Speed 127.58 samples/sec Loss 3.4764 Epoch: 12 Global Step: 127280 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:27:16,347-Speed 127.65 samples/sec Loss 3.4293 Epoch: 12 Global Step: 127285 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:27:35,933-Speed 130.72 samples/sec Loss 3.3450 Epoch: 12 Global Step: 127290 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:27:55,748-Speed 129.20 samples/sec Loss 3.3034 Epoch: 12 Global Step: 127295 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:28:15,181-Speed 131.74 samples/sec Loss 3.3963 Epoch: 12 Global Step: 127300 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:28:35,007-Speed 129.13 samples/sec Loss 3.3096 Epoch: 12 Global Step: 127305 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:28:54,758-Speed 129.62 samples/sec Loss 3.5546 Epoch: 12 Global Step: 127310 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:29:14,367-Speed 130.56 samples/sec Loss 3.4369 Epoch: 12 Global Step: 127315 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:29:33,971-Speed 130.59 samples/sec Loss 3.3032 Epoch: 12 Global Step: 127320 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:29:53,907-Speed 128.42 samples/sec Loss 3.5328 Epoch: 12 Global Step: 127325 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:30:13,941-Speed 127.79 samples/sec Loss 3.4262 Epoch: 12 Global Step: 127330 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:30:33,374-Speed 131.74 samples/sec Loss 3.5308 Epoch: 12 Global Step: 127335 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:30:53,268-Speed 128.69 samples/sec Loss 3.4156 Epoch: 12 Global Step: 127340 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:31:12,817-Speed 130.96 samples/sec Loss 3.3410 Epoch: 12 Global Step: 127345 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:31:32,715-Speed 128.66 samples/sec Loss 3.2988 Epoch: 12 Global Step: 127350 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:31:52,323-Speed 130.56 samples/sec Loss 3.5910 Epoch: 12 Global Step: 127355 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:32:11,975-Speed 130.28 samples/sec Loss 3.2905 Epoch: 12 Global Step: 127360 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:32:31,591-Speed 130.51 samples/sec Loss 3.5891 Epoch: 12 Global Step: 127365 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:32:51,680-Speed 127.44 samples/sec Loss 3.3157 Epoch: 12 Global Step: 127370 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:33:11,426-Speed 129.65 samples/sec Loss 3.3358 Epoch: 12 Global Step: 127375 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:33:30,907-Speed 131.42 samples/sec Loss 3.3794 Epoch: 12 Global Step: 127380 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:33:50,890-Speed 128.12 samples/sec Loss 3.6968 Epoch: 12 Global Step: 127385 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:34:10,440-Speed 130.95 samples/sec Loss 3.4123 Epoch: 12 Global Step: 127390 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:34:30,355-Speed 128.55 samples/sec Loss 3.3757 Epoch: 12 Global Step: 127395 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:34:49,878-Speed 131.13 samples/sec Loss 3.5304 Epoch: 12 Global Step: 127400 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:35:09,769-Speed 128.71 samples/sec Loss 3.3389 Epoch: 12 Global Step: 127405 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:35:29,323-Speed 130.93 samples/sec Loss 3.3187 Epoch: 12 Global Step: 127410 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:35:49,392-Speed 127.56 samples/sec Loss 3.2635 Epoch: 12 Global Step: 127415 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:36:09,182-Speed 129.37 samples/sec Loss 3.5300 Epoch: 12 Global Step: 127420 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:36:28,774-Speed 130.67 samples/sec Loss 3.4536 Epoch: 12 Global Step: 127425 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:36:48,386-Speed 130.54 samples/sec Loss 3.3455 Epoch: 12 Global Step: 127430 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:37:07,928-Speed 131.00 samples/sec Loss 3.4763 Epoch: 12 Global Step: 127435 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:37:27,858-Speed 128.46 samples/sec Loss 3.1970 Epoch: 12 Global Step: 127440 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:37:47,305-Speed 131.65 samples/sec Loss 3.4669 Epoch: 12 Global Step: 127445 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:38:07,198-Speed 128.69 samples/sec Loss 3.5204 Epoch: 12 Global Step: 127450 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:38:26,754-Speed 130.92 samples/sec Loss 3.2323 Epoch: 12 Global Step: 127455 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:38:46,826-Speed 127.54 samples/sec Loss 3.3366 Epoch: 12 Global Step: 127460 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:39:06,738-Speed 128.57 samples/sec Loss 3.3162 Epoch: 12 Global Step: 127465 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:39:26,308-Speed 130.82 samples/sec Loss 3.3594 Epoch: 12 Global Step: 127470 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:39:46,034-Speed 129.78 samples/sec Loss 3.1939 Epoch: 12 Global Step: 127475 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:40:05,644-Speed 130.55 samples/sec Loss 3.3563 Epoch: 12 Global Step: 127480 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:40:25,354-Speed 129.89 samples/sec Loss 3.5470 Epoch: 12 Global Step: 127485 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:40:44,898-Speed 130.99 samples/sec Loss 3.3515 Epoch: 12 Global Step: 127490 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:41:04,909-Speed 127.94 samples/sec Loss 3.3755 Epoch: 12 Global Step: 127495 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:41:24,467-Speed 130.90 samples/sec Loss 3.2176 Epoch: 12 Global Step: 127500 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:41:44,608-Speed 127.11 samples/sec Loss 3.2646 Epoch: 12 Global Step: 127505 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:42:04,481-Speed 128.82 samples/sec Loss 3.3130 Epoch: 12 Global Step: 127510 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:42:24,031-Speed 130.95 samples/sec Loss 3.6899 Epoch: 12 Global Step: 127515 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:42:43,977-Speed 128.36 samples/sec Loss 3.4107 Epoch: 12 Global Step: 127520 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:43:03,596-Speed 130.49 samples/sec Loss 3.4523 Epoch: 12 Global Step: 127525 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:43:23,229-Speed 130.40 samples/sec Loss 3.4021 Epoch: 12 Global Step: 127530 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:43:42,840-Speed 130.55 samples/sec Loss 3.5277 Epoch: 12 Global Step: 127535 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:44:02,459-Speed 130.49 samples/sec Loss 3.3431 Epoch: 12 Global Step: 127540 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:44:22,084-Speed 130.45 samples/sec Loss 3.2853 Epoch: 12 Global Step: 127545 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:44:42,101-Speed 127.90 samples/sec Loss 3.4145 Epoch: 12 Global Step: 127550 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:45:02,056-Speed 128.30 samples/sec Loss 3.3355 Epoch: 12 Global Step: 127555 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:45:21,594-Speed 131.03 samples/sec Loss 3.7050 Epoch: 12 Global Step: 127560 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:45:41,506-Speed 128.57 samples/sec Loss 3.4679 Epoch: 12 Global Step: 127565 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:46:01,054-Speed 130.97 samples/sec Loss 3.4498 Epoch: 12 Global Step: 127570 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:46:20,908-Speed 128.95 samples/sec Loss 3.3367 Epoch: 12 Global Step: 127575 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:46:40,558-Speed 130.29 samples/sec Loss 3.3320 Epoch: 12 Global Step: 127580 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:47:00,364-Speed 129.26 samples/sec Loss 3.2698 Epoch: 12 Global Step: 127585 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:47:19,983-Speed 130.50 samples/sec Loss 3.4138 Epoch: 12 Global Step: 127590 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:47:39,924-Speed 128.38 samples/sec Loss 3.4019 Epoch: 12 Global Step: 127595 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:47:59,724-Speed 129.30 samples/sec Loss 3.3952 Epoch: 12 Global Step: 127600 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:48:19,371-Speed 130.30 samples/sec Loss 3.4106 Epoch: 12 Global Step: 127605 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:48:39,366-Speed 128.04 samples/sec Loss 3.2642 Epoch: 12 Global Step: 127610 Fp16 Grad Scale: 8192 Required: 144 hours Training: 2021-06-13 01:48:58,933-Speed 130.84 samples/sec Loss 3.4121 Epoch: 12 Global Step: 127615 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:49:18,876-Speed 128.37 samples/sec Loss 3.3634 Epoch: 12 Global Step: 127620 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:49:38,415-Speed 131.03 samples/sec Loss 3.2423 Epoch: 12 Global Step: 127625 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:49:58,141-Speed 129.78 samples/sec Loss 3.2538 Epoch: 12 Global Step: 127630 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:50:17,764-Speed 130.47 samples/sec Loss 3.3278 Epoch: 12 Global Step: 127635 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:50:37,844-Speed 127.49 samples/sec Loss 3.4727 Epoch: 12 Global Step: 127640 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:50:57,562-Speed 129.84 samples/sec Loss 3.3058 Epoch: 12 Global Step: 127645 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:51:17,127-Speed 130.85 samples/sec Loss 3.2112 Epoch: 12 Global Step: 127650 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:51:37,135-Speed 127.96 samples/sec Loss 3.5318 Epoch: 12 Global Step: 127655 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:51:56,686-Speed 130.95 samples/sec Loss 3.3482 Epoch: 12 Global Step: 127660 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:52:16,479-Speed 129.35 samples/sec Loss 3.4932 Epoch: 12 Global Step: 127665 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:52:36,008-Speed 131.09 samples/sec Loss 3.3760 Epoch: 12 Global Step: 127670 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:52:55,968-Speed 128.27 samples/sec Loss 3.1631 Epoch: 12 Global Step: 127675 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:53:15,542-Speed 130.80 samples/sec Loss 3.3760 Epoch: 12 Global Step: 127680 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:53:35,620-Speed 127.51 samples/sec Loss 3.5445 Epoch: 12 Global Step: 127685 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:53:55,269-Speed 130.29 samples/sec Loss 3.3956 Epoch: 12 Global Step: 127690 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:54:14,897-Speed 130.43 samples/sec Loss 3.2398 Epoch: 12 Global Step: 127695 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:54:34,629-Speed 129.75 samples/sec Loss 3.2052 Epoch: 12 Global Step: 127700 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:54:54,181-Speed 130.94 samples/sec Loss 3.3185 Epoch: 12 Global Step: 127705 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:55:14,069-Speed 128.73 samples/sec Loss 3.4378 Epoch: 12 Global Step: 127710 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:55:33,641-Speed 130.80 samples/sec Loss 3.4294 Epoch: 12 Global Step: 127715 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:55:53,613-Speed 128.19 samples/sec Loss 3.3337 Epoch: 12 Global Step: 127720 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:56:13,418-Speed 129.27 samples/sec Loss 3.2415 Epoch: 12 Global Step: 127725 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:56:33,501-Speed 127.47 samples/sec Loss 3.6729 Epoch: 12 Global Step: 127730 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:56:53,665-Speed 126.97 samples/sec Loss 3.6928 Epoch: 12 Global Step: 127735 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:57:13,263-Speed 130.63 samples/sec Loss 3.2583 Epoch: 12 Global Step: 127740 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:57:32,983-Speed 129.83 samples/sec Loss 3.1698 Epoch: 12 Global Step: 127745 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:57:52,595-Speed 130.54 samples/sec Loss 3.2690 Epoch: 12 Global Step: 127750 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:58:12,422-Speed 129.12 samples/sec Loss 3.5384 Epoch: 12 Global Step: 127755 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:58:32,021-Speed 130.63 samples/sec Loss 3.3180 Epoch: 12 Global Step: 127760 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:58:51,829-Speed 129.25 samples/sec Loss 3.5422 Epoch: 12 Global Step: 127765 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:59:11,366-Speed 131.04 samples/sec Loss 3.4310 Epoch: 12 Global Step: 127770 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:59:31,576-Speed 126.68 samples/sec Loss 3.3964 Epoch: 12 Global Step: 127775 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 01:59:51,657-Speed 127.49 samples/sec Loss 3.2659 Epoch: 12 Global Step: 127780 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:00:11,202-Speed 130.99 samples/sec Loss 3.5057 Epoch: 12 Global Step: 127785 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:00:31,097-Speed 128.68 samples/sec Loss 3.2898 Epoch: 12 Global Step: 127790 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:00:50,572-Speed 131.46 samples/sec Loss 3.3634 Epoch: 12 Global Step: 127795 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:01:10,342-Speed 129.49 samples/sec Loss 3.4615 Epoch: 12 Global Step: 127800 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:01:29,956-Speed 130.53 samples/sec Loss 3.4565 Epoch: 12 Global Step: 127805 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:01:49,640-Speed 130.06 samples/sec Loss 3.3206 Epoch: 12 Global Step: 127810 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:02:09,259-Speed 130.49 samples/sec Loss 3.2074 Epoch: 12 Global Step: 127815 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:02:29,241-Speed 128.13 samples/sec Loss 3.3624 Epoch: 12 Global Step: 127820 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:02:49,062-Speed 129.16 samples/sec Loss 3.5010 Epoch: 12 Global Step: 127825 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:03:08,561-Speed 131.30 samples/sec Loss 3.3167 Epoch: 12 Global Step: 127830 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:03:28,361-Speed 129.30 samples/sec Loss 3.5521 Epoch: 12 Global Step: 127835 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:03:47,884-Speed 131.14 samples/sec Loss 3.5948 Epoch: 12 Global Step: 127840 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:04:07,791-Speed 128.60 samples/sec Loss 3.2880 Epoch: 12 Global Step: 127845 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:04:27,411-Speed 130.48 samples/sec Loss 3.5088 Epoch: 12 Global Step: 127850 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:04:47,298-Speed 128.73 samples/sec Loss 3.4073 Epoch: 12 Global Step: 127855 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:05:06,747-Speed 131.64 samples/sec Loss 3.2231 Epoch: 12 Global Step: 127860 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:05:26,849-Speed 127.35 samples/sec Loss 3.2202 Epoch: 12 Global Step: 127865 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:05:46,541-Speed 130.01 samples/sec Loss 3.3066 Epoch: 12 Global Step: 127870 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:06:06,373-Speed 129.09 samples/sec Loss 3.3329 Epoch: 12 Global Step: 127875 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:06:26,003-Speed 130.42 samples/sec Loss 3.2429 Epoch: 12 Global Step: 127880 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:06:45,577-Speed 130.80 samples/sec Loss 3.3498 Epoch: 12 Global Step: 127885 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:07:05,627-Speed 127.69 samples/sec Loss 3.5584 Epoch: 12 Global Step: 127890 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:07:25,177-Speed 130.95 samples/sec Loss 3.3326 Epoch: 12 Global Step: 127895 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:07:45,091-Speed 128.56 samples/sec Loss 3.2652 Epoch: 12 Global Step: 127900 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:08:04,541-Speed 131.62 samples/sec Loss 3.2549 Epoch: 12 Global Step: 127905 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:08:24,649-Speed 127.32 samples/sec Loss 3.4571 Epoch: 12 Global Step: 127910 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:08:44,387-Speed 129.70 samples/sec Loss 3.3940 Epoch: 12 Global Step: 127915 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:09:03,991-Speed 130.59 samples/sec Loss 3.5180 Epoch: 12 Global Step: 127920 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:09:23,633-Speed 130.34 samples/sec Loss 3.2931 Epoch: 12 Global Step: 127925 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:09:43,221-Speed 130.70 samples/sec Loss 3.4767 Epoch: 12 Global Step: 127930 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:10:03,142-Speed 128.51 samples/sec Loss 3.2786 Epoch: 12 Global Step: 127935 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:10:22,679-Speed 131.04 samples/sec Loss 3.1882 Epoch: 12 Global Step: 127940 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:10:42,462-Speed 129.42 samples/sec Loss 3.3658 Epoch: 12 Global Step: 127945 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:11:02,015-Speed 130.93 samples/sec Loss 3.4539 Epoch: 12 Global Step: 127950 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:11:22,087-Speed 127.55 samples/sec Loss 3.1994 Epoch: 12 Global Step: 127955 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:11:41,882-Speed 129.33 samples/sec Loss 3.2018 Epoch: 12 Global Step: 127960 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:12:01,469-Speed 130.70 samples/sec Loss 3.2672 Epoch: 12 Global Step: 127965 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:12:21,166-Speed 129.97 samples/sec Loss 3.3388 Epoch: 12 Global Step: 127970 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:12:40,789-Speed 130.47 samples/sec Loss 3.4489 Epoch: 12 Global Step: 127975 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:13:00,497-Speed 129.91 samples/sec Loss 3.4061 Epoch: 12 Global Step: 127980 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:13:19,942-Speed 131.66 samples/sec Loss 3.5280 Epoch: 12 Global Step: 127985 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:13:39,752-Speed 129.24 samples/sec Loss 3.3422 Epoch: 12 Global Step: 127990 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:13:59,289-Speed 131.04 samples/sec Loss 3.6994 Epoch: 12 Global Step: 127995 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:14:19,475-Speed 126.83 samples/sec Loss 3.5312 Epoch: 12 Global Step: 128000 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:14:39,289-Speed 129.21 samples/sec Loss 3.4621 Epoch: 12 Global Step: 128005 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:14:58,814-Speed 131.12 samples/sec Loss 3.3264 Epoch: 12 Global Step: 128010 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:15:18,498-Speed 130.06 samples/sec Loss 3.4842 Epoch: 12 Global Step: 128015 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:15:38,101-Speed 130.60 samples/sec Loss 3.2809 Epoch: 12 Global Step: 128020 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:15:57,721-Speed 130.49 samples/sec Loss 3.3845 Epoch: 12 Global Step: 128025 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:16:17,208-Speed 131.38 samples/sec Loss 3.3601 Epoch: 12 Global Step: 128030 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:16:37,086-Speed 128.79 samples/sec Loss 3.3304 Epoch: 12 Global Step: 128035 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:16:56,693-Speed 130.57 samples/sec Loss 3.1420 Epoch: 12 Global Step: 128040 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:17:16,788-Speed 127.40 samples/sec Loss 3.2476 Epoch: 12 Global Step: 128045 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:17:36,591-Speed 129.28 samples/sec Loss 3.2866 Epoch: 12 Global Step: 128050 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:17:56,271-Speed 130.09 samples/sec Loss 3.4408 Epoch: 12 Global Step: 128055 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:18:16,110-Speed 129.05 samples/sec Loss 3.3323 Epoch: 12 Global Step: 128060 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:18:35,780-Speed 130.15 samples/sec Loss 3.4837 Epoch: 12 Global Step: 128065 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:18:55,395-Speed 130.52 samples/sec Loss 3.4198 Epoch: 12 Global Step: 128070 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:19:14,995-Speed 130.62 samples/sec Loss 3.5564 Epoch: 12 Global Step: 128075 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:19:34,705-Speed 129.89 samples/sec Loss 3.4310 Epoch: 12 Global Step: 128080 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:19:54,289-Speed 130.73 samples/sec Loss 3.3290 Epoch: 12 Global Step: 128085 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:20:14,288-Speed 128.01 samples/sec Loss 3.3497 Epoch: 12 Global Step: 128090 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:20:34,166-Speed 128.79 samples/sec Loss 3.5147 Epoch: 12 Global Step: 128095 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:20:53,896-Speed 129.76 samples/sec Loss 3.3443 Epoch: 12 Global Step: 128100 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:21:13,804-Speed 128.60 samples/sec Loss 3.4780 Epoch: 12 Global Step: 128105 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:21:33,350-Speed 130.98 samples/sec Loss 3.5295 Epoch: 12 Global Step: 128110 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:21:53,190-Speed 129.04 samples/sec Loss 3.3340 Epoch: 12 Global Step: 128115 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:22:12,713-Speed 131.14 samples/sec Loss 3.2493 Epoch: 12 Global Step: 128120 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:22:32,537-Speed 129.15 samples/sec Loss 3.4169 Epoch: 12 Global Step: 128125 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:22:52,125-Speed 130.70 samples/sec Loss 3.4425 Epoch: 12 Global Step: 128130 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:23:12,219-Speed 127.41 samples/sec Loss 3.2183 Epoch: 12 Global Step: 128135 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:23:31,947-Speed 129.77 samples/sec Loss 3.4066 Epoch: 12 Global Step: 128140 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:23:51,517-Speed 130.82 samples/sec Loss 3.3944 Epoch: 12 Global Step: 128145 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:24:11,286-Speed 129.51 samples/sec Loss 3.3321 Epoch: 12 Global Step: 128150 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:24:30,781-Speed 131.32 samples/sec Loss 3.3880 Epoch: 12 Global Step: 128155 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:24:50,674-Speed 128.70 samples/sec Loss 3.2445 Epoch: 12 Global Step: 128160 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:25:10,224-Speed 130.96 samples/sec Loss 3.3223 Epoch: 12 Global Step: 128165 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:25:30,118-Speed 128.69 samples/sec Loss 3.4911 Epoch: 12 Global Step: 128170 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:25:49,668-Speed 130.95 samples/sec Loss 3.3644 Epoch: 12 Global Step: 128175 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:26:09,648-Speed 128.13 samples/sec Loss 3.2419 Epoch: 12 Global Step: 128180 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:26:29,369-Speed 129.82 samples/sec Loss 3.3269 Epoch: 12 Global Step: 128185 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:26:48,946-Speed 130.78 samples/sec Loss 3.2664 Epoch: 12 Global Step: 128190 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:27:08,759-Speed 129.21 samples/sec Loss 3.4419 Epoch: 12 Global Step: 128195 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:27:28,339-Speed 130.76 samples/sec Loss 3.4288 Epoch: 12 Global Step: 128200 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:27:48,196-Speed 128.93 samples/sec Loss 3.5523 Epoch: 12 Global Step: 128205 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:28:07,817-Speed 130.48 samples/sec Loss 3.4750 Epoch: 12 Global Step: 128210 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:28:27,604-Speed 129.38 samples/sec Loss 3.3342 Epoch: 12 Global Step: 128215 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:28:47,171-Speed 130.84 samples/sec Loss 3.4976 Epoch: 12 Global Step: 128220 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:29:07,252-Speed 127.49 samples/sec Loss 3.2609 Epoch: 12 Global Step: 128225 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:29:27,048-Speed 129.32 samples/sec Loss 3.3629 Epoch: 12 Global Step: 128230 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:29:46,674-Speed 130.44 samples/sec Loss 3.4005 Epoch: 12 Global Step: 128235 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:30:06,340-Speed 130.18 samples/sec Loss 3.4427 Epoch: 12 Global Step: 128240 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:30:25,937-Speed 130.64 samples/sec Loss 3.2234 Epoch: 12 Global Step: 128245 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:30:45,673-Speed 129.72 samples/sec Loss 3.4490 Epoch: 12 Global Step: 128250 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:31:05,295-Speed 130.47 samples/sec Loss 3.2380 Epoch: 12 Global Step: 128255 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:31:25,093-Speed 129.31 samples/sec Loss 3.4350 Epoch: 12 Global Step: 128260 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:31:44,624-Speed 131.08 samples/sec Loss 3.2607 Epoch: 12 Global Step: 128265 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:32:04,683-Speed 127.63 samples/sec Loss 3.3150 Epoch: 12 Global Step: 128270 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:32:24,786-Speed 127.35 samples/sec Loss 3.3546 Epoch: 12 Global Step: 128275 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:32:44,342-Speed 130.91 samples/sec Loss 3.5123 Epoch: 12 Global Step: 128280 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:33:04,094-Speed 129.62 samples/sec Loss 3.5132 Epoch: 12 Global Step: 128285 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:33:23,661-Speed 130.84 samples/sec Loss 3.2077 Epoch: 12 Global Step: 128290 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:33:43,370-Speed 129.90 samples/sec Loss 3.2231 Epoch: 12 Global Step: 128295 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:34:02,992-Speed 130.47 samples/sec Loss 3.4475 Epoch: 12 Global Step: 128300 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:34:22,707-Speed 129.86 samples/sec Loss 3.3003 Epoch: 12 Global Step: 128305 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:34:42,268-Speed 130.88 samples/sec Loss 3.6381 Epoch: 12 Global Step: 128310 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:35:02,307-Speed 127.76 samples/sec Loss 3.4124 Epoch: 12 Global Step: 128315 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:35:22,275-Speed 128.21 samples/sec Loss 3.3353 Epoch: 12 Global Step: 128320 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:35:41,819-Speed 131.00 samples/sec Loss 3.4086 Epoch: 12 Global Step: 128325 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:36:01,712-Speed 128.69 samples/sec Loss 3.3632 Epoch: 12 Global Step: 128330 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:36:21,253-Speed 131.02 samples/sec Loss 3.4801 Epoch: 12 Global Step: 128335 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:36:41,203-Speed 128.33 samples/sec Loss 3.3792 Epoch: 12 Global Step: 128340 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:37:00,664-Speed 131.55 samples/sec Loss 3.2441 Epoch: 12 Global Step: 128345 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:37:20,416-Speed 129.62 samples/sec Loss 3.5100 Epoch: 12 Global Step: 128350 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:37:40,111-Speed 129.99 samples/sec Loss 3.3841 Epoch: 12 Global Step: 128355 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:38:00,180-Speed 127.57 samples/sec Loss 3.2716 Epoch: 12 Global Step: 128360 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:38:19,789-Speed 130.55 samples/sec Loss 3.4458 Epoch: 12 Global Step: 128365 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:38:39,336-Speed 130.97 samples/sec Loss 3.2371 Epoch: 12 Global Step: 128370 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:38:59,324-Speed 128.08 samples/sec Loss 3.4378 Epoch: 12 Global Step: 128375 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:39:18,909-Speed 130.72 samples/sec Loss 3.4803 Epoch: 12 Global Step: 128380 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:39:38,803-Speed 128.69 samples/sec Loss 3.2737 Epoch: 12 Global Step: 128385 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:39:58,362-Speed 130.89 samples/sec Loss 3.7297 Epoch: 12 Global Step: 128390 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:40:18,190-Speed 129.12 samples/sec Loss 3.3099 Epoch: 12 Global Step: 128395 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:40:37,656-Speed 131.52 samples/sec Loss 3.6659 Epoch: 12 Global Step: 128400 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:40:57,756-Speed 127.37 samples/sec Loss 3.4926 Epoch: 12 Global Step: 128405 Fp16 Grad Scale: 8192 Required: 143 hours Training: 2021-06-13 02:41:17,485-Speed 129.77 samples/sec Loss 3.4983 Epoch: 12 Global Step: 128410 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:41:37,100-Speed 130.52 samples/sec Loss 3.3357 Epoch: 12 Global Step: 128415 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:41:56,704-Speed 130.59 samples/sec Loss 3.2494 Epoch: 12 Global Step: 128420 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:42:16,307-Speed 130.60 samples/sec Loss 3.4557 Epoch: 12 Global Step: 128425 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:42:36,087-Speed 129.43 samples/sec Loss 3.4242 Epoch: 12 Global Step: 128430 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:42:55,642-Speed 130.92 samples/sec Loss 3.2688 Epoch: 12 Global Step: 128435 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:43:15,536-Speed 128.69 samples/sec Loss 3.5972 Epoch: 12 Global Step: 128440 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:43:35,186-Speed 130.29 samples/sec Loss 3.6804 Epoch: 12 Global Step: 128445 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:43:55,338-Speed 127.04 samples/sec Loss 3.2182 Epoch: 12 Global Step: 128450 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:44:15,069-Speed 129.75 samples/sec Loss 3.4887 Epoch: 12 Global Step: 128455 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:44:34,639-Speed 130.82 samples/sec Loss 3.4247 Epoch: 12 Global Step: 128460 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:44:54,343-Speed 129.93 samples/sec Loss 3.3827 Epoch: 12 Global Step: 128465 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:45:13,959-Speed 130.51 samples/sec Loss 3.4060 Epoch: 12 Global Step: 128470 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:45:33,669-Speed 129.89 samples/sec Loss 3.4351 Epoch: 12 Global Step: 128475 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:45:53,228-Speed 130.89 samples/sec Loss 3.4263 Epoch: 12 Global Step: 128480 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:46:12,985-Speed 129.58 samples/sec Loss 3.2473 Epoch: 12 Global Step: 128485 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:46:32,527-Speed 131.01 samples/sec Loss 3.4650 Epoch: 12 Global Step: 128490 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:46:52,675-Speed 127.06 samples/sec Loss 3.5543 Epoch: 12 Global Step: 128495 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:47:12,586-Speed 128.58 samples/sec Loss 3.3819 Epoch: 12 Global Step: 128500 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:47:32,107-Speed 131.14 samples/sec Loss 3.5433 Epoch: 12 Global Step: 128505 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:47:51,873-Speed 129.52 samples/sec Loss 3.2049 Epoch: 12 Global Step: 128510 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:48:11,447-Speed 130.79 samples/sec Loss 3.4622 Epoch: 12 Global Step: 128515 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:48:31,193-Speed 129.65 samples/sec Loss 3.4344 Epoch: 12 Global Step: 128520 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:48:50,794-Speed 130.62 samples/sec Loss 3.2561 Epoch: 12 Global Step: 128525 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:49:10,493-Speed 129.96 samples/sec Loss 3.4873 Epoch: 12 Global Step: 128530 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:49:30,116-Speed 130.47 samples/sec Loss 3.2655 Epoch: 12 Global Step: 128535 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:49:50,070-Speed 128.30 samples/sec Loss 3.3681 Epoch: 12 Global Step: 128540 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:50:09,951-Speed 128.77 samples/sec Loss 3.3422 Epoch: 12 Global Step: 128545 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:50:29,465-Speed 131.20 samples/sec Loss 3.4628 Epoch: 12 Global Step: 128550 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:50:49,432-Speed 128.22 samples/sec Loss 3.2900 Epoch: 12 Global Step: 128555 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:51:08,978-Speed 130.98 samples/sec Loss 3.2904 Epoch: 12 Global Step: 128560 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:51:28,793-Speed 129.20 samples/sec Loss 3.4883 Epoch: 12 Global Step: 128565 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:51:48,379-Speed 130.72 samples/sec Loss 3.2389 Epoch: 12 Global Step: 128570 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:52:08,101-Speed 129.81 samples/sec Loss 3.5846 Epoch: 12 Global Step: 128575 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:52:27,722-Speed 130.49 samples/sec Loss 3.3272 Epoch: 12 Global Step: 128580 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:52:47,657-Speed 128.42 samples/sec Loss 3.2271 Epoch: 12 Global Step: 128585 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:53:07,432-Speed 129.47 samples/sec Loss 3.4767 Epoch: 12 Global Step: 128590 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:53:27,039-Speed 130.57 samples/sec Loss 3.3144 Epoch: 12 Global Step: 128595 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:53:46,940-Speed 128.65 samples/sec Loss 3.2665 Epoch: 12 Global Step: 128600 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:54:06,483-Speed 131.00 samples/sec Loss 3.3461 Epoch: 12 Global Step: 128605 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:54:26,286-Speed 129.28 samples/sec Loss 3.4248 Epoch: 12 Global Step: 128610 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:54:45,815-Speed 131.09 samples/sec Loss 3.2803 Epoch: 12 Global Step: 128615 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:55:05,643-Speed 129.12 samples/sec Loss 3.4069 Epoch: 12 Global Step: 128620 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:55:25,224-Speed 130.75 samples/sec Loss 3.2007 Epoch: 12 Global Step: 128625 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:55:45,314-Speed 127.43 samples/sec Loss 3.5717 Epoch: 12 Global Step: 128630 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:56:05,024-Speed 129.89 samples/sec Loss 3.4876 Epoch: 12 Global Step: 128635 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:56:24,639-Speed 130.52 samples/sec Loss 3.3572 Epoch: 12 Global Step: 128640 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:56:44,515-Speed 128.80 samples/sec Loss 3.3191 Epoch: 12 Global Step: 128645 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:57:04,085-Speed 130.82 samples/sec Loss 3.5300 Epoch: 12 Global Step: 128650 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:57:23,911-Speed 129.13 samples/sec Loss 3.4618 Epoch: 12 Global Step: 128655 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:57:43,450-Speed 131.03 samples/sec Loss 3.3167 Epoch: 12 Global Step: 128660 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:58:03,239-Speed 129.37 samples/sec Loss 3.5312 Epoch: 12 Global Step: 128665 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:58:22,819-Speed 130.75 samples/sec Loss 3.4518 Epoch: 12 Global Step: 128670 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:58:42,821-Speed 127.99 samples/sec Loss 3.3410 Epoch: 12 Global Step: 128675 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:59:02,543-Speed 129.82 samples/sec Loss 3.3232 Epoch: 12 Global Step: 128680 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:59:22,081-Speed 131.03 samples/sec Loss 3.3832 Epoch: 12 Global Step: 128685 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 02:59:41,926-Speed 129.01 samples/sec Loss 3.3862 Epoch: 12 Global Step: 128690 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:00:01,364-Speed 131.71 samples/sec Loss 3.4666 Epoch: 12 Global Step: 128695 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:00:21,242-Speed 128.80 samples/sec Loss 3.3848 Epoch: 12 Global Step: 128700 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:00:40,773-Speed 131.08 samples/sec Loss 3.4126 Epoch: 12 Global Step: 128705 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:01:00,653-Speed 128.78 samples/sec Loss 3.3684 Epoch: 12 Global Step: 128710 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:01:20,224-Speed 130.81 samples/sec Loss 3.4373 Epoch: 12 Global Step: 128715 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:01:40,203-Speed 128.14 samples/sec Loss 3.4123 Epoch: 12 Global Step: 128720 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:02:00,150-Speed 128.34 samples/sec Loss 3.5263 Epoch: 12 Global Step: 128725 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:02:19,821-Speed 130.15 samples/sec Loss 3.2956 Epoch: 12 Global Step: 128730 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:02:39,484-Speed 130.20 samples/sec Loss 3.4763 Epoch: 12 Global Step: 128735 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:02:59,085-Speed 130.62 samples/sec Loss 3.3930 Epoch: 12 Global Step: 128740 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:03:18,713-Speed 130.43 samples/sec Loss 3.4025 Epoch: 12 Global Step: 128745 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:03:38,327-Speed 130.52 samples/sec Loss 3.5165 Epoch: 12 Global Step: 128750 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:03:58,144-Speed 129.19 samples/sec Loss 3.5428 Epoch: 12 Global Step: 128755 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:04:17,661-Speed 131.17 samples/sec Loss 3.4134 Epoch: 12 Global Step: 128760 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:04:37,730-Speed 127.57 samples/sec Loss 3.3514 Epoch: 12 Global Step: 128765 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:04:57,711-Speed 128.13 samples/sec Loss 3.3780 Epoch: 12 Global Step: 128770 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:05:17,262-Speed 130.95 samples/sec Loss 3.4184 Epoch: 12 Global Step: 128775 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:05:36,961-Speed 129.96 samples/sec Loss 3.5975 Epoch: 12 Global Step: 128780 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:05:56,516-Speed 130.92 samples/sec Loss 3.3667 Epoch: 12 Global Step: 128785 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:06:16,213-Speed 129.98 samples/sec Loss 3.6372 Epoch: 12 Global Step: 128790 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:06:35,888-Speed 130.12 samples/sec Loss 3.4201 Epoch: 12 Global Step: 128795 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:06:55,634-Speed 129.65 samples/sec Loss 3.3652 Epoch: 12 Global Step: 128800 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:07:15,131-Speed 131.31 samples/sec Loss 3.4753 Epoch: 12 Global Step: 128805 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:07:35,172-Speed 127.74 samples/sec Loss 3.0930 Epoch: 12 Global Step: 128810 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:07:55,262-Speed 127.43 samples/sec Loss 3.4230 Epoch: 12 Global Step: 128815 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:08:14,813-Speed 130.95 samples/sec Loss 3.4679 Epoch: 12 Global Step: 128820 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:08:34,709-Speed 128.68 samples/sec Loss 3.4035 Epoch: 12 Global Step: 128825 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:08:54,169-Speed 131.56 samples/sec Loss 3.3212 Epoch: 12 Global Step: 128830 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:09:14,031-Speed 128.90 samples/sec Loss 3.6539 Epoch: 12 Global Step: 128835 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:09:33,610-Speed 130.76 samples/sec Loss 3.3773 Epoch: 12 Global Step: 128840 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:09:53,236-Speed 130.45 samples/sec Loss 3.4859 Epoch: 12 Global Step: 128845 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:10:12,869-Speed 130.40 samples/sec Loss 3.5138 Epoch: 12 Global Step: 128850 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:10:32,924-Speed 127.65 samples/sec Loss 3.5715 Epoch: 12 Global Step: 128855 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:10:52,659-Speed 129.73 samples/sec Loss 3.3056 Epoch: 12 Global Step: 128860 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:11:12,149-Speed 131.35 samples/sec Loss 3.2438 Epoch: 12 Global Step: 128865 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:11:31,999-Speed 128.98 samples/sec Loss 3.4878 Epoch: 12 Global Step: 128870 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:11:51,630-Speed 130.41 samples/sec Loss 3.2481 Epoch: 12 Global Step: 128875 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:12:11,484-Speed 128.95 samples/sec Loss 3.5511 Epoch: 12 Global Step: 128880 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:12:31,033-Speed 130.96 samples/sec Loss 3.4009 Epoch: 12 Global Step: 128885 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:12:50,856-Speed 129.15 samples/sec Loss 3.4211 Epoch: 12 Global Step: 128890 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:13:10,400-Speed 130.99 samples/sec Loss 3.1405 Epoch: 12 Global Step: 128895 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:13:30,560-Speed 126.99 samples/sec Loss 3.6615 Epoch: 12 Global Step: 128900 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:13:50,221-Speed 130.21 samples/sec Loss 3.4162 Epoch: 12 Global Step: 128905 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:14:09,783-Speed 130.88 samples/sec Loss 3.2706 Epoch: 12 Global Step: 128910 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:14:29,562-Speed 129.44 samples/sec Loss 3.3991 Epoch: 12 Global Step: 128915 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:14:49,184-Speed 130.47 samples/sec Loss 3.5256 Epoch: 12 Global Step: 128920 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:15:09,078-Speed 128.69 samples/sec Loss 3.2387 Epoch: 12 Global Step: 128925 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:15:28,570-Speed 131.34 samples/sec Loss 3.3604 Epoch: 12 Global Step: 128930 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:15:48,373-Speed 129.28 samples/sec Loss 3.4018 Epoch: 12 Global Step: 128935 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:16:07,996-Speed 130.47 samples/sec Loss 3.4787 Epoch: 12 Global Step: 128940 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:16:28,025-Speed 127.82 samples/sec Loss 3.4948 Epoch: 12 Global Step: 128945 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:16:47,749-Speed 129.80 samples/sec Loss 3.5883 Epoch: 12 Global Step: 128950 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:17:07,279-Speed 131.09 samples/sec Loss 3.6350 Epoch: 12 Global Step: 128955 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:17:27,164-Speed 128.75 samples/sec Loss 3.3849 Epoch: 12 Global Step: 128960 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:17:46,620-Speed 131.58 samples/sec Loss 3.6427 Epoch: 12 Global Step: 128965 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:18:06,533-Speed 128.57 samples/sec Loss 3.4540 Epoch: 12 Global Step: 128970 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:18:26,084-Speed 130.94 samples/sec Loss 3.4063 Epoch: 12 Global Step: 128975 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:18:46,032-Speed 128.35 samples/sec Loss 3.2648 Epoch: 12 Global Step: 128980 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:19:05,637-Speed 130.59 samples/sec Loss 3.3441 Epoch: 12 Global Step: 128985 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:19:25,624-Speed 128.09 samples/sec Loss 3.3430 Epoch: 12 Global Step: 128990 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:19:45,354-Speed 129.75 samples/sec Loss 3.5506 Epoch: 12 Global Step: 128995 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:20:04,949-Speed 130.65 samples/sec Loss 3.4501 Epoch: 12 Global Step: 129000 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:20:24,730-Speed 129.43 samples/sec Loss 3.3642 Epoch: 12 Global Step: 129005 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:20:44,289-Speed 130.89 samples/sec Loss 3.4269 Epoch: 12 Global Step: 129010 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:21:04,128-Speed 129.05 samples/sec Loss 3.3577 Epoch: 12 Global Step: 129015 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:21:23,619-Speed 131.35 samples/sec Loss 3.5809 Epoch: 12 Global Step: 129020 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:21:43,503-Speed 128.76 samples/sec Loss 3.3975 Epoch: 12 Global Step: 129025 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:22:03,131-Speed 130.43 samples/sec Loss 3.3294 Epoch: 12 Global Step: 129030 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:22:23,297-Speed 126.95 samples/sec Loss 3.3105 Epoch: 12 Global Step: 129035 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:22:43,207-Speed 128.59 samples/sec Loss 3.4409 Epoch: 12 Global Step: 129040 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:23:02,741-Speed 131.06 samples/sec Loss 3.3897 Epoch: 12 Global Step: 129045 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:23:22,496-Speed 129.59 samples/sec Loss 3.3687 Epoch: 12 Global Step: 129050 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:23:42,118-Speed 130.47 samples/sec Loss 3.3529 Epoch: 12 Global Step: 129055 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:24:01,940-Speed 129.16 samples/sec Loss 3.3290 Epoch: 12 Global Step: 129060 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:24:21,486-Speed 130.98 samples/sec Loss 3.4899 Epoch: 12 Global Step: 129065 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:24:41,356-Speed 128.85 samples/sec Loss 3.4830 Epoch: 12 Global Step: 129070 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:25:00,916-Speed 130.89 samples/sec Loss 3.3613 Epoch: 12 Global Step: 129075 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:25:20,960-Speed 127.73 samples/sec Loss 3.7597 Epoch: 12 Global Step: 129080 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:25:40,956-Speed 128.03 samples/sec Loss 3.3232 Epoch: 12 Global Step: 129085 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:26:00,575-Speed 130.49 samples/sec Loss 3.4084 Epoch: 12 Global Step: 129090 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:26:20,433-Speed 128.92 samples/sec Loss 3.4680 Epoch: 12 Global Step: 129095 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:26:39,971-Speed 131.04 samples/sec Loss 3.4885 Epoch: 12 Global Step: 129100 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:26:59,711-Speed 129.69 samples/sec Loss 3.4188 Epoch: 12 Global Step: 129105 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:27:19,357-Speed 130.31 samples/sec Loss 3.3801 Epoch: 12 Global Step: 129110 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:27:39,593-Speed 126.52 samples/sec Loss 3.2563 Epoch: 12 Global Step: 129115 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:27:59,156-Speed 130.87 samples/sec Loss 3.5309 Epoch: 12 Global Step: 129120 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:28:19,107-Speed 128.32 samples/sec Loss 3.3864 Epoch: 12 Global Step: 129125 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:28:39,120-Speed 127.92 samples/sec Loss 3.3350 Epoch: 12 Global Step: 129130 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:28:58,705-Speed 130.72 samples/sec Loss 3.4779 Epoch: 12 Global Step: 129135 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:29:18,636-Speed 128.45 samples/sec Loss 3.3535 Epoch: 12 Global Step: 129140 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:29:38,175-Speed 131.03 samples/sec Loss 3.3715 Epoch: 12 Global Step: 129145 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:29:57,802-Speed 130.44 samples/sec Loss 3.4837 Epoch: 12 Global Step: 129150 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:30:17,382-Speed 130.76 samples/sec Loss 3.4241 Epoch: 12 Global Step: 129155 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:30:37,154-Speed 129.48 samples/sec Loss 3.3878 Epoch: 12 Global Step: 129160 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:30:56,787-Speed 130.40 samples/sec Loss 3.4826 Epoch: 12 Global Step: 129165 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:31:16,739-Speed 128.31 samples/sec Loss 3.4510 Epoch: 12 Global Step: 129170 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:31:36,576-Speed 129.06 samples/sec Loss 3.4732 Epoch: 12 Global Step: 129175 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:31:56,190-Speed 130.52 samples/sec Loss 3.2741 Epoch: 12 Global Step: 129180 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:32:16,005-Speed 129.21 samples/sec Loss 3.2853 Epoch: 12 Global Step: 129185 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:32:35,537-Speed 131.07 samples/sec Loss 3.6169 Epoch: 12 Global Step: 129190 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:32:55,303-Speed 129.52 samples/sec Loss 3.4077 Epoch: 12 Global Step: 129195 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:33:14,881-Speed 130.77 samples/sec Loss 3.4197 Epoch: 12 Global Step: 129200 Fp16 Grad Scale: 8192 Required: 142 hours Training: 2021-06-13 03:33:34,612-Speed 129.75 samples/sec Loss 3.3979 Epoch: 12 Global Step: 129205 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:33:54,226-Speed 130.53 samples/sec Loss 3.4704 Epoch: 12 Global Step: 129210 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:34:14,211-Speed 128.10 samples/sec Loss 3.3734 Epoch: 12 Global Step: 129215 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:34:34,103-Speed 128.70 samples/sec Loss 3.2093 Epoch: 12 Global Step: 129220 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:34:53,645-Speed 131.01 samples/sec Loss 3.4471 Epoch: 12 Global Step: 129225 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:35:13,562-Speed 128.54 samples/sec Loss 3.5047 Epoch: 12 Global Step: 129230 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:35:33,164-Speed 130.61 samples/sec Loss 3.3853 Epoch: 12 Global Step: 129235 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:35:52,945-Speed 129.43 samples/sec Loss 3.6250 Epoch: 12 Global Step: 129240 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:36:12,557-Speed 130.54 samples/sec Loss 3.4826 Epoch: 12 Global Step: 129245 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:36:32,362-Speed 129.27 samples/sec Loss 3.4378 Epoch: 12 Global Step: 129250 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:36:51,954-Speed 130.67 samples/sec Loss 3.3626 Epoch: 12 Global Step: 129255 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:37:12,138-Speed 126.84 samples/sec Loss 3.4971 Epoch: 12 Global Step: 129260 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:37:32,014-Speed 128.80 samples/sec Loss 3.5558 Epoch: 12 Global Step: 129265 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:37:51,556-Speed 131.01 samples/sec Loss 3.5991 Epoch: 12 Global Step: 129270 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:38:11,512-Speed 128.29 samples/sec Loss 3.3344 Epoch: 12 Global Step: 129275 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:38:31,016-Speed 131.27 samples/sec Loss 3.5064 Epoch: 12 Global Step: 129280 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:38:50,863-Speed 129.00 samples/sec Loss 3.2702 Epoch: 12 Global Step: 129285 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:39:10,435-Speed 130.81 samples/sec Loss 3.5294 Epoch: 12 Global Step: 129290 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:39:30,324-Speed 128.72 samples/sec Loss 3.3738 Epoch: 12 Global Step: 129295 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:39:49,884-Speed 130.89 samples/sec Loss 3.1977 Epoch: 12 Global Step: 129300 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:40:09,851-Speed 128.21 samples/sec Loss 3.3441 Epoch: 12 Global Step: 129305 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:40:29,780-Speed 128.47 samples/sec Loss 3.3841 Epoch: 12 Global Step: 129310 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:40:49,357-Speed 130.77 samples/sec Loss 3.3400 Epoch: 12 Global Step: 129315 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:41:09,110-Speed 129.61 samples/sec Loss 3.5330 Epoch: 12 Global Step: 129320 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:41:29,877-Speed 123.28 samples/sec Loss 3.3559 Epoch: 12 Global Step: 129325 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:41:49,660-Speed 129.41 samples/sec Loss 3.4810 Epoch: 12 Global Step: 129330 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:42:09,203-Speed 131.01 samples/sec Loss 3.4733 Epoch: 12 Global Step: 129335 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:42:29,036-Speed 129.08 samples/sec Loss 3.4130 Epoch: 12 Global Step: 129340 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:42:48,569-Speed 131.07 samples/sec Loss 3.1570 Epoch: 12 Global Step: 129345 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:43:08,585-Speed 127.90 samples/sec Loss 3.3306 Epoch: 12 Global Step: 129350 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:43:28,612-Speed 127.83 samples/sec Loss 3.3813 Epoch: 12 Global Step: 129355 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:43:48,158-Speed 130.98 samples/sec Loss 3.4475 Epoch: 12 Global Step: 129360 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:44:07,938-Speed 129.43 samples/sec Loss 3.4102 Epoch: 12 Global Step: 129365 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:44:27,477-Speed 131.03 samples/sec Loss 3.6609 Epoch: 12 Global Step: 129370 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:44:47,216-Speed 129.70 samples/sec Loss 3.5231 Epoch: 12 Global Step: 129375 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:45:06,834-Speed 130.50 samples/sec Loss 3.3242 Epoch: 12 Global Step: 129380 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:45:26,495-Speed 130.21 samples/sec Loss 3.6198 Epoch: 12 Global Step: 129385 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:45:46,094-Speed 130.63 samples/sec Loss 3.6383 Epoch: 12 Global Step: 129390 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:46:06,104-Speed 127.94 samples/sec Loss 3.4884 Epoch: 12 Global Step: 129395 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:46:26,061-Speed 128.28 samples/sec Loss 3.3659 Epoch: 12 Global Step: 129400 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:46:45,569-Speed 131.23 samples/sec Loss 3.2205 Epoch: 12 Global Step: 129405 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:47:05,379-Speed 129.23 samples/sec Loss 3.4527 Epoch: 12 Global Step: 129410 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:47:24,998-Speed 130.50 samples/sec Loss 3.6704 Epoch: 12 Global Step: 129415 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:47:44,772-Speed 129.46 samples/sec Loss 3.5752 Epoch: 12 Global Step: 129420 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:48:04,338-Speed 130.85 samples/sec Loss 3.4496 Epoch: 12 Global Step: 129425 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:48:24,096-Speed 129.58 samples/sec Loss 3.4812 Epoch: 12 Global Step: 129430 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:48:43,735-Speed 130.36 samples/sec Loss 3.3706 Epoch: 12 Global Step: 129435 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:49:03,913-Speed 126.87 samples/sec Loss 3.4412 Epoch: 12 Global Step: 129440 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:49:23,616-Speed 129.94 samples/sec Loss 3.3119 Epoch: 12 Global Step: 129445 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:49:43,117-Speed 131.28 samples/sec Loss 3.4574 Epoch: 12 Global Step: 129450 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:50:02,907-Speed 129.36 samples/sec Loss 3.4507 Epoch: 12 Global Step: 129455 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:50:22,532-Speed 130.46 samples/sec Loss 3.4231 Epoch: 12 Global Step: 129460 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:50:42,429-Speed 128.67 samples/sec Loss 3.4561 Epoch: 12 Global Step: 129465 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:51:02,038-Speed 130.56 samples/sec Loss 3.2968 Epoch: 12 Global Step: 129470 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:51:21,730-Speed 130.01 samples/sec Loss 3.2432 Epoch: 12 Global Step: 129475 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:51:41,305-Speed 130.79 samples/sec Loss 3.5069 Epoch: 12 Global Step: 129480 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:52:01,482-Speed 126.89 samples/sec Loss 3.4194 Epoch: 12 Global Step: 129485 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:52:21,202-Speed 129.82 samples/sec Loss 3.3603 Epoch: 12 Global Step: 129490 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:52:40,813-Speed 130.54 samples/sec Loss 3.5965 Epoch: 12 Global Step: 129495 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:53:00,578-Speed 129.53 samples/sec Loss 3.3568 Epoch: 12 Global Step: 129500 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:53:20,162-Speed 130.72 samples/sec Loss 3.5594 Epoch: 12 Global Step: 129505 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:53:39,982-Speed 129.17 samples/sec Loss 3.3221 Epoch: 12 Global Step: 129510 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:53:59,621-Speed 130.36 samples/sec Loss 3.5114 Epoch: 12 Global Step: 129515 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:54:19,420-Speed 129.31 samples/sec Loss 3.5286 Epoch: 12 Global Step: 129520 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:54:39,232-Speed 129.22 samples/sec Loss 3.3016 Epoch: 12 Global Step: 129525 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:54:59,161-Speed 128.46 samples/sec Loss 3.3840 Epoch: 12 Global Step: 129530 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:55:42,554-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-13 03:56:01,478-Speed 41.08 samples/sec Loss 3.0858 Epoch: 13 Global Step: 129535 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:56:21,807-Speed 125.94 samples/sec Loss 2.6888 Epoch: 13 Global Step: 129540 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:56:41,470-Speed 130.20 samples/sec Loss 2.7448 Epoch: 13 Global Step: 129545 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:57:01,639-Speed 126.93 samples/sec Loss 2.8285 Epoch: 13 Global Step: 129550 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:57:21,367-Speed 129.77 samples/sec Loss 2.7720 Epoch: 13 Global Step: 129555 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:57:41,262-Speed 128.68 samples/sec Loss 2.5779 Epoch: 13 Global Step: 129560 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:58:01,039-Speed 129.45 samples/sec Loss 2.6629 Epoch: 13 Global Step: 129565 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:58:20,963-Speed 128.50 samples/sec Loss 2.9667 Epoch: 13 Global Step: 129570 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:58:40,574-Speed 130.54 samples/sec Loss 2.6767 Epoch: 13 Global Step: 129575 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:59:00,634-Speed 127.62 samples/sec Loss 2.7055 Epoch: 13 Global Step: 129580 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:59:20,723-Speed 127.44 samples/sec Loss 2.8299 Epoch: 13 Global Step: 129585 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 03:59:40,651-Speed 128.47 samples/sec Loss 2.7448 Epoch: 13 Global Step: 129590 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:00:00,401-Speed 129.63 samples/sec Loss 2.7801 Epoch: 13 Global Step: 129595 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:00:20,299-Speed 128.66 samples/sec Loss 2.8537 Epoch: 13 Global Step: 129600 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:00:40,160-Speed 128.90 samples/sec Loss 2.8273 Epoch: 13 Global Step: 129605 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:00:59,971-Speed 129.23 samples/sec Loss 2.6563 Epoch: 13 Global Step: 129610 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:01:19,856-Speed 128.75 samples/sec Loss 2.8119 Epoch: 13 Global Step: 129615 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:01:39,569-Speed 129.87 samples/sec Loss 2.7584 Epoch: 13 Global Step: 129620 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:01:59,712-Speed 127.09 samples/sec Loss 2.6255 Epoch: 13 Global Step: 129625 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:02:19,755-Speed 127.74 samples/sec Loss 2.7389 Epoch: 13 Global Step: 129630 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:02:39,640-Speed 128.74 samples/sec Loss 2.6758 Epoch: 13 Global Step: 129635 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:02:59,374-Speed 129.74 samples/sec Loss 2.7606 Epoch: 13 Global Step: 129640 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:03:19,324-Speed 128.33 samples/sec Loss 2.6568 Epoch: 13 Global Step: 129645 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:03:38,989-Speed 130.18 samples/sec Loss 2.6949 Epoch: 13 Global Step: 129650 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:03:58,782-Speed 129.35 samples/sec Loss 2.5020 Epoch: 13 Global Step: 129655 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:04:18,534-Speed 129.61 samples/sec Loss 2.5975 Epoch: 13 Global Step: 129660 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:04:38,444-Speed 128.58 samples/sec Loss 2.8750 Epoch: 13 Global Step: 129665 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:04:58,388-Speed 128.36 samples/sec Loss 2.7614 Epoch: 13 Global Step: 129670 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:05:18,485-Speed 127.39 samples/sec Loss 2.9112 Epoch: 13 Global Step: 129675 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:05:38,166-Speed 130.08 samples/sec Loss 2.7357 Epoch: 13 Global Step: 129680 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:05:57,984-Speed 129.18 samples/sec Loss 2.8119 Epoch: 13 Global Step: 129685 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:06:17,831-Speed 128.99 samples/sec Loss 2.6169 Epoch: 13 Global Step: 129690 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:06:37,465-Speed 130.40 samples/sec Loss 2.8668 Epoch: 13 Global Step: 129695 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:06:57,369-Speed 128.62 samples/sec Loss 2.7367 Epoch: 13 Global Step: 129700 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:07:17,107-Speed 129.71 samples/sec Loss 2.7851 Epoch: 13 Global Step: 129705 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:07:37,265-Speed 127.00 samples/sec Loss 2.6559 Epoch: 13 Global Step: 129710 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:07:57,220-Speed 128.29 samples/sec Loss 2.7469 Epoch: 13 Global Step: 129715 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:08:17,308-Speed 127.45 samples/sec Loss 3.0211 Epoch: 13 Global Step: 129720 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:08:37,059-Speed 129.61 samples/sec Loss 2.5668 Epoch: 13 Global Step: 129725 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:08:56,906-Speed 128.99 samples/sec Loss 2.7508 Epoch: 13 Global Step: 129730 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:09:16,503-Speed 130.64 samples/sec Loss 2.7216 Epoch: 13 Global Step: 129735 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:09:36,449-Speed 128.35 samples/sec Loss 2.9848 Epoch: 13 Global Step: 129740 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:09:56,175-Speed 129.78 samples/sec Loss 2.7190 Epoch: 13 Global Step: 129745 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:10:16,025-Speed 128.98 samples/sec Loss 2.7365 Epoch: 13 Global Step: 129750 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:10:35,881-Speed 128.93 samples/sec Loss 2.7278 Epoch: 13 Global Step: 129755 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:10:55,724-Speed 129.01 samples/sec Loss 2.9657 Epoch: 13 Global Step: 129760 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:11:15,818-Speed 127.41 samples/sec Loss 2.6216 Epoch: 13 Global Step: 129765 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:11:35,695-Speed 128.79 samples/sec Loss 2.6446 Epoch: 13 Global Step: 129770 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:11:55,499-Speed 129.27 samples/sec Loss 2.7949 Epoch: 13 Global Step: 129775 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:12:15,170-Speed 130.15 samples/sec Loss 2.8565 Epoch: 13 Global Step: 129780 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:12:35,053-Speed 128.76 samples/sec Loss 2.6366 Epoch: 13 Global Step: 129785 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:12:54,782-Speed 129.77 samples/sec Loss 2.7718 Epoch: 13 Global Step: 129790 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:13:14,746-Speed 128.24 samples/sec Loss 2.7286 Epoch: 13 Global Step: 129795 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:13:34,400-Speed 130.25 samples/sec Loss 2.7047 Epoch: 13 Global Step: 129800 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:13:54,589-Speed 126.81 samples/sec Loss 2.9684 Epoch: 13 Global Step: 129805 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:14:14,411-Speed 129.16 samples/sec Loss 2.8184 Epoch: 13 Global Step: 129810 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:14:34,311-Speed 128.65 samples/sec Loss 2.7429 Epoch: 13 Global Step: 129815 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:14:54,102-Speed 129.35 samples/sec Loss 2.7871 Epoch: 13 Global Step: 129820 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:15:14,010-Speed 128.60 samples/sec Loss 2.8134 Epoch: 13 Global Step: 129825 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:15:33,752-Speed 129.68 samples/sec Loss 2.6896 Epoch: 13 Global Step: 129830 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:15:53,461-Speed 129.90 samples/sec Loss 2.7444 Epoch: 13 Global Step: 129835 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:16:13,362-Speed 128.64 samples/sec Loss 2.8474 Epoch: 13 Global Step: 129840 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:16:33,132-Speed 129.50 samples/sec Loss 2.7607 Epoch: 13 Global Step: 129845 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:16:53,297-Speed 126.96 samples/sec Loss 2.7500 Epoch: 13 Global Step: 129850 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:17:13,241-Speed 128.36 samples/sec Loss 2.8021 Epoch: 13 Global Step: 129855 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:17:33,041-Speed 129.30 samples/sec Loss 2.6691 Epoch: 13 Global Step: 129860 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:17:52,809-Speed 129.51 samples/sec Loss 2.8918 Epoch: 13 Global Step: 129865 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:18:12,686-Speed 128.80 samples/sec Loss 2.7218 Epoch: 13 Global Step: 129870 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:18:32,351-Speed 130.19 samples/sec Loss 2.7155 Epoch: 13 Global Step: 129875 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:18:52,296-Speed 128.36 samples/sec Loss 2.8320 Epoch: 13 Global Step: 129880 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:19:11,976-Speed 130.09 samples/sec Loss 2.7544 Epoch: 13 Global Step: 129885 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:19:31,845-Speed 128.85 samples/sec Loss 2.8211 Epoch: 13 Global Step: 129890 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:19:51,755-Speed 128.58 samples/sec Loss 2.8420 Epoch: 13 Global Step: 129895 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:20:11,904-Speed 127.06 samples/sec Loss 2.6489 Epoch: 13 Global Step: 129900 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:20:32,010-Speed 127.33 samples/sec Loss 2.6925 Epoch: 13 Global Step: 129905 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:20:51,876-Speed 128.87 samples/sec Loss 2.7992 Epoch: 13 Global Step: 129910 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:21:11,605-Speed 129.76 samples/sec Loss 3.0084 Epoch: 13 Global Step: 129915 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:21:31,449-Speed 129.02 samples/sec Loss 2.8120 Epoch: 13 Global Step: 129920 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:21:51,180-Speed 129.75 samples/sec Loss 2.7802 Epoch: 13 Global Step: 129925 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:22:11,096-Speed 128.54 samples/sec Loss 2.8766 Epoch: 13 Global Step: 129930 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:22:30,807-Speed 129.88 samples/sec Loss 2.7879 Epoch: 13 Global Step: 129935 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:22:50,712-Speed 128.62 samples/sec Loss 2.6683 Epoch: 13 Global Step: 129940 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:23:10,815-Speed 127.35 samples/sec Loss 2.5740 Epoch: 13 Global Step: 129945 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:23:30,718-Speed 128.63 samples/sec Loss 2.9046 Epoch: 13 Global Step: 129950 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:23:50,424-Speed 129.91 samples/sec Loss 2.8424 Epoch: 13 Global Step: 129955 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:24:10,264-Speed 129.04 samples/sec Loss 2.8308 Epoch: 13 Global Step: 129960 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:24:29,943-Speed 130.10 samples/sec Loss 2.8483 Epoch: 13 Global Step: 129965 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:24:49,728-Speed 129.40 samples/sec Loss 2.8262 Epoch: 13 Global Step: 129970 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:25:09,593-Speed 128.88 samples/sec Loss 2.9198 Epoch: 13 Global Step: 129975 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:25:29,259-Speed 130.18 samples/sec Loss 2.6662 Epoch: 13 Global Step: 129980 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:25:49,564-Speed 126.08 samples/sec Loss 2.7563 Epoch: 13 Global Step: 129985 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:26:09,632-Speed 127.57 samples/sec Loss 2.8480 Epoch: 13 Global Step: 129990 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:26:29,347-Speed 129.86 samples/sec Loss 2.7972 Epoch: 13 Global Step: 129995 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:26:49,002-Speed 130.25 samples/sec Loss 2.7874 Epoch: 13 Global Step: 130000 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 04:41:32,349-[lfw][130000]XNorm: 22.214082 Training: 2021-06-13 04:41:32,350-[lfw][130000]Accuracy-Flip: 0.99783+-0.00224 Training: 2021-06-13 04:41:32,350-[lfw][130000]Accuracy-Highest: 0.99833 Training: 2021-06-13 04:58:43,630-[cfp_fp][130000]XNorm: 21.996674 Training: 2021-06-13 04:58:43,631-[cfp_fp][130000]Accuracy-Flip: 0.99071+-0.00321 Training: 2021-06-13 04:58:43,631-[cfp_fp][130000]Accuracy-Highest: 0.99071 Training: 2021-06-13 05:13:26,548-[agedb_30][130000]XNorm: 22.577875 Training: 2021-06-13 05:13:26,548-[agedb_30][130000]Accuracy-Flip: 0.98233+-0.00688 Training: 2021-06-13 05:13:26,548-[agedb_30][130000]Accuracy-Highest: 0.98233 Training: 2021-06-13 05:13:46,565-Speed 0.91 samples/sec Loss 2.8035 Epoch: 13 Global Step: 130005 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:14:06,331-Speed 129.52 samples/sec Loss 2.8118 Epoch: 13 Global Step: 130010 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:14:25,955-Speed 130.46 samples/sec Loss 2.7452 Epoch: 13 Global Step: 130015 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:14:45,779-Speed 129.14 samples/sec Loss 2.7650 Epoch: 13 Global Step: 130020 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:15:05,484-Speed 129.93 samples/sec Loss 2.7764 Epoch: 13 Global Step: 130025 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:15:25,725-Speed 126.48 samples/sec Loss 2.9530 Epoch: 13 Global Step: 130030 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:15:45,436-Speed 129.88 samples/sec Loss 2.9937 Epoch: 13 Global Step: 130035 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:16:05,567-Speed 127.18 samples/sec Loss 2.8397 Epoch: 13 Global Step: 130040 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:16:25,269-Speed 129.94 samples/sec Loss 2.8004 Epoch: 13 Global Step: 130045 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:16:45,088-Speed 129.17 samples/sec Loss 2.7473 Epoch: 13 Global Step: 130050 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:17:04,720-Speed 130.41 samples/sec Loss 2.9437 Epoch: 13 Global Step: 130055 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:17:24,530-Speed 129.23 samples/sec Loss 2.8182 Epoch: 13 Global Step: 130060 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:17:44,265-Speed 129.73 samples/sec Loss 2.5838 Epoch: 13 Global Step: 130065 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:18:04,076-Speed 129.22 samples/sec Loss 2.7319 Epoch: 13 Global Step: 130070 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:18:23,945-Speed 128.85 samples/sec Loss 2.9113 Epoch: 13 Global Step: 130075 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:18:43,595-Speed 130.29 samples/sec Loss 2.8686 Epoch: 13 Global Step: 130080 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:19:03,692-Speed 127.39 samples/sec Loss 2.7274 Epoch: 13 Global Step: 130085 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:19:23,605-Speed 128.57 samples/sec Loss 2.9628 Epoch: 13 Global Step: 130090 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:19:43,251-Speed 130.31 samples/sec Loss 2.8729 Epoch: 13 Global Step: 130095 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:20:03,077-Speed 129.13 samples/sec Loss 2.7070 Epoch: 13 Global Step: 130100 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:20:22,704-Speed 130.44 samples/sec Loss 2.8355 Epoch: 13 Global Step: 130105 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:20:42,609-Speed 128.62 samples/sec Loss 2.7446 Epoch: 13 Global Step: 130110 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:21:02,493-Speed 128.75 samples/sec Loss 2.7464 Epoch: 13 Global Step: 130115 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:21:22,423-Speed 128.46 samples/sec Loss 2.7043 Epoch: 13 Global Step: 130120 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:21:42,380-Speed 128.28 samples/sec Loss 2.9494 Epoch: 13 Global Step: 130125 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:22:02,457-Speed 127.52 samples/sec Loss 2.8564 Epoch: 13 Global Step: 130130 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:22:22,190-Speed 129.74 samples/sec Loss 2.8397 Epoch: 13 Global Step: 130135 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:22:41,866-Speed 130.11 samples/sec Loss 2.7839 Epoch: 13 Global Step: 130140 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:23:01,654-Speed 129.38 samples/sec Loss 3.0055 Epoch: 13 Global Step: 130145 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:23:21,569-Speed 128.55 samples/sec Loss 2.9407 Epoch: 13 Global Step: 130150 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:23:41,457-Speed 128.73 samples/sec Loss 2.8103 Epoch: 13 Global Step: 130155 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:24:01,183-Speed 129.78 samples/sec Loss 3.0483 Epoch: 13 Global Step: 130160 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:24:21,268-Speed 127.46 samples/sec Loss 2.9298 Epoch: 13 Global Step: 130165 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:24:41,073-Speed 129.27 samples/sec Loss 2.6989 Epoch: 13 Global Step: 130170 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:25:01,092-Speed 127.89 samples/sec Loss 2.7784 Epoch: 13 Global Step: 130175 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:25:20,814-Speed 129.81 samples/sec Loss 2.8515 Epoch: 13 Global Step: 130180 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:25:40,611-Speed 129.32 samples/sec Loss 2.8846 Epoch: 13 Global Step: 130185 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:26:00,260-Speed 130.29 samples/sec Loss 3.1349 Epoch: 13 Global Step: 130190 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:26:20,114-Speed 128.95 samples/sec Loss 2.8708 Epoch: 13 Global Step: 130195 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:26:39,941-Speed 129.13 samples/sec Loss 2.7788 Epoch: 13 Global Step: 130200 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:26:59,648-Speed 129.91 samples/sec Loss 2.7708 Epoch: 13 Global Step: 130205 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:27:19,726-Speed 127.51 samples/sec Loss 2.9198 Epoch: 13 Global Step: 130210 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:27:39,462-Speed 129.72 samples/sec Loss 2.9879 Epoch: 13 Global Step: 130215 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:27:59,502-Speed 127.75 samples/sec Loss 2.9735 Epoch: 13 Global Step: 130220 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:28:19,301-Speed 129.31 samples/sec Loss 3.0627 Epoch: 13 Global Step: 130225 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:28:38,974-Speed 130.14 samples/sec Loss 3.0309 Epoch: 13 Global Step: 130230 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:28:58,810-Speed 129.06 samples/sec Loss 2.8085 Epoch: 13 Global Step: 130235 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:29:18,686-Speed 128.81 samples/sec Loss 2.9292 Epoch: 13 Global Step: 130240 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:29:38,404-Speed 129.83 samples/sec Loss 3.0684 Epoch: 13 Global Step: 130245 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:29:58,182-Speed 129.45 samples/sec Loss 2.8993 Epoch: 13 Global Step: 130250 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:30:18,173-Speed 128.06 samples/sec Loss 2.9788 Epoch: 13 Global Step: 130255 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:30:37,949-Speed 129.46 samples/sec Loss 2.9397 Epoch: 13 Global Step: 130260 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:30:57,876-Speed 128.48 samples/sec Loss 2.6956 Epoch: 13 Global Step: 130265 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:31:17,663-Speed 129.38 samples/sec Loss 2.9848 Epoch: 13 Global Step: 130270 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:31:37,364-Speed 129.95 samples/sec Loss 2.7800 Epoch: 13 Global Step: 130275 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:31:57,294-Speed 128.45 samples/sec Loss 2.9270 Epoch: 13 Global Step: 130280 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:32:17,028-Speed 129.74 samples/sec Loss 2.6808 Epoch: 13 Global Step: 130285 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:32:36,689-Speed 130.21 samples/sec Loss 2.7710 Epoch: 13 Global Step: 130290 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:32:56,477-Speed 129.38 samples/sec Loss 2.7904 Epoch: 13 Global Step: 130295 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:33:16,542-Speed 127.59 samples/sec Loss 3.0593 Epoch: 13 Global Step: 130300 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:33:36,584-Speed 127.74 samples/sec Loss 2.7874 Epoch: 13 Global Step: 130305 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:33:56,404-Speed 129.17 samples/sec Loss 2.9531 Epoch: 13 Global Step: 130310 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:34:16,141-Speed 129.71 samples/sec Loss 3.0038 Epoch: 13 Global Step: 130315 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:34:35,936-Speed 129.33 samples/sec Loss 2.8578 Epoch: 13 Global Step: 130320 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:34:55,845-Speed 128.59 samples/sec Loss 2.8850 Epoch: 13 Global Step: 130325 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:35:15,440-Speed 130.66 samples/sec Loss 2.9004 Epoch: 13 Global Step: 130330 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:35:35,242-Speed 129.28 samples/sec Loss 3.0276 Epoch: 13 Global Step: 130335 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:35:54,962-Speed 129.82 samples/sec Loss 2.9131 Epoch: 13 Global Step: 130340 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:36:15,212-Speed 126.42 samples/sec Loss 2.9819 Epoch: 13 Global Step: 130345 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:36:34,991-Speed 129.44 samples/sec Loss 3.0068 Epoch: 13 Global Step: 130350 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:36:54,937-Speed 128.35 samples/sec Loss 2.8687 Epoch: 13 Global Step: 130355 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:37:14,667-Speed 129.76 samples/sec Loss 2.9006 Epoch: 13 Global Step: 130360 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:37:34,639-Speed 128.18 samples/sec Loss 2.9645 Epoch: 13 Global Step: 130365 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:37:54,252-Speed 130.53 samples/sec Loss 2.9237 Epoch: 13 Global Step: 130370 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:38:13,964-Speed 129.87 samples/sec Loss 2.8002 Epoch: 13 Global Step: 130375 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:38:33,776-Speed 129.23 samples/sec Loss 2.8426 Epoch: 13 Global Step: 130380 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:38:53,608-Speed 129.09 samples/sec Loss 2.9530 Epoch: 13 Global Step: 130385 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:39:13,675-Speed 127.58 samples/sec Loss 2.8744 Epoch: 13 Global Step: 130390 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:39:33,278-Speed 130.60 samples/sec Loss 2.8991 Epoch: 13 Global Step: 130395 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:39:53,350-Speed 127.55 samples/sec Loss 2.9179 Epoch: 13 Global Step: 130400 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:40:13,159-Speed 129.24 samples/sec Loss 2.8333 Epoch: 13 Global Step: 130405 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:40:33,036-Speed 128.80 samples/sec Loss 2.8579 Epoch: 13 Global Step: 130410 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:40:52,657-Speed 130.48 samples/sec Loss 2.9474 Epoch: 13 Global Step: 130415 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:41:12,524-Speed 128.87 samples/sec Loss 2.9484 Epoch: 13 Global Step: 130420 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:41:32,310-Speed 129.39 samples/sec Loss 2.7653 Epoch: 13 Global Step: 130425 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:41:52,282-Speed 128.19 samples/sec Loss 3.0318 Epoch: 13 Global Step: 130430 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:42:12,154-Speed 128.83 samples/sec Loss 2.7770 Epoch: 13 Global Step: 130435 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:42:31,928-Speed 129.47 samples/sec Loss 2.8916 Epoch: 13 Global Step: 130440 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:42:52,041-Speed 127.29 samples/sec Loss 3.1084 Epoch: 13 Global Step: 130445 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:43:11,960-Speed 128.53 samples/sec Loss 2.7736 Epoch: 13 Global Step: 130450 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:43:31,604-Speed 130.33 samples/sec Loss 3.0491 Epoch: 13 Global Step: 130455 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:43:51,512-Speed 128.60 samples/sec Loss 2.8578 Epoch: 13 Global Step: 130460 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:44:11,247-Speed 129.73 samples/sec Loss 2.9030 Epoch: 13 Global Step: 130465 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:44:31,085-Speed 129.05 samples/sec Loss 2.8962 Epoch: 13 Global Step: 130470 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:44:51,049-Speed 128.24 samples/sec Loss 3.0733 Epoch: 13 Global Step: 130475 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:45:10,956-Speed 128.60 samples/sec Loss 3.1182 Epoch: 13 Global Step: 130480 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:45:30,899-Speed 128.37 samples/sec Loss 2.7903 Epoch: 13 Global Step: 130485 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:45:50,812-Speed 128.57 samples/sec Loss 2.9927 Epoch: 13 Global Step: 130490 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:46:10,619-Speed 129.25 samples/sec Loss 2.9586 Epoch: 13 Global Step: 130495 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:46:30,221-Speed 130.61 samples/sec Loss 3.0133 Epoch: 13 Global Step: 130500 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:46:50,126-Speed 128.62 samples/sec Loss 2.8396 Epoch: 13 Global Step: 130505 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:47:09,832-Speed 129.91 samples/sec Loss 2.8202 Epoch: 13 Global Step: 130510 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:47:29,627-Speed 129.33 samples/sec Loss 2.8318 Epoch: 13 Global Step: 130515 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:47:49,388-Speed 129.55 samples/sec Loss 3.0342 Epoch: 13 Global Step: 130520 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:48:09,407-Speed 127.88 samples/sec Loss 3.0747 Epoch: 13 Global Step: 130525 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:48:29,337-Speed 128.46 samples/sec Loss 2.9188 Epoch: 13 Global Step: 130530 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:48:49,429-Speed 127.42 samples/sec Loss 2.9837 Epoch: 13 Global Step: 130535 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:49:09,028-Speed 130.62 samples/sec Loss 3.1381 Epoch: 13 Global Step: 130540 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:49:28,729-Speed 129.95 samples/sec Loss 3.0226 Epoch: 13 Global Step: 130545 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:49:48,674-Speed 128.36 samples/sec Loss 2.8313 Epoch: 13 Global Step: 130550 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:50:08,393-Speed 129.83 samples/sec Loss 2.8382 Epoch: 13 Global Step: 130555 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:50:28,300-Speed 128.60 samples/sec Loss 3.0598 Epoch: 13 Global Step: 130560 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:50:48,054-Speed 129.60 samples/sec Loss 2.9632 Epoch: 13 Global Step: 130565 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:51:08,182-Speed 127.19 samples/sec Loss 2.9007 Epoch: 13 Global Step: 130570 Fp16 Grad Scale: 8192 Required: 141 hours Training: 2021-06-13 05:51:27,908-Speed 129.78 samples/sec Loss 3.0298 Epoch: 13 Global Step: 130575 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:51:47,979-Speed 127.56 samples/sec Loss 2.9685 Epoch: 13 Global Step: 130580 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:52:07,721-Speed 129.68 samples/sec Loss 2.8113 Epoch: 13 Global Step: 130585 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:52:27,541-Speed 129.16 samples/sec Loss 2.9283 Epoch: 13 Global Step: 130590 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:52:47,514-Speed 128.18 samples/sec Loss 3.1308 Epoch: 13 Global Step: 130595 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:53:07,122-Speed 130.56 samples/sec Loss 2.9596 Epoch: 13 Global Step: 130600 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:53:27,041-Speed 128.53 samples/sec Loss 3.0185 Epoch: 13 Global Step: 130605 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:53:46,760-Speed 129.83 samples/sec Loss 2.9032 Epoch: 13 Global Step: 130610 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:54:07,080-Speed 125.99 samples/sec Loss 2.9881 Epoch: 13 Global Step: 130615 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:54:26,815-Speed 129.73 samples/sec Loss 3.0965 Epoch: 13 Global Step: 130620 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:54:46,814-Speed 128.01 samples/sec Loss 2.9950 Epoch: 13 Global Step: 130625 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:55:06,627-Speed 129.21 samples/sec Loss 3.0925 Epoch: 13 Global Step: 130630 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:55:26,403-Speed 129.46 samples/sec Loss 2.8380 Epoch: 13 Global Step: 130635 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:55:46,426-Speed 127.86 samples/sec Loss 3.0897 Epoch: 13 Global Step: 130640 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:56:06,320-Speed 128.69 samples/sec Loss 2.9745 Epoch: 13 Global Step: 130645 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:56:26,040-Speed 129.82 samples/sec Loss 2.8742 Epoch: 13 Global Step: 130650 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:56:46,006-Speed 128.22 samples/sec Loss 2.8416 Epoch: 13 Global Step: 130655 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:57:05,829-Speed 129.15 samples/sec Loss 2.9297 Epoch: 13 Global Step: 130660 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:57:25,608-Speed 129.44 samples/sec Loss 2.8987 Epoch: 13 Global Step: 130665 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:57:45,748-Speed 127.12 samples/sec Loss 2.8555 Epoch: 13 Global Step: 130670 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:58:05,646-Speed 128.66 samples/sec Loss 3.0088 Epoch: 13 Global Step: 130675 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:58:25,363-Speed 129.84 samples/sec Loss 2.9615 Epoch: 13 Global Step: 130680 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:58:45,080-Speed 129.85 samples/sec Loss 2.9983 Epoch: 13 Global Step: 130685 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:59:04,882-Speed 129.29 samples/sec Loss 3.0494 Epoch: 13 Global Step: 130690 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:59:24,649-Speed 129.52 samples/sec Loss 3.1924 Epoch: 13 Global Step: 130695 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 05:59:44,520-Speed 128.84 samples/sec Loss 2.9469 Epoch: 13 Global Step: 130700 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:00:04,527-Speed 127.96 samples/sec Loss 2.8756 Epoch: 13 Global Step: 130705 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:00:24,495-Speed 128.21 samples/sec Loss 2.9518 Epoch: 13 Global Step: 130710 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:00:44,431-Speed 128.42 samples/sec Loss 2.9857 Epoch: 13 Global Step: 130715 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:01:04,269-Speed 129.05 samples/sec Loss 2.8816 Epoch: 13 Global Step: 130720 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:01:24,003-Speed 129.73 samples/sec Loss 2.8484 Epoch: 13 Global Step: 130725 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:01:43,915-Speed 128.57 samples/sec Loss 2.9231 Epoch: 13 Global Step: 130730 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:02:03,786-Speed 128.84 samples/sec Loss 3.1490 Epoch: 13 Global Step: 130735 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:02:23,684-Speed 128.67 samples/sec Loss 3.1223 Epoch: 13 Global Step: 130740 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:02:43,327-Speed 130.33 samples/sec Loss 2.7511 Epoch: 13 Global Step: 130745 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:03:03,393-Speed 127.58 samples/sec Loss 3.0432 Epoch: 13 Global Step: 130750 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:03:23,390-Speed 128.03 samples/sec Loss 3.1578 Epoch: 13 Global Step: 130755 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:03:43,300-Speed 128.59 samples/sec Loss 3.1955 Epoch: 13 Global Step: 130760 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:04:03,011-Speed 129.88 samples/sec Loss 3.0862 Epoch: 13 Global Step: 130765 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:04:22,722-Speed 129.88 samples/sec Loss 2.8186 Epoch: 13 Global Step: 130770 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:04:42,687-Speed 128.23 samples/sec Loss 3.0234 Epoch: 13 Global Step: 130775 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:05:02,424-Speed 129.71 samples/sec Loss 3.0755 Epoch: 13 Global Step: 130780 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:05:22,081-Speed 130.24 samples/sec Loss 3.1085 Epoch: 13 Global Step: 130785 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:05:41,942-Speed 128.90 samples/sec Loss 3.0530 Epoch: 13 Global Step: 130790 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:06:01,988-Speed 127.71 samples/sec Loss 2.8811 Epoch: 13 Global Step: 130795 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:06:21,833-Speed 129.00 samples/sec Loss 3.1382 Epoch: 13 Global Step: 130800 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:06:41,725-Speed 128.70 samples/sec Loss 3.0226 Epoch: 13 Global Step: 130805 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:07:01,488-Speed 129.54 samples/sec Loss 3.0832 Epoch: 13 Global Step: 130810 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:07:21,217-Speed 129.76 samples/sec Loss 2.8698 Epoch: 13 Global Step: 130815 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:07:41,116-Speed 128.66 samples/sec Loss 3.0075 Epoch: 13 Global Step: 130820 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:08:00,730-Speed 130.53 samples/sec Loss 2.9789 Epoch: 13 Global Step: 130825 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:08:20,632-Speed 128.64 samples/sec Loss 3.0617 Epoch: 13 Global Step: 130830 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:08:40,334-Speed 129.94 samples/sec Loss 3.1360 Epoch: 13 Global Step: 130835 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:09:00,422-Speed 127.44 samples/sec Loss 3.0034 Epoch: 13 Global Step: 130840 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:09:20,344-Speed 128.51 samples/sec Loss 3.1133 Epoch: 13 Global Step: 130845 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:09:40,300-Speed 128.29 samples/sec Loss 3.0630 Epoch: 13 Global Step: 130850 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:09:59,988-Speed 130.03 samples/sec Loss 3.0425 Epoch: 13 Global Step: 130855 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:10:20,296-Speed 126.07 samples/sec Loss 3.0236 Epoch: 13 Global Step: 130860 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:10:40,155-Speed 128.91 samples/sec Loss 3.1053 Epoch: 13 Global Step: 130865 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:10:59,870-Speed 129.86 samples/sec Loss 3.1538 Epoch: 13 Global Step: 130870 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:11:19,666-Speed 129.32 samples/sec Loss 2.8966 Epoch: 13 Global Step: 130875 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:11:39,451-Speed 129.40 samples/sec Loss 2.8718 Epoch: 13 Global Step: 130880 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:11:59,641-Speed 126.80 samples/sec Loss 3.0729 Epoch: 13 Global Step: 130885 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:12:19,361-Speed 129.83 samples/sec Loss 2.9874 Epoch: 13 Global Step: 130890 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:12:39,339-Speed 128.14 samples/sec Loss 2.8712 Epoch: 13 Global Step: 130895 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:12:59,118-Speed 129.44 samples/sec Loss 3.0457 Epoch: 13 Global Step: 130900 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:13:18,926-Speed 129.25 samples/sec Loss 2.9034 Epoch: 13 Global Step: 130905 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:13:38,648-Speed 129.81 samples/sec Loss 3.1024 Epoch: 13 Global Step: 130910 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:13:58,283-Speed 130.39 samples/sec Loss 2.9081 Epoch: 13 Global Step: 130915 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:14:18,228-Speed 128.36 samples/sec Loss 3.1625 Epoch: 13 Global Step: 130920 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:14:37,930-Speed 129.94 samples/sec Loss 3.1031 Epoch: 13 Global Step: 130925 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:14:58,110-Speed 126.87 samples/sec Loss 3.0951 Epoch: 13 Global Step: 130930 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:15:17,708-Speed 130.64 samples/sec Loss 3.1945 Epoch: 13 Global Step: 130935 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:15:37,863-Speed 127.02 samples/sec Loss 3.1092 Epoch: 13 Global Step: 130940 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:15:57,571-Speed 129.90 samples/sec Loss 3.1110 Epoch: 13 Global Step: 130945 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:16:17,483-Speed 128.57 samples/sec Loss 3.1518 Epoch: 13 Global Step: 130950 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:16:37,371-Speed 128.73 samples/sec Loss 3.0023 Epoch: 13 Global Step: 130955 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:16:57,008-Speed 130.37 samples/sec Loss 2.9230 Epoch: 13 Global Step: 130960 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:17:16,880-Speed 128.83 samples/sec Loss 3.0516 Epoch: 13 Global Step: 130965 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:17:36,565-Speed 130.05 samples/sec Loss 2.9672 Epoch: 13 Global Step: 130970 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:17:56,634-Speed 127.57 samples/sec Loss 3.1154 Epoch: 13 Global Step: 130975 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:18:16,359-Speed 129.79 samples/sec Loss 3.0240 Epoch: 13 Global Step: 130980 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:18:36,480-Speed 127.24 samples/sec Loss 3.0434 Epoch: 13 Global Step: 130985 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:18:56,826-Speed 125.83 samples/sec Loss 3.0215 Epoch: 13 Global Step: 130990 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:19:16,700-Speed 128.82 samples/sec Loss 3.0689 Epoch: 13 Global Step: 130995 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:19:36,397-Speed 129.97 samples/sec Loss 2.9091 Epoch: 13 Global Step: 131000 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:19:56,352-Speed 128.29 samples/sec Loss 2.9938 Epoch: 13 Global Step: 131005 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:20:16,109-Speed 129.59 samples/sec Loss 3.0199 Epoch: 13 Global Step: 131010 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:20:36,005-Speed 128.67 samples/sec Loss 2.9985 Epoch: 13 Global Step: 131015 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:20:55,942-Speed 128.41 samples/sec Loss 2.9903 Epoch: 13 Global Step: 131020 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:21:15,762-Speed 129.17 samples/sec Loss 3.2345 Epoch: 13 Global Step: 131025 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:21:35,802-Speed 127.75 samples/sec Loss 3.0863 Epoch: 13 Global Step: 131030 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:21:55,702-Speed 128.64 samples/sec Loss 3.1925 Epoch: 13 Global Step: 131035 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:22:15,407-Speed 129.92 samples/sec Loss 2.8862 Epoch: 13 Global Step: 131040 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:22:35,360-Speed 128.31 samples/sec Loss 3.1563 Epoch: 13 Global Step: 131045 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:22:55,084-Speed 129.80 samples/sec Loss 2.8215 Epoch: 13 Global Step: 131050 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:23:14,796-Speed 129.87 samples/sec Loss 2.8436 Epoch: 13 Global Step: 131055 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:23:34,593-Speed 129.32 samples/sec Loss 3.3240 Epoch: 13 Global Step: 131060 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:23:54,655-Speed 127.61 samples/sec Loss 3.0513 Epoch: 13 Global Step: 131065 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:24:14,610-Speed 128.30 samples/sec Loss 2.9789 Epoch: 13 Global Step: 131070 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:24:34,588-Speed 128.15 samples/sec Loss 3.0230 Epoch: 13 Global Step: 131075 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:24:54,399-Speed 129.23 samples/sec Loss 3.1237 Epoch: 13 Global Step: 131080 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:25:14,129-Speed 129.76 samples/sec Loss 3.3084 Epoch: 13 Global Step: 131085 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:25:34,102-Speed 128.18 samples/sec Loss 3.1015 Epoch: 13 Global Step: 131090 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:25:53,838-Speed 129.72 samples/sec Loss 3.1617 Epoch: 13 Global Step: 131095 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:26:13,745-Speed 128.61 samples/sec Loss 3.1917 Epoch: 13 Global Step: 131100 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:26:33,466-Speed 129.82 samples/sec Loss 3.1129 Epoch: 13 Global Step: 131105 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:26:53,569-Speed 127.35 samples/sec Loss 3.1842 Epoch: 13 Global Step: 131110 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:27:13,430-Speed 128.90 samples/sec Loss 2.9100 Epoch: 13 Global Step: 131115 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:27:33,404-Speed 128.17 samples/sec Loss 3.1240 Epoch: 13 Global Step: 131120 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:27:53,132-Speed 129.77 samples/sec Loss 3.1030 Epoch: 13 Global Step: 131125 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:28:12,946-Speed 129.21 samples/sec Loss 3.0483 Epoch: 13 Global Step: 131130 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:28:33,448-Speed 124.88 samples/sec Loss 2.9965 Epoch: 13 Global Step: 131135 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:28:54,064-Speed 124.18 samples/sec Loss 2.9414 Epoch: 13 Global Step: 131140 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:29:13,968-Speed 128.62 samples/sec Loss 2.9901 Epoch: 13 Global Step: 131145 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:29:33,751-Speed 129.41 samples/sec Loss 2.8982 Epoch: 13 Global Step: 131150 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:29:53,904-Speed 127.04 samples/sec Loss 3.1430 Epoch: 13 Global Step: 131155 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:30:13,582-Speed 130.10 samples/sec Loss 3.2300 Epoch: 13 Global Step: 131160 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:30:33,673-Speed 127.43 samples/sec Loss 3.3426 Epoch: 13 Global Step: 131165 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:30:53,446-Speed 129.47 samples/sec Loss 3.2975 Epoch: 13 Global Step: 131170 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:31:13,353-Speed 128.61 samples/sec Loss 2.9728 Epoch: 13 Global Step: 131175 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:31:33,068-Speed 129.85 samples/sec Loss 3.0270 Epoch: 13 Global Step: 131180 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:31:52,952-Speed 128.76 samples/sec Loss 3.1710 Epoch: 13 Global Step: 131185 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:32:12,751-Speed 129.30 samples/sec Loss 3.1724 Epoch: 13 Global Step: 131190 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:32:32,418-Speed 130.17 samples/sec Loss 2.9843 Epoch: 13 Global Step: 131195 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:32:52,593-Speed 126.89 samples/sec Loss 2.9936 Epoch: 13 Global Step: 131200 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:33:12,323-Speed 129.76 samples/sec Loss 2.9659 Epoch: 13 Global Step: 131205 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:33:32,427-Speed 127.34 samples/sec Loss 3.1130 Epoch: 13 Global Step: 131210 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:33:52,370-Speed 128.37 samples/sec Loss 3.2134 Epoch: 13 Global Step: 131215 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:34:12,126-Speed 129.58 samples/sec Loss 3.3648 Epoch: 13 Global Step: 131220 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:34:31,834-Speed 129.90 samples/sec Loss 2.9505 Epoch: 13 Global Step: 131225 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:34:51,830-Speed 128.03 samples/sec Loss 3.0649 Epoch: 13 Global Step: 131230 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:35:11,640-Speed 129.24 samples/sec Loss 3.2428 Epoch: 13 Global Step: 131235 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:35:31,419-Speed 129.44 samples/sec Loss 3.0347 Epoch: 13 Global Step: 131240 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:35:51,448-Speed 127.82 samples/sec Loss 2.9796 Epoch: 13 Global Step: 131245 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:36:11,199-Speed 129.62 samples/sec Loss 2.9045 Epoch: 13 Global Step: 131250 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:36:31,154-Speed 128.30 samples/sec Loss 3.2458 Epoch: 13 Global Step: 131255 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:36:51,044-Speed 128.71 samples/sec Loss 2.9990 Epoch: 13 Global Step: 131260 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:37:10,739-Speed 129.99 samples/sec Loss 2.9520 Epoch: 13 Global Step: 131265 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:37:30,723-Speed 128.10 samples/sec Loss 3.0157 Epoch: 13 Global Step: 131270 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:37:50,344-Speed 130.48 samples/sec Loss 3.1849 Epoch: 13 Global Step: 131275 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:38:10,125-Speed 129.43 samples/sec Loss 3.0714 Epoch: 13 Global Step: 131280 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:38:29,983-Speed 128.92 samples/sec Loss 3.0928 Epoch: 13 Global Step: 131285 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:38:50,078-Speed 127.40 samples/sec Loss 3.1392 Epoch: 13 Global Step: 131290 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:39:09,893-Speed 129.20 samples/sec Loss 2.8823 Epoch: 13 Global Step: 131295 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:39:29,818-Speed 128.49 samples/sec Loss 3.0448 Epoch: 13 Global Step: 131300 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:39:49,735-Speed 128.54 samples/sec Loss 3.0547 Epoch: 13 Global Step: 131305 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:40:09,453-Speed 129.84 samples/sec Loss 3.2454 Epoch: 13 Global Step: 131310 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:40:29,423-Speed 128.20 samples/sec Loss 3.1593 Epoch: 13 Global Step: 131315 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:40:49,447-Speed 127.85 samples/sec Loss 2.9178 Epoch: 13 Global Step: 131320 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:41:09,117-Speed 130.16 samples/sec Loss 3.1630 Epoch: 13 Global Step: 131325 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:41:28,992-Speed 128.81 samples/sec Loss 2.9717 Epoch: 13 Global Step: 131330 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:41:49,172-Speed 126.86 samples/sec Loss 3.3036 Epoch: 13 Global Step: 131335 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:42:09,184-Speed 127.93 samples/sec Loss 3.1052 Epoch: 13 Global Step: 131340 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:42:29,221-Speed 127.77 samples/sec Loss 2.9431 Epoch: 13 Global Step: 131345 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:42:49,131-Speed 128.59 samples/sec Loss 2.9511 Epoch: 13 Global Step: 131350 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:43:08,902-Speed 129.49 samples/sec Loss 2.9576 Epoch: 13 Global Step: 131355 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:43:28,795-Speed 128.69 samples/sec Loss 3.1504 Epoch: 13 Global Step: 131360 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:43:48,460-Speed 130.19 samples/sec Loss 3.1014 Epoch: 13 Global Step: 131365 Fp16 Grad Scale: 8192 Required: 140 hours Training: 2021-06-13 06:44:08,367-Speed 128.61 samples/sec Loss 3.0369 Epoch: 13 Global Step: 131370 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:44:28,072-Speed 129.92 samples/sec Loss 3.2735 Epoch: 13 Global Step: 131375 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:44:48,199-Speed 127.20 samples/sec Loss 3.1992 Epoch: 13 Global Step: 131380 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:45:07,951-Speed 129.61 samples/sec Loss 3.2214 Epoch: 13 Global Step: 131385 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:45:28,042-Speed 127.43 samples/sec Loss 3.2250 Epoch: 13 Global Step: 131390 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:45:47,810-Speed 129.51 samples/sec Loss 3.1153 Epoch: 13 Global Step: 131395 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:46:07,507-Speed 129.97 samples/sec Loss 3.1079 Epoch: 13 Global Step: 131400 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:46:28,124-Speed 124.18 samples/sec Loss 3.2653 Epoch: 13 Global Step: 131405 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:46:47,744-Speed 130.48 samples/sec Loss 3.1593 Epoch: 13 Global Step: 131410 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:47:07,668-Speed 128.50 samples/sec Loss 3.3798 Epoch: 13 Global Step: 131415 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:47:27,421-Speed 129.61 samples/sec Loss 3.0319 Epoch: 13 Global Step: 131420 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:47:47,638-Speed 126.63 samples/sec Loss 3.1031 Epoch: 13 Global Step: 131425 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:48:07,346-Speed 129.90 samples/sec Loss 2.9777 Epoch: 13 Global Step: 131430 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:48:27,482-Speed 127.14 samples/sec Loss 3.0599 Epoch: 13 Global Step: 131435 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:48:47,104-Speed 130.47 samples/sec Loss 3.0699 Epoch: 13 Global Step: 131440 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:49:07,010-Speed 128.61 samples/sec Loss 3.0448 Epoch: 13 Global Step: 131445 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:49:26,670-Speed 130.22 samples/sec Loss 3.1870 Epoch: 13 Global Step: 131450 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:49:46,396-Speed 129.79 samples/sec Loss 3.1049 Epoch: 13 Global Step: 131455 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:50:06,374-Speed 128.15 samples/sec Loss 3.1119 Epoch: 13 Global Step: 131460 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:50:25,990-Speed 130.52 samples/sec Loss 3.0509 Epoch: 13 Global Step: 131465 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:50:46,092-Speed 127.35 samples/sec Loss 3.0417 Epoch: 13 Global Step: 131470 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:51:05,803-Speed 129.89 samples/sec Loss 3.2236 Epoch: 13 Global Step: 131475 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:51:25,902-Speed 127.37 samples/sec Loss 3.0652 Epoch: 13 Global Step: 131480 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:51:45,605-Speed 129.93 samples/sec Loss 2.8539 Epoch: 13 Global Step: 131485 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:52:05,506-Speed 128.64 samples/sec Loss 3.0654 Epoch: 13 Global Step: 131490 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:52:25,224-Speed 129.84 samples/sec Loss 3.1603 Epoch: 13 Global Step: 131495 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:52:45,194-Speed 128.20 samples/sec Loss 3.2257 Epoch: 13 Global Step: 131500 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:53:04,817-Speed 130.47 samples/sec Loss 3.2478 Epoch: 13 Global Step: 131505 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:53:24,572-Speed 129.59 samples/sec Loss 3.1081 Epoch: 13 Global Step: 131510 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:53:44,649-Speed 127.52 samples/sec Loss 3.0974 Epoch: 13 Global Step: 131515 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:54:04,489-Speed 129.04 samples/sec Loss 3.0162 Epoch: 13 Global Step: 131520 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:54:24,465-Speed 128.16 samples/sec Loss 3.1856 Epoch: 13 Global Step: 131525 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:54:44,312-Speed 128.99 samples/sec Loss 3.2581 Epoch: 13 Global Step: 131530 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:55:04,242-Speed 128.46 samples/sec Loss 3.1914 Epoch: 13 Global Step: 131535 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:55:23,993-Speed 129.62 samples/sec Loss 3.1655 Epoch: 13 Global Step: 131540 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:55:43,869-Speed 128.80 samples/sec Loss 3.2692 Epoch: 13 Global Step: 131545 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:56:03,601-Speed 129.75 samples/sec Loss 3.3004 Epoch: 13 Global Step: 131550 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:56:23,501-Speed 128.64 samples/sec Loss 3.1308 Epoch: 13 Global Step: 131555 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:56:43,517-Speed 127.91 samples/sec Loss 3.1895 Epoch: 13 Global Step: 131560 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:57:03,243-Speed 129.78 samples/sec Loss 3.0357 Epoch: 13 Global Step: 131565 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:57:23,279-Speed 127.78 samples/sec Loss 3.1508 Epoch: 13 Global Step: 131570 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:57:44,061-Speed 123.19 samples/sec Loss 2.9674 Epoch: 13 Global Step: 131575 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:58:03,794-Speed 129.74 samples/sec Loss 3.1657 Epoch: 13 Global Step: 131580 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:58:23,742-Speed 128.34 samples/sec Loss 2.9915 Epoch: 13 Global Step: 131585 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:58:43,453-Speed 129.88 samples/sec Loss 3.0664 Epoch: 13 Global Step: 131590 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:59:03,143-Speed 130.02 samples/sec Loss 3.0409 Epoch: 13 Global Step: 131595 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:59:22,950-Speed 129.25 samples/sec Loss 3.1551 Epoch: 13 Global Step: 131600 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 06:59:42,953-Speed 127.99 samples/sec Loss 3.2294 Epoch: 13 Global Step: 131605 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:00:02,963-Speed 127.94 samples/sec Loss 3.1448 Epoch: 13 Global Step: 131610 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:00:22,907-Speed 128.37 samples/sec Loss 3.1408 Epoch: 13 Global Step: 131615 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:00:42,749-Speed 129.02 samples/sec Loss 3.0913 Epoch: 13 Global Step: 131620 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:01:02,521-Speed 129.48 samples/sec Loss 3.1121 Epoch: 13 Global Step: 131625 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:01:22,420-Speed 128.66 samples/sec Loss 3.0327 Epoch: 13 Global Step: 131630 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:01:42,017-Speed 130.64 samples/sec Loss 2.9962 Epoch: 13 Global Step: 131635 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:02:01,780-Speed 129.54 samples/sec Loss 3.1118 Epoch: 13 Global Step: 131640 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:02:21,647-Speed 128.86 samples/sec Loss 3.2877 Epoch: 13 Global Step: 131645 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:02:41,595-Speed 128.34 samples/sec Loss 3.0516 Epoch: 13 Global Step: 131650 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:03:01,506-Speed 128.58 samples/sec Loss 3.2300 Epoch: 13 Global Step: 131655 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:03:21,553-Speed 127.70 samples/sec Loss 3.1625 Epoch: 13 Global Step: 131660 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:03:41,332-Speed 129.43 samples/sec Loss 3.0989 Epoch: 13 Global Step: 131665 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:04:01,097-Speed 129.53 samples/sec Loss 2.9460 Epoch: 13 Global Step: 131670 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:04:20,908-Speed 129.22 samples/sec Loss 2.9916 Epoch: 13 Global Step: 131675 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:04:40,543-Speed 130.39 samples/sec Loss 3.0246 Epoch: 13 Global Step: 131680 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:05:00,424-Speed 128.77 samples/sec Loss 3.0163 Epoch: 13 Global Step: 131685 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:05:20,279-Speed 128.94 samples/sec Loss 3.3326 Epoch: 13 Global Step: 131690 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:05:40,505-Speed 126.58 samples/sec Loss 3.0521 Epoch: 13 Global Step: 131695 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:06:00,402-Speed 128.67 samples/sec Loss 3.2647 Epoch: 13 Global Step: 131700 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:06:20,507-Speed 127.34 samples/sec Loss 3.1855 Epoch: 13 Global Step: 131705 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:06:40,203-Speed 129.98 samples/sec Loss 3.0754 Epoch: 13 Global Step: 131710 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:07:00,032-Speed 129.11 samples/sec Loss 3.0425 Epoch: 13 Global Step: 131715 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:07:19,784-Speed 129.62 samples/sec Loss 3.1610 Epoch: 13 Global Step: 131720 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:07:39,521-Speed 129.71 samples/sec Loss 3.2197 Epoch: 13 Global Step: 131725 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:07:59,466-Speed 128.36 samples/sec Loss 3.1361 Epoch: 13 Global Step: 131730 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:08:19,167-Speed 129.95 samples/sec Loss 3.2995 Epoch: 13 Global Step: 131735 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:08:39,366-Speed 126.75 samples/sec Loss 3.1184 Epoch: 13 Global Step: 131740 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:08:59,100-Speed 129.74 samples/sec Loss 3.0481 Epoch: 13 Global Step: 131745 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:09:19,309-Speed 126.68 samples/sec Loss 3.1830 Epoch: 13 Global Step: 131750 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:09:39,060-Speed 129.62 samples/sec Loss 3.0747 Epoch: 13 Global Step: 131755 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:09:58,930-Speed 128.84 samples/sec Loss 3.0746 Epoch: 13 Global Step: 131760 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:10:18,538-Speed 130.57 samples/sec Loss 2.9855 Epoch: 13 Global Step: 131765 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:10:38,810-Speed 126.29 samples/sec Loss 3.2097 Epoch: 13 Global Step: 131770 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:10:58,527-Speed 129.85 samples/sec Loss 3.0106 Epoch: 13 Global Step: 131775 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:11:18,180-Speed 130.26 samples/sec Loss 3.1538 Epoch: 13 Global Step: 131780 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:11:38,346-Speed 126.95 samples/sec Loss 3.0846 Epoch: 13 Global Step: 131785 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:11:58,113-Speed 129.51 samples/sec Loss 3.0845 Epoch: 13 Global Step: 131790 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:12:18,192-Speed 127.50 samples/sec Loss 3.0020 Epoch: 13 Global Step: 131795 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:12:38,075-Speed 128.76 samples/sec Loss 3.2252 Epoch: 13 Global Step: 131800 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:12:57,799-Speed 129.80 samples/sec Loss 3.1335 Epoch: 13 Global Step: 131805 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:13:17,923-Speed 127.22 samples/sec Loss 3.2220 Epoch: 13 Global Step: 131810 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:13:37,717-Speed 129.34 samples/sec Loss 3.0796 Epoch: 13 Global Step: 131815 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:13:57,433-Speed 129.85 samples/sec Loss 3.1886 Epoch: 13 Global Step: 131820 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:14:17,569-Speed 127.14 samples/sec Loss 3.4383 Epoch: 13 Global Step: 131825 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:14:37,588-Speed 127.88 samples/sec Loss 3.1843 Epoch: 13 Global Step: 131830 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:14:57,376-Speed 129.38 samples/sec Loss 3.1755 Epoch: 13 Global Step: 131835 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:15:17,398-Speed 127.87 samples/sec Loss 3.1995 Epoch: 13 Global Step: 131840 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:15:37,303-Speed 128.62 samples/sec Loss 3.1075 Epoch: 13 Global Step: 131845 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:15:57,075-Speed 129.48 samples/sec Loss 3.2362 Epoch: 13 Global Step: 131850 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:16:16,774-Speed 129.96 samples/sec Loss 3.2438 Epoch: 13 Global Step: 131855 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:16:36,634-Speed 128.91 samples/sec Loss 2.9403 Epoch: 13 Global Step: 131860 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:16:56,406-Speed 129.48 samples/sec Loss 3.1371 Epoch: 13 Global Step: 131865 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:17:16,238-Speed 129.09 samples/sec Loss 3.1814 Epoch: 13 Global Step: 131870 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:17:36,253-Speed 127.91 samples/sec Loss 3.2302 Epoch: 13 Global Step: 131875 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:17:56,155-Speed 128.64 samples/sec Loss 3.2592 Epoch: 13 Global Step: 131880 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:18:16,091-Speed 128.41 samples/sec Loss 3.0165 Epoch: 13 Global Step: 131885 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:18:36,026-Speed 128.42 samples/sec Loss 3.0544 Epoch: 13 Global Step: 131890 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:18:55,730-Speed 129.93 samples/sec Loss 3.2690 Epoch: 13 Global Step: 131895 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:19:15,569-Speed 129.04 samples/sec Loss 3.2340 Epoch: 13 Global Step: 131900 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:19:35,270-Speed 129.95 samples/sec Loss 3.1260 Epoch: 13 Global Step: 131905 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:19:55,234-Speed 128.23 samples/sec Loss 3.0580 Epoch: 13 Global Step: 131910 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:20:14,954-Speed 129.83 samples/sec Loss 3.3497 Epoch: 13 Global Step: 131915 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:20:34,987-Speed 127.80 samples/sec Loss 3.2961 Epoch: 13 Global Step: 131920 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:20:54,748-Speed 129.55 samples/sec Loss 3.1979 Epoch: 13 Global Step: 131925 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:21:14,942-Speed 126.77 samples/sec Loss 3.1677 Epoch: 13 Global Step: 131930 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:21:34,663-Speed 129.82 samples/sec Loss 3.1396 Epoch: 13 Global Step: 131935 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:21:54,392-Speed 129.76 samples/sec Loss 3.5298 Epoch: 13 Global Step: 131940 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:22:14,275-Speed 128.76 samples/sec Loss 3.2915 Epoch: 13 Global Step: 131945 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:22:33,971-Speed 129.98 samples/sec Loss 3.1795 Epoch: 13 Global Step: 131950 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:22:53,877-Speed 128.61 samples/sec Loss 3.1041 Epoch: 13 Global Step: 131955 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:23:13,596-Speed 129.83 samples/sec Loss 3.0496 Epoch: 13 Global Step: 131960 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:23:33,749-Speed 127.04 samples/sec Loss 3.1500 Epoch: 13 Global Step: 131965 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:23:53,613-Speed 128.89 samples/sec Loss 3.2667 Epoch: 13 Global Step: 131970 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:24:13,692-Speed 127.50 samples/sec Loss 2.8897 Epoch: 13 Global Step: 131975 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:24:33,421-Speed 129.76 samples/sec Loss 2.9786 Epoch: 13 Global Step: 131980 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:24:53,231-Speed 129.24 samples/sec Loss 3.1248 Epoch: 13 Global Step: 131985 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:25:13,034-Speed 129.28 samples/sec Loss 3.2312 Epoch: 13 Global Step: 131990 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:25:32,728-Speed 129.99 samples/sec Loss 3.1729 Epoch: 13 Global Step: 131995 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:25:52,646-Speed 128.54 samples/sec Loss 3.1117 Epoch: 13 Global Step: 132000 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:26:12,357-Speed 129.88 samples/sec Loss 3.1141 Epoch: 13 Global Step: 132005 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:26:32,624-Speed 126.32 samples/sec Loss 3.1785 Epoch: 13 Global Step: 132010 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:26:52,279-Speed 130.25 samples/sec Loss 2.9720 Epoch: 13 Global Step: 132015 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:27:12,367-Speed 127.45 samples/sec Loss 3.1604 Epoch: 13 Global Step: 132020 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:27:32,063-Speed 129.98 samples/sec Loss 3.2749 Epoch: 13 Global Step: 132025 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:27:52,018-Speed 128.29 samples/sec Loss 3.2643 Epoch: 13 Global Step: 132030 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:28:11,636-Speed 130.50 samples/sec Loss 3.2242 Epoch: 13 Global Step: 132035 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:28:31,473-Speed 129.06 samples/sec Loss 3.3046 Epoch: 13 Global Step: 132040 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:28:51,259-Speed 129.39 samples/sec Loss 3.2636 Epoch: 13 Global Step: 132045 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:29:10,952-Speed 130.00 samples/sec Loss 3.0927 Epoch: 13 Global Step: 132050 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:29:31,062-Speed 127.31 samples/sec Loss 3.1324 Epoch: 13 Global Step: 132055 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:29:50,790-Speed 129.77 samples/sec Loss 3.0744 Epoch: 13 Global Step: 132060 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:30:10,835-Speed 127.72 samples/sec Loss 3.0325 Epoch: 13 Global Step: 132065 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:30:30,586-Speed 129.62 samples/sec Loss 3.1495 Epoch: 13 Global Step: 132070 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:30:50,480-Speed 128.69 samples/sec Loss 3.1696 Epoch: 13 Global Step: 132075 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:31:10,236-Speed 129.59 samples/sec Loss 3.2883 Epoch: 13 Global Step: 132080 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:31:30,148-Speed 128.57 samples/sec Loss 3.2902 Epoch: 13 Global Step: 132085 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:31:49,861-Speed 129.87 samples/sec Loss 3.0575 Epoch: 13 Global Step: 132090 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:32:09,553-Speed 130.01 samples/sec Loss 3.2581 Epoch: 13 Global Step: 132095 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:32:29,752-Speed 126.75 samples/sec Loss 3.2015 Epoch: 13 Global Step: 132100 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:32:49,481-Speed 129.77 samples/sec Loss 3.2255 Epoch: 13 Global Step: 132105 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:33:09,562-Speed 127.49 samples/sec Loss 3.0275 Epoch: 13 Global Step: 132110 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:33:29,348-Speed 129.39 samples/sec Loss 2.9932 Epoch: 13 Global Step: 132115 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:33:49,137-Speed 129.37 samples/sec Loss 3.2048 Epoch: 13 Global Step: 132120 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:34:08,896-Speed 129.57 samples/sec Loss 3.4724 Epoch: 13 Global Step: 132125 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:34:28,790-Speed 128.69 samples/sec Loss 3.2196 Epoch: 13 Global Step: 132130 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:34:49,143-Speed 125.79 samples/sec Loss 3.3142 Epoch: 13 Global Step: 132135 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:35:08,974-Speed 129.09 samples/sec Loss 2.9707 Epoch: 13 Global Step: 132140 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:35:28,907-Speed 128.44 samples/sec Loss 3.2127 Epoch: 13 Global Step: 132145 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:35:48,819-Speed 128.58 samples/sec Loss 3.2259 Epoch: 13 Global Step: 132150 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:36:08,651-Speed 129.09 samples/sec Loss 3.2079 Epoch: 13 Global Step: 132155 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:36:28,582-Speed 128.45 samples/sec Loss 3.0585 Epoch: 13 Global Step: 132160 Fp16 Grad Scale: 8192 Required: 139 hours Training: 2021-06-13 07:36:48,292-Speed 129.89 samples/sec Loss 3.3168 Epoch: 13 Global Step: 132165 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:37:08,198-Speed 128.61 samples/sec Loss 3.2424 Epoch: 13 Global Step: 132170 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:37:27,884-Speed 130.05 samples/sec Loss 3.2241 Epoch: 13 Global Step: 132175 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:37:47,668-Speed 129.40 samples/sec Loss 3.1939 Epoch: 13 Global Step: 132180 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:38:07,493-Speed 129.14 samples/sec Loss 3.3783 Epoch: 13 Global Step: 132185 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:38:27,471-Speed 128.14 samples/sec Loss 3.0529 Epoch: 13 Global Step: 132190 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:38:47,380-Speed 128.59 samples/sec Loss 3.0427 Epoch: 13 Global Step: 132195 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:39:07,426-Speed 127.71 samples/sec Loss 3.2007 Epoch: 13 Global Step: 132200 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:39:27,154-Speed 129.77 samples/sec Loss 3.2344 Epoch: 13 Global Step: 132205 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:39:47,089-Speed 128.42 samples/sec Loss 3.2195 Epoch: 13 Global Step: 132210 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:40:06,988-Speed 128.66 samples/sec Loss 3.2147 Epoch: 13 Global Step: 132215 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:40:26,637-Speed 130.29 samples/sec Loss 3.2269 Epoch: 13 Global Step: 132220 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:40:46,578-Speed 128.39 samples/sec Loss 3.3134 Epoch: 13 Global Step: 132225 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:41:06,319-Speed 129.68 samples/sec Loss 3.2401 Epoch: 13 Global Step: 132230 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:41:26,421-Speed 127.36 samples/sec Loss 3.2185 Epoch: 13 Global Step: 132235 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:41:46,300-Speed 128.78 samples/sec Loss 3.3679 Epoch: 13 Global Step: 132240 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:42:06,297-Speed 128.03 samples/sec Loss 3.1109 Epoch: 13 Global Step: 132245 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:42:26,021-Speed 129.79 samples/sec Loss 3.2659 Epoch: 13 Global Step: 132250 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:42:45,916-Speed 128.68 samples/sec Loss 3.2207 Epoch: 13 Global Step: 132255 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:43:05,685-Speed 129.50 samples/sec Loss 3.3040 Epoch: 13 Global Step: 132260 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:43:25,446-Speed 129.55 samples/sec Loss 3.2543 Epoch: 13 Global Step: 132265 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:43:45,319-Speed 128.83 samples/sec Loss 3.1652 Epoch: 13 Global Step: 132270 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:44:04,964-Speed 130.32 samples/sec Loss 3.2234 Epoch: 13 Global Step: 132275 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:44:25,251-Speed 126.20 samples/sec Loss 3.0268 Epoch: 13 Global Step: 132280 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:44:44,915-Speed 130.19 samples/sec Loss 3.2351 Epoch: 13 Global Step: 132285 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:45:05,002-Speed 127.46 samples/sec Loss 3.1651 Epoch: 13 Global Step: 132290 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:45:24,789-Speed 129.38 samples/sec Loss 3.2282 Epoch: 13 Global Step: 132295 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:45:44,615-Speed 129.12 samples/sec Loss 3.3368 Epoch: 13 Global Step: 132300 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:46:04,414-Speed 129.31 samples/sec Loss 3.3665 Epoch: 13 Global Step: 132305 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:46:24,279-Speed 128.87 samples/sec Loss 3.3420 Epoch: 13 Global Step: 132310 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:46:44,025-Speed 129.66 samples/sec Loss 3.3676 Epoch: 13 Global Step: 132315 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:47:03,902-Speed 128.80 samples/sec Loss 3.3132 Epoch: 13 Global Step: 132320 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:47:23,859-Speed 128.28 samples/sec Loss 3.0416 Epoch: 13 Global Step: 132325 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:47:43,645-Speed 129.39 samples/sec Loss 3.1788 Epoch: 13 Global Step: 132330 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:48:03,679-Speed 127.79 samples/sec Loss 3.1374 Epoch: 13 Global Step: 132335 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:48:23,656-Speed 128.15 samples/sec Loss 3.2489 Epoch: 13 Global Step: 132340 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:48:43,374-Speed 129.84 samples/sec Loss 3.1396 Epoch: 13 Global Step: 132345 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:49:03,165-Speed 129.36 samples/sec Loss 3.1428 Epoch: 13 Global Step: 132350 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:49:22,830-Speed 130.19 samples/sec Loss 3.1648 Epoch: 13 Global Step: 132355 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:49:42,634-Speed 129.27 samples/sec Loss 3.3216 Epoch: 13 Global Step: 132360 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:50:02,533-Speed 128.65 samples/sec Loss 3.0318 Epoch: 13 Global Step: 132365 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:50:22,503-Speed 128.20 samples/sec Loss 3.3562 Epoch: 13 Global Step: 132370 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:50:42,386-Speed 128.76 samples/sec Loss 3.1935 Epoch: 13 Global Step: 132375 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:51:02,494-Speed 127.32 samples/sec Loss 3.3950 Epoch: 13 Global Step: 132380 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:51:22,207-Speed 129.87 samples/sec Loss 3.1080 Epoch: 13 Global Step: 132385 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:51:41,963-Speed 129.58 samples/sec Loss 3.2043 Epoch: 13 Global Step: 132390 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:52:01,830-Speed 128.87 samples/sec Loss 3.3453 Epoch: 13 Global Step: 132395 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:52:21,532-Speed 129.94 samples/sec Loss 3.1569 Epoch: 13 Global Step: 132400 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:52:41,479-Speed 128.34 samples/sec Loss 3.2140 Epoch: 13 Global Step: 132405 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:53:01,197-Speed 129.84 samples/sec Loss 3.2925 Epoch: 13 Global Step: 132410 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:53:21,171-Speed 128.17 samples/sec Loss 3.2710 Epoch: 13 Global Step: 132415 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:53:41,040-Speed 128.85 samples/sec Loss 3.3404 Epoch: 13 Global Step: 132420 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:54:01,368-Speed 125.95 samples/sec Loss 3.3995 Epoch: 13 Global Step: 132425 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:54:20,976-Speed 130.56 samples/sec Loss 3.2264 Epoch: 13 Global Step: 132430 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:54:40,729-Speed 129.60 samples/sec Loss 3.2873 Epoch: 13 Global Step: 132435 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:55:00,692-Speed 128.25 samples/sec Loss 3.3391 Epoch: 13 Global Step: 132440 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:55:20,476-Speed 129.40 samples/sec Loss 3.2160 Epoch: 13 Global Step: 132445 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:55:40,389-Speed 128.57 samples/sec Loss 3.2392 Epoch: 13 Global Step: 132450 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:56:00,014-Speed 130.45 samples/sec Loss 3.0797 Epoch: 13 Global Step: 132455 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:56:20,223-Speed 126.68 samples/sec Loss 3.1950 Epoch: 13 Global Step: 132460 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:56:40,012-Speed 129.37 samples/sec Loss 3.1754 Epoch: 13 Global Step: 132465 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:56:59,962-Speed 128.33 samples/sec Loss 3.1287 Epoch: 13 Global Step: 132470 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:57:19,689-Speed 129.78 samples/sec Loss 3.2924 Epoch: 13 Global Step: 132475 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:57:39,612-Speed 128.50 samples/sec Loss 3.4034 Epoch: 13 Global Step: 132480 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:57:59,374-Speed 129.55 samples/sec Loss 3.2418 Epoch: 13 Global Step: 132485 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:58:19,365-Speed 128.06 samples/sec Loss 3.3515 Epoch: 13 Global Step: 132490 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:58:39,012-Speed 130.31 samples/sec Loss 3.1241 Epoch: 13 Global Step: 132495 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:58:58,754-Speed 129.68 samples/sec Loss 3.2423 Epoch: 13 Global Step: 132500 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:59:18,988-Speed 126.53 samples/sec Loss 3.0971 Epoch: 13 Global Step: 132505 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:59:38,593-Speed 130.58 samples/sec Loss 3.2139 Epoch: 13 Global Step: 132510 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 07:59:58,674-Speed 127.49 samples/sec Loss 3.1669 Epoch: 13 Global Step: 132515 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:00:18,352-Speed 130.10 samples/sec Loss 3.1125 Epoch: 13 Global Step: 132520 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:00:38,221-Speed 128.85 samples/sec Loss 3.4953 Epoch: 13 Global Step: 132525 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:00:57,933-Speed 129.88 samples/sec Loss 3.0423 Epoch: 13 Global Step: 132530 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:01:17,746-Speed 129.22 samples/sec Loss 3.1579 Epoch: 13 Global Step: 132535 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:01:37,888-Speed 127.11 samples/sec Loss 3.2287 Epoch: 13 Global Step: 132540 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:01:57,641-Speed 129.60 samples/sec Loss 3.2563 Epoch: 13 Global Step: 132545 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:02:17,793-Speed 127.04 samples/sec Loss 3.0995 Epoch: 13 Global Step: 132550 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:02:37,401-Speed 130.57 samples/sec Loss 3.2009 Epoch: 13 Global Step: 132555 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:02:57,494-Speed 127.41 samples/sec Loss 3.1880 Epoch: 13 Global Step: 132560 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:03:17,401-Speed 128.60 samples/sec Loss 3.2989 Epoch: 13 Global Step: 132565 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:03:37,195-Speed 129.34 samples/sec Loss 3.0342 Epoch: 13 Global Step: 132570 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:03:56,849-Speed 130.27 samples/sec Loss 3.2307 Epoch: 13 Global Step: 132575 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:04:16,615-Speed 129.52 samples/sec Loss 3.1244 Epoch: 13 Global Step: 132580 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:04:36,314-Speed 129.97 samples/sec Loss 3.1496 Epoch: 13 Global Step: 132585 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:04:56,228-Speed 128.56 samples/sec Loss 3.3130 Epoch: 13 Global Step: 132590 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:05:16,143-Speed 128.56 samples/sec Loss 3.2847 Epoch: 13 Global Step: 132595 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:05:35,921-Speed 129.45 samples/sec Loss 3.2589 Epoch: 13 Global Step: 132600 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:05:55,956-Speed 127.78 samples/sec Loss 3.3477 Epoch: 13 Global Step: 132605 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:06:15,924-Speed 128.21 samples/sec Loss 3.0656 Epoch: 13 Global Step: 132610 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:06:35,537-Speed 130.53 samples/sec Loss 3.1121 Epoch: 13 Global Step: 132615 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:06:55,355-Speed 129.18 samples/sec Loss 3.3550 Epoch: 13 Global Step: 132620 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:07:14,991-Speed 130.38 samples/sec Loss 3.1546 Epoch: 13 Global Step: 132625 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:07:34,819-Speed 129.12 samples/sec Loss 3.2557 Epoch: 13 Global Step: 132630 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:07:54,712-Speed 128.70 samples/sec Loss 3.0407 Epoch: 13 Global Step: 132635 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:08:14,677-Speed 128.23 samples/sec Loss 3.0857 Epoch: 13 Global Step: 132640 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:08:34,572-Speed 128.68 samples/sec Loss 3.2974 Epoch: 13 Global Step: 132645 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:08:54,640-Speed 127.58 samples/sec Loss 3.1943 Epoch: 13 Global Step: 132650 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:09:14,584-Speed 128.36 samples/sec Loss 3.3251 Epoch: 13 Global Step: 132655 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:09:34,267-Speed 130.07 samples/sec Loss 3.2820 Epoch: 13 Global Step: 132660 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:09:54,102-Speed 129.07 samples/sec Loss 3.2956 Epoch: 13 Global Step: 132665 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:10:13,804-Speed 129.94 samples/sec Loss 3.1893 Epoch: 13 Global Step: 132670 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:10:33,643-Speed 129.05 samples/sec Loss 3.2016 Epoch: 13 Global Step: 132675 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:10:53,431-Speed 129.38 samples/sec Loss 3.1887 Epoch: 13 Global Step: 132680 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:11:13,378-Speed 128.35 samples/sec Loss 3.1822 Epoch: 13 Global Step: 132685 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:11:33,246-Speed 128.86 samples/sec Loss 3.1117 Epoch: 13 Global Step: 132690 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:11:53,389-Speed 127.09 samples/sec Loss 3.3148 Epoch: 13 Global Step: 132695 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:12:13,004-Speed 130.52 samples/sec Loss 3.5002 Epoch: 13 Global Step: 132700 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:12:32,649-Speed 130.32 samples/sec Loss 3.3020 Epoch: 13 Global Step: 132705 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:12:52,519-Speed 128.85 samples/sec Loss 2.9066 Epoch: 13 Global Step: 132710 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:13:12,239-Speed 129.82 samples/sec Loss 3.2154 Epoch: 13 Global Step: 132715 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:13:32,137-Speed 128.66 samples/sec Loss 3.4985 Epoch: 13 Global Step: 132720 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:13:51,742-Speed 130.59 samples/sec Loss 3.4343 Epoch: 13 Global Step: 132725 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:14:11,765-Speed 127.86 samples/sec Loss 3.2789 Epoch: 13 Global Step: 132730 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:14:31,623-Speed 128.92 samples/sec Loss 3.3237 Epoch: 13 Global Step: 132735 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:14:51,665-Speed 127.74 samples/sec Loss 3.3719 Epoch: 13 Global Step: 132740 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:15:11,390-Speed 129.79 samples/sec Loss 3.0825 Epoch: 13 Global Step: 132745 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:15:31,240-Speed 128.97 samples/sec Loss 3.4293 Epoch: 13 Global Step: 132750 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:15:50,956-Speed 129.86 samples/sec Loss 3.0900 Epoch: 13 Global Step: 132755 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:16:10,800-Speed 129.01 samples/sec Loss 3.2247 Epoch: 13 Global Step: 132760 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:16:30,539-Speed 129.70 samples/sec Loss 3.3087 Epoch: 13 Global Step: 132765 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:16:50,171-Speed 130.40 samples/sec Loss 3.4821 Epoch: 13 Global Step: 132770 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:17:10,365-Speed 126.78 samples/sec Loss 3.1913 Epoch: 13 Global Step: 132775 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:17:30,144-Speed 129.44 samples/sec Loss 3.2095 Epoch: 13 Global Step: 132780 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:17:50,123-Speed 128.14 samples/sec Loss 3.5136 Epoch: 13 Global Step: 132785 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:18:09,734-Speed 130.55 samples/sec Loss 3.1107 Epoch: 13 Global Step: 132790 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:18:29,638-Speed 128.62 samples/sec Loss 3.1866 Epoch: 13 Global Step: 132795 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:18:49,432-Speed 129.33 samples/sec Loss 3.1573 Epoch: 13 Global Step: 132800 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:19:09,385-Speed 128.31 samples/sec Loss 3.4287 Epoch: 13 Global Step: 132805 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:19:28,988-Speed 130.60 samples/sec Loss 3.2552 Epoch: 13 Global Step: 132810 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:19:48,670-Speed 130.07 samples/sec Loss 3.4053 Epoch: 13 Global Step: 132815 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:20:08,927-Speed 126.38 samples/sec Loss 3.3635 Epoch: 13 Global Step: 132820 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:20:28,650-Speed 129.80 samples/sec Loss 3.2870 Epoch: 13 Global Step: 132825 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:20:48,634-Speed 128.11 samples/sec Loss 3.2011 Epoch: 13 Global Step: 132830 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:21:08,512-Speed 128.79 samples/sec Loss 3.3395 Epoch: 13 Global Step: 132835 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:21:28,266-Speed 129.60 samples/sec Loss 3.2332 Epoch: 13 Global Step: 132840 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:21:48,031-Speed 129.53 samples/sec Loss 3.3258 Epoch: 13 Global Step: 132845 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:22:07,860-Speed 129.11 samples/sec Loss 3.2478 Epoch: 13 Global Step: 132850 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:22:27,589-Speed 129.76 samples/sec Loss 3.2929 Epoch: 13 Global Step: 132855 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:22:47,470-Speed 128.77 samples/sec Loss 3.2975 Epoch: 13 Global Step: 132860 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:23:07,454-Speed 128.11 samples/sec Loss 3.3635 Epoch: 13 Global Step: 132865 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:23:27,106-Speed 130.27 samples/sec Loss 3.0883 Epoch: 13 Global Step: 132870 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:23:47,026-Speed 128.52 samples/sec Loss 3.2988 Epoch: 13 Global Step: 132875 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:24:06,949-Speed 128.49 samples/sec Loss 3.3344 Epoch: 13 Global Step: 132880 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:24:26,732-Speed 129.42 samples/sec Loss 3.2298 Epoch: 13 Global Step: 132885 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:24:46,537-Speed 129.27 samples/sec Loss 3.1946 Epoch: 13 Global Step: 132890 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:25:06,158-Speed 130.48 samples/sec Loss 3.2680 Epoch: 13 Global Step: 132895 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:25:25,944-Speed 129.39 samples/sec Loss 3.1929 Epoch: 13 Global Step: 132900 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:25:45,821-Speed 128.79 samples/sec Loss 3.3436 Epoch: 13 Global Step: 132905 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:26:05,738-Speed 128.54 samples/sec Loss 3.0616 Epoch: 13 Global Step: 132910 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:26:25,528-Speed 129.37 samples/sec Loss 3.2652 Epoch: 13 Global Step: 132915 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:26:45,600-Speed 127.55 samples/sec Loss 3.4535 Epoch: 13 Global Step: 132920 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:27:05,388-Speed 129.38 samples/sec Loss 3.1944 Epoch: 13 Global Step: 132925 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:27:25,055-Speed 130.17 samples/sec Loss 3.3447 Epoch: 13 Global Step: 132930 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:27:44,903-Speed 128.99 samples/sec Loss 3.2742 Epoch: 13 Global Step: 132935 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:28:04,698-Speed 129.33 samples/sec Loss 3.2131 Epoch: 13 Global Step: 132940 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:28:24,621-Speed 128.50 samples/sec Loss 3.4496 Epoch: 13 Global Step: 132945 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:28:44,422-Speed 129.30 samples/sec Loss 3.2396 Epoch: 13 Global Step: 132950 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:29:04,412-Speed 128.07 samples/sec Loss 3.2685 Epoch: 13 Global Step: 132955 Fp16 Grad Scale: 8192 Required: 138 hours Training: 2021-06-13 08:29:24,129-Speed 129.84 samples/sec Loss 3.3037 Epoch: 13 Global Step: 132960 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:29:44,365-Speed 126.51 samples/sec Loss 3.2254 Epoch: 13 Global Step: 132965 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:30:04,093-Speed 129.77 samples/sec Loss 3.2291 Epoch: 13 Global Step: 132970 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:30:23,889-Speed 129.32 samples/sec Loss 3.3935 Epoch: 13 Global Step: 132975 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:30:43,601-Speed 129.88 samples/sec Loss 3.2768 Epoch: 13 Global Step: 132980 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:31:03,416-Speed 129.20 samples/sec Loss 3.2476 Epoch: 13 Global Step: 132985 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:31:23,265-Speed 128.98 samples/sec Loss 3.2783 Epoch: 13 Global Step: 132990 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:31:42,978-Speed 129.87 samples/sec Loss 3.2683 Epoch: 13 Global Step: 132995 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:32:03,038-Speed 127.62 samples/sec Loss 3.2878 Epoch: 13 Global Step: 133000 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:32:22,844-Speed 129.26 samples/sec Loss 3.1104 Epoch: 13 Global Step: 133005 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:32:42,941-Speed 127.39 samples/sec Loss 3.1849 Epoch: 13 Global Step: 133010 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:33:02,741-Speed 129.29 samples/sec Loss 3.0837 Epoch: 13 Global Step: 133015 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:33:22,450-Speed 129.89 samples/sec Loss 3.3262 Epoch: 13 Global Step: 133020 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:33:42,281-Speed 129.11 samples/sec Loss 3.3775 Epoch: 13 Global Step: 133025 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:34:02,166-Speed 128.75 samples/sec Loss 3.1895 Epoch: 13 Global Step: 133030 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:34:21,902-Speed 129.72 samples/sec Loss 3.4240 Epoch: 13 Global Step: 133035 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:34:41,705-Speed 129.28 samples/sec Loss 3.3292 Epoch: 13 Global Step: 133040 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:35:01,805-Speed 127.37 samples/sec Loss 3.2153 Epoch: 13 Global Step: 133045 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:35:21,652-Speed 128.99 samples/sec Loss 3.1832 Epoch: 13 Global Step: 133050 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:35:41,686-Speed 127.79 samples/sec Loss 3.3920 Epoch: 13 Global Step: 133055 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:36:01,442-Speed 129.59 samples/sec Loss 3.2284 Epoch: 13 Global Step: 133060 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:36:21,205-Speed 129.54 samples/sec Loss 3.3226 Epoch: 13 Global Step: 133065 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:36:41,143-Speed 128.40 samples/sec Loss 3.2715 Epoch: 13 Global Step: 133070 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:37:00,830-Speed 130.04 samples/sec Loss 3.2389 Epoch: 13 Global Step: 133075 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:37:20,716-Speed 128.74 samples/sec Loss 3.3333 Epoch: 13 Global Step: 133080 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:37:40,434-Speed 129.84 samples/sec Loss 3.3550 Epoch: 13 Global Step: 133085 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:38:00,426-Speed 128.05 samples/sec Loss 3.0554 Epoch: 13 Global Step: 133090 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:38:20,307-Speed 128.78 samples/sec Loss 3.1815 Epoch: 13 Global Step: 133095 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:38:40,304-Speed 128.03 samples/sec Loss 3.2173 Epoch: 13 Global Step: 133100 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:39:00,073-Speed 129.50 samples/sec Loss 3.1308 Epoch: 13 Global Step: 133105 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:39:19,739-Speed 130.18 samples/sec Loss 3.3171 Epoch: 13 Global Step: 133110 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:39:39,590-Speed 128.97 samples/sec Loss 3.3017 Epoch: 13 Global Step: 133115 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:39:59,474-Speed 128.76 samples/sec Loss 3.3800 Epoch: 13 Global Step: 133120 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:40:19,377-Speed 128.63 samples/sec Loss 3.5494 Epoch: 13 Global Step: 133125 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:40:39,113-Speed 129.72 samples/sec Loss 3.6378 Epoch: 13 Global Step: 133130 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:40:59,488-Speed 125.65 samples/sec Loss 3.2341 Epoch: 13 Global Step: 133135 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:41:19,159-Speed 130.15 samples/sec Loss 3.1784 Epoch: 13 Global Step: 133140 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:41:39,234-Speed 127.53 samples/sec Loss 3.2827 Epoch: 13 Global Step: 133145 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:41:59,008-Speed 129.47 samples/sec Loss 3.2509 Epoch: 13 Global Step: 133150 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:42:18,926-Speed 128.53 samples/sec Loss 3.4892 Epoch: 13 Global Step: 133155 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:42:38,616-Speed 130.02 samples/sec Loss 3.2899 Epoch: 13 Global Step: 133160 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:42:58,314-Speed 129.97 samples/sec Loss 2.9776 Epoch: 13 Global Step: 133165 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:43:18,232-Speed 128.54 samples/sec Loss 3.2844 Epoch: 13 Global Step: 133170 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:43:37,872-Speed 130.35 samples/sec Loss 3.0595 Epoch: 13 Global Step: 133175 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:43:58,007-Speed 127.14 samples/sec Loss 3.4297 Epoch: 13 Global Step: 133180 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:44:17,727-Speed 129.83 samples/sec Loss 3.3299 Epoch: 13 Global Step: 133185 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:44:37,869-Speed 127.11 samples/sec Loss 3.2538 Epoch: 13 Global Step: 133190 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:44:57,577-Speed 129.90 samples/sec Loss 3.2055 Epoch: 13 Global Step: 133195 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:45:17,457-Speed 128.78 samples/sec Loss 3.5880 Epoch: 13 Global Step: 133200 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:45:37,171-Speed 129.86 samples/sec Loss 3.4231 Epoch: 13 Global Step: 133205 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:45:56,879-Speed 129.91 samples/sec Loss 3.0000 Epoch: 13 Global Step: 133210 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:46:16,782-Speed 128.63 samples/sec Loss 3.3146 Epoch: 13 Global Step: 133215 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:46:36,526-Speed 129.67 samples/sec Loss 3.4243 Epoch: 13 Global Step: 133220 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:46:56,716-Speed 126.80 samples/sec Loss 3.3075 Epoch: 13 Global Step: 133225 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:47:16,435-Speed 129.83 samples/sec Loss 3.3295 Epoch: 13 Global Step: 133230 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:47:36,504-Speed 127.57 samples/sec Loss 3.2683 Epoch: 13 Global Step: 133235 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:47:56,251-Speed 129.65 samples/sec Loss 3.2937 Epoch: 13 Global Step: 133240 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:48:16,175-Speed 128.49 samples/sec Loss 3.2462 Epoch: 13 Global Step: 133245 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:48:35,913-Speed 129.71 samples/sec Loss 3.4703 Epoch: 13 Global Step: 133250 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:48:55,757-Speed 129.01 samples/sec Loss 3.2811 Epoch: 13 Global Step: 133255 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:49:15,582-Speed 129.13 samples/sec Loss 3.3061 Epoch: 13 Global Step: 133260 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:49:35,326-Speed 129.67 samples/sec Loss 3.3239 Epoch: 13 Global Step: 133265 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:49:55,526-Speed 126.74 samples/sec Loss 3.3041 Epoch: 13 Global Step: 133270 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:50:15,318-Speed 129.35 samples/sec Loss 3.4150 Epoch: 13 Global Step: 133275 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:50:35,393-Speed 127.53 samples/sec Loss 3.3300 Epoch: 13 Global Step: 133280 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:50:55,123-Speed 129.76 samples/sec Loss 3.3580 Epoch: 13 Global Step: 133285 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:51:15,022-Speed 128.66 samples/sec Loss 3.3508 Epoch: 13 Global Step: 133290 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:51:34,732-Speed 129.89 samples/sec Loss 3.4151 Epoch: 13 Global Step: 133295 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:51:54,602-Speed 128.85 samples/sec Loss 3.3454 Epoch: 13 Global Step: 133300 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:52:14,282-Speed 130.09 samples/sec Loss 3.3556 Epoch: 13 Global Step: 133305 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:52:34,667-Speed 125.59 samples/sec Loss 3.2780 Epoch: 13 Global Step: 133310 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:52:54,656-Speed 128.08 samples/sec Loss 3.2422 Epoch: 13 Global Step: 133315 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:53:14,290-Speed 130.39 samples/sec Loss 3.3495 Epoch: 13 Global Step: 133320 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:53:34,409-Speed 127.25 samples/sec Loss 3.5045 Epoch: 13 Global Step: 133325 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:53:54,279-Speed 128.85 samples/sec Loss 3.5672 Epoch: 13 Global Step: 133330 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:54:14,001-Speed 129.81 samples/sec Loss 3.2469 Epoch: 13 Global Step: 133335 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:54:33,832-Speed 129.10 samples/sec Loss 3.1981 Epoch: 13 Global Step: 133340 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:54:53,679-Speed 128.99 samples/sec Loss 3.1838 Epoch: 13 Global Step: 133345 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:55:13,404-Speed 129.80 samples/sec Loss 3.4804 Epoch: 13 Global Step: 133350 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:55:33,294-Speed 128.71 samples/sec Loss 3.1791 Epoch: 13 Global Step: 133355 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:55:53,374-Speed 127.50 samples/sec Loss 3.1754 Epoch: 13 Global Step: 133360 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:56:13,113-Speed 129.70 samples/sec Loss 3.1680 Epoch: 13 Global Step: 133365 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:56:33,158-Speed 127.72 samples/sec Loss 3.2687 Epoch: 13 Global Step: 133370 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:56:53,021-Speed 128.89 samples/sec Loss 3.4666 Epoch: 13 Global Step: 133375 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:57:12,684-Speed 130.20 samples/sec Loss 3.3754 Epoch: 13 Global Step: 133380 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:57:32,648-Speed 128.24 samples/sec Loss 3.1762 Epoch: 13 Global Step: 133385 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:57:52,426-Speed 129.45 samples/sec Loss 3.3158 Epoch: 13 Global Step: 133390 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:58:12,184-Speed 129.58 samples/sec Loss 3.1203 Epoch: 13 Global Step: 133395 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:58:32,047-Speed 128.89 samples/sec Loss 3.4894 Epoch: 13 Global Step: 133400 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:58:52,099-Speed 127.67 samples/sec Loss 3.3767 Epoch: 13 Global Step: 133405 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:59:12,020-Speed 128.52 samples/sec Loss 3.1842 Epoch: 13 Global Step: 133410 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:59:31,987-Speed 128.22 samples/sec Loss 3.3326 Epoch: 13 Global Step: 133415 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 08:59:51,634-Speed 130.30 samples/sec Loss 3.3806 Epoch: 13 Global Step: 133420 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:00:11,363-Speed 129.76 samples/sec Loss 3.3954 Epoch: 13 Global Step: 133425 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:00:31,325-Speed 128.25 samples/sec Loss 3.2044 Epoch: 13 Global Step: 133430 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:00:51,020-Speed 129.99 samples/sec Loss 3.1933 Epoch: 13 Global Step: 133435 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:01:10,868-Speed 128.99 samples/sec Loss 3.3282 Epoch: 13 Global Step: 133440 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:01:30,531-Speed 130.20 samples/sec Loss 3.4087 Epoch: 13 Global Step: 133445 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:01:50,652-Speed 127.24 samples/sec Loss 3.4252 Epoch: 13 Global Step: 133450 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:02:10,517-Speed 128.88 samples/sec Loss 3.2434 Epoch: 13 Global Step: 133455 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:02:30,638-Speed 127.24 samples/sec Loss 3.2595 Epoch: 13 Global Step: 133460 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:02:50,348-Speed 129.89 samples/sec Loss 3.3391 Epoch: 13 Global Step: 133465 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:03:10,094-Speed 129.66 samples/sec Loss 3.1792 Epoch: 13 Global Step: 133470 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:03:30,034-Speed 128.39 samples/sec Loss 3.2298 Epoch: 13 Global Step: 133475 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:03:49,695-Speed 130.22 samples/sec Loss 3.3415 Epoch: 13 Global Step: 133480 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:04:09,648-Speed 128.31 samples/sec Loss 3.2398 Epoch: 13 Global Step: 133485 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:04:29,418-Speed 129.49 samples/sec Loss 3.0849 Epoch: 13 Global Step: 133490 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:04:49,629-Speed 126.67 samples/sec Loss 3.2388 Epoch: 13 Global Step: 133495 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:05:09,242-Speed 130.53 samples/sec Loss 3.2758 Epoch: 13 Global Step: 133500 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:05:29,298-Speed 127.65 samples/sec Loss 3.2442 Epoch: 13 Global Step: 133505 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:05:49,526-Speed 126.56 samples/sec Loss 3.4585 Epoch: 13 Global Step: 133510 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:06:09,416-Speed 128.71 samples/sec Loss 3.2868 Epoch: 13 Global Step: 133515 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:06:29,161-Speed 129.66 samples/sec Loss 3.3867 Epoch: 13 Global Step: 133520 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:06:48,963-Speed 129.29 samples/sec Loss 3.1183 Epoch: 13 Global Step: 133525 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:07:08,683-Speed 129.82 samples/sec Loss 3.2887 Epoch: 13 Global Step: 133530 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:07:28,475-Speed 129.35 samples/sec Loss 3.1322 Epoch: 13 Global Step: 133535 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:07:48,674-Speed 126.74 samples/sec Loss 3.3094 Epoch: 13 Global Step: 133540 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:08:08,315-Speed 130.35 samples/sec Loss 3.3052 Epoch: 13 Global Step: 133545 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:08:28,403-Speed 127.45 samples/sec Loss 3.4112 Epoch: 13 Global Step: 133550 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:08:48,305-Speed 128.63 samples/sec Loss 3.0612 Epoch: 13 Global Step: 133555 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:09:08,030-Speed 129.79 samples/sec Loss 3.2453 Epoch: 13 Global Step: 133560 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:09:27,736-Speed 129.92 samples/sec Loss 3.3874 Epoch: 13 Global Step: 133565 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:09:47,553-Speed 129.19 samples/sec Loss 3.2325 Epoch: 13 Global Step: 133570 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:10:07,491-Speed 128.40 samples/sec Loss 3.2921 Epoch: 13 Global Step: 133575 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:10:27,369-Speed 128.79 samples/sec Loss 3.2902 Epoch: 13 Global Step: 133580 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:10:47,242-Speed 128.82 samples/sec Loss 3.4331 Epoch: 13 Global Step: 133585 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:11:07,147-Speed 128.62 samples/sec Loss 3.4813 Epoch: 13 Global Step: 133590 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:11:27,062-Speed 128.56 samples/sec Loss 3.2428 Epoch: 13 Global Step: 133595 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:11:47,002-Speed 128.39 samples/sec Loss 3.2307 Epoch: 13 Global Step: 133600 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:12:06,619-Speed 130.50 samples/sec Loss 3.2289 Epoch: 13 Global Step: 133605 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:12:26,400-Speed 129.42 samples/sec Loss 3.1768 Epoch: 13 Global Step: 133610 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:12:46,090-Speed 130.02 samples/sec Loss 3.2685 Epoch: 13 Global Step: 133615 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:13:06,007-Speed 128.54 samples/sec Loss 3.1697 Epoch: 13 Global Step: 133620 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:13:25,777-Speed 129.49 samples/sec Loss 3.3963 Epoch: 13 Global Step: 133625 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:13:45,694-Speed 128.54 samples/sec Loss 3.1719 Epoch: 13 Global Step: 133630 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:14:05,577-Speed 128.77 samples/sec Loss 3.4475 Epoch: 13 Global Step: 133635 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:14:25,658-Speed 127.49 samples/sec Loss 3.2873 Epoch: 13 Global Step: 133640 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:14:45,455-Speed 129.32 samples/sec Loss 3.2502 Epoch: 13 Global Step: 133645 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:15:05,249-Speed 129.34 samples/sec Loss 3.3828 Epoch: 13 Global Step: 133650 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:15:24,979-Speed 129.76 samples/sec Loss 3.1127 Epoch: 13 Global Step: 133655 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:15:44,733-Speed 129.60 samples/sec Loss 3.2559 Epoch: 13 Global Step: 133660 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:16:04,609-Speed 128.81 samples/sec Loss 3.3252 Epoch: 13 Global Step: 133665 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:16:24,327-Speed 129.84 samples/sec Loss 3.3795 Epoch: 13 Global Step: 133670 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:16:44,513-Speed 126.82 samples/sec Loss 3.2286 Epoch: 13 Global Step: 133675 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:17:04,231-Speed 129.84 samples/sec Loss 3.1712 Epoch: 13 Global Step: 133680 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:17:24,327-Speed 127.40 samples/sec Loss 3.2266 Epoch: 13 Global Step: 133685 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:17:44,060-Speed 129.74 samples/sec Loss 3.5570 Epoch: 13 Global Step: 133690 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:18:03,902-Speed 129.02 samples/sec Loss 3.4714 Epoch: 13 Global Step: 133695 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:18:23,621-Speed 129.83 samples/sec Loss 3.2156 Epoch: 13 Global Step: 133700 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:18:43,496-Speed 128.81 samples/sec Loss 3.3213 Epoch: 13 Global Step: 133705 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:19:03,297-Speed 129.29 samples/sec Loss 3.3777 Epoch: 13 Global Step: 133710 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:19:23,059-Speed 129.55 samples/sec Loss 3.2928 Epoch: 13 Global Step: 133715 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:19:43,097-Speed 127.76 samples/sec Loss 3.3328 Epoch: 13 Global Step: 133720 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:20:02,781-Speed 130.06 samples/sec Loss 3.2339 Epoch: 13 Global Step: 133725 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:20:22,939-Speed 127.00 samples/sec Loss 3.2639 Epoch: 13 Global Step: 133730 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:20:42,840-Speed 128.64 samples/sec Loss 3.4380 Epoch: 13 Global Step: 133735 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:21:02,663-Speed 129.15 samples/sec Loss 3.0746 Epoch: 13 Global Step: 133740 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:21:22,431-Speed 129.51 samples/sec Loss 3.1598 Epoch: 13 Global Step: 133745 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:21:42,181-Speed 129.63 samples/sec Loss 3.3439 Epoch: 13 Global Step: 133750 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:22:01,966-Speed 129.40 samples/sec Loss 3.1430 Epoch: 13 Global Step: 133755 Fp16 Grad Scale: 8192 Required: 137 hours Training: 2021-06-13 09:22:21,937-Speed 128.19 samples/sec Loss 3.3982 Epoch: 13 Global Step: 133760 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:22:41,831-Speed 128.69 samples/sec Loss 3.1668 Epoch: 13 Global Step: 133765 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:23:01,493-Speed 130.21 samples/sec Loss 3.2612 Epoch: 13 Global Step: 133770 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:23:21,612-Speed 127.25 samples/sec Loss 3.1972 Epoch: 13 Global Step: 133775 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:23:41,520-Speed 128.60 samples/sec Loss 3.4127 Epoch: 13 Global Step: 133780 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:24:01,136-Speed 130.51 samples/sec Loss 3.1589 Epoch: 13 Global Step: 133785 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:24:21,032-Speed 128.68 samples/sec Loss 3.1371 Epoch: 13 Global Step: 133790 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:24:40,723-Speed 130.02 samples/sec Loss 3.2569 Epoch: 13 Global Step: 133795 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:25:00,538-Speed 129.20 samples/sec Loss 3.3344 Epoch: 13 Global Step: 133800 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:25:20,392-Speed 128.95 samples/sec Loss 3.2957 Epoch: 13 Global Step: 133805 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:25:40,310-Speed 128.53 samples/sec Loss 3.1864 Epoch: 13 Global Step: 133810 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:26:00,251-Speed 128.39 samples/sec Loss 3.2383 Epoch: 13 Global Step: 133815 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:26:20,181-Speed 128.46 samples/sec Loss 3.1839 Epoch: 13 Global Step: 133820 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:26:40,085-Speed 128.62 samples/sec Loss 3.4063 Epoch: 13 Global Step: 133825 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:26:59,794-Speed 129.90 samples/sec Loss 3.2845 Epoch: 13 Global Step: 133830 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:27:19,748-Speed 128.30 samples/sec Loss 3.3468 Epoch: 13 Global Step: 133835 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:27:39,397-Speed 130.29 samples/sec Loss 3.3118 Epoch: 13 Global Step: 133840 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:27:59,276-Speed 128.79 samples/sec Loss 3.3034 Epoch: 13 Global Step: 133845 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:28:18,985-Speed 129.90 samples/sec Loss 3.3914 Epoch: 13 Global Step: 133850 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:28:39,080-Speed 127.40 samples/sec Loss 3.3860 Epoch: 13 Global Step: 133855 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:28:58,864-Speed 129.40 samples/sec Loss 3.2677 Epoch: 13 Global Step: 133860 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:29:18,951-Speed 127.45 samples/sec Loss 3.1983 Epoch: 13 Global Step: 133865 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:29:38,669-Speed 129.84 samples/sec Loss 3.3534 Epoch: 13 Global Step: 133870 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:29:58,490-Speed 129.16 samples/sec Loss 3.5964 Epoch: 13 Global Step: 133875 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:30:18,402-Speed 128.57 samples/sec Loss 3.3989 Epoch: 13 Global Step: 133880 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:30:38,144-Speed 129.67 samples/sec Loss 3.4191 Epoch: 13 Global Step: 133885 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:30:58,156-Speed 127.93 samples/sec Loss 3.4064 Epoch: 13 Global Step: 133890 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:31:17,900-Speed 129.67 samples/sec Loss 3.1044 Epoch: 13 Global Step: 133895 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:31:37,997-Speed 127.39 samples/sec Loss 3.1738 Epoch: 13 Global Step: 133900 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:31:57,834-Speed 129.06 samples/sec Loss 3.3001 Epoch: 13 Global Step: 133905 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:32:17,863-Speed 127.82 samples/sec Loss 3.4554 Epoch: 13 Global Step: 133910 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:32:37,630-Speed 129.51 samples/sec Loss 3.1809 Epoch: 13 Global Step: 133915 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:32:57,518-Speed 128.73 samples/sec Loss 3.4522 Epoch: 13 Global Step: 133920 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:33:17,240-Speed 129.81 samples/sec Loss 3.3213 Epoch: 13 Global Step: 133925 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:33:37,039-Speed 129.31 samples/sec Loss 3.1949 Epoch: 13 Global Step: 133930 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:33:56,952-Speed 128.57 samples/sec Loss 3.1534 Epoch: 13 Global Step: 133935 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:34:16,603-Speed 130.28 samples/sec Loss 3.3012 Epoch: 13 Global Step: 133940 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:34:36,668-Speed 127.59 samples/sec Loss 3.2961 Epoch: 13 Global Step: 133945 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:34:56,356-Speed 130.03 samples/sec Loss 3.3620 Epoch: 13 Global Step: 133950 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:35:16,496-Speed 127.12 samples/sec Loss 3.3959 Epoch: 13 Global Step: 133955 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:35:36,305-Speed 129.24 samples/sec Loss 3.3281 Epoch: 13 Global Step: 133960 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:35:56,140-Speed 129.06 samples/sec Loss 3.4547 Epoch: 13 Global Step: 133965 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:36:15,820-Speed 130.09 samples/sec Loss 3.4188 Epoch: 13 Global Step: 133970 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:36:35,778-Speed 128.28 samples/sec Loss 3.3418 Epoch: 13 Global Step: 133975 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:36:55,499-Speed 129.82 samples/sec Loss 3.2115 Epoch: 13 Global Step: 133980 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:37:15,195-Speed 129.98 samples/sec Loss 3.2910 Epoch: 13 Global Step: 133985 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:37:35,611-Speed 125.40 samples/sec Loss 3.2273 Epoch: 13 Global Step: 133990 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:37:55,405-Speed 129.34 samples/sec Loss 3.3264 Epoch: 13 Global Step: 133995 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:38:15,485-Speed 127.50 samples/sec Loss 3.4481 Epoch: 13 Global Step: 134000 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:38:35,367-Speed 128.76 samples/sec Loss 3.4463 Epoch: 13 Global Step: 134005 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:38:55,023-Speed 130.25 samples/sec Loss 3.4911 Epoch: 13 Global Step: 134010 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:39:14,833-Speed 129.23 samples/sec Loss 3.2590 Epoch: 13 Global Step: 134015 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:39:34,676-Speed 129.02 samples/sec Loss 3.4688 Epoch: 13 Global Step: 134020 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:39:54,459-Speed 129.41 samples/sec Loss 3.2221 Epoch: 13 Global Step: 134025 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:40:14,352-Speed 128.69 samples/sec Loss 3.1575 Epoch: 13 Global Step: 134030 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:40:34,324-Speed 128.19 samples/sec Loss 3.3599 Epoch: 13 Global Step: 134035 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:40:54,084-Speed 129.56 samples/sec Loss 3.1924 Epoch: 13 Global Step: 134040 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:41:14,026-Speed 128.38 samples/sec Loss 3.4880 Epoch: 13 Global Step: 134045 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:41:33,976-Speed 128.33 samples/sec Loss 3.3436 Epoch: 13 Global Step: 134050 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:41:53,682-Speed 129.91 samples/sec Loss 3.4418 Epoch: 13 Global Step: 134055 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:42:13,563-Speed 128.78 samples/sec Loss 3.0738 Epoch: 13 Global Step: 134060 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:42:33,230-Speed 130.17 samples/sec Loss 3.2379 Epoch: 13 Global Step: 134065 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:42:53,025-Speed 129.33 samples/sec Loss 3.3991 Epoch: 13 Global Step: 134070 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:43:12,742-Speed 129.84 samples/sec Loss 3.3483 Epoch: 13 Global Step: 134075 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:43:32,803-Speed 127.62 samples/sec Loss 3.4119 Epoch: 13 Global Step: 134080 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:43:52,671-Speed 128.86 samples/sec Loss 3.2336 Epoch: 13 Global Step: 134085 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:44:12,644-Speed 128.18 samples/sec Loss 3.3917 Epoch: 13 Global Step: 134090 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:44:32,604-Speed 128.26 samples/sec Loss 3.3842 Epoch: 13 Global Step: 134095 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:44:52,432-Speed 129.12 samples/sec Loss 3.2915 Epoch: 13 Global Step: 134100 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:45:12,301-Speed 128.85 samples/sec Loss 3.4018 Epoch: 13 Global Step: 134105 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:45:31,918-Speed 130.51 samples/sec Loss 3.2101 Epoch: 13 Global Step: 134110 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:45:51,812-Speed 128.68 samples/sec Loss 3.4493 Epoch: 13 Global Step: 134115 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:46:11,497-Speed 130.05 samples/sec Loss 3.2659 Epoch: 13 Global Step: 134120 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:46:31,606-Speed 127.32 samples/sec Loss 3.3640 Epoch: 13 Global Step: 134125 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:46:51,433-Speed 129.12 samples/sec Loss 3.4059 Epoch: 13 Global Step: 134130 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:47:11,409-Speed 128.16 samples/sec Loss 3.3969 Epoch: 13 Global Step: 134135 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:47:31,198-Speed 129.37 samples/sec Loss 3.3338 Epoch: 13 Global Step: 134140 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:47:51,027-Speed 129.11 samples/sec Loss 3.3076 Epoch: 13 Global Step: 134145 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:48:10,738-Speed 129.89 samples/sec Loss 3.1549 Epoch: 13 Global Step: 134150 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:48:30,508-Speed 129.50 samples/sec Loss 3.3233 Epoch: 13 Global Step: 134155 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:48:50,356-Speed 128.98 samples/sec Loss 3.4607 Epoch: 13 Global Step: 134160 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:49:10,086-Speed 129.76 samples/sec Loss 3.2050 Epoch: 13 Global Step: 134165 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:49:30,185-Speed 127.37 samples/sec Loss 3.1867 Epoch: 13 Global Step: 134170 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:49:49,911-Speed 129.78 samples/sec Loss 3.1100 Epoch: 13 Global Step: 134175 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:50:10,033-Speed 127.23 samples/sec Loss 3.3693 Epoch: 13 Global Step: 134180 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:50:29,759-Speed 129.78 samples/sec Loss 3.3355 Epoch: 13 Global Step: 134185 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:50:49,658-Speed 128.65 samples/sec Loss 3.1385 Epoch: 13 Global Step: 134190 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:51:09,259-Speed 130.62 samples/sec Loss 3.3379 Epoch: 13 Global Step: 134195 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:51:29,261-Speed 127.99 samples/sec Loss 3.2652 Epoch: 13 Global Step: 134200 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:51:48,982-Speed 129.81 samples/sec Loss 3.2813 Epoch: 13 Global Step: 134205 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:52:08,802-Speed 129.17 samples/sec Loss 3.3917 Epoch: 13 Global Step: 134210 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:52:28,821-Speed 127.88 samples/sec Loss 3.3070 Epoch: 13 Global Step: 134215 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:52:48,596-Speed 129.46 samples/sec Loss 3.4170 Epoch: 13 Global Step: 134220 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:53:08,693-Speed 127.39 samples/sec Loss 3.4616 Epoch: 13 Global Step: 134225 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:53:28,601-Speed 128.60 samples/sec Loss 3.1081 Epoch: 13 Global Step: 134230 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:53:48,205-Speed 130.59 samples/sec Loss 3.2624 Epoch: 13 Global Step: 134235 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:54:08,186-Speed 128.13 samples/sec Loss 3.3980 Epoch: 13 Global Step: 134240 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:54:27,885-Speed 129.96 samples/sec Loss 3.2377 Epoch: 13 Global Step: 134245 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:54:47,799-Speed 128.56 samples/sec Loss 3.3116 Epoch: 13 Global Step: 134250 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:55:07,401-Speed 130.60 samples/sec Loss 3.3537 Epoch: 13 Global Step: 134255 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:55:27,376-Speed 128.16 samples/sec Loss 3.3111 Epoch: 13 Global Step: 134260 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:55:47,301-Speed 128.49 samples/sec Loss 3.2469 Epoch: 13 Global Step: 134265 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:56:07,290-Speed 128.08 samples/sec Loss 3.3424 Epoch: 13 Global Step: 134270 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:56:27,019-Speed 129.76 samples/sec Loss 3.3960 Epoch: 13 Global Step: 134275 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:56:46,792-Speed 129.48 samples/sec Loss 3.2973 Epoch: 13 Global Step: 134280 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:57:06,675-Speed 128.75 samples/sec Loss 3.3301 Epoch: 13 Global Step: 134285 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:57:26,641-Speed 128.23 samples/sec Loss 3.3145 Epoch: 13 Global Step: 134290 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:57:46,441-Speed 129.30 samples/sec Loss 3.4492 Epoch: 13 Global Step: 134295 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:58:06,044-Speed 130.60 samples/sec Loss 3.1380 Epoch: 13 Global Step: 134300 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:58:26,226-Speed 126.85 samples/sec Loss 3.4211 Epoch: 13 Global Step: 134305 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:58:45,954-Speed 129.78 samples/sec Loss 3.3454 Epoch: 13 Global Step: 134310 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:59:05,922-Speed 128.21 samples/sec Loss 3.4633 Epoch: 13 Global Step: 134315 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:59:25,619-Speed 129.97 samples/sec Loss 3.2334 Epoch: 13 Global Step: 134320 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 09:59:45,583-Speed 128.24 samples/sec Loss 3.2653 Epoch: 13 Global Step: 134325 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:00:05,308-Speed 129.79 samples/sec Loss 3.1701 Epoch: 13 Global Step: 134330 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:00:25,062-Speed 129.60 samples/sec Loss 3.2447 Epoch: 13 Global Step: 134335 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:00:44,913-Speed 128.97 samples/sec Loss 3.4196 Epoch: 13 Global Step: 134340 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:01:04,512-Speed 130.63 samples/sec Loss 3.4160 Epoch: 13 Global Step: 134345 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:01:24,772-Speed 126.36 samples/sec Loss 3.3728 Epoch: 13 Global Step: 134350 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:01:44,365-Speed 130.66 samples/sec Loss 3.2713 Epoch: 13 Global Step: 134355 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:02:04,493-Speed 127.20 samples/sec Loss 3.4429 Epoch: 13 Global Step: 134360 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:02:24,455-Speed 128.25 samples/sec Loss 3.3204 Epoch: 13 Global Step: 134365 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:02:44,321-Speed 128.87 samples/sec Loss 3.2579 Epoch: 13 Global Step: 134370 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:03:04,046-Speed 129.79 samples/sec Loss 3.2395 Epoch: 13 Global Step: 134375 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:03:23,866-Speed 129.17 samples/sec Loss 3.3691 Epoch: 13 Global Step: 134380 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:03:43,670-Speed 129.27 samples/sec Loss 3.3794 Epoch: 13 Global Step: 134385 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:04:03,600-Speed 128.46 samples/sec Loss 3.3195 Epoch: 13 Global Step: 134390 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:04:23,593-Speed 128.05 samples/sec Loss 3.3771 Epoch: 13 Global Step: 134395 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:04:43,351-Speed 129.58 samples/sec Loss 3.1881 Epoch: 13 Global Step: 134400 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:05:03,326-Speed 128.17 samples/sec Loss 3.3123 Epoch: 13 Global Step: 134405 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:05:23,279-Speed 128.30 samples/sec Loss 3.3969 Epoch: 13 Global Step: 134410 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:05:43,042-Speed 129.55 samples/sec Loss 3.1878 Epoch: 13 Global Step: 134415 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:06:02,940-Speed 128.66 samples/sec Loss 3.4058 Epoch: 13 Global Step: 134420 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:06:22,663-Speed 129.80 samples/sec Loss 3.3194 Epoch: 13 Global Step: 134425 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:06:42,542-Speed 128.79 samples/sec Loss 3.2736 Epoch: 13 Global Step: 134430 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:07:02,407-Speed 128.88 samples/sec Loss 3.3283 Epoch: 13 Global Step: 134435 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:07:22,452-Speed 127.72 samples/sec Loss 3.2773 Epoch: 13 Global Step: 134440 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:07:42,283-Speed 129.10 samples/sec Loss 3.4692 Epoch: 13 Global Step: 134445 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:08:02,262-Speed 128.14 samples/sec Loss 3.5559 Epoch: 13 Global Step: 134450 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:08:22,167-Speed 128.61 samples/sec Loss 3.3647 Epoch: 13 Global Step: 134455 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:08:41,903-Speed 129.72 samples/sec Loss 3.3167 Epoch: 13 Global Step: 134460 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:09:01,783-Speed 128.77 samples/sec Loss 3.3930 Epoch: 13 Global Step: 134465 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:09:21,399-Speed 130.52 samples/sec Loss 3.7089 Epoch: 13 Global Step: 134470 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:09:41,176-Speed 129.45 samples/sec Loss 3.2331 Epoch: 13 Global Step: 134475 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:10:01,025-Speed 128.98 samples/sec Loss 3.5114 Epoch: 13 Global Step: 134480 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:10:21,031-Speed 127.97 samples/sec Loss 3.3254 Epoch: 13 Global Step: 134485 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:10:40,867-Speed 129.07 samples/sec Loss 3.4765 Epoch: 13 Global Step: 134490 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:11:00,876-Speed 127.95 samples/sec Loss 3.1318 Epoch: 13 Global Step: 134495 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:11:20,857-Speed 128.12 samples/sec Loss 3.3458 Epoch: 13 Global Step: 134500 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:11:40,902-Speed 127.72 samples/sec Loss 3.2557 Epoch: 13 Global Step: 134505 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:12:00,802-Speed 128.65 samples/sec Loss 3.2428 Epoch: 13 Global Step: 134510 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:12:20,492-Speed 130.02 samples/sec Loss 3.4517 Epoch: 13 Global Step: 134515 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:12:40,395-Speed 128.63 samples/sec Loss 3.2583 Epoch: 13 Global Step: 134520 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:13:00,088-Speed 130.00 samples/sec Loss 3.4344 Epoch: 13 Global Step: 134525 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:13:20,045-Speed 128.28 samples/sec Loss 3.3958 Epoch: 13 Global Step: 134530 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:13:39,913-Speed 128.86 samples/sec Loss 3.6288 Epoch: 13 Global Step: 134535 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:13:59,981-Speed 127.57 samples/sec Loss 3.1802 Epoch: 13 Global Step: 134540 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:14:19,725-Speed 129.67 samples/sec Loss 3.3870 Epoch: 13 Global Step: 134545 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:14:39,445-Speed 129.82 samples/sec Loss 3.2317 Epoch: 13 Global Step: 134550 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:14:59,210-Speed 129.53 samples/sec Loss 3.3037 Epoch: 13 Global Step: 134555 Fp16 Grad Scale: 8192 Required: 136 hours Training: 2021-06-13 10:15:19,020-Speed 129.23 samples/sec Loss 3.2696 Epoch: 13 Global Step: 134560 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:15:38,925-Speed 128.62 samples/sec Loss 3.3328 Epoch: 13 Global Step: 134565 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:15:58,658-Speed 129.74 samples/sec Loss 3.2944 Epoch: 13 Global Step: 134570 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:16:18,847-Speed 126.80 samples/sec Loss 3.4701 Epoch: 13 Global Step: 134575 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:16:38,615-Speed 129.51 samples/sec Loss 3.4036 Epoch: 13 Global Step: 134580 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:16:58,698-Speed 127.48 samples/sec Loss 3.4172 Epoch: 13 Global Step: 134585 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:17:18,393-Speed 129.99 samples/sec Loss 3.5136 Epoch: 13 Global Step: 134590 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:17:38,362-Speed 128.21 samples/sec Loss 3.4271 Epoch: 13 Global Step: 134595 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:17:58,134-Speed 129.48 samples/sec Loss 3.3793 Epoch: 13 Global Step: 134600 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:18:17,841-Speed 129.91 samples/sec Loss 3.2571 Epoch: 13 Global Step: 134605 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:18:37,717-Speed 128.81 samples/sec Loss 3.3180 Epoch: 13 Global Step: 134610 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:18:57,317-Speed 130.62 samples/sec Loss 3.3431 Epoch: 13 Global Step: 134615 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:19:17,576-Speed 126.37 samples/sec Loss 3.4881 Epoch: 13 Global Step: 134620 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:19:37,342-Speed 129.53 samples/sec Loss 3.2959 Epoch: 13 Global Step: 134625 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:19:57,433-Speed 127.43 samples/sec Loss 3.2364 Epoch: 13 Global Step: 134630 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:20:17,211-Speed 129.44 samples/sec Loss 3.4046 Epoch: 13 Global Step: 134635 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:20:37,015-Speed 129.27 samples/sec Loss 3.1971 Epoch: 13 Global Step: 134640 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:20:56,696-Speed 130.08 samples/sec Loss 3.4111 Epoch: 13 Global Step: 134645 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:21:16,617-Speed 128.52 samples/sec Loss 3.4402 Epoch: 13 Global Step: 134650 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:21:36,350-Speed 129.74 samples/sec Loss 3.3867 Epoch: 13 Global Step: 134655 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:21:56,201-Speed 128.97 samples/sec Loss 3.1706 Epoch: 13 Global Step: 134660 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:22:16,367-Speed 126.95 samples/sec Loss 3.4760 Epoch: 13 Global Step: 134665 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:22:36,062-Speed 129.99 samples/sec Loss 3.4481 Epoch: 13 Global Step: 134670 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:22:56,118-Speed 127.65 samples/sec Loss 3.4903 Epoch: 13 Global Step: 134675 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:23:16,096-Speed 128.15 samples/sec Loss 3.2907 Epoch: 13 Global Step: 134680 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:23:35,704-Speed 130.56 samples/sec Loss 3.3607 Epoch: 13 Global Step: 134685 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:23:55,446-Speed 129.68 samples/sec Loss 3.3446 Epoch: 13 Global Step: 134690 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:24:15,311-Speed 128.88 samples/sec Loss 3.4133 Epoch: 13 Global Step: 134695 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:24:35,030-Speed 129.83 samples/sec Loss 3.4733 Epoch: 13 Global Step: 134700 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:24:54,985-Speed 128.29 samples/sec Loss 3.4250 Epoch: 13 Global Step: 134705 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:25:14,931-Speed 128.35 samples/sec Loss 3.3697 Epoch: 13 Global Step: 134710 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:25:34,707-Speed 129.46 samples/sec Loss 3.5093 Epoch: 13 Global Step: 134715 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:25:54,798-Speed 127.43 samples/sec Loss 3.5735 Epoch: 13 Global Step: 134720 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:26:14,580-Speed 129.41 samples/sec Loss 3.4305 Epoch: 13 Global Step: 134725 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:26:34,292-Speed 129.87 samples/sec Loss 3.4441 Epoch: 13 Global Step: 134730 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:26:54,425-Speed 127.16 samples/sec Loss 3.4706 Epoch: 13 Global Step: 134735 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:27:14,266-Speed 129.03 samples/sec Loss 3.3972 Epoch: 13 Global Step: 134740 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:27:34,077-Speed 129.23 samples/sec Loss 3.3877 Epoch: 13 Global Step: 134745 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:27:53,950-Speed 128.82 samples/sec Loss 3.2857 Epoch: 13 Global Step: 134750 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:28:13,991-Speed 127.75 samples/sec Loss 3.3752 Epoch: 13 Global Step: 134755 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:28:33,774-Speed 129.41 samples/sec Loss 3.5229 Epoch: 13 Global Step: 134760 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:28:53,789-Speed 127.91 samples/sec Loss 3.2820 Epoch: 13 Global Step: 134765 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:29:13,680-Speed 128.70 samples/sec Loss 3.2866 Epoch: 13 Global Step: 134770 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:29:33,311-Speed 130.41 samples/sec Loss 3.1739 Epoch: 13 Global Step: 134775 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:29:53,210-Speed 128.66 samples/sec Loss 3.4101 Epoch: 13 Global Step: 134780 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:30:12,983-Speed 129.48 samples/sec Loss 3.1581 Epoch: 13 Global Step: 134785 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:30:32,794-Speed 129.23 samples/sec Loss 3.3168 Epoch: 13 Global Step: 134790 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:30:52,479-Speed 130.05 samples/sec Loss 3.2915 Epoch: 13 Global Step: 134795 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:31:12,505-Speed 127.84 samples/sec Loss 3.4326 Epoch: 13 Global Step: 134800 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:31:32,480-Speed 128.17 samples/sec Loss 3.6573 Epoch: 13 Global Step: 134805 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:31:52,409-Speed 128.46 samples/sec Loss 3.4550 Epoch: 13 Global Step: 134810 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:32:12,186-Speed 129.45 samples/sec Loss 3.1459 Epoch: 13 Global Step: 134815 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:32:31,913-Speed 129.78 samples/sec Loss 3.3387 Epoch: 13 Global Step: 134820 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:32:51,751-Speed 129.05 samples/sec Loss 3.3607 Epoch: 13 Global Step: 134825 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:33:11,506-Speed 129.59 samples/sec Loss 3.5225 Epoch: 13 Global Step: 134830 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:33:31,423-Speed 128.54 samples/sec Loss 3.3177 Epoch: 13 Global Step: 134835 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:33:51,025-Speed 130.60 samples/sec Loss 3.3484 Epoch: 13 Global Step: 134840 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:34:11,132-Speed 127.33 samples/sec Loss 3.4513 Epoch: 13 Global Step: 134845 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:34:30,900-Speed 129.51 samples/sec Loss 3.4474 Epoch: 13 Global Step: 134850 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:34:50,911-Speed 127.94 samples/sec Loss 3.2440 Epoch: 13 Global Step: 134855 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:35:10,628-Speed 129.84 samples/sec Loss 3.5209 Epoch: 13 Global Step: 134860 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:35:30,443-Speed 129.20 samples/sec Loss 3.2835 Epoch: 13 Global Step: 134865 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:35:50,361-Speed 128.54 samples/sec Loss 3.4274 Epoch: 13 Global Step: 134870 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:36:09,975-Speed 130.53 samples/sec Loss 3.4000 Epoch: 13 Global Step: 134875 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:36:29,896-Speed 128.51 samples/sec Loss 3.3340 Epoch: 13 Global Step: 134880 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:36:49,596-Speed 129.96 samples/sec Loss 3.4334 Epoch: 13 Global Step: 134885 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:37:09,831-Speed 126.52 samples/sec Loss 3.4719 Epoch: 13 Global Step: 134890 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:37:29,629-Speed 129.31 samples/sec Loss 3.3350 Epoch: 13 Global Step: 134895 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:37:49,652-Speed 127.86 samples/sec Loss 3.3745 Epoch: 13 Global Step: 134900 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:38:09,508-Speed 128.93 samples/sec Loss 3.2904 Epoch: 13 Global Step: 134905 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:38:29,456-Speed 128.34 samples/sec Loss 3.3568 Epoch: 13 Global Step: 134910 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:38:49,099-Speed 130.33 samples/sec Loss 3.3029 Epoch: 13 Global Step: 134915 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:39:09,000-Speed 128.65 samples/sec Loss 3.4271 Epoch: 13 Global Step: 134920 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:39:28,713-Speed 129.87 samples/sec Loss 3.6305 Epoch: 13 Global Step: 134925 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:39:48,388-Speed 130.12 samples/sec Loss 3.4454 Epoch: 13 Global Step: 134930 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:40:08,492-Speed 127.35 samples/sec Loss 3.5095 Epoch: 13 Global Step: 134935 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:40:28,317-Speed 129.14 samples/sec Loss 3.3502 Epoch: 13 Global Step: 134940 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:40:48,405-Speed 127.45 samples/sec Loss 3.5037 Epoch: 13 Global Step: 134945 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:41:08,224-Speed 129.17 samples/sec Loss 3.0582 Epoch: 13 Global Step: 134950 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:41:28,047-Speed 129.15 samples/sec Loss 3.3724 Epoch: 13 Global Step: 134955 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:41:47,762-Speed 129.86 samples/sec Loss 3.2256 Epoch: 13 Global Step: 134960 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:42:07,585-Speed 129.15 samples/sec Loss 3.3952 Epoch: 13 Global Step: 134965 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:42:27,308-Speed 129.80 samples/sec Loss 3.1835 Epoch: 13 Global Step: 134970 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:42:47,090-Speed 129.41 samples/sec Loss 3.5523 Epoch: 13 Global Step: 134975 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:43:07,151-Speed 127.62 samples/sec Loss 3.3666 Epoch: 13 Global Step: 134980 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:43:26,920-Speed 129.51 samples/sec Loss 3.4146 Epoch: 13 Global Step: 134985 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:43:47,041-Speed 127.24 samples/sec Loss 3.1056 Epoch: 13 Global Step: 134990 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:44:06,935-Speed 128.69 samples/sec Loss 3.2864 Epoch: 13 Global Step: 134995 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:44:26,721-Speed 129.38 samples/sec Loss 3.2754 Epoch: 13 Global Step: 135000 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:44:46,598-Speed 128.80 samples/sec Loss 3.3947 Epoch: 13 Global Step: 135005 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:45:06,324-Speed 129.78 samples/sec Loss 3.3845 Epoch: 13 Global Step: 135010 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:45:26,146-Speed 129.16 samples/sec Loss 3.2818 Epoch: 13 Global Step: 135015 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:45:46,015-Speed 128.85 samples/sec Loss 3.0971 Epoch: 13 Global Step: 135020 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:46:05,975-Speed 128.26 samples/sec Loss 3.2067 Epoch: 13 Global Step: 135025 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:46:25,837-Speed 128.90 samples/sec Loss 3.4780 Epoch: 13 Global Step: 135030 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:46:45,880-Speed 127.73 samples/sec Loss 3.3812 Epoch: 13 Global Step: 135035 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:47:05,746-Speed 128.87 samples/sec Loss 3.3397 Epoch: 13 Global Step: 135040 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:47:25,377-Speed 130.41 samples/sec Loss 3.1456 Epoch: 13 Global Step: 135045 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:47:45,276-Speed 128.66 samples/sec Loss 3.3172 Epoch: 13 Global Step: 135050 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:48:04,910-Speed 130.39 samples/sec Loss 3.4457 Epoch: 13 Global Step: 135055 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:48:24,865-Speed 128.30 samples/sec Loss 3.4025 Epoch: 13 Global Step: 135060 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:48:44,704-Speed 129.05 samples/sec Loss 3.3880 Epoch: 13 Global Step: 135065 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:49:04,790-Speed 127.46 samples/sec Loss 3.1443 Epoch: 13 Global Step: 135070 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:49:24,713-Speed 128.50 samples/sec Loss 3.1371 Epoch: 13 Global Step: 135075 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:49:44,799-Speed 127.46 samples/sec Loss 3.4854 Epoch: 13 Global Step: 135080 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:50:04,522-Speed 129.81 samples/sec Loss 3.5344 Epoch: 13 Global Step: 135085 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:50:24,277-Speed 129.59 samples/sec Loss 3.4597 Epoch: 13 Global Step: 135090 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:50:44,179-Speed 128.64 samples/sec Loss 3.4695 Epoch: 13 Global Step: 135095 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:51:03,944-Speed 129.53 samples/sec Loss 3.3523 Epoch: 13 Global Step: 135100 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:51:23,751-Speed 129.25 samples/sec Loss 3.4104 Epoch: 13 Global Step: 135105 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:51:43,472-Speed 129.82 samples/sec Loss 3.5356 Epoch: 13 Global Step: 135110 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:52:03,735-Speed 126.34 samples/sec Loss 3.5962 Epoch: 13 Global Step: 135115 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:52:23,455-Speed 129.82 samples/sec Loss 3.2774 Epoch: 13 Global Step: 135120 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:52:43,549-Speed 127.41 samples/sec Loss 3.2366 Epoch: 13 Global Step: 135125 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:53:03,289-Speed 129.69 samples/sec Loss 3.4361 Epoch: 13 Global Step: 135130 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:53:23,203-Speed 128.56 samples/sec Loss 3.2306 Epoch: 13 Global Step: 135135 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:53:42,978-Speed 129.46 samples/sec Loss 3.3117 Epoch: 13 Global Step: 135140 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:54:02,850-Speed 128.83 samples/sec Loss 3.3221 Epoch: 13 Global Step: 135145 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:54:22,554-Speed 129.93 samples/sec Loss 3.5495 Epoch: 13 Global Step: 135150 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:54:42,300-Speed 129.65 samples/sec Loss 3.5213 Epoch: 13 Global Step: 135155 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:55:02,480-Speed 126.87 samples/sec Loss 3.2485 Epoch: 13 Global Step: 135160 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:55:22,193-Speed 129.87 samples/sec Loss 3.4904 Epoch: 13 Global Step: 135165 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:55:42,252-Speed 127.63 samples/sec Loss 3.4560 Epoch: 13 Global Step: 135170 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:56:01,928-Speed 130.11 samples/sec Loss 3.3930 Epoch: 13 Global Step: 135175 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:56:21,856-Speed 128.47 samples/sec Loss 3.4385 Epoch: 13 Global Step: 135180 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:56:41,485-Speed 130.43 samples/sec Loss 3.3563 Epoch: 13 Global Step: 135185 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:57:01,398-Speed 128.57 samples/sec Loss 3.4349 Epoch: 13 Global Step: 135190 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:57:21,133-Speed 129.73 samples/sec Loss 3.1346 Epoch: 13 Global Step: 135195 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:57:40,984-Speed 128.97 samples/sec Loss 3.3182 Epoch: 13 Global Step: 135200 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:58:01,020-Speed 127.77 samples/sec Loss 3.4908 Epoch: 13 Global Step: 135205 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:58:20,717-Speed 129.98 samples/sec Loss 3.4120 Epoch: 13 Global Step: 135210 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:58:40,820-Speed 127.35 samples/sec Loss 3.5858 Epoch: 13 Global Step: 135215 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:59:00,590-Speed 129.49 samples/sec Loss 3.4300 Epoch: 13 Global Step: 135220 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:59:20,358-Speed 129.51 samples/sec Loss 3.3625 Epoch: 13 Global Step: 135225 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:59:40,063-Speed 129.92 samples/sec Loss 3.3012 Epoch: 13 Global Step: 135230 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 10:59:59,950-Speed 128.74 samples/sec Loss 3.4047 Epoch: 13 Global Step: 135235 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 11:00:19,736-Speed 129.39 samples/sec Loss 3.4623 Epoch: 13 Global Step: 135240 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 11:00:39,607-Speed 128.83 samples/sec Loss 3.4407 Epoch: 13 Global Step: 135245 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 11:00:59,662-Speed 127.66 samples/sec Loss 3.5321 Epoch: 13 Global Step: 135250 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 11:01:19,456-Speed 129.34 samples/sec Loss 3.3476 Epoch: 13 Global Step: 135255 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 11:01:39,557-Speed 127.36 samples/sec Loss 3.3609 Epoch: 13 Global Step: 135260 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 11:01:59,349-Speed 129.35 samples/sec Loss 3.3530 Epoch: 13 Global Step: 135265 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 11:02:19,050-Speed 129.94 samples/sec Loss 3.4902 Epoch: 13 Global Step: 135270 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 11:02:38,947-Speed 128.67 samples/sec Loss 3.3644 Epoch: 13 Global Step: 135275 Fp16 Grad Scale: 8192 Required: 135 hours Training: 2021-06-13 11:02:58,493-Speed 130.98 samples/sec Loss 3.3223 Epoch: 13 Global Step: 135280 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:03:18,241-Speed 129.64 samples/sec Loss 3.3507 Epoch: 13 Global Step: 135285 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:03:38,022-Speed 129.43 samples/sec Loss 3.5026 Epoch: 13 Global Step: 135290 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:03:58,013-Speed 128.06 samples/sec Loss 3.4270 Epoch: 13 Global Step: 135295 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:04:17,811-Speed 129.31 samples/sec Loss 3.5661 Epoch: 13 Global Step: 135300 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:04:37,760-Speed 128.34 samples/sec Loss 3.4373 Epoch: 13 Global Step: 135305 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:04:57,569-Speed 129.24 samples/sec Loss 3.3575 Epoch: 13 Global Step: 135310 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:05:17,224-Speed 130.25 samples/sec Loss 3.3566 Epoch: 13 Global Step: 135315 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:05:37,205-Speed 128.13 samples/sec Loss 3.5197 Epoch: 13 Global Step: 135320 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:05:56,926-Speed 129.82 samples/sec Loss 3.4787 Epoch: 13 Global Step: 135325 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:06:16,571-Speed 130.32 samples/sec Loss 3.4627 Epoch: 13 Global Step: 135330 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:06:36,349-Speed 129.44 samples/sec Loss 3.4805 Epoch: 13 Global Step: 135335 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:06:56,418-Speed 127.56 samples/sec Loss 3.4061 Epoch: 13 Global Step: 135340 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:07:16,349-Speed 128.45 samples/sec Loss 3.4413 Epoch: 13 Global Step: 135345 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:07:36,182-Speed 129.08 samples/sec Loss 3.4002 Epoch: 13 Global Step: 135350 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:07:56,040-Speed 128.92 samples/sec Loss 3.3758 Epoch: 13 Global Step: 135355 Fp16 Grad Scale: 4096 Required: 135 hours Training: 2021-06-13 11:08:15,822-Speed 129.41 samples/sec Loss 3.5088 Epoch: 13 Global Step: 135360 Fp16 Grad Scale: 4096 Required: 134 hours Training: 2021-06-13 11:08:35,628-Speed 129.26 samples/sec Loss 3.5111 Epoch: 13 Global Step: 135365 Fp16 Grad Scale: 4096 Required: 134 hours Training: 2021-06-13 11:08:55,368-Speed 129.69 samples/sec Loss 3.1823 Epoch: 13 Global Step: 135370 Fp16 Grad Scale: 4096 Required: 134 hours Training: 2021-06-13 11:09:15,205-Speed 129.06 samples/sec Loss 3.4884 Epoch: 13 Global Step: 135375 Fp16 Grad Scale: 4096 Required: 134 hours Training: 2021-06-13 11:09:34,920-Speed 129.86 samples/sec Loss 3.3880 Epoch: 13 Global Step: 135380 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:09:54,921-Speed 128.00 samples/sec Loss 3.4865 Epoch: 13 Global Step: 135385 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:10:14,799-Speed 128.79 samples/sec Loss 3.4457 Epoch: 13 Global Step: 135390 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:10:34,870-Speed 127.56 samples/sec Loss 3.1933 Epoch: 13 Global Step: 135395 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:10:54,666-Speed 129.32 samples/sec Loss 3.4121 Epoch: 13 Global Step: 135400 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:11:14,305-Speed 130.36 samples/sec Loss 3.5640 Epoch: 13 Global Step: 135405 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:11:34,061-Speed 129.59 samples/sec Loss 3.3595 Epoch: 13 Global Step: 135410 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:11:53,778-Speed 129.84 samples/sec Loss 3.2094 Epoch: 13 Global Step: 135415 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:12:13,737-Speed 128.27 samples/sec Loss 3.1583 Epoch: 13 Global Step: 135420 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:12:33,470-Speed 129.74 samples/sec Loss 3.3026 Epoch: 13 Global Step: 135425 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:12:53,498-Speed 127.82 samples/sec Loss 3.6453 Epoch: 13 Global Step: 135430 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:13:13,211-Speed 129.88 samples/sec Loss 3.5037 Epoch: 13 Global Step: 135435 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:13:33,369-Speed 127.00 samples/sec Loss 3.3881 Epoch: 13 Global Step: 135440 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:13:53,075-Speed 129.91 samples/sec Loss 3.5358 Epoch: 13 Global Step: 135445 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:14:12,888-Speed 129.21 samples/sec Loss 3.3952 Epoch: 13 Global Step: 135450 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:14:32,600-Speed 129.88 samples/sec Loss 3.5153 Epoch: 13 Global Step: 135455 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:14:52,499-Speed 128.66 samples/sec Loss 3.1619 Epoch: 13 Global Step: 135460 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:15:12,313-Speed 129.21 samples/sec Loss 3.4397 Epoch: 13 Global Step: 135465 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:15:32,018-Speed 129.92 samples/sec Loss 3.6022 Epoch: 13 Global Step: 135470 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:15:52,042-Speed 127.85 samples/sec Loss 3.5541 Epoch: 13 Global Step: 135475 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:16:11,845-Speed 129.29 samples/sec Loss 3.4457 Epoch: 13 Global Step: 135480 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:16:31,929-Speed 127.47 samples/sec Loss 3.4574 Epoch: 13 Global Step: 135485 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:16:51,571-Speed 130.33 samples/sec Loss 3.5251 Epoch: 13 Global Step: 135490 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:17:11,441-Speed 128.84 samples/sec Loss 3.5094 Epoch: 13 Global Step: 135495 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:17:31,241-Speed 129.30 samples/sec Loss 3.2847 Epoch: 13 Global Step: 135500 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:17:51,151-Speed 128.59 samples/sec Loss 3.5227 Epoch: 13 Global Step: 135505 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:18:10,850-Speed 129.96 samples/sec Loss 3.3632 Epoch: 13 Global Step: 135510 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:18:30,534-Speed 130.06 samples/sec Loss 3.2808 Epoch: 13 Global Step: 135515 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:18:50,717-Speed 126.85 samples/sec Loss 3.2529 Epoch: 13 Global Step: 135520 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:19:10,473-Speed 129.59 samples/sec Loss 3.2569 Epoch: 13 Global Step: 135525 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:19:30,576-Speed 127.35 samples/sec Loss 3.4764 Epoch: 13 Global Step: 135530 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:19:50,485-Speed 128.59 samples/sec Loss 3.1974 Epoch: 13 Global Step: 135535 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:20:10,218-Speed 129.74 samples/sec Loss 3.3102 Epoch: 13 Global Step: 135540 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:20:29,986-Speed 129.51 samples/sec Loss 3.2013 Epoch: 13 Global Step: 135545 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:20:49,930-Speed 128.36 samples/sec Loss 3.4848 Epoch: 13 Global Step: 135550 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:21:09,858-Speed 128.47 samples/sec Loss 3.3804 Epoch: 13 Global Step: 135555 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:21:29,750-Speed 128.70 samples/sec Loss 3.3752 Epoch: 13 Global Step: 135560 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:21:49,807-Speed 127.64 samples/sec Loss 3.5639 Epoch: 13 Global Step: 135565 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:22:09,581-Speed 129.47 samples/sec Loss 3.4650 Epoch: 13 Global Step: 135570 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:22:29,739-Speed 127.01 samples/sec Loss 3.5260 Epoch: 13 Global Step: 135575 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:22:49,636-Speed 128.66 samples/sec Loss 3.3394 Epoch: 13 Global Step: 135580 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:23:09,416-Speed 129.43 samples/sec Loss 3.4512 Epoch: 13 Global Step: 135585 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:23:29,404-Speed 128.08 samples/sec Loss 3.4463 Epoch: 13 Global Step: 135590 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:23:49,023-Speed 130.49 samples/sec Loss 3.4691 Epoch: 13 Global Step: 135595 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:24:08,922-Speed 128.66 samples/sec Loss 3.2967 Epoch: 13 Global Step: 135600 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:24:28,655-Speed 129.74 samples/sec Loss 3.4849 Epoch: 13 Global Step: 135605 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:24:48,707-Speed 127.67 samples/sec Loss 3.2362 Epoch: 13 Global Step: 135610 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:25:08,586-Speed 128.79 samples/sec Loss 3.5077 Epoch: 13 Global Step: 135615 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:25:28,676-Speed 127.44 samples/sec Loss 3.5158 Epoch: 13 Global Step: 135620 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:25:48,461-Speed 129.39 samples/sec Loss 3.4881 Epoch: 13 Global Step: 135625 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:26:08,197-Speed 129.72 samples/sec Loss 3.2049 Epoch: 13 Global Step: 135630 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:26:28,003-Speed 129.26 samples/sec Loss 3.4694 Epoch: 13 Global Step: 135635 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:26:47,625-Speed 130.47 samples/sec Loss 3.5098 Epoch: 13 Global Step: 135640 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:27:07,521-Speed 128.68 samples/sec Loss 3.5882 Epoch: 13 Global Step: 135645 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:27:27,378-Speed 128.93 samples/sec Loss 3.4080 Epoch: 13 Global Step: 135650 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:27:47,450-Speed 127.54 samples/sec Loss 3.3647 Epoch: 13 Global Step: 135655 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:28:07,162-Speed 129.89 samples/sec Loss 3.4335 Epoch: 13 Global Step: 135660 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:28:27,326-Speed 126.97 samples/sec Loss 3.4364 Epoch: 13 Global Step: 135665 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:28:47,026-Speed 129.95 samples/sec Loss 3.2600 Epoch: 13 Global Step: 135670 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:29:06,821-Speed 129.33 samples/sec Loss 3.3090 Epoch: 13 Global Step: 135675 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:29:26,555-Speed 129.73 samples/sec Loss 3.5707 Epoch: 13 Global Step: 135680 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:29:46,397-Speed 129.03 samples/sec Loss 3.4957 Epoch: 13 Global Step: 135685 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:30:06,284-Speed 128.73 samples/sec Loss 3.2769 Epoch: 13 Global Step: 135690 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:30:26,283-Speed 128.01 samples/sec Loss 3.4655 Epoch: 13 Global Step: 135695 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:30:46,258-Speed 128.16 samples/sec Loss 3.2951 Epoch: 13 Global Step: 135700 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:31:06,079-Speed 129.17 samples/sec Loss 3.6057 Epoch: 13 Global Step: 135705 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:31:26,144-Speed 127.59 samples/sec Loss 3.2972 Epoch: 13 Global Step: 135710 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:31:45,902-Speed 129.57 samples/sec Loss 3.3131 Epoch: 13 Global Step: 135715 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:32:05,776-Speed 128.82 samples/sec Loss 3.3543 Epoch: 13 Global Step: 135720 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:32:25,564-Speed 129.38 samples/sec Loss 3.3962 Epoch: 13 Global Step: 135725 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:32:45,457-Speed 128.70 samples/sec Loss 3.3138 Epoch: 13 Global Step: 135730 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:33:05,172-Speed 129.86 samples/sec Loss 3.2845 Epoch: 13 Global Step: 135735 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:33:24,890-Speed 129.83 samples/sec Loss 3.4316 Epoch: 13 Global Step: 135740 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:33:45,033-Speed 127.10 samples/sec Loss 3.4632 Epoch: 13 Global Step: 135745 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:34:04,759-Speed 129.78 samples/sec Loss 3.3565 Epoch: 13 Global Step: 135750 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:34:24,873-Speed 127.28 samples/sec Loss 3.5642 Epoch: 13 Global Step: 135755 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:34:44,771-Speed 128.66 samples/sec Loss 3.4444 Epoch: 13 Global Step: 135760 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:35:04,486-Speed 129.86 samples/sec Loss 3.3106 Epoch: 13 Global Step: 135765 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:35:24,289-Speed 129.28 samples/sec Loss 3.3293 Epoch: 13 Global Step: 135770 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:35:44,104-Speed 129.20 samples/sec Loss 3.5181 Epoch: 13 Global Step: 135775 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:36:03,890-Speed 129.39 samples/sec Loss 3.4007 Epoch: 13 Global Step: 135780 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:36:23,641-Speed 129.62 samples/sec Loss 3.3385 Epoch: 13 Global Step: 135785 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:36:43,720-Speed 127.51 samples/sec Loss 3.5008 Epoch: 13 Global Step: 135790 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:37:03,623-Speed 128.63 samples/sec Loss 3.3739 Epoch: 13 Global Step: 135795 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:37:23,480-Speed 128.93 samples/sec Loss 3.4025 Epoch: 13 Global Step: 135800 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:37:43,343-Speed 128.89 samples/sec Loss 3.5858 Epoch: 13 Global Step: 135805 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:38:03,049-Speed 129.92 samples/sec Loss 3.5386 Epoch: 13 Global Step: 135810 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:38:22,940-Speed 128.71 samples/sec Loss 3.3527 Epoch: 13 Global Step: 135815 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:38:42,622-Speed 130.08 samples/sec Loss 3.3975 Epoch: 13 Global Step: 135820 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:39:02,490-Speed 128.86 samples/sec Loss 3.4988 Epoch: 13 Global Step: 135825 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:39:22,284-Speed 129.33 samples/sec Loss 3.5089 Epoch: 13 Global Step: 135830 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:39:42,235-Speed 128.32 samples/sec Loss 3.5160 Epoch: 13 Global Step: 135835 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:40:02,114-Speed 128.79 samples/sec Loss 3.3878 Epoch: 13 Global Step: 135840 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:40:22,280-Speed 126.95 samples/sec Loss 3.4305 Epoch: 13 Global Step: 135845 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:40:41,969-Speed 130.03 samples/sec Loss 3.3270 Epoch: 13 Global Step: 135850 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:41:01,880-Speed 128.58 samples/sec Loss 3.3825 Epoch: 13 Global Step: 135855 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:41:21,615-Speed 129.73 samples/sec Loss 3.3351 Epoch: 13 Global Step: 135860 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:41:41,987-Speed 125.67 samples/sec Loss 3.2694 Epoch: 13 Global Step: 135865 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:42:01,957-Speed 128.20 samples/sec Loss 3.2265 Epoch: 13 Global Step: 135870 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:42:21,696-Speed 129.70 samples/sec Loss 3.4149 Epoch: 13 Global Step: 135875 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:42:41,835-Speed 127.12 samples/sec Loss 3.5899 Epoch: 13 Global Step: 135880 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:43:01,691-Speed 128.94 samples/sec Loss 3.3183 Epoch: 13 Global Step: 135885 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:43:21,860-Speed 126.93 samples/sec Loss 3.4757 Epoch: 13 Global Step: 135890 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:43:41,611-Speed 129.62 samples/sec Loss 3.3243 Epoch: 13 Global Step: 135895 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:44:01,446-Speed 129.07 samples/sec Loss 3.2584 Epoch: 13 Global Step: 135900 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:44:21,129-Speed 130.07 samples/sec Loss 3.4708 Epoch: 13 Global Step: 135905 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:44:41,010-Speed 128.77 samples/sec Loss 3.2909 Epoch: 13 Global Step: 135910 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:45:00,823-Speed 129.22 samples/sec Loss 3.5889 Epoch: 13 Global Step: 135915 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:45:20,585-Speed 129.55 samples/sec Loss 3.4375 Epoch: 13 Global Step: 135920 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:45:40,696-Speed 127.30 samples/sec Loss 3.4660 Epoch: 13 Global Step: 135925 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:46:00,480-Speed 129.40 samples/sec Loss 3.5237 Epoch: 13 Global Step: 135930 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:46:20,533-Speed 127.67 samples/sec Loss 3.3799 Epoch: 13 Global Step: 135935 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:46:40,255-Speed 129.81 samples/sec Loss 3.5886 Epoch: 13 Global Step: 135940 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:47:00,148-Speed 128.69 samples/sec Loss 3.3865 Epoch: 13 Global Step: 135945 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:47:19,921-Speed 129.48 samples/sec Loss 3.3199 Epoch: 13 Global Step: 135950 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:47:39,934-Speed 127.93 samples/sec Loss 3.2539 Epoch: 13 Global Step: 135955 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:47:59,687-Speed 129.61 samples/sec Loss 3.4926 Epoch: 13 Global Step: 135960 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:48:19,326-Speed 130.35 samples/sec Loss 3.4938 Epoch: 13 Global Step: 135965 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:48:39,550-Speed 126.59 samples/sec Loss 3.3539 Epoch: 13 Global Step: 135970 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:48:59,315-Speed 129.53 samples/sec Loss 3.5117 Epoch: 13 Global Step: 135975 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:49:19,345-Speed 127.81 samples/sec Loss 3.1939 Epoch: 13 Global Step: 135980 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:49:39,169-Speed 129.14 samples/sec Loss 3.5706 Epoch: 13 Global Step: 135985 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:49:58,896-Speed 129.78 samples/sec Loss 3.3534 Epoch: 13 Global Step: 135990 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:50:18,663-Speed 129.51 samples/sec Loss 3.5010 Epoch: 13 Global Step: 135995 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:50:38,537-Speed 128.82 samples/sec Loss 3.6216 Epoch: 13 Global Step: 136000 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:50:58,156-Speed 130.49 samples/sec Loss 3.4690 Epoch: 13 Global Step: 136005 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:51:18,090-Speed 128.43 samples/sec Loss 3.1835 Epoch: 13 Global Step: 136010 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:51:38,173-Speed 127.48 samples/sec Loss 3.3862 Epoch: 13 Global Step: 136015 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:51:57,918-Speed 129.66 samples/sec Loss 3.3063 Epoch: 13 Global Step: 136020 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:52:17,860-Speed 128.38 samples/sec Loss 3.4193 Epoch: 13 Global Step: 136025 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:52:37,767-Speed 128.60 samples/sec Loss 3.2991 Epoch: 13 Global Step: 136030 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:52:57,475-Speed 129.91 samples/sec Loss 3.5427 Epoch: 13 Global Step: 136035 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:53:17,456-Speed 128.13 samples/sec Loss 3.5692 Epoch: 13 Global Step: 136040 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:53:37,069-Speed 130.53 samples/sec Loss 3.3830 Epoch: 13 Global Step: 136045 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:53:56,740-Speed 130.15 samples/sec Loss 3.5361 Epoch: 13 Global Step: 136050 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:54:16,599-Speed 128.92 samples/sec Loss 3.4814 Epoch: 13 Global Step: 136055 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:54:36,691-Speed 127.41 samples/sec Loss 3.6073 Epoch: 13 Global Step: 136060 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:54:56,495-Speed 129.28 samples/sec Loss 3.4216 Epoch: 13 Global Step: 136065 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:55:16,518-Speed 127.86 samples/sec Loss 3.5067 Epoch: 13 Global Step: 136070 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:55:36,311-Speed 129.34 samples/sec Loss 3.4035 Epoch: 13 Global Step: 136075 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:55:55,975-Speed 130.19 samples/sec Loss 3.4626 Epoch: 13 Global Step: 136080 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:56:15,947-Speed 128.19 samples/sec Loss 3.3279 Epoch: 13 Global Step: 136085 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:56:35,555-Speed 130.57 samples/sec Loss 3.2478 Epoch: 13 Global Step: 136090 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:56:55,473-Speed 128.54 samples/sec Loss 3.2403 Epoch: 13 Global Step: 136095 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:57:15,135-Speed 130.20 samples/sec Loss 3.3680 Epoch: 13 Global Step: 136100 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:57:35,252-Speed 127.26 samples/sec Loss 3.3664 Epoch: 13 Global Step: 136105 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:57:55,084-Speed 129.09 samples/sec Loss 3.4472 Epoch: 13 Global Step: 136110 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:58:15,425-Speed 125.86 samples/sec Loss 3.3480 Epoch: 13 Global Step: 136115 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:58:35,139-Speed 129.86 samples/sec Loss 3.4541 Epoch: 13 Global Step: 136120 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:58:55,014-Speed 128.81 samples/sec Loss 3.2790 Epoch: 13 Global Step: 136125 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:59:14,814-Speed 129.30 samples/sec Loss 3.4309 Epoch: 13 Global Step: 136130 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:59:34,831-Speed 127.90 samples/sec Loss 3.1220 Epoch: 13 Global Step: 136135 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 11:59:54,743-Speed 128.58 samples/sec Loss 3.5134 Epoch: 13 Global Step: 136140 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 12:00:14,458-Speed 129.85 samples/sec Loss 3.2116 Epoch: 13 Global Step: 136145 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 12:00:34,691-Speed 126.53 samples/sec Loss 3.4029 Epoch: 13 Global Step: 136150 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 12:00:54,440-Speed 129.63 samples/sec Loss 3.4385 Epoch: 13 Global Step: 136155 Fp16 Grad Scale: 8192 Required: 134 hours Training: 2021-06-13 12:01:14,518-Speed 127.51 samples/sec Loss 3.2627 Epoch: 13 Global Step: 136160 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:01:34,299-Speed 129.42 samples/sec Loss 3.2941 Epoch: 13 Global Step: 136165 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:01:54,094-Speed 129.33 samples/sec Loss 3.3866 Epoch: 13 Global Step: 136170 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:02:13,813-Speed 129.83 samples/sec Loss 3.3138 Epoch: 13 Global Step: 136175 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:02:33,544-Speed 129.76 samples/sec Loss 3.3923 Epoch: 13 Global Step: 136180 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:02:53,395-Speed 128.96 samples/sec Loss 3.4752 Epoch: 13 Global Step: 136185 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:03:13,101-Speed 129.92 samples/sec Loss 3.3784 Epoch: 13 Global Step: 136190 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:03:33,223-Speed 127.23 samples/sec Loss 3.4162 Epoch: 13 Global Step: 136195 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:03:52,921-Speed 129.96 samples/sec Loss 3.5328 Epoch: 13 Global Step: 136200 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:04:12,996-Speed 127.53 samples/sec Loss 3.5406 Epoch: 13 Global Step: 136205 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:04:32,738-Speed 129.68 samples/sec Loss 3.3061 Epoch: 13 Global Step: 136210 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:04:52,518-Speed 129.43 samples/sec Loss 3.2709 Epoch: 13 Global Step: 136215 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:05:12,157-Speed 130.35 samples/sec Loss 3.2787 Epoch: 13 Global Step: 136220 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:05:32,035-Speed 128.80 samples/sec Loss 3.3701 Epoch: 13 Global Step: 136225 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:05:51,887-Speed 128.96 samples/sec Loss 3.4866 Epoch: 13 Global Step: 136230 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:06:11,776-Speed 128.72 samples/sec Loss 3.4754 Epoch: 13 Global Step: 136235 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:06:31,807-Speed 127.81 samples/sec Loss 3.2786 Epoch: 13 Global Step: 136240 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:06:51,633-Speed 129.13 samples/sec Loss 3.5522 Epoch: 13 Global Step: 136245 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:07:11,674-Speed 127.75 samples/sec Loss 3.4797 Epoch: 13 Global Step: 136250 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:07:31,581-Speed 128.60 samples/sec Loss 3.3691 Epoch: 13 Global Step: 136255 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:07:51,194-Speed 130.53 samples/sec Loss 3.5052 Epoch: 13 Global Step: 136260 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:08:11,102-Speed 128.60 samples/sec Loss 3.5486 Epoch: 13 Global Step: 136265 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:08:31,015-Speed 128.57 samples/sec Loss 3.4712 Epoch: 13 Global Step: 136270 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:08:50,774-Speed 129.57 samples/sec Loss 3.3273 Epoch: 13 Global Step: 136275 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:09:10,658-Speed 128.76 samples/sec Loss 3.5522 Epoch: 13 Global Step: 136280 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:09:30,681-Speed 127.86 samples/sec Loss 3.3500 Epoch: 13 Global Step: 136285 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:09:50,655-Speed 128.18 samples/sec Loss 3.2503 Epoch: 13 Global Step: 136290 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:10:10,573-Speed 128.53 samples/sec Loss 3.3383 Epoch: 13 Global Step: 136295 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:10:30,344-Speed 129.49 samples/sec Loss 3.3198 Epoch: 13 Global Step: 136300 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:10:50,078-Speed 129.73 samples/sec Loss 3.3295 Epoch: 13 Global Step: 136305 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:11:10,279-Speed 126.73 samples/sec Loss 3.5244 Epoch: 13 Global Step: 136310 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:11:30,035-Speed 129.59 samples/sec Loss 3.5682 Epoch: 13 Global Step: 136315 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:11:49,912-Speed 128.80 samples/sec Loss 3.4460 Epoch: 13 Global Step: 136320 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:12:09,533-Speed 130.48 samples/sec Loss 3.5430 Epoch: 13 Global Step: 136325 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:12:29,601-Speed 127.57 samples/sec Loss 3.5166 Epoch: 13 Global Step: 136330 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:12:49,463-Speed 128.89 samples/sec Loss 3.4452 Epoch: 13 Global Step: 136335 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:13:09,450-Speed 128.10 samples/sec Loss 3.6067 Epoch: 13 Global Step: 136340 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:13:29,133-Speed 130.07 samples/sec Loss 3.4603 Epoch: 13 Global Step: 136345 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:13:49,042-Speed 128.59 samples/sec Loss 3.6209 Epoch: 13 Global Step: 136350 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:14:08,772-Speed 129.75 samples/sec Loss 3.2214 Epoch: 13 Global Step: 136355 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:14:28,422-Speed 130.29 samples/sec Loss 3.4146 Epoch: 13 Global Step: 136360 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:14:48,239-Speed 129.19 samples/sec Loss 3.3827 Epoch: 13 Global Step: 136365 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:15:07,961-Speed 129.81 samples/sec Loss 3.2948 Epoch: 13 Global Step: 136370 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:15:28,306-Speed 125.83 samples/sec Loss 3.2907 Epoch: 13 Global Step: 136375 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:15:47,920-Speed 130.52 samples/sec Loss 3.4347 Epoch: 13 Global Step: 136380 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:16:07,976-Speed 127.65 samples/sec Loss 3.6406 Epoch: 13 Global Step: 136385 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:16:27,863-Speed 128.73 samples/sec Loss 3.3958 Epoch: 13 Global Step: 136390 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:16:47,792-Speed 128.46 samples/sec Loss 3.4520 Epoch: 13 Global Step: 136395 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:17:07,512-Speed 129.83 samples/sec Loss 3.3381 Epoch: 13 Global Step: 136400 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:17:27,183-Speed 130.14 samples/sec Loss 3.3561 Epoch: 13 Global Step: 136405 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:17:47,059-Speed 128.81 samples/sec Loss 3.7306 Epoch: 13 Global Step: 136410 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:18:06,805-Speed 129.66 samples/sec Loss 3.5174 Epoch: 13 Global Step: 136415 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:18:27,007-Speed 126.73 samples/sec Loss 3.1901 Epoch: 13 Global Step: 136420 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:18:46,801-Speed 129.34 samples/sec Loss 3.2580 Epoch: 13 Global Step: 136425 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:19:06,886-Speed 127.47 samples/sec Loss 3.2775 Epoch: 13 Global Step: 136430 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:19:26,682-Speed 129.32 samples/sec Loss 3.4200 Epoch: 13 Global Step: 136435 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:19:46,423-Speed 129.69 samples/sec Loss 3.4712 Epoch: 13 Global Step: 136440 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:20:06,149-Speed 129.79 samples/sec Loss 3.5048 Epoch: 13 Global Step: 136445 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:20:26,044-Speed 128.68 samples/sec Loss 3.4635 Epoch: 13 Global Step: 136450 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:20:45,765-Speed 129.82 samples/sec Loss 3.3831 Epoch: 13 Global Step: 136455 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:21:05,589-Speed 129.14 samples/sec Loss 3.3393 Epoch: 13 Global Step: 136460 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:21:25,585-Speed 128.03 samples/sec Loss 3.3161 Epoch: 13 Global Step: 136465 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:21:45,310-Speed 129.79 samples/sec Loss 3.3563 Epoch: 13 Global Step: 136470 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:22:05,435-Speed 127.21 samples/sec Loss 3.5113 Epoch: 13 Global Step: 136475 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:22:25,399-Speed 128.24 samples/sec Loss 3.4161 Epoch: 13 Global Step: 136480 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:22:45,130-Speed 129.75 samples/sec Loss 3.4897 Epoch: 13 Global Step: 136485 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:23:04,891-Speed 129.56 samples/sec Loss 3.4864 Epoch: 13 Global Step: 136490 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:23:24,754-Speed 128.89 samples/sec Loss 3.4359 Epoch: 13 Global Step: 136495 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:23:44,688-Speed 128.43 samples/sec Loss 3.3569 Epoch: 13 Global Step: 136500 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:24:04,475-Speed 129.38 samples/sec Loss 3.4443 Epoch: 13 Global Step: 136505 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:24:24,402-Speed 128.47 samples/sec Loss 3.4355 Epoch: 13 Global Step: 136510 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:24:44,303-Speed 128.64 samples/sec Loss 3.3837 Epoch: 13 Global Step: 136515 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:25:04,305-Speed 128.00 samples/sec Loss 3.2900 Epoch: 13 Global Step: 136520 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:25:24,228-Speed 128.50 samples/sec Loss 3.1953 Epoch: 13 Global Step: 136525 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:25:43,887-Speed 130.23 samples/sec Loss 3.3291 Epoch: 13 Global Step: 136530 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:26:03,809-Speed 128.50 samples/sec Loss 3.4124 Epoch: 13 Global Step: 136535 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:26:23,726-Speed 128.54 samples/sec Loss 3.5391 Epoch: 13 Global Step: 136540 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:26:43,573-Speed 129.00 samples/sec Loss 3.4375 Epoch: 13 Global Step: 136545 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:27:03,313-Speed 129.69 samples/sec Loss 3.5021 Epoch: 13 Global Step: 136550 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:27:23,378-Speed 127.59 samples/sec Loss 3.3157 Epoch: 13 Global Step: 136555 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:27:43,312-Speed 128.43 samples/sec Loss 3.5399 Epoch: 13 Global Step: 136560 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:28:03,375-Speed 127.61 samples/sec Loss 3.3659 Epoch: 13 Global Step: 136565 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:28:22,993-Speed 130.50 samples/sec Loss 3.2678 Epoch: 13 Global Step: 136570 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:28:42,811-Speed 129.19 samples/sec Loss 3.2845 Epoch: 13 Global Step: 136575 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:29:02,605-Speed 129.34 samples/sec Loss 3.4467 Epoch: 13 Global Step: 136580 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:29:22,310-Speed 129.92 samples/sec Loss 3.3157 Epoch: 13 Global Step: 136585 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:29:42,133-Speed 129.15 samples/sec Loss 3.2806 Epoch: 13 Global Step: 136590 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:30:01,816-Speed 130.07 samples/sec Loss 3.3785 Epoch: 13 Global Step: 136595 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:30:21,989-Speed 126.91 samples/sec Loss 3.6624 Epoch: 13 Global Step: 136600 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:30:41,695-Speed 129.92 samples/sec Loss 3.2185 Epoch: 13 Global Step: 136605 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:31:01,690-Speed 128.04 samples/sec Loss 3.2525 Epoch: 13 Global Step: 136610 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:31:21,432-Speed 129.67 samples/sec Loss 3.3276 Epoch: 13 Global Step: 136615 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:31:41,334-Speed 128.64 samples/sec Loss 3.3103 Epoch: 13 Global Step: 136620 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:32:01,104-Speed 129.49 samples/sec Loss 3.4517 Epoch: 13 Global Step: 136625 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:32:20,969-Speed 128.88 samples/sec Loss 3.4592 Epoch: 13 Global Step: 136630 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:32:40,662-Speed 130.00 samples/sec Loss 3.4790 Epoch: 13 Global Step: 136635 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:33:00,431-Speed 129.50 samples/sec Loss 3.5699 Epoch: 13 Global Step: 136640 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:33:20,660-Speed 126.55 samples/sec Loss 3.2000 Epoch: 13 Global Step: 136645 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:33:40,323-Speed 130.20 samples/sec Loss 3.4429 Epoch: 13 Global Step: 136650 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:34:00,296-Speed 128.18 samples/sec Loss 3.3958 Epoch: 13 Global Step: 136655 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:34:20,156-Speed 128.90 samples/sec Loss 3.6796 Epoch: 13 Global Step: 136660 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:34:39,821-Speed 130.19 samples/sec Loss 3.4893 Epoch: 13 Global Step: 136665 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:34:59,762-Speed 128.38 samples/sec Loss 3.3300 Epoch: 13 Global Step: 136670 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:35:19,647-Speed 128.75 samples/sec Loss 3.3645 Epoch: 13 Global Step: 136675 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:35:39,373-Speed 129.78 samples/sec Loss 3.5541 Epoch: 13 Global Step: 136680 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:35:59,269-Speed 128.68 samples/sec Loss 3.3743 Epoch: 13 Global Step: 136685 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:36:19,347-Speed 127.51 samples/sec Loss 3.5063 Epoch: 13 Global Step: 136690 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:36:39,197-Speed 128.97 samples/sec Loss 3.2971 Epoch: 13 Global Step: 136695 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:36:59,151-Speed 128.31 samples/sec Loss 3.6347 Epoch: 13 Global Step: 136700 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:37:19,005-Speed 128.94 samples/sec Loss 3.3599 Epoch: 13 Global Step: 136705 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:37:38,787-Speed 129.42 samples/sec Loss 3.5500 Epoch: 13 Global Step: 136710 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:37:58,684-Speed 128.67 samples/sec Loss 3.3387 Epoch: 13 Global Step: 136715 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:38:18,354-Speed 130.15 samples/sec Loss 3.4043 Epoch: 13 Global Step: 136720 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:38:38,184-Speed 129.10 samples/sec Loss 3.4316 Epoch: 13 Global Step: 136725 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:38:57,986-Speed 129.29 samples/sec Loss 3.5700 Epoch: 13 Global Step: 136730 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:39:18,027-Speed 127.75 samples/sec Loss 3.3889 Epoch: 13 Global Step: 136735 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:39:37,872-Speed 129.01 samples/sec Loss 3.2951 Epoch: 13 Global Step: 136740 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:39:57,918-Speed 127.71 samples/sec Loss 3.2916 Epoch: 13 Global Step: 136745 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:40:17,697-Speed 129.43 samples/sec Loss 3.2324 Epoch: 13 Global Step: 136750 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:40:37,351-Speed 130.26 samples/sec Loss 3.2932 Epoch: 13 Global Step: 136755 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:40:57,222-Speed 128.84 samples/sec Loss 3.4143 Epoch: 13 Global Step: 136760 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:41:16,829-Speed 130.57 samples/sec Loss 3.5361 Epoch: 13 Global Step: 136765 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:41:36,803-Speed 128.17 samples/sec Loss 3.2873 Epoch: 13 Global Step: 136770 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:41:56,449-Speed 130.31 samples/sec Loss 3.4288 Epoch: 13 Global Step: 136775 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:42:16,553-Speed 127.35 samples/sec Loss 3.4100 Epoch: 13 Global Step: 136780 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:42:36,235-Speed 130.08 samples/sec Loss 3.3865 Epoch: 13 Global Step: 136785 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:42:56,408-Speed 126.91 samples/sec Loss 3.4703 Epoch: 13 Global Step: 136790 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:43:16,017-Speed 130.55 samples/sec Loss 3.5195 Epoch: 13 Global Step: 136795 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:43:35,923-Speed 128.61 samples/sec Loss 3.4106 Epoch: 13 Global Step: 136800 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:43:55,630-Speed 129.91 samples/sec Loss 3.4060 Epoch: 13 Global Step: 136805 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:44:15,501-Speed 128.84 samples/sec Loss 3.3099 Epoch: 13 Global Step: 136810 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:44:35,332-Speed 129.10 samples/sec Loss 3.2933 Epoch: 13 Global Step: 136815 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:44:55,110-Speed 129.44 samples/sec Loss 3.5406 Epoch: 13 Global Step: 136820 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:45:15,142-Speed 127.80 samples/sec Loss 3.5685 Epoch: 13 Global Step: 136825 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:45:34,943-Speed 129.29 samples/sec Loss 3.3626 Epoch: 13 Global Step: 136830 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:45:54,978-Speed 127.79 samples/sec Loss 3.6051 Epoch: 13 Global Step: 136835 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:46:14,880-Speed 128.64 samples/sec Loss 3.5952 Epoch: 13 Global Step: 136840 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:46:34,574-Speed 129.99 samples/sec Loss 3.3667 Epoch: 13 Global Step: 136845 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:46:54,217-Speed 130.33 samples/sec Loss 3.3048 Epoch: 13 Global Step: 136850 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:47:14,155-Speed 128.41 samples/sec Loss 3.6194 Epoch: 13 Global Step: 136855 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:47:33,884-Speed 129.77 samples/sec Loss 3.3534 Epoch: 13 Global Step: 136860 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:47:53,784-Speed 128.65 samples/sec Loss 3.4264 Epoch: 13 Global Step: 136865 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:48:13,679-Speed 128.68 samples/sec Loss 3.6096 Epoch: 13 Global Step: 136870 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:48:33,482-Speed 129.28 samples/sec Loss 3.4177 Epoch: 13 Global Step: 136875 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:48:53,482-Speed 128.01 samples/sec Loss 3.3171 Epoch: 13 Global Step: 136880 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:49:13,409-Speed 128.47 samples/sec Loss 3.3356 Epoch: 13 Global Step: 136885 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:49:33,013-Speed 130.59 samples/sec Loss 3.2612 Epoch: 13 Global Step: 136890 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:49:52,817-Speed 129.27 samples/sec Loss 3.7216 Epoch: 13 Global Step: 136895 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:50:12,701-Speed 128.76 samples/sec Loss 3.5947 Epoch: 13 Global Step: 136900 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:50:32,458-Speed 129.58 samples/sec Loss 3.4488 Epoch: 13 Global Step: 136905 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:50:52,223-Speed 129.53 samples/sec Loss 3.4434 Epoch: 13 Global Step: 136910 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:51:12,093-Speed 128.84 samples/sec Loss 3.6406 Epoch: 13 Global Step: 136915 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:51:32,044-Speed 128.32 samples/sec Loss 3.4239 Epoch: 13 Global Step: 136920 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:51:51,964-Speed 128.52 samples/sec Loss 3.3081 Epoch: 13 Global Step: 136925 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:52:11,768-Speed 129.27 samples/sec Loss 3.2284 Epoch: 13 Global Step: 136930 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:52:31,402-Speed 130.39 samples/sec Loss 3.4468 Epoch: 13 Global Step: 136935 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:52:51,317-Speed 128.55 samples/sec Loss 3.3362 Epoch: 13 Global Step: 136940 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:53:11,104-Speed 129.39 samples/sec Loss 3.3155 Epoch: 13 Global Step: 136945 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:53:31,021-Speed 128.54 samples/sec Loss 3.6663 Epoch: 13 Global Step: 136950 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:53:50,645-Speed 130.46 samples/sec Loss 3.4743 Epoch: 13 Global Step: 136955 Fp16 Grad Scale: 8192 Required: 133 hours Training: 2021-06-13 12:54:10,708-Speed 127.60 samples/sec Loss 3.4855 Epoch: 13 Global Step: 136960 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:54:30,570-Speed 128.90 samples/sec Loss 3.4371 Epoch: 13 Global Step: 136965 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:54:50,731-Speed 126.98 samples/sec Loss 3.5014 Epoch: 13 Global Step: 136970 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:55:10,348-Speed 130.51 samples/sec Loss 3.4665 Epoch: 13 Global Step: 136975 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:55:30,101-Speed 129.60 samples/sec Loss 3.3028 Epoch: 13 Global Step: 136980 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:55:50,002-Speed 128.65 samples/sec Loss 3.3888 Epoch: 13 Global Step: 136985 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:56:09,876-Speed 128.82 samples/sec Loss 3.5104 Epoch: 13 Global Step: 136990 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:56:29,731-Speed 128.94 samples/sec Loss 3.4281 Epoch: 13 Global Step: 136995 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:56:49,424-Speed 130.00 samples/sec Loss 3.3408 Epoch: 13 Global Step: 137000 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:57:09,522-Speed 127.38 samples/sec Loss 3.2617 Epoch: 13 Global Step: 137005 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:57:29,255-Speed 129.74 samples/sec Loss 3.5266 Epoch: 13 Global Step: 137010 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:57:49,233-Speed 128.15 samples/sec Loss 3.3355 Epoch: 13 Global Step: 137015 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:58:08,940-Speed 129.90 samples/sec Loss 3.5149 Epoch: 13 Global Step: 137020 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:58:28,837-Speed 128.67 samples/sec Loss 3.3944 Epoch: 13 Global Step: 137025 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:58:48,561-Speed 129.80 samples/sec Loss 3.4848 Epoch: 13 Global Step: 137030 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:59:08,381-Speed 129.17 samples/sec Loss 3.4755 Epoch: 13 Global Step: 137035 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:59:28,250-Speed 128.85 samples/sec Loss 3.4385 Epoch: 13 Global Step: 137040 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 12:59:47,990-Speed 129.69 samples/sec Loss 3.1911 Epoch: 13 Global Step: 137045 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:00:08,172-Speed 126.85 samples/sec Loss 3.3325 Epoch: 13 Global Step: 137050 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:00:27,945-Speed 129.48 samples/sec Loss 3.3641 Epoch: 13 Global Step: 137055 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:00:47,934-Speed 128.07 samples/sec Loss 3.7028 Epoch: 13 Global Step: 137060 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:01:07,693-Speed 129.57 samples/sec Loss 3.5546 Epoch: 13 Global Step: 137065 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:01:27,557-Speed 128.88 samples/sec Loss 3.4121 Epoch: 13 Global Step: 137070 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:01:47,280-Speed 129.81 samples/sec Loss 3.3134 Epoch: 13 Global Step: 137075 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:02:07,161-Speed 128.77 samples/sec Loss 3.5139 Epoch: 13 Global Step: 137080 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:02:26,865-Speed 129.93 samples/sec Loss 3.3850 Epoch: 13 Global Step: 137085 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:02:46,671-Speed 129.26 samples/sec Loss 3.3731 Epoch: 13 Global Step: 137090 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:03:06,763-Speed 127.42 samples/sec Loss 3.4223 Epoch: 13 Global Step: 137095 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:03:26,517-Speed 129.60 samples/sec Loss 3.4079 Epoch: 13 Global Step: 137100 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:03:46,555-Speed 127.77 samples/sec Loss 3.4468 Epoch: 13 Global Step: 137105 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:04:06,474-Speed 128.52 samples/sec Loss 3.2944 Epoch: 13 Global Step: 137110 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:04:26,192-Speed 129.84 samples/sec Loss 3.5199 Epoch: 13 Global Step: 137115 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:04:46,020-Speed 129.12 samples/sec Loss 3.6343 Epoch: 13 Global Step: 137120 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:05:05,999-Speed 128.14 samples/sec Loss 3.5506 Epoch: 13 Global Step: 137125 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:05:25,706-Speed 129.91 samples/sec Loss 3.4202 Epoch: 13 Global Step: 137130 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:05:45,608-Speed 128.63 samples/sec Loss 3.5806 Epoch: 13 Global Step: 137135 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:06:05,600-Speed 128.06 samples/sec Loss 3.4787 Epoch: 13 Global Step: 137140 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:06:25,474-Speed 128.82 samples/sec Loss 3.5650 Epoch: 13 Global Step: 137145 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:06:45,424-Speed 128.33 samples/sec Loss 3.3458 Epoch: 13 Global Step: 137150 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:07:05,315-Speed 128.71 samples/sec Loss 3.4545 Epoch: 13 Global Step: 137155 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:07:25,248-Speed 128.44 samples/sec Loss 3.3586 Epoch: 13 Global Step: 137160 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:07:45,040-Speed 129.35 samples/sec Loss 3.4435 Epoch: 13 Global Step: 137165 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:08:04,769-Speed 129.76 samples/sec Loss 3.3499 Epoch: 13 Global Step: 137170 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:08:24,901-Speed 127.17 samples/sec Loss 3.4345 Epoch: 13 Global Step: 137175 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:08:44,574-Speed 130.14 samples/sec Loss 3.6141 Epoch: 13 Global Step: 137180 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:09:04,584-Speed 127.94 samples/sec Loss 3.4586 Epoch: 13 Global Step: 137185 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:09:24,451-Speed 128.87 samples/sec Loss 3.6369 Epoch: 13 Global Step: 137190 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:09:44,536-Speed 127.47 samples/sec Loss 3.4248 Epoch: 13 Global Step: 137195 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:10:04,247-Speed 129.88 samples/sec Loss 3.5628 Epoch: 13 Global Step: 137200 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:10:23,976-Speed 129.77 samples/sec Loss 3.3668 Epoch: 13 Global Step: 137205 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:10:43,827-Speed 128.97 samples/sec Loss 3.2777 Epoch: 13 Global Step: 137210 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:11:03,430-Speed 130.60 samples/sec Loss 3.3903 Epoch: 13 Global Step: 137215 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:11:23,326-Speed 128.67 samples/sec Loss 3.3240 Epoch: 13 Global Step: 137220 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:11:43,029-Speed 129.94 samples/sec Loss 3.7295 Epoch: 13 Global Step: 137225 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:12:03,222-Speed 126.78 samples/sec Loss 3.4250 Epoch: 13 Global Step: 137230 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:12:22,961-Speed 129.70 samples/sec Loss 3.5425 Epoch: 13 Global Step: 137235 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:12:43,047-Speed 127.46 samples/sec Loss 3.2644 Epoch: 13 Global Step: 137240 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:13:02,817-Speed 129.50 samples/sec Loss 3.3089 Epoch: 13 Global Step: 137245 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:13:22,637-Speed 129.17 samples/sec Loss 3.4075 Epoch: 13 Global Step: 137250 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:13:42,402-Speed 129.52 samples/sec Loss 3.5122 Epoch: 13 Global Step: 137255 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:14:02,304-Speed 128.64 samples/sec Loss 3.5039 Epoch: 13 Global Step: 137260 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:14:22,026-Speed 129.81 samples/sec Loss 3.3208 Epoch: 13 Global Step: 137265 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:14:41,727-Speed 129.95 samples/sec Loss 3.4218 Epoch: 13 Global Step: 137270 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:15:01,904-Speed 126.88 samples/sec Loss 3.3392 Epoch: 13 Global Step: 137275 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:15:21,623-Speed 129.83 samples/sec Loss 3.4100 Epoch: 13 Global Step: 137280 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:15:41,596-Speed 128.18 samples/sec Loss 3.3967 Epoch: 13 Global Step: 137285 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:16:01,310-Speed 129.86 samples/sec Loss 3.4148 Epoch: 13 Global Step: 137290 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:16:21,463-Speed 127.04 samples/sec Loss 3.2968 Epoch: 13 Global Step: 137295 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:16:41,193-Speed 129.76 samples/sec Loss 3.4885 Epoch: 13 Global Step: 137300 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:17:01,100-Speed 128.61 samples/sec Loss 3.4426 Epoch: 13 Global Step: 137305 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:17:20,708-Speed 130.56 samples/sec Loss 3.4352 Epoch: 13 Global Step: 137310 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:17:40,713-Speed 127.97 samples/sec Loss 3.5557 Epoch: 13 Global Step: 137315 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:18:00,713-Speed 128.01 samples/sec Loss 3.4250 Epoch: 13 Global Step: 137320 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:18:20,328-Speed 130.52 samples/sec Loss 3.3893 Epoch: 13 Global Step: 137325 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:18:40,325-Speed 128.02 samples/sec Loss 3.4480 Epoch: 13 Global Step: 137330 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:19:00,302-Speed 128.15 samples/sec Loss 3.4269 Epoch: 13 Global Step: 137335 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:19:20,049-Speed 129.65 samples/sec Loss 3.3773 Epoch: 13 Global Step: 137340 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:19:39,782-Speed 129.74 samples/sec Loss 3.3060 Epoch: 13 Global Step: 137345 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:19:59,576-Speed 129.33 samples/sec Loss 3.7184 Epoch: 13 Global Step: 137350 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:20:19,371-Speed 129.34 samples/sec Loss 3.4353 Epoch: 13 Global Step: 137355 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:20:39,195-Speed 129.14 samples/sec Loss 3.3764 Epoch: 13 Global Step: 137360 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:20:59,272-Speed 127.52 samples/sec Loss 3.2442 Epoch: 13 Global Step: 137365 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:21:18,901-Speed 130.43 samples/sec Loss 3.4849 Epoch: 13 Global Step: 137370 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:21:39,003-Speed 127.36 samples/sec Loss 3.3683 Epoch: 13 Global Step: 137375 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:21:58,916-Speed 128.56 samples/sec Loss 3.2504 Epoch: 13 Global Step: 137380 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:22:18,619-Speed 129.94 samples/sec Loss 3.4138 Epoch: 13 Global Step: 137385 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:22:38,613-Speed 128.04 samples/sec Loss 3.4777 Epoch: 13 Global Step: 137390 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:22:58,240-Speed 130.44 samples/sec Loss 3.3332 Epoch: 13 Global Step: 137395 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:23:18,077-Speed 129.06 samples/sec Loss 3.4285 Epoch: 13 Global Step: 137400 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:23:37,950-Speed 128.83 samples/sec Loss 3.6249 Epoch: 13 Global Step: 137405 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:23:57,842-Speed 128.69 samples/sec Loss 3.2477 Epoch: 13 Global Step: 137410 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:24:17,732-Speed 128.72 samples/sec Loss 3.2569 Epoch: 13 Global Step: 137415 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:24:37,756-Speed 127.85 samples/sec Loss 3.2803 Epoch: 13 Global Step: 137420 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:24:57,629-Speed 128.82 samples/sec Loss 3.4008 Epoch: 13 Global Step: 137425 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:25:17,295-Speed 130.18 samples/sec Loss 3.5155 Epoch: 13 Global Step: 137430 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:25:37,101-Speed 129.26 samples/sec Loss 3.4948 Epoch: 13 Global Step: 137435 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:25:56,813-Speed 129.88 samples/sec Loss 3.4922 Epoch: 13 Global Step: 137440 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:26:16,812-Speed 128.01 samples/sec Loss 3.2045 Epoch: 13 Global Step: 137445 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:26:36,517-Speed 129.92 samples/sec Loss 3.4857 Epoch: 13 Global Step: 137450 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:26:56,365-Speed 128.99 samples/sec Loss 3.5184 Epoch: 13 Global Step: 137455 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:27:16,397-Speed 127.80 samples/sec Loss 3.4787 Epoch: 13 Global Step: 137460 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:27:36,495-Speed 127.38 samples/sec Loss 3.4381 Epoch: 13 Global Step: 137465 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:27:56,212-Speed 129.84 samples/sec Loss 3.4131 Epoch: 13 Global Step: 137470 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:28:15,855-Speed 130.33 samples/sec Loss 3.4528 Epoch: 13 Global Step: 137475 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:28:35,723-Speed 128.86 samples/sec Loss 3.3959 Epoch: 13 Global Step: 137480 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:28:55,490-Speed 129.52 samples/sec Loss 3.3065 Epoch: 13 Global Step: 137485 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:29:15,394-Speed 128.63 samples/sec Loss 3.5023 Epoch: 13 Global Step: 137490 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:29:35,026-Speed 130.40 samples/sec Loss 3.4099 Epoch: 13 Global Step: 137495 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:29:55,097-Speed 127.56 samples/sec Loss 3.2628 Epoch: 13 Global Step: 137500 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:30:14,893-Speed 129.33 samples/sec Loss 3.6464 Epoch: 13 Global Step: 137505 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:30:34,976-Speed 127.47 samples/sec Loss 3.4659 Epoch: 13 Global Step: 137510 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:30:54,580-Speed 130.59 samples/sec Loss 3.4054 Epoch: 13 Global Step: 137515 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:31:14,457-Speed 128.80 samples/sec Loss 3.3510 Epoch: 13 Global Step: 137520 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:31:34,248-Speed 129.36 samples/sec Loss 3.3631 Epoch: 13 Global Step: 137525 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:31:53,969-Speed 129.82 samples/sec Loss 3.4034 Epoch: 13 Global Step: 137530 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:32:13,843-Speed 128.82 samples/sec Loss 3.3693 Epoch: 13 Global Step: 137535 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:32:33,472-Speed 130.42 samples/sec Loss 3.3618 Epoch: 13 Global Step: 137540 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:32:53,651-Speed 126.87 samples/sec Loss 3.4804 Epoch: 13 Global Step: 137545 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:33:13,382-Speed 129.75 samples/sec Loss 3.5240 Epoch: 13 Global Step: 137550 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:33:33,364-Speed 128.12 samples/sec Loss 3.3754 Epoch: 13 Global Step: 137555 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:33:53,020-Speed 130.24 samples/sec Loss 3.5619 Epoch: 13 Global Step: 137560 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:34:12,782-Speed 129.55 samples/sec Loss 3.5084 Epoch: 13 Global Step: 137565 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:34:32,561-Speed 129.44 samples/sec Loss 3.3188 Epoch: 13 Global Step: 137570 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:34:52,467-Speed 128.61 samples/sec Loss 3.3732 Epoch: 13 Global Step: 137575 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:35:12,071-Speed 130.59 samples/sec Loss 3.3570 Epoch: 13 Global Step: 137580 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:35:32,093-Speed 127.87 samples/sec Loss 3.5338 Epoch: 13 Global Step: 137585 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:35:52,155-Speed 127.61 samples/sec Loss 3.5403 Epoch: 13 Global Step: 137590 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:36:11,923-Speed 129.51 samples/sec Loss 3.5413 Epoch: 13 Global Step: 137595 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:36:31,915-Speed 128.06 samples/sec Loss 3.4327 Epoch: 13 Global Step: 137600 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:36:51,801-Speed 128.74 samples/sec Loss 3.3136 Epoch: 13 Global Step: 137605 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:37:11,603-Speed 129.28 samples/sec Loss 3.2811 Epoch: 13 Global Step: 137610 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:37:31,373-Speed 129.50 samples/sec Loss 3.6544 Epoch: 13 Global Step: 137615 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:37:51,230-Speed 128.93 samples/sec Loss 3.3186 Epoch: 13 Global Step: 137620 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:38:10,944-Speed 129.86 samples/sec Loss 3.4921 Epoch: 13 Global Step: 137625 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:38:30,909-Speed 128.23 samples/sec Loss 3.4802 Epoch: 13 Global Step: 137630 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:38:50,912-Speed 127.98 samples/sec Loss 3.4461 Epoch: 13 Global Step: 137635 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:39:10,671-Speed 129.57 samples/sec Loss 3.4906 Epoch: 13 Global Step: 137640 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:39:30,729-Speed 127.64 samples/sec Loss 3.5257 Epoch: 13 Global Step: 137645 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:39:50,639-Speed 128.59 samples/sec Loss 3.4252 Epoch: 13 Global Step: 137650 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:40:10,424-Speed 129.40 samples/sec Loss 3.5710 Epoch: 13 Global Step: 137655 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:40:30,261-Speed 129.06 samples/sec Loss 3.2390 Epoch: 13 Global Step: 137660 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:40:50,027-Speed 129.52 samples/sec Loss 3.5138 Epoch: 13 Global Step: 137665 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:41:09,851-Speed 129.14 samples/sec Loss 3.4327 Epoch: 13 Global Step: 137670 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:41:29,626-Speed 129.46 samples/sec Loss 3.4342 Epoch: 13 Global Step: 137675 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:41:49,548-Speed 128.50 samples/sec Loss 3.6799 Epoch: 13 Global Step: 137680 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:42:09,468-Speed 128.53 samples/sec Loss 3.4738 Epoch: 13 Global Step: 137685 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:42:29,463-Speed 128.04 samples/sec Loss 3.3614 Epoch: 13 Global Step: 137690 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:42:49,290-Speed 129.12 samples/sec Loss 3.4887 Epoch: 13 Global Step: 137695 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:43:09,076-Speed 129.39 samples/sec Loss 3.4566 Epoch: 13 Global Step: 137700 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:43:28,958-Speed 128.77 samples/sec Loss 3.6199 Epoch: 13 Global Step: 137705 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:43:48,766-Speed 129.25 samples/sec Loss 3.5548 Epoch: 13 Global Step: 137710 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:44:08,634-Speed 128.86 samples/sec Loss 3.4703 Epoch: 13 Global Step: 137715 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:44:28,352-Speed 129.83 samples/sec Loss 3.3899 Epoch: 13 Global Step: 137720 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:44:48,486-Speed 127.16 samples/sec Loss 3.4726 Epoch: 13 Global Step: 137725 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:45:08,351-Speed 128.88 samples/sec Loss 3.3585 Epoch: 13 Global Step: 137730 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:45:28,341-Speed 128.07 samples/sec Loss 3.5544 Epoch: 13 Global Step: 137735 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:45:48,048-Speed 129.91 samples/sec Loss 3.4831 Epoch: 13 Global Step: 137740 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:46:07,830-Speed 129.41 samples/sec Loss 3.3053 Epoch: 13 Global Step: 137745 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:46:27,630-Speed 129.30 samples/sec Loss 3.4136 Epoch: 13 Global Step: 137750 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:46:47,298-Speed 130.17 samples/sec Loss 3.5257 Epoch: 13 Global Step: 137755 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:47:07,184-Speed 128.74 samples/sec Loss 3.1787 Epoch: 13 Global Step: 137760 Fp16 Grad Scale: 8192 Required: 132 hours Training: 2021-06-13 13:47:26,879-Speed 129.99 samples/sec Loss 3.5098 Epoch: 13 Global Step: 137765 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:47:47,230-Speed 125.80 samples/sec Loss 3.3416 Epoch: 13 Global Step: 137770 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:48:06,923-Speed 130.00 samples/sec Loss 3.4081 Epoch: 13 Global Step: 137775 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:48:26,986-Speed 127.61 samples/sec Loss 3.4828 Epoch: 13 Global Step: 137780 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:48:46,771-Speed 129.39 samples/sec Loss 3.4829 Epoch: 13 Global Step: 137785 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:49:06,565-Speed 129.34 samples/sec Loss 3.4869 Epoch: 13 Global Step: 137790 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:49:26,321-Speed 129.59 samples/sec Loss 3.3687 Epoch: 13 Global Step: 137795 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:49:46,196-Speed 128.81 samples/sec Loss 3.1308 Epoch: 13 Global Step: 137800 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:50:05,914-Speed 129.84 samples/sec Loss 3.6450 Epoch: 13 Global Step: 137805 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:50:25,810-Speed 128.67 samples/sec Loss 3.2995 Epoch: 13 Global Step: 137810 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:50:45,833-Speed 127.86 samples/sec Loss 3.3909 Epoch: 13 Global Step: 137815 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:51:05,594-Speed 129.55 samples/sec Loss 3.4136 Epoch: 13 Global Step: 137820 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:51:25,572-Speed 128.15 samples/sec Loss 3.5545 Epoch: 13 Global Step: 137825 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:51:45,590-Speed 127.89 samples/sec Loss 3.4548 Epoch: 13 Global Step: 137830 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:52:05,317-Speed 129.78 samples/sec Loss 3.2873 Epoch: 13 Global Step: 137835 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:52:25,188-Speed 128.83 samples/sec Loss 3.3781 Epoch: 13 Global Step: 137840 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:52:44,904-Speed 129.86 samples/sec Loss 3.3505 Epoch: 13 Global Step: 137845 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:53:04,659-Speed 129.59 samples/sec Loss 3.4985 Epoch: 13 Global Step: 137850 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:53:24,510-Speed 128.97 samples/sec Loss 3.4168 Epoch: 13 Global Step: 137855 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:53:44,534-Speed 127.85 samples/sec Loss 3.4918 Epoch: 13 Global Step: 137860 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:54:04,396-Speed 128.90 samples/sec Loss 3.5391 Epoch: 13 Global Step: 137865 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:54:24,380-Speed 128.11 samples/sec Loss 3.5887 Epoch: 13 Global Step: 137870 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:54:44,252-Speed 128.83 samples/sec Loss 3.2417 Epoch: 13 Global Step: 137875 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:55:03,963-Speed 129.88 samples/sec Loss 3.4523 Epoch: 13 Global Step: 137880 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:55:23,843-Speed 128.78 samples/sec Loss 3.5254 Epoch: 13 Global Step: 137885 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:55:43,583-Speed 129.69 samples/sec Loss 3.4794 Epoch: 13 Global Step: 137890 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:56:03,503-Speed 128.52 samples/sec Loss 3.5565 Epoch: 13 Global Step: 137895 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:56:23,223-Speed 129.83 samples/sec Loss 3.4889 Epoch: 13 Global Step: 137900 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:56:43,349-Speed 127.20 samples/sec Loss 3.4970 Epoch: 13 Global Step: 137905 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:57:03,180-Speed 129.10 samples/sec Loss 3.6644 Epoch: 13 Global Step: 137910 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:57:23,254-Speed 127.53 samples/sec Loss 3.6086 Epoch: 13 Global Step: 137915 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:57:43,051-Speed 129.32 samples/sec Loss 3.5605 Epoch: 13 Global Step: 137920 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:58:03,094-Speed 127.73 samples/sec Loss 3.3296 Epoch: 13 Global Step: 137925 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:58:22,879-Speed 129.40 samples/sec Loss 3.7494 Epoch: 13 Global Step: 137930 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:58:42,592-Speed 129.87 samples/sec Loss 3.5154 Epoch: 13 Global Step: 137935 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:59:02,516-Speed 128.49 samples/sec Loss 3.3688 Epoch: 13 Global Step: 137940 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:59:22,228-Speed 129.88 samples/sec Loss 3.4568 Epoch: 13 Global Step: 137945 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 13:59:42,442-Speed 126.65 samples/sec Loss 3.3539 Epoch: 13 Global Step: 137950 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:00:02,160-Speed 129.83 samples/sec Loss 3.4454 Epoch: 13 Global Step: 137955 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:00:22,316-Speed 127.02 samples/sec Loss 3.2601 Epoch: 13 Global Step: 137960 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:00:42,038-Speed 129.80 samples/sec Loss 3.6005 Epoch: 13 Global Step: 137965 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:01:01,947-Speed 128.59 samples/sec Loss 3.5172 Epoch: 13 Global Step: 137970 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:01:21,553-Speed 130.58 samples/sec Loss 3.4915 Epoch: 13 Global Step: 137975 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:01:41,456-Speed 128.63 samples/sec Loss 3.4176 Epoch: 13 Global Step: 137980 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:02:01,252-Speed 129.33 samples/sec Loss 3.6756 Epoch: 13 Global Step: 137985 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:02:21,003-Speed 129.62 samples/sec Loss 3.3980 Epoch: 13 Global Step: 137990 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:02:41,193-Speed 126.80 samples/sec Loss 3.2372 Epoch: 13 Global Step: 137995 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:03:00,911-Speed 129.84 samples/sec Loss 3.2792 Epoch: 13 Global Step: 138000 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:03:20,980-Speed 127.57 samples/sec Loss 3.3171 Epoch: 13 Global Step: 138005 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:03:40,614-Speed 130.39 samples/sec Loss 3.5663 Epoch: 13 Global Step: 138010 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:04:00,499-Speed 128.74 samples/sec Loss 3.2214 Epoch: 13 Global Step: 138015 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:04:20,280-Speed 129.42 samples/sec Loss 3.2359 Epoch: 13 Global Step: 138020 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:04:40,063-Speed 129.41 samples/sec Loss 3.3045 Epoch: 13 Global Step: 138025 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:04:59,797-Speed 129.74 samples/sec Loss 3.4349 Epoch: 13 Global Step: 138030 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:05:19,548-Speed 129.62 samples/sec Loss 3.4705 Epoch: 13 Global Step: 138035 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:05:39,714-Speed 126.95 samples/sec Loss 3.3457 Epoch: 13 Global Step: 138040 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:05:59,434-Speed 129.82 samples/sec Loss 3.4907 Epoch: 13 Global Step: 138045 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:06:19,517-Speed 127.48 samples/sec Loss 3.1922 Epoch: 13 Global Step: 138050 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:06:39,418-Speed 128.64 samples/sec Loss 3.3351 Epoch: 13 Global Step: 138055 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:06:59,084-Speed 130.18 samples/sec Loss 3.5735 Epoch: 13 Global Step: 138060 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:07:18,884-Speed 129.30 samples/sec Loss 3.5909 Epoch: 13 Global Step: 138065 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:07:38,735-Speed 128.97 samples/sec Loss 3.5274 Epoch: 13 Global Step: 138070 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:07:58,353-Speed 130.50 samples/sec Loss 3.3362 Epoch: 13 Global Step: 138075 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:08:18,341-Speed 128.08 samples/sec Loss 3.3703 Epoch: 13 Global Step: 138080 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:08:38,317-Speed 128.16 samples/sec Loss 3.4860 Epoch: 13 Global Step: 138085 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:08:58,065-Speed 129.64 samples/sec Loss 3.2647 Epoch: 13 Global Step: 138090 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:09:18,097-Speed 127.80 samples/sec Loss 3.5250 Epoch: 13 Global Step: 138095 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:09:38,001-Speed 128.63 samples/sec Loss 3.3418 Epoch: 13 Global Step: 138100 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:09:57,791-Speed 129.36 samples/sec Loss 3.4033 Epoch: 13 Global Step: 138105 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:10:17,599-Speed 129.24 samples/sec Loss 3.6332 Epoch: 13 Global Step: 138110 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:10:37,330-Speed 129.75 samples/sec Loss 3.2677 Epoch: 13 Global Step: 138115 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:10:57,147-Speed 129.19 samples/sec Loss 3.4322 Epoch: 13 Global Step: 138120 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:11:16,970-Speed 129.14 samples/sec Loss 3.3568 Epoch: 13 Global Step: 138125 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:11:37,006-Speed 127.78 samples/sec Loss 3.4747 Epoch: 13 Global Step: 138130 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:11:56,920-Speed 128.56 samples/sec Loss 3.5363 Epoch: 13 Global Step: 138135 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:12:16,911-Speed 128.07 samples/sec Loss 3.4258 Epoch: 13 Global Step: 138140 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:12:36,759-Speed 128.99 samples/sec Loss 3.3756 Epoch: 13 Global Step: 138145 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:12:56,433-Speed 130.12 samples/sec Loss 3.4512 Epoch: 13 Global Step: 138150 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:13:16,322-Speed 128.72 samples/sec Loss 3.5303 Epoch: 13 Global Step: 138155 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:13:36,026-Speed 129.93 samples/sec Loss 3.5100 Epoch: 13 Global Step: 138160 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:13:55,994-Speed 128.21 samples/sec Loss 3.5201 Epoch: 13 Global Step: 138165 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:14:15,607-Speed 130.53 samples/sec Loss 3.3684 Epoch: 13 Global Step: 138170 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:14:35,548-Speed 128.39 samples/sec Loss 3.2924 Epoch: 13 Global Step: 138175 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:14:55,721-Speed 126.91 samples/sec Loss 3.4804 Epoch: 13 Global Step: 138180 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:15:15,858-Speed 127.14 samples/sec Loss 3.3992 Epoch: 13 Global Step: 138185 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:15:35,580-Speed 129.80 samples/sec Loss 3.4383 Epoch: 13 Global Step: 138190 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:15:55,305-Speed 129.79 samples/sec Loss 3.3947 Epoch: 13 Global Step: 138195 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:16:15,150-Speed 129.01 samples/sec Loss 3.5351 Epoch: 13 Global Step: 138200 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:16:34,882-Speed 129.74 samples/sec Loss 3.2479 Epoch: 13 Global Step: 138205 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:16:54,844-Speed 128.25 samples/sec Loss 3.5731 Epoch: 13 Global Step: 138210 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:17:14,518-Speed 130.13 samples/sec Loss 3.4154 Epoch: 13 Global Step: 138215 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:17:34,627-Speed 127.32 samples/sec Loss 3.5491 Epoch: 13 Global Step: 138220 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:17:54,489-Speed 128.90 samples/sec Loss 3.3815 Epoch: 13 Global Step: 138225 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:18:14,620-Speed 127.17 samples/sec Loss 3.4832 Epoch: 13 Global Step: 138230 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:18:34,227-Speed 130.57 samples/sec Loss 3.6179 Epoch: 13 Global Step: 138235 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:18:54,130-Speed 128.63 samples/sec Loss 3.4073 Epoch: 13 Global Step: 138240 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:19:13,826-Speed 129.98 samples/sec Loss 3.7127 Epoch: 13 Global Step: 138245 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:19:33,614-Speed 129.38 samples/sec Loss 3.3472 Epoch: 13 Global Step: 138250 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:19:53,485-Speed 128.84 samples/sec Loss 3.5635 Epoch: 13 Global Step: 138255 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:20:13,115-Speed 130.42 samples/sec Loss 3.3483 Epoch: 13 Global Step: 138260 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:20:33,643-Speed 124.71 samples/sec Loss 3.4218 Epoch: 13 Global Step: 138265 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:20:53,442-Speed 129.31 samples/sec Loss 3.5563 Epoch: 13 Global Step: 138270 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:21:13,445-Speed 127.99 samples/sec Loss 3.5633 Epoch: 13 Global Step: 138275 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:21:33,146-Speed 129.95 samples/sec Loss 3.2940 Epoch: 13 Global Step: 138280 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:21:53,028-Speed 128.77 samples/sec Loss 3.6301 Epoch: 13 Global Step: 138285 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:22:12,811-Speed 129.41 samples/sec Loss 3.4947 Epoch: 13 Global Step: 138290 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:22:32,622-Speed 129.23 samples/sec Loss 3.2421 Epoch: 13 Global Step: 138295 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:22:52,354-Speed 129.75 samples/sec Loss 3.5431 Epoch: 13 Global Step: 138300 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:23:12,091-Speed 129.71 samples/sec Loss 3.3986 Epoch: 13 Global Step: 138305 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:23:32,369-Speed 126.25 samples/sec Loss 3.2634 Epoch: 13 Global Step: 138310 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:23:52,095-Speed 129.79 samples/sec Loss 3.3088 Epoch: 13 Global Step: 138315 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:24:12,164-Speed 127.56 samples/sec Loss 3.5017 Epoch: 13 Global Step: 138320 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:24:31,925-Speed 129.55 samples/sec Loss 3.4818 Epoch: 13 Global Step: 138325 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:24:51,732-Speed 129.25 samples/sec Loss 3.6592 Epoch: 13 Global Step: 138330 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:25:11,504-Speed 129.49 samples/sec Loss 3.2808 Epoch: 13 Global Step: 138335 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:25:31,316-Speed 129.22 samples/sec Loss 3.3908 Epoch: 13 Global Step: 138340 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:25:51,024-Speed 129.90 samples/sec Loss 3.4563 Epoch: 13 Global Step: 138345 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:26:10,777-Speed 129.61 samples/sec Loss 3.3488 Epoch: 13 Global Step: 138350 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:26:30,899-Speed 127.23 samples/sec Loss 3.4907 Epoch: 13 Global Step: 138355 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:26:50,615-Speed 129.85 samples/sec Loss 3.5652 Epoch: 13 Global Step: 138360 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:27:10,697-Speed 127.49 samples/sec Loss 3.4609 Epoch: 13 Global Step: 138365 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:27:30,485-Speed 129.37 samples/sec Loss 3.3696 Epoch: 13 Global Step: 138370 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:27:50,203-Speed 129.84 samples/sec Loss 3.2930 Epoch: 13 Global Step: 138375 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:28:09,977-Speed 129.47 samples/sec Loss 3.5125 Epoch: 13 Global Step: 138380 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:28:29,860-Speed 128.76 samples/sec Loss 3.4325 Epoch: 13 Global Step: 138385 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:28:49,625-Speed 129.53 samples/sec Loss 3.2597 Epoch: 13 Global Step: 138390 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:29:09,409-Speed 129.40 samples/sec Loss 3.6782 Epoch: 13 Global Step: 138395 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:29:29,474-Speed 127.59 samples/sec Loss 3.5197 Epoch: 13 Global Step: 138400 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:29:49,405-Speed 128.45 samples/sec Loss 3.4664 Epoch: 13 Global Step: 138405 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:30:09,221-Speed 129.19 samples/sec Loss 3.5611 Epoch: 13 Global Step: 138410 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:30:29,135-Speed 128.56 samples/sec Loss 3.5005 Epoch: 13 Global Step: 138415 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:30:48,848-Speed 129.87 samples/sec Loss 3.4360 Epoch: 13 Global Step: 138420 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:31:08,735-Speed 128.73 samples/sec Loss 3.5583 Epoch: 13 Global Step: 138425 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:31:28,448-Speed 129.88 samples/sec Loss 3.2884 Epoch: 13 Global Step: 138430 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:31:48,304-Speed 128.93 samples/sec Loss 3.5374 Epoch: 13 Global Step: 138435 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:32:08,062-Speed 129.58 samples/sec Loss 3.4500 Epoch: 13 Global Step: 138440 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:32:28,050-Speed 128.08 samples/sec Loss 3.4341 Epoch: 13 Global Step: 138445 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:32:47,906-Speed 128.94 samples/sec Loss 3.5815 Epoch: 13 Global Step: 138450 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:33:07,978-Speed 127.55 samples/sec Loss 3.2393 Epoch: 13 Global Step: 138455 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:33:27,688-Speed 129.89 samples/sec Loss 3.5049 Epoch: 13 Global Step: 138460 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:33:47,435-Speed 129.65 samples/sec Loss 3.6709 Epoch: 13 Global Step: 138465 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:34:07,295-Speed 128.90 samples/sec Loss 3.5066 Epoch: 13 Global Step: 138470 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:34:26,979-Speed 130.06 samples/sec Loss 3.3406 Epoch: 13 Global Step: 138475 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:34:46,946-Speed 128.23 samples/sec Loss 3.5655 Epoch: 13 Global Step: 138480 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:35:06,621-Speed 130.12 samples/sec Loss 3.6516 Epoch: 13 Global Step: 138485 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:35:26,578-Speed 128.28 samples/sec Loss 3.2906 Epoch: 13 Global Step: 138490 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:35:46,370-Speed 129.35 samples/sec Loss 3.4037 Epoch: 13 Global Step: 138495 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:36:06,479-Speed 127.32 samples/sec Loss 3.6196 Epoch: 13 Global Step: 138500 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:36:26,199-Speed 129.82 samples/sec Loss 3.5228 Epoch: 13 Global Step: 138505 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:36:45,963-Speed 129.53 samples/sec Loss 3.3981 Epoch: 13 Global Step: 138510 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:37:05,725-Speed 129.55 samples/sec Loss 3.3259 Epoch: 13 Global Step: 138515 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:37:25,503-Speed 129.44 samples/sec Loss 3.3768 Epoch: 13 Global Step: 138520 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:37:45,432-Speed 128.47 samples/sec Loss 3.4126 Epoch: 13 Global Step: 138525 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:38:05,161-Speed 129.76 samples/sec Loss 3.7428 Epoch: 13 Global Step: 138530 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:38:25,238-Speed 127.51 samples/sec Loss 3.3669 Epoch: 13 Global Step: 138535 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:38:45,015-Speed 129.45 samples/sec Loss 3.5179 Epoch: 13 Global Step: 138540 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:39:05,080-Speed 127.59 samples/sec Loss 3.4835 Epoch: 13 Global Step: 138545 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:39:24,836-Speed 129.59 samples/sec Loss 3.4938 Epoch: 13 Global Step: 138550 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:39:44,714-Speed 128.79 samples/sec Loss 3.3416 Epoch: 13 Global Step: 138555 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:40:04,500-Speed 129.39 samples/sec Loss 3.3301 Epoch: 13 Global Step: 138560 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:40:24,336-Speed 129.07 samples/sec Loss 3.5138 Epoch: 13 Global Step: 138565 Fp16 Grad Scale: 8192 Required: 131 hours Training: 2021-06-13 14:40:44,162-Speed 129.13 samples/sec Loss 3.1784 Epoch: 13 Global Step: 138570 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:41:03,901-Speed 129.70 samples/sec Loss 3.4862 Epoch: 13 Global Step: 138575 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:41:24,088-Speed 126.82 samples/sec Loss 3.5409 Epoch: 13 Global Step: 138580 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:41:43,806-Speed 129.84 samples/sec Loss 3.4330 Epoch: 13 Global Step: 138585 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:42:03,787-Speed 128.13 samples/sec Loss 3.4064 Epoch: 13 Global Step: 138590 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:42:23,554-Speed 129.51 samples/sec Loss 3.3818 Epoch: 13 Global Step: 138595 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:42:43,474-Speed 128.52 samples/sec Loss 3.4454 Epoch: 13 Global Step: 138600 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:43:03,212-Speed 129.71 samples/sec Loss 3.4638 Epoch: 13 Global Step: 138605 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:43:23,076-Speed 128.88 samples/sec Loss 3.5192 Epoch: 13 Global Step: 138610 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:43:42,698-Speed 130.48 samples/sec Loss 3.4243 Epoch: 13 Global Step: 138615 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:44:02,536-Speed 129.05 samples/sec Loss 3.6471 Epoch: 13 Global Step: 138620 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:44:22,629-Speed 127.41 samples/sec Loss 3.6919 Epoch: 13 Global Step: 138625 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:44:42,273-Speed 130.33 samples/sec Loss 3.6053 Epoch: 13 Global Step: 138630 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:45:02,331-Speed 127.63 samples/sec Loss 3.5600 Epoch: 13 Global Step: 138635 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:45:22,195-Speed 128.88 samples/sec Loss 3.5232 Epoch: 13 Global Step: 138640 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:45:41,979-Speed 129.40 samples/sec Loss 3.4968 Epoch: 13 Global Step: 138645 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:46:01,758-Speed 129.44 samples/sec Loss 3.4441 Epoch: 13 Global Step: 138650 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:46:21,527-Speed 129.51 samples/sec Loss 3.5317 Epoch: 13 Global Step: 138655 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:46:41,264-Speed 129.71 samples/sec Loss 3.5141 Epoch: 13 Global Step: 138660 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:47:01,239-Speed 128.17 samples/sec Loss 3.5812 Epoch: 13 Global Step: 138665 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:47:20,693-Speed 131.60 samples/sec Loss 3.3483 Epoch: 13 Global Step: 138670 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:47:40,502-Speed 129.24 samples/sec Loss 3.6083 Epoch: 13 Global Step: 138675 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:48:00,224-Speed 129.81 samples/sec Loss 3.4414 Epoch: 13 Global Step: 138680 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:48:20,042-Speed 129.18 samples/sec Loss 3.5591 Epoch: 13 Global Step: 138685 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:48:40,082-Speed 127.75 samples/sec Loss 3.5126 Epoch: 13 Global Step: 138690 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:48:59,855-Speed 129.48 samples/sec Loss 3.2681 Epoch: 13 Global Step: 138695 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:49:19,557-Speed 129.94 samples/sec Loss 3.4978 Epoch: 13 Global Step: 138700 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:49:39,532-Speed 128.17 samples/sec Loss 3.3619 Epoch: 13 Global Step: 138705 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:49:59,164-Speed 130.41 samples/sec Loss 3.5178 Epoch: 13 Global Step: 138710 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:50:19,137-Speed 128.18 samples/sec Loss 3.4693 Epoch: 13 Global Step: 138715 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:50:39,006-Speed 128.86 samples/sec Loss 3.5704 Epoch: 13 Global Step: 138720 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:50:58,749-Speed 129.67 samples/sec Loss 3.5075 Epoch: 13 Global Step: 138725 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:51:18,641-Speed 128.71 samples/sec Loss 3.5961 Epoch: 13 Global Step: 138730 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:51:38,484-Speed 129.02 samples/sec Loss 3.4401 Epoch: 13 Global Step: 138735 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:51:58,383-Speed 128.66 samples/sec Loss 3.3960 Epoch: 13 Global Step: 138740 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:52:18,105-Speed 129.81 samples/sec Loss 3.3267 Epoch: 13 Global Step: 138745 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:52:38,006-Speed 128.64 samples/sec Loss 3.6835 Epoch: 13 Global Step: 138750 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:52:57,731-Speed 129.79 samples/sec Loss 3.4473 Epoch: 13 Global Step: 138755 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:53:17,827-Speed 127.39 samples/sec Loss 3.5037 Epoch: 13 Global Step: 138760 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:53:37,593-Speed 129.52 samples/sec Loss 3.2739 Epoch: 13 Global Step: 138765 Fp16 Grad Scale: 4096 Required: 130 hours Training: 2021-06-13 14:53:57,294-Speed 129.95 samples/sec Loss 3.4663 Epoch: 13 Global Step: 138770 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:54:17,114-Speed 129.17 samples/sec Loss 3.4685 Epoch: 13 Global Step: 138775 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:54:37,129-Speed 127.91 samples/sec Loss 3.1994 Epoch: 13 Global Step: 138780 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:54:56,830-Speed 129.95 samples/sec Loss 3.5972 Epoch: 13 Global Step: 138785 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:55:16,774-Speed 128.36 samples/sec Loss 3.5423 Epoch: 13 Global Step: 138790 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:55:36,420-Speed 130.31 samples/sec Loss 3.3612 Epoch: 13 Global Step: 138795 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:55:56,183-Speed 129.54 samples/sec Loss 3.5002 Epoch: 13 Global Step: 138800 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:56:16,273-Speed 127.44 samples/sec Loss 3.5002 Epoch: 13 Global Step: 138805 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:56:36,020-Speed 129.64 samples/sec Loss 3.4147 Epoch: 13 Global Step: 138810 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:56:55,910-Speed 128.72 samples/sec Loss 3.2905 Epoch: 13 Global Step: 138815 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:57:15,562-Speed 130.28 samples/sec Loss 3.6778 Epoch: 13 Global Step: 138820 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:57:35,614-Speed 127.67 samples/sec Loss 3.4289 Epoch: 13 Global Step: 138825 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:57:55,372-Speed 129.57 samples/sec Loss 3.6197 Epoch: 13 Global Step: 138830 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:58:15,218-Speed 129.00 samples/sec Loss 3.2627 Epoch: 13 Global Step: 138835 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:58:34,964-Speed 129.65 samples/sec Loss 3.4070 Epoch: 13 Global Step: 138840 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:58:54,848-Speed 128.75 samples/sec Loss 3.3544 Epoch: 13 Global Step: 138845 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:59:14,797-Speed 128.33 samples/sec Loss 3.4751 Epoch: 13 Global Step: 138850 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:59:34,557-Speed 129.57 samples/sec Loss 3.2159 Epoch: 13 Global Step: 138855 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 14:59:54,390-Speed 129.08 samples/sec Loss 3.3565 Epoch: 13 Global Step: 138860 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:00:14,101-Speed 129.88 samples/sec Loss 3.2708 Epoch: 13 Global Step: 138865 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:00:34,180-Speed 127.51 samples/sec Loss 3.5970 Epoch: 13 Global Step: 138870 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:00:54,124-Speed 128.36 samples/sec Loss 3.1543 Epoch: 13 Global Step: 138875 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:01:13,849-Speed 129.79 samples/sec Loss 3.3054 Epoch: 13 Global Step: 138880 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:01:33,762-Speed 128.57 samples/sec Loss 3.6789 Epoch: 13 Global Step: 138885 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:01:53,699-Speed 128.41 samples/sec Loss 3.5092 Epoch: 13 Global Step: 138890 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:02:13,846-Speed 127.07 samples/sec Loss 3.3925 Epoch: 13 Global Step: 138895 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:02:33,842-Speed 128.03 samples/sec Loss 3.6525 Epoch: 13 Global Step: 138900 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:02:53,538-Speed 129.99 samples/sec Loss 3.4369 Epoch: 13 Global Step: 138905 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:03:13,445-Speed 128.61 samples/sec Loss 3.4992 Epoch: 13 Global Step: 138910 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:03:33,406-Speed 128.25 samples/sec Loss 3.4864 Epoch: 13 Global Step: 138915 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:03:53,221-Speed 129.20 samples/sec Loss 3.2995 Epoch: 13 Global Step: 138920 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:04:12,889-Speed 130.17 samples/sec Loss 3.2046 Epoch: 13 Global Step: 138925 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:04:32,968-Speed 127.51 samples/sec Loss 3.4411 Epoch: 13 Global Step: 138930 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:04:52,666-Speed 129.97 samples/sec Loss 3.4253 Epoch: 13 Global Step: 138935 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:05:12,916-Speed 126.43 samples/sec Loss 3.4186 Epoch: 13 Global Step: 138940 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:05:32,623-Speed 129.91 samples/sec Loss 3.4532 Epoch: 13 Global Step: 138945 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:05:52,366-Speed 129.67 samples/sec Loss 3.6743 Epoch: 13 Global Step: 138950 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:06:12,211-Speed 129.01 samples/sec Loss 3.3485 Epoch: 13 Global Step: 138955 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:06:32,464-Speed 126.41 samples/sec Loss 3.5388 Epoch: 13 Global Step: 138960 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:06:52,158-Speed 129.99 samples/sec Loss 3.4391 Epoch: 13 Global Step: 138965 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:07:11,935-Speed 129.45 samples/sec Loss 3.7228 Epoch: 13 Global Step: 138970 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:07:31,753-Speed 129.18 samples/sec Loss 3.5470 Epoch: 13 Global Step: 138975 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:07:51,526-Speed 129.48 samples/sec Loss 3.3264 Epoch: 13 Global Step: 138980 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:08:11,693-Speed 126.94 samples/sec Loss 3.7715 Epoch: 13 Global Step: 138985 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:08:31,369-Speed 130.12 samples/sec Loss 3.3191 Epoch: 13 Global Step: 138990 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:08:51,315-Speed 128.35 samples/sec Loss 3.2528 Epoch: 13 Global Step: 138995 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:09:10,955-Speed 130.36 samples/sec Loss 3.3672 Epoch: 13 Global Step: 139000 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:09:30,972-Speed 127.90 samples/sec Loss 3.4682 Epoch: 13 Global Step: 139005 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:09:50,690-Speed 129.83 samples/sec Loss 3.5478 Epoch: 13 Global Step: 139010 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:10:10,595-Speed 128.62 samples/sec Loss 3.3895 Epoch: 13 Global Step: 139015 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:10:30,371-Speed 129.46 samples/sec Loss 3.6527 Epoch: 13 Global Step: 139020 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:10:50,113-Speed 129.68 samples/sec Loss 3.6251 Epoch: 13 Global Step: 139025 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:11:10,137-Speed 127.85 samples/sec Loss 3.5816 Epoch: 13 Global Step: 139030 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:11:29,929-Speed 129.36 samples/sec Loss 3.4795 Epoch: 13 Global Step: 139035 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:11:49,828-Speed 128.65 samples/sec Loss 3.8620 Epoch: 13 Global Step: 139040 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:12:09,552-Speed 129.81 samples/sec Loss 3.4668 Epoch: 13 Global Step: 139045 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:12:29,568-Speed 127.91 samples/sec Loss 3.5110 Epoch: 13 Global Step: 139050 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:12:49,320-Speed 129.61 samples/sec Loss 3.4902 Epoch: 13 Global Step: 139055 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:13:09,170-Speed 128.97 samples/sec Loss 3.5302 Epoch: 13 Global Step: 139060 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:13:29,026-Speed 128.93 samples/sec Loss 3.4936 Epoch: 13 Global Step: 139065 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:13:48,878-Speed 128.96 samples/sec Loss 3.3983 Epoch: 13 Global Step: 139070 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:14:08,898-Speed 127.88 samples/sec Loss 3.4025 Epoch: 13 Global Step: 139075 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:14:28,786-Speed 128.72 samples/sec Loss 3.4682 Epoch: 13 Global Step: 139080 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:14:48,391-Speed 130.59 samples/sec Loss 3.3996 Epoch: 13 Global Step: 139085 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:15:08,118-Speed 129.78 samples/sec Loss 3.4229 Epoch: 13 Global Step: 139090 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:15:28,087-Speed 128.20 samples/sec Loss 3.6120 Epoch: 13 Global Step: 139095 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:15:48,058-Speed 128.19 samples/sec Loss 3.4693 Epoch: 13 Global Step: 139100 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:16:07,697-Speed 130.36 samples/sec Loss 3.5331 Epoch: 13 Global Step: 139105 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:16:27,515-Speed 129.19 samples/sec Loss 3.4665 Epoch: 13 Global Step: 139110 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:16:47,292-Speed 129.45 samples/sec Loss 3.3786 Epoch: 13 Global Step: 139115 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:17:07,357-Speed 127.59 samples/sec Loss 3.3858 Epoch: 13 Global Step: 139120 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:17:27,133-Speed 129.46 samples/sec Loss 3.7045 Epoch: 13 Global Step: 139125 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:17:46,838-Speed 129.92 samples/sec Loss 3.4302 Epoch: 13 Global Step: 139130 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:18:06,647-Speed 129.24 samples/sec Loss 3.5142 Epoch: 13 Global Step: 139135 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:18:26,634-Speed 128.09 samples/sec Loss 3.5379 Epoch: 13 Global Step: 139140 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:18:46,592-Speed 128.27 samples/sec Loss 3.3866 Epoch: 13 Global Step: 139145 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:19:06,380-Speed 129.38 samples/sec Loss 3.3871 Epoch: 13 Global Step: 139150 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:19:26,465-Speed 127.46 samples/sec Loss 3.5034 Epoch: 13 Global Step: 139155 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:19:46,138-Speed 130.13 samples/sec Loss 3.5138 Epoch: 13 Global Step: 139160 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:20:06,309-Speed 126.92 samples/sec Loss 3.1233 Epoch: 13 Global Step: 139165 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:20:25,927-Speed 130.50 samples/sec Loss 3.3391 Epoch: 13 Global Step: 139170 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:20:45,609-Speed 130.07 samples/sec Loss 3.3663 Epoch: 13 Global Step: 139175 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:21:05,594-Speed 128.11 samples/sec Loss 3.5001 Epoch: 13 Global Step: 139180 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:21:25,711-Speed 127.26 samples/sec Loss 3.2850 Epoch: 13 Global Step: 139185 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:21:45,321-Speed 130.55 samples/sec Loss 3.4875 Epoch: 13 Global Step: 139190 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:22:05,097-Speed 129.46 samples/sec Loss 3.5989 Epoch: 13 Global Step: 139195 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:22:24,908-Speed 129.23 samples/sec Loss 3.4600 Epoch: 13 Global Step: 139200 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:22:44,671-Speed 129.54 samples/sec Loss 3.5091 Epoch: 13 Global Step: 139205 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:23:04,735-Speed 127.60 samples/sec Loss 3.5956 Epoch: 13 Global Step: 139210 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:23:24,503-Speed 129.51 samples/sec Loss 3.5234 Epoch: 13 Global Step: 139215 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:23:44,427-Speed 128.50 samples/sec Loss 3.5107 Epoch: 13 Global Step: 139220 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:24:04,211-Speed 129.40 samples/sec Loss 3.4003 Epoch: 13 Global Step: 139225 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:24:24,186-Speed 128.17 samples/sec Loss 3.3625 Epoch: 13 Global Step: 139230 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:24:43,913-Speed 129.78 samples/sec Loss 3.4745 Epoch: 13 Global Step: 139235 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:25:03,700-Speed 129.38 samples/sec Loss 3.4950 Epoch: 13 Global Step: 139240 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:25:23,499-Speed 129.31 samples/sec Loss 3.3887 Epoch: 13 Global Step: 139245 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:25:43,427-Speed 128.47 samples/sec Loss 3.4395 Epoch: 13 Global Step: 139250 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:26:03,345-Speed 128.53 samples/sec Loss 3.1669 Epoch: 13 Global Step: 139255 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:26:23,263-Speed 128.53 samples/sec Loss 3.6181 Epoch: 13 Global Step: 139260 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:26:43,147-Speed 128.76 samples/sec Loss 3.4116 Epoch: 13 Global Step: 139265 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:27:02,844-Speed 129.98 samples/sec Loss 3.3801 Epoch: 13 Global Step: 139270 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:27:22,798-Speed 128.30 samples/sec Loss 3.4850 Epoch: 13 Global Step: 139275 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:27:43,150-Speed 125.79 samples/sec Loss 3.4633 Epoch: 13 Global Step: 139280 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:28:02,942-Speed 129.35 samples/sec Loss 3.6499 Epoch: 13 Global Step: 139285 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:28:22,742-Speed 129.30 samples/sec Loss 3.4845 Epoch: 13 Global Step: 139290 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:28:42,525-Speed 129.41 samples/sec Loss 3.5272 Epoch: 13 Global Step: 139295 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:29:02,532-Speed 127.96 samples/sec Loss 3.3857 Epoch: 13 Global Step: 139300 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:29:22,274-Speed 129.68 samples/sec Loss 3.4308 Epoch: 13 Global Step: 139305 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:29:42,116-Speed 129.03 samples/sec Loss 3.4071 Epoch: 13 Global Step: 139310 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:30:01,757-Speed 130.35 samples/sec Loss 3.3991 Epoch: 13 Global Step: 139315 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:30:21,828-Speed 127.55 samples/sec Loss 3.8427 Epoch: 13 Global Step: 139320 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:30:41,645-Speed 129.19 samples/sec Loss 3.3783 Epoch: 13 Global Step: 139325 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:31:01,457-Speed 129.22 samples/sec Loss 3.7163 Epoch: 13 Global Step: 139330 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:31:21,206-Speed 129.63 samples/sec Loss 3.4645 Epoch: 13 Global Step: 139335 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:31:41,053-Speed 128.99 samples/sec Loss 3.3473 Epoch: 13 Global Step: 139340 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:32:01,047-Speed 128.04 samples/sec Loss 3.4051 Epoch: 13 Global Step: 139345 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:32:20,968-Speed 128.52 samples/sec Loss 3.6783 Epoch: 13 Global Step: 139350 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:32:40,875-Speed 128.61 samples/sec Loss 3.2657 Epoch: 13 Global Step: 139355 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:33:00,633-Speed 129.57 samples/sec Loss 3.5316 Epoch: 13 Global Step: 139360 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:33:20,654-Speed 127.88 samples/sec Loss 3.6591 Epoch: 13 Global Step: 139365 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:33:40,558-Speed 128.62 samples/sec Loss 3.4348 Epoch: 13 Global Step: 139370 Fp16 Grad Scale: 8192 Required: 130 hours Training: 2021-06-13 15:34:00,283-Speed 129.79 samples/sec Loss 3.5819 Epoch: 13 Global Step: 139375 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:34:20,244-Speed 128.26 samples/sec Loss 3.4402 Epoch: 13 Global Step: 139380 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:34:39,977-Speed 129.74 samples/sec Loss 3.4204 Epoch: 13 Global Step: 139385 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:35:00,029-Speed 127.68 samples/sec Loss 3.6044 Epoch: 13 Global Step: 139390 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:35:19,942-Speed 128.57 samples/sec Loss 3.3196 Epoch: 13 Global Step: 139395 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:35:39,654-Speed 129.88 samples/sec Loss 3.3218 Epoch: 13 Global Step: 139400 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:35:59,545-Speed 128.71 samples/sec Loss 3.5054 Epoch: 13 Global Step: 139405 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:36:19,488-Speed 128.37 samples/sec Loss 3.3499 Epoch: 13 Global Step: 139410 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:36:39,368-Speed 128.77 samples/sec Loss 3.5277 Epoch: 13 Global Step: 139415 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:36:59,146-Speed 129.45 samples/sec Loss 3.4811 Epoch: 13 Global Step: 139420 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:37:19,045-Speed 128.65 samples/sec Loss 3.4190 Epoch: 13 Global Step: 139425 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:37:38,644-Speed 130.62 samples/sec Loss 3.4827 Epoch: 13 Global Step: 139430 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:37:58,842-Speed 126.75 samples/sec Loss 3.5051 Epoch: 13 Global Step: 139435 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:38:18,656-Speed 129.21 samples/sec Loss 3.5355 Epoch: 13 Global Step: 139440 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:38:38,460-Speed 129.27 samples/sec Loss 3.6470 Epoch: 13 Global Step: 139445 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:38:58,310-Speed 128.98 samples/sec Loss 3.4879 Epoch: 13 Global Step: 139450 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:39:18,265-Speed 128.29 samples/sec Loss 3.6786 Epoch: 13 Global Step: 139455 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:39:37,997-Speed 129.74 samples/sec Loss 3.4084 Epoch: 13 Global Step: 139460 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:39:57,813-Speed 129.19 samples/sec Loss 3.5753 Epoch: 13 Global Step: 139465 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:40:17,597-Speed 129.40 samples/sec Loss 3.4545 Epoch: 13 Global Step: 139470 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:40:37,310-Speed 129.87 samples/sec Loss 3.3384 Epoch: 13 Global Step: 139475 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:40:57,489-Speed 126.87 samples/sec Loss 3.4332 Epoch: 13 Global Step: 139480 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:41:17,302-Speed 129.21 samples/sec Loss 3.6912 Epoch: 13 Global Step: 139485 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:41:37,006-Speed 129.93 samples/sec Loss 3.2996 Epoch: 13 Global Step: 139490 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:41:56,575-Speed 130.82 samples/sec Loss 3.2947 Epoch: 13 Global Step: 139495 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:42:35,763-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-13 15:42:58,952-Speed 41.04 samples/sec Loss 2.9956 Epoch: 14 Global Step: 139500 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:43:18,885-Speed 128.44 samples/sec Loss 2.6981 Epoch: 14 Global Step: 139505 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:43:38,711-Speed 129.13 samples/sec Loss 2.7495 Epoch: 14 Global Step: 139510 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:43:58,509-Speed 129.31 samples/sec Loss 2.9049 Epoch: 14 Global Step: 139515 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:44:18,090-Speed 130.75 samples/sec Loss 2.7372 Epoch: 14 Global Step: 139520 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:44:37,929-Speed 129.04 samples/sec Loss 2.7596 Epoch: 14 Global Step: 139525 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:44:57,524-Speed 130.66 samples/sec Loss 2.9394 Epoch: 14 Global Step: 139530 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:45:17,134-Speed 130.55 samples/sec Loss 2.5760 Epoch: 14 Global Step: 139535 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:45:36,849-Speed 129.86 samples/sec Loss 2.8039 Epoch: 14 Global Step: 139540 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:45:56,392-Speed 131.00 samples/sec Loss 2.5807 Epoch: 14 Global Step: 139545 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:46:16,147-Speed 129.60 samples/sec Loss 2.6727 Epoch: 14 Global Step: 139550 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:46:36,066-Speed 128.53 samples/sec Loss 2.6599 Epoch: 14 Global Step: 139555 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:46:55,859-Speed 129.35 samples/sec Loss 2.7826 Epoch: 14 Global Step: 139560 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:47:15,312-Speed 131.60 samples/sec Loss 2.8519 Epoch: 14 Global Step: 139565 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:47:35,172-Speed 128.91 samples/sec Loss 3.1201 Epoch: 14 Global Step: 139570 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:47:55,072-Speed 128.65 samples/sec Loss 2.8919 Epoch: 14 Global Step: 139575 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:48:14,717-Speed 130.32 samples/sec Loss 2.8287 Epoch: 14 Global Step: 139580 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:48:34,351-Speed 130.39 samples/sec Loss 2.6596 Epoch: 14 Global Step: 139585 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:48:53,965-Speed 130.53 samples/sec Loss 3.0467 Epoch: 14 Global Step: 139590 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:49:13,611-Speed 130.31 samples/sec Loss 2.9054 Epoch: 14 Global Step: 139595 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:49:33,449-Speed 129.05 samples/sec Loss 2.7796 Epoch: 14 Global Step: 139600 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:49:53,179-Speed 129.76 samples/sec Loss 2.7051 Epoch: 14 Global Step: 139605 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:50:12,807-Speed 130.44 samples/sec Loss 2.7797 Epoch: 14 Global Step: 139610 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:50:32,558-Speed 129.62 samples/sec Loss 2.9860 Epoch: 14 Global Step: 139615 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:50:52,535-Speed 128.16 samples/sec Loss 2.7154 Epoch: 14 Global Step: 139620 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:51:12,063-Speed 131.10 samples/sec Loss 2.6704 Epoch: 14 Global Step: 139625 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:51:31,945-Speed 128.77 samples/sec Loss 3.0050 Epoch: 14 Global Step: 139630 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:51:51,546-Speed 130.61 samples/sec Loss 2.7586 Epoch: 14 Global Step: 139635 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:52:11,259-Speed 129.87 samples/sec Loss 2.7580 Epoch: 14 Global Step: 139640 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:52:31,158-Speed 128.66 samples/sec Loss 2.5500 Epoch: 14 Global Step: 139645 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:52:50,836-Speed 130.10 samples/sec Loss 2.7782 Epoch: 14 Global Step: 139650 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:53:10,441-Speed 130.58 samples/sec Loss 2.7642 Epoch: 14 Global Step: 139655 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:53:30,291-Speed 128.97 samples/sec Loss 2.8255 Epoch: 14 Global Step: 139660 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:53:50,199-Speed 128.60 samples/sec Loss 3.0560 Epoch: 14 Global Step: 139665 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:54:09,723-Speed 131.13 samples/sec Loss 2.8878 Epoch: 14 Global Step: 139670 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:54:29,612-Speed 128.72 samples/sec Loss 2.6795 Epoch: 14 Global Step: 139675 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:54:49,119-Speed 131.24 samples/sec Loss 2.8924 Epoch: 14 Global Step: 139680 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:55:08,932-Speed 129.22 samples/sec Loss 2.7281 Epoch: 14 Global Step: 139685 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:55:28,692-Speed 129.56 samples/sec Loss 2.5891 Epoch: 14 Global Step: 139690 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:55:48,532-Speed 129.04 samples/sec Loss 2.7931 Epoch: 14 Global Step: 139695 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:56:07,951-Speed 131.83 samples/sec Loss 2.8058 Epoch: 14 Global Step: 139700 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:56:27,862-Speed 128.58 samples/sec Loss 2.7513 Epoch: 14 Global Step: 139705 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:56:47,578-Speed 129.85 samples/sec Loss 2.8792 Epoch: 14 Global Step: 139710 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:57:07,126-Speed 130.96 samples/sec Loss 2.9113 Epoch: 14 Global Step: 139715 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:57:26,990-Speed 128.89 samples/sec Loss 2.7505 Epoch: 14 Global Step: 139720 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:57:46,548-Speed 130.90 samples/sec Loss 2.8176 Epoch: 14 Global Step: 139725 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:58:06,504-Speed 128.29 samples/sec Loss 2.7051 Epoch: 14 Global Step: 139730 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:58:26,447-Speed 128.37 samples/sec Loss 2.6463 Epoch: 14 Global Step: 139735 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:58:46,278-Speed 129.10 samples/sec Loss 2.9220 Epoch: 14 Global Step: 139740 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:59:05,706-Speed 131.78 samples/sec Loss 2.7127 Epoch: 14 Global Step: 139745 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:59:25,609-Speed 128.63 samples/sec Loss 2.9707 Epoch: 14 Global Step: 139750 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 15:59:45,326-Speed 129.84 samples/sec Loss 2.8491 Epoch: 14 Global Step: 139755 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:00:04,926-Speed 130.62 samples/sec Loss 2.6060 Epoch: 14 Global Step: 139760 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:00:24,638-Speed 129.88 samples/sec Loss 2.8081 Epoch: 14 Global Step: 139765 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:00:44,247-Speed 130.56 samples/sec Loss 2.9119 Epoch: 14 Global Step: 139770 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:01:03,998-Speed 129.62 samples/sec Loss 2.7542 Epoch: 14 Global Step: 139775 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:01:23,827-Speed 129.11 samples/sec Loss 2.8513 Epoch: 14 Global Step: 139780 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:01:43,553-Speed 129.78 samples/sec Loss 2.7827 Epoch: 14 Global Step: 139785 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:02:03,251-Speed 129.97 samples/sec Loss 2.7693 Epoch: 14 Global Step: 139790 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:02:23,061-Speed 129.23 samples/sec Loss 2.7803 Epoch: 14 Global Step: 139795 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:02:42,802-Speed 129.68 samples/sec Loss 2.8017 Epoch: 14 Global Step: 139800 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:03:02,372-Speed 130.82 samples/sec Loss 2.8481 Epoch: 14 Global Step: 139805 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:03:22,159-Speed 129.38 samples/sec Loss 2.6616 Epoch: 14 Global Step: 139810 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:03:41,762-Speed 130.60 samples/sec Loss 2.8361 Epoch: 14 Global Step: 139815 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:04:01,389-Speed 130.44 samples/sec Loss 2.7213 Epoch: 14 Global Step: 139820 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:04:21,193-Speed 129.27 samples/sec Loss 2.6948 Epoch: 14 Global Step: 139825 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:04:40,972-Speed 129.44 samples/sec Loss 2.9907 Epoch: 14 Global Step: 139830 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:05:00,667-Speed 129.99 samples/sec Loss 2.8524 Epoch: 14 Global Step: 139835 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:05:20,395-Speed 129.77 samples/sec Loss 2.9064 Epoch: 14 Global Step: 139840 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:05:40,290-Speed 128.68 samples/sec Loss 2.6558 Epoch: 14 Global Step: 139845 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:05:59,815-Speed 131.12 samples/sec Loss 2.8030 Epoch: 14 Global Step: 139850 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:06:19,599-Speed 129.41 samples/sec Loss 2.7833 Epoch: 14 Global Step: 139855 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:06:39,206-Speed 130.57 samples/sec Loss 2.7807 Epoch: 14 Global Step: 139860 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:06:58,922-Speed 129.85 samples/sec Loss 2.7665 Epoch: 14 Global Step: 139865 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:07:18,711-Speed 129.37 samples/sec Loss 2.9022 Epoch: 14 Global Step: 139870 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:07:38,613-Speed 128.63 samples/sec Loss 2.7751 Epoch: 14 Global Step: 139875 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:07:58,031-Speed 131.85 samples/sec Loss 2.8052 Epoch: 14 Global Step: 139880 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:08:17,752-Speed 129.81 samples/sec Loss 2.8443 Epoch: 14 Global Step: 139885 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:08:37,627-Speed 128.81 samples/sec Loss 3.0694 Epoch: 14 Global Step: 139890 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:08:57,309-Speed 130.07 samples/sec Loss 2.8600 Epoch: 14 Global Step: 139895 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:09:17,213-Speed 128.63 samples/sec Loss 2.8467 Epoch: 14 Global Step: 139900 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:09:36,660-Speed 131.64 samples/sec Loss 3.0163 Epoch: 14 Global Step: 139905 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:09:56,445-Speed 129.40 samples/sec Loss 2.6692 Epoch: 14 Global Step: 139910 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:10:16,307-Speed 128.89 samples/sec Loss 2.8086 Epoch: 14 Global Step: 139915 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:10:36,289-Speed 128.12 samples/sec Loss 2.9453 Epoch: 14 Global Step: 139920 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:10:55,781-Speed 131.34 samples/sec Loss 2.8016 Epoch: 14 Global Step: 139925 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:11:15,502-Speed 129.81 samples/sec Loss 2.9057 Epoch: 14 Global Step: 139930 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:11:35,553-Speed 127.69 samples/sec Loss 2.7243 Epoch: 14 Global Step: 139935 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:11:55,108-Speed 130.92 samples/sec Loss 2.8167 Epoch: 14 Global Step: 139940 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:12:15,077-Speed 128.21 samples/sec Loss 2.7612 Epoch: 14 Global Step: 139945 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:12:34,635-Speed 130.90 samples/sec Loss 2.9865 Epoch: 14 Global Step: 139950 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:12:54,541-Speed 128.61 samples/sec Loss 2.8790 Epoch: 14 Global Step: 139955 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:13:14,500-Speed 128.27 samples/sec Loss 2.7886 Epoch: 14 Global Step: 139960 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:13:34,218-Speed 129.84 samples/sec Loss 2.7521 Epoch: 14 Global Step: 139965 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:13:53,661-Speed 131.68 samples/sec Loss 2.7320 Epoch: 14 Global Step: 139970 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:14:13,510-Speed 128.98 samples/sec Loss 2.7529 Epoch: 14 Global Step: 139975 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:14:33,307-Speed 129.32 samples/sec Loss 2.9521 Epoch: 14 Global Step: 139980 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:14:52,918-Speed 130.54 samples/sec Loss 2.8734 Epoch: 14 Global Step: 139985 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:15:12,534-Speed 130.51 samples/sec Loss 2.9973 Epoch: 14 Global Step: 139990 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:15:32,072-Speed 131.04 samples/sec Loss 2.6837 Epoch: 14 Global Step: 139995 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:15:51,870-Speed 129.31 samples/sec Loss 3.0121 Epoch: 14 Global Step: 140000 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 16:30:37,175-[lfw][140000]XNorm: 22.434025 Training: 2021-06-13 16:30:37,176-[lfw][140000]Accuracy-Flip: 0.99783+-0.00183 Training: 2021-06-13 16:30:37,176-[lfw][140000]Accuracy-Highest: 0.99833 Training: 2021-06-13 16:47:49,157-[cfp_fp][140000]XNorm: 22.167316 Training: 2021-06-13 16:47:49,157-[cfp_fp][140000]Accuracy-Flip: 0.98971+-0.00298 Training: 2021-06-13 16:47:49,157-[cfp_fp][140000]Accuracy-Highest: 0.99071 Training: 2021-06-13 17:02:37,624-[agedb_30][140000]XNorm: 23.029533 Training: 2021-06-13 17:02:37,625-[agedb_30][140000]Accuracy-Flip: 0.98417+-0.00692 Training: 2021-06-13 17:02:37,625-[agedb_30][140000]Accuracy-Highest: 0.98417 Training: 2021-06-13 17:02:57,279-Speed 0.91 samples/sec Loss 2.7769 Epoch: 14 Global Step: 140005 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:03:17,068-Speed 129.37 samples/sec Loss 3.0436 Epoch: 14 Global Step: 140010 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:03:36,501-Speed 131.74 samples/sec Loss 3.1119 Epoch: 14 Global Step: 140015 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:03:55,998-Speed 131.31 samples/sec Loss 2.9739 Epoch: 14 Global Step: 140020 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:04:15,574-Speed 130.78 samples/sec Loss 2.8859 Epoch: 14 Global Step: 140025 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:04:35,130-Speed 130.91 samples/sec Loss 3.1463 Epoch: 14 Global Step: 140030 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:04:54,743-Speed 130.53 samples/sec Loss 3.0378 Epoch: 14 Global Step: 140035 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:05:14,178-Speed 131.73 samples/sec Loss 2.9191 Epoch: 14 Global Step: 140040 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:05:33,701-Speed 131.14 samples/sec Loss 2.7238 Epoch: 14 Global Step: 140045 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:05:53,221-Speed 131.15 samples/sec Loss 3.0299 Epoch: 14 Global Step: 140050 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:06:13,135-Speed 128.56 samples/sec Loss 3.0414 Epoch: 14 Global Step: 140055 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:06:32,561-Speed 131.79 samples/sec Loss 2.9423 Epoch: 14 Global Step: 140060 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:06:52,165-Speed 130.59 samples/sec Loss 2.9276 Epoch: 14 Global Step: 140065 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:07:11,692-Speed 131.11 samples/sec Loss 3.0251 Epoch: 14 Global Step: 140070 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:07:31,293-Speed 130.61 samples/sec Loss 2.9092 Epoch: 14 Global Step: 140075 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:07:50,859-Speed 130.85 samples/sec Loss 2.9689 Epoch: 14 Global Step: 140080 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:08:10,310-Speed 131.62 samples/sec Loss 2.7684 Epoch: 14 Global Step: 140085 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:08:29,909-Speed 130.62 samples/sec Loss 3.0063 Epoch: 14 Global Step: 140090 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:08:49,464-Speed 130.92 samples/sec Loss 3.1038 Epoch: 14 Global Step: 140095 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:09:09,293-Speed 129.11 samples/sec Loss 3.0051 Epoch: 14 Global Step: 140100 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:09:28,750-Speed 131.58 samples/sec Loss 2.8718 Epoch: 14 Global Step: 140105 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:09:48,463-Speed 129.87 samples/sec Loss 2.9898 Epoch: 14 Global Step: 140110 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:10:07,999-Speed 131.05 samples/sec Loss 2.6602 Epoch: 14 Global Step: 140115 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:10:27,544-Speed 130.99 samples/sec Loss 2.7976 Epoch: 14 Global Step: 140120 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:10:47,077-Speed 131.07 samples/sec Loss 2.8019 Epoch: 14 Global Step: 140125 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:11:06,508-Speed 131.76 samples/sec Loss 2.9802 Epoch: 14 Global Step: 140130 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:11:26,107-Speed 130.63 samples/sec Loss 2.8708 Epoch: 14 Global Step: 140135 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:11:45,633-Speed 131.11 samples/sec Loss 3.0670 Epoch: 14 Global Step: 140140 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:12:05,432-Speed 129.31 samples/sec Loss 2.8366 Epoch: 14 Global Step: 140145 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:12:24,881-Speed 131.63 samples/sec Loss 2.8687 Epoch: 14 Global Step: 140150 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:12:44,558-Speed 130.11 samples/sec Loss 3.0623 Epoch: 14 Global Step: 140155 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:13:04,096-Speed 131.03 samples/sec Loss 2.8240 Epoch: 14 Global Step: 140160 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:13:23,698-Speed 130.61 samples/sec Loss 2.8334 Epoch: 14 Global Step: 140165 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:13:43,137-Speed 131.70 samples/sec Loss 2.9689 Epoch: 14 Global Step: 140170 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:14:02,744-Speed 130.57 samples/sec Loss 2.5935 Epoch: 14 Global Step: 140175 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:14:22,368-Speed 130.46 samples/sec Loss 2.9443 Epoch: 14 Global Step: 140180 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:14:41,816-Speed 131.64 samples/sec Loss 2.9352 Epoch: 14 Global Step: 140185 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:15:01,596-Speed 129.43 samples/sec Loss 2.9925 Epoch: 14 Global Step: 140190 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:15:21,171-Speed 130.79 samples/sec Loss 2.9061 Epoch: 14 Global Step: 140195 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:15:40,889-Speed 129.84 samples/sec Loss 2.8594 Epoch: 14 Global Step: 140200 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:16:00,441-Speed 130.94 samples/sec Loss 2.8928 Epoch: 14 Global Step: 140205 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:16:20,014-Speed 130.80 samples/sec Loss 2.8909 Epoch: 14 Global Step: 140210 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:16:39,517-Speed 131.27 samples/sec Loss 2.8833 Epoch: 14 Global Step: 140215 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:16:59,221-Speed 129.93 samples/sec Loss 3.0388 Epoch: 14 Global Step: 140220 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:17:18,785-Speed 130.86 samples/sec Loss 2.8649 Epoch: 14 Global Step: 140225 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:17:38,344-Speed 130.89 samples/sec Loss 3.1374 Epoch: 14 Global Step: 140230 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:17:58,135-Speed 129.36 samples/sec Loss 3.0073 Epoch: 14 Global Step: 140235 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:18:17,748-Speed 130.53 samples/sec Loss 3.0113 Epoch: 14 Global Step: 140240 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:18:37,213-Speed 131.53 samples/sec Loss 2.9584 Epoch: 14 Global Step: 140245 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:18:56,781-Speed 130.83 samples/sec Loss 3.0070 Epoch: 14 Global Step: 140250 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:19:16,373-Speed 130.68 samples/sec Loss 2.8925 Epoch: 14 Global Step: 140255 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:19:35,809-Speed 131.72 samples/sec Loss 2.9770 Epoch: 14 Global Step: 140260 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:19:55,415-Speed 130.58 samples/sec Loss 3.0613 Epoch: 14 Global Step: 140265 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:20:15,080-Speed 130.19 samples/sec Loss 2.8950 Epoch: 14 Global Step: 140270 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:20:34,675-Speed 130.65 samples/sec Loss 2.8729 Epoch: 14 Global Step: 140275 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:20:54,336-Speed 130.21 samples/sec Loss 2.9604 Epoch: 14 Global Step: 140280 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:21:14,092-Speed 129.59 samples/sec Loss 2.8951 Epoch: 14 Global Step: 140285 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:21:33,682-Speed 130.69 samples/sec Loss 2.8676 Epoch: 14 Global Step: 140290 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:21:53,293-Speed 130.55 samples/sec Loss 2.8788 Epoch: 14 Global Step: 140295 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:22:12,728-Speed 131.72 samples/sec Loss 3.0732 Epoch: 14 Global Step: 140300 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:22:32,332-Speed 130.60 samples/sec Loss 2.8646 Epoch: 14 Global Step: 140305 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:22:51,914-Speed 130.74 samples/sec Loss 3.0940 Epoch: 14 Global Step: 140310 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:23:11,476-Speed 130.88 samples/sec Loss 2.8610 Epoch: 14 Global Step: 140315 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:23:31,039-Speed 130.87 samples/sec Loss 3.0148 Epoch: 14 Global Step: 140320 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:23:50,844-Speed 129.27 samples/sec Loss 3.0814 Epoch: 14 Global Step: 140325 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:24:10,274-Speed 131.76 samples/sec Loss 3.0910 Epoch: 14 Global Step: 140330 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:24:30,166-Speed 128.70 samples/sec Loss 2.8331 Epoch: 14 Global Step: 140335 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:24:49,704-Speed 131.03 samples/sec Loss 2.9809 Epoch: 14 Global Step: 140340 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:25:09,249-Speed 130.99 samples/sec Loss 2.9652 Epoch: 14 Global Step: 140345 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:25:28,863-Speed 130.52 samples/sec Loss 2.8849 Epoch: 14 Global Step: 140350 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:25:48,585-Speed 129.81 samples/sec Loss 3.0194 Epoch: 14 Global Step: 140355 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:26:08,189-Speed 130.60 samples/sec Loss 2.9948 Epoch: 14 Global Step: 140360 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:26:27,615-Speed 131.79 samples/sec Loss 2.9456 Epoch: 14 Global Step: 140365 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:26:47,494-Speed 128.79 samples/sec Loss 2.9588 Epoch: 14 Global Step: 140370 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:27:06,998-Speed 131.26 samples/sec Loss 3.1273 Epoch: 14 Global Step: 140375 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:27:26,643-Speed 130.32 samples/sec Loss 2.9552 Epoch: 14 Global Step: 140380 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:27:46,074-Speed 131.76 samples/sec Loss 3.0514 Epoch: 14 Global Step: 140385 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:28:05,700-Speed 130.45 samples/sec Loss 2.9741 Epoch: 14 Global Step: 140390 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:28:25,284-Speed 130.72 samples/sec Loss 2.9541 Epoch: 14 Global Step: 140395 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:28:44,830-Speed 130.98 samples/sec Loss 2.8877 Epoch: 14 Global Step: 140400 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:29:04,381-Speed 130.95 samples/sec Loss 3.0254 Epoch: 14 Global Step: 140405 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:29:23,850-Speed 131.50 samples/sec Loss 3.0121 Epoch: 14 Global Step: 140410 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:29:43,621-Speed 129.49 samples/sec Loss 2.9435 Epoch: 14 Global Step: 140415 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:30:03,328-Speed 129.91 samples/sec Loss 2.9263 Epoch: 14 Global Step: 140420 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:30:22,767-Speed 131.70 samples/sec Loss 3.0215 Epoch: 14 Global Step: 140425 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:30:42,305-Speed 131.03 samples/sec Loss 3.1588 Epoch: 14 Global Step: 140430 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:31:01,929-Speed 130.46 samples/sec Loss 2.9095 Epoch: 14 Global Step: 140435 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:31:21,456-Speed 131.10 samples/sec Loss 2.9449 Epoch: 14 Global Step: 140440 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:31:41,058-Speed 130.61 samples/sec Loss 2.9337 Epoch: 14 Global Step: 140445 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:32:00,506-Speed 131.64 samples/sec Loss 3.0463 Epoch: 14 Global Step: 140450 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:32:20,121-Speed 130.52 samples/sec Loss 2.8767 Epoch: 14 Global Step: 140455 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:32:40,049-Speed 128.47 samples/sec Loss 3.1165 Epoch: 14 Global Step: 140460 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:32:59,698-Speed 130.29 samples/sec Loss 3.1498 Epoch: 14 Global Step: 140465 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:33:19,134-Speed 131.72 samples/sec Loss 2.9288 Epoch: 14 Global Step: 140470 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:33:38,757-Speed 130.47 samples/sec Loss 3.0107 Epoch: 14 Global Step: 140475 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:33:58,355-Speed 130.63 samples/sec Loss 2.8946 Epoch: 14 Global Step: 140480 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:34:17,961-Speed 130.58 samples/sec Loss 2.8437 Epoch: 14 Global Step: 140485 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:34:37,586-Speed 130.45 samples/sec Loss 2.9525 Epoch: 14 Global Step: 140490 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:34:57,011-Speed 131.80 samples/sec Loss 2.7739 Epoch: 14 Global Step: 140495 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:35:16,618-Speed 130.57 samples/sec Loss 2.8864 Epoch: 14 Global Step: 140500 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:35:36,356-Speed 129.71 samples/sec Loss 2.8960 Epoch: 14 Global Step: 140505 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:35:56,000-Speed 130.33 samples/sec Loss 3.2514 Epoch: 14 Global Step: 140510 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:36:15,454-Speed 131.60 samples/sec Loss 3.0353 Epoch: 14 Global Step: 140515 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:36:35,242-Speed 129.38 samples/sec Loss 2.9270 Epoch: 14 Global Step: 140520 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:36:54,696-Speed 131.60 samples/sec Loss 2.9984 Epoch: 14 Global Step: 140525 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:37:14,304-Speed 130.57 samples/sec Loss 2.8925 Epoch: 14 Global Step: 140530 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:37:33,724-Speed 131.83 samples/sec Loss 3.0453 Epoch: 14 Global Step: 140535 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:37:53,256-Speed 131.07 samples/sec Loss 3.2259 Epoch: 14 Global Step: 140540 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:38:12,964-Speed 129.91 samples/sec Loss 3.1055 Epoch: 14 Global Step: 140545 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:38:32,861-Speed 128.67 samples/sec Loss 3.1036 Epoch: 14 Global Step: 140550 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:38:52,280-Speed 131.83 samples/sec Loss 3.0401 Epoch: 14 Global Step: 140555 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:39:11,934-Speed 130.26 samples/sec Loss 3.0109 Epoch: 14 Global Step: 140560 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:39:31,474-Speed 131.02 samples/sec Loss 2.9765 Epoch: 14 Global Step: 140565 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:39:51,137-Speed 130.20 samples/sec Loss 3.1304 Epoch: 14 Global Step: 140570 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:40:10,915-Speed 129.45 samples/sec Loss 3.2225 Epoch: 14 Global Step: 140575 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:40:30,452-Speed 131.04 samples/sec Loss 3.1251 Epoch: 14 Global Step: 140580 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:40:50,063-Speed 130.55 samples/sec Loss 3.0042 Epoch: 14 Global Step: 140585 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:41:09,549-Speed 131.38 samples/sec Loss 3.0063 Epoch: 14 Global Step: 140590 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:41:29,441-Speed 128.71 samples/sec Loss 3.1303 Epoch: 14 Global Step: 140595 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:41:48,870-Speed 131.77 samples/sec Loss 2.9566 Epoch: 14 Global Step: 140600 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:42:08,582-Speed 129.87 samples/sec Loss 3.0901 Epoch: 14 Global Step: 140605 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:42:28,203-Speed 130.48 samples/sec Loss 3.2213 Epoch: 14 Global Step: 140610 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:42:47,699-Speed 131.31 samples/sec Loss 2.9516 Epoch: 14 Global Step: 140615 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:43:07,295-Speed 130.65 samples/sec Loss 2.9962 Epoch: 14 Global Step: 140620 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:43:26,959-Speed 130.19 samples/sec Loss 2.8140 Epoch: 14 Global Step: 140625 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:43:46,568-Speed 130.56 samples/sec Loss 2.9420 Epoch: 14 Global Step: 140630 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:44:06,100-Speed 131.08 samples/sec Loss 3.2106 Epoch: 14 Global Step: 140635 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:44:25,915-Speed 129.20 samples/sec Loss 3.1661 Epoch: 14 Global Step: 140640 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:44:45,336-Speed 131.82 samples/sec Loss 3.0421 Epoch: 14 Global Step: 140645 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:45:05,118-Speed 129.42 samples/sec Loss 2.8803 Epoch: 14 Global Step: 140650 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:45:24,703-Speed 130.72 samples/sec Loss 3.1089 Epoch: 14 Global Step: 140655 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:45:44,296-Speed 130.67 samples/sec Loss 2.9359 Epoch: 14 Global Step: 140660 Fp16 Grad Scale: 8192 Required: 129 hours Training: 2021-06-13 17:46:03,823-Speed 131.11 samples/sec Loss 3.0821 Epoch: 14 Global Step: 140665 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:46:23,439-Speed 130.51 samples/sec Loss 3.0242 Epoch: 14 Global Step: 140670 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:46:43,018-Speed 130.76 samples/sec Loss 2.9893 Epoch: 14 Global Step: 140675 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:47:02,769-Speed 129.62 samples/sec Loss 3.0565 Epoch: 14 Global Step: 140680 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:47:22,559-Speed 129.36 samples/sec Loss 3.0129 Epoch: 14 Global Step: 140685 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:47:42,281-Speed 129.81 samples/sec Loss 2.9939 Epoch: 14 Global Step: 140690 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:48:01,796-Speed 131.18 samples/sec Loss 3.0128 Epoch: 14 Global Step: 140695 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:48:21,354-Speed 130.90 samples/sec Loss 2.7656 Epoch: 14 Global Step: 140700 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:48:40,938-Speed 130.73 samples/sec Loss 3.0830 Epoch: 14 Global Step: 140705 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:49:00,533-Speed 130.66 samples/sec Loss 3.1387 Epoch: 14 Global Step: 140710 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:49:20,157-Speed 130.46 samples/sec Loss 2.9970 Epoch: 14 Global Step: 140715 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:49:39,721-Speed 130.86 samples/sec Loss 3.0626 Epoch: 14 Global Step: 140720 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:49:59,363-Speed 130.34 samples/sec Loss 3.0262 Epoch: 14 Global Step: 140725 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:50:19,025-Speed 130.21 samples/sec Loss 3.1879 Epoch: 14 Global Step: 140730 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:50:38,727-Speed 129.94 samples/sec Loss 3.0632 Epoch: 14 Global Step: 140735 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:50:58,281-Speed 130.93 samples/sec Loss 3.1387 Epoch: 14 Global Step: 140740 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:51:17,958-Speed 130.11 samples/sec Loss 2.9483 Epoch: 14 Global Step: 140745 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:51:37,727-Speed 129.50 samples/sec Loss 3.2000 Epoch: 14 Global Step: 140750 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:51:57,362-Speed 130.38 samples/sec Loss 2.7817 Epoch: 14 Global Step: 140755 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:52:16,933-Speed 130.81 samples/sec Loss 3.1890 Epoch: 14 Global Step: 140760 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:52:36,471-Speed 131.03 samples/sec Loss 3.1566 Epoch: 14 Global Step: 140765 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:52:56,098-Speed 130.45 samples/sec Loss 3.1107 Epoch: 14 Global Step: 140770 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:53:15,922-Speed 129.14 samples/sec Loss 2.8917 Epoch: 14 Global Step: 140775 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:53:35,510-Speed 130.70 samples/sec Loss 3.1885 Epoch: 14 Global Step: 140780 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:53:55,059-Speed 130.96 samples/sec Loss 3.0634 Epoch: 14 Global Step: 140785 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:54:14,619-Speed 130.89 samples/sec Loss 3.1029 Epoch: 14 Global Step: 140790 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:54:34,171-Speed 130.94 samples/sec Loss 3.0118 Epoch: 14 Global Step: 140795 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:54:53,778-Speed 130.58 samples/sec Loss 3.1241 Epoch: 14 Global Step: 140800 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:55:13,330-Speed 130.94 samples/sec Loss 3.0524 Epoch: 14 Global Step: 140805 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:55:33,064-Speed 129.74 samples/sec Loss 3.0016 Epoch: 14 Global Step: 140810 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:55:52,622-Speed 130.90 samples/sec Loss 2.9460 Epoch: 14 Global Step: 140815 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:56:12,598-Speed 128.16 samples/sec Loss 2.9894 Epoch: 14 Global Step: 140820 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:56:32,038-Speed 131.69 samples/sec Loss 3.2189 Epoch: 14 Global Step: 140825 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:56:51,747-Speed 129.89 samples/sec Loss 2.9618 Epoch: 14 Global Step: 140830 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:57:11,284-Speed 131.04 samples/sec Loss 3.1594 Epoch: 14 Global Step: 140835 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:57:30,868-Speed 130.72 samples/sec Loss 2.9961 Epoch: 14 Global Step: 140840 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:57:50,376-Speed 131.24 samples/sec Loss 3.1513 Epoch: 14 Global Step: 140845 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:58:09,920-Speed 130.99 samples/sec Loss 2.9510 Epoch: 14 Global Step: 140850 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:58:29,516-Speed 130.65 samples/sec Loss 2.9533 Epoch: 14 Global Step: 140855 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:58:49,113-Speed 130.64 samples/sec Loss 3.1558 Epoch: 14 Global Step: 140860 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:59:08,916-Speed 129.28 samples/sec Loss 3.0027 Epoch: 14 Global Step: 140865 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:59:28,428-Speed 131.21 samples/sec Loss 2.9590 Epoch: 14 Global Step: 140870 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 17:59:48,043-Speed 130.52 samples/sec Loss 2.8658 Epoch: 14 Global Step: 140875 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:00:07,656-Speed 130.53 samples/sec Loss 3.0007 Epoch: 14 Global Step: 140880 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:00:27,233-Speed 130.77 samples/sec Loss 3.0188 Epoch: 14 Global Step: 140885 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:00:46,796-Speed 130.86 samples/sec Loss 3.0172 Epoch: 14 Global Step: 140890 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:01:06,413-Speed 130.51 samples/sec Loss 2.8867 Epoch: 14 Global Step: 140895 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:01:25,959-Speed 130.98 samples/sec Loss 3.0084 Epoch: 14 Global Step: 140900 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:01:45,569-Speed 130.55 samples/sec Loss 3.0909 Epoch: 14 Global Step: 140905 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:02:05,250-Speed 130.08 samples/sec Loss 2.9557 Epoch: 14 Global Step: 140910 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:02:24,904-Speed 130.26 samples/sec Loss 3.0306 Epoch: 14 Global Step: 140915 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:02:44,444-Speed 131.02 samples/sec Loss 3.1566 Epoch: 14 Global Step: 140920 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:03:04,060-Speed 130.51 samples/sec Loss 2.9690 Epoch: 14 Global Step: 140925 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:03:23,488-Speed 131.78 samples/sec Loss 3.3381 Epoch: 14 Global Step: 140930 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:03:43,045-Speed 130.91 samples/sec Loss 3.0327 Epoch: 14 Global Step: 140935 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:04:02,619-Speed 130.79 samples/sec Loss 3.0260 Epoch: 14 Global Step: 140940 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:04:22,231-Speed 130.54 samples/sec Loss 3.1618 Epoch: 14 Global Step: 140945 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:04:41,839-Speed 130.56 samples/sec Loss 3.1280 Epoch: 14 Global Step: 140950 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:05:01,516-Speed 130.11 samples/sec Loss 2.8806 Epoch: 14 Global Step: 140955 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:05:21,175-Speed 130.23 samples/sec Loss 2.8194 Epoch: 14 Global Step: 140960 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:05:40,798-Speed 130.47 samples/sec Loss 2.9862 Epoch: 14 Global Step: 140965 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:06:00,411-Speed 130.53 samples/sec Loss 3.0985 Epoch: 14 Global Step: 140970 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:06:19,964-Speed 130.94 samples/sec Loss 3.1046 Epoch: 14 Global Step: 140975 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:06:39,492-Speed 131.10 samples/sec Loss 3.1405 Epoch: 14 Global Step: 140980 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:06:59,053-Speed 130.88 samples/sec Loss 2.8554 Epoch: 14 Global Step: 140985 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:07:18,673-Speed 130.49 samples/sec Loss 3.0663 Epoch: 14 Global Step: 140990 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:07:38,283-Speed 130.55 samples/sec Loss 2.9779 Epoch: 14 Global Step: 140995 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:07:58,030-Speed 129.65 samples/sec Loss 3.2515 Epoch: 14 Global Step: 141000 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:08:17,764-Speed 129.73 samples/sec Loss 2.9972 Epoch: 14 Global Step: 141005 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:08:37,343-Speed 130.76 samples/sec Loss 3.0107 Epoch: 14 Global Step: 141010 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:08:56,923-Speed 130.75 samples/sec Loss 3.0386 Epoch: 14 Global Step: 141015 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:09:16,394-Speed 131.48 samples/sec Loss 3.2805 Epoch: 14 Global Step: 141020 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:09:35,933-Speed 131.03 samples/sec Loss 3.0532 Epoch: 14 Global Step: 141025 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:09:55,530-Speed 130.64 samples/sec Loss 2.9314 Epoch: 14 Global Step: 141030 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:10:15,257-Speed 129.78 samples/sec Loss 3.0442 Epoch: 14 Global Step: 141035 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:10:34,873-Speed 130.52 samples/sec Loss 3.0058 Epoch: 14 Global Step: 141040 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:10:54,530-Speed 130.24 samples/sec Loss 3.0690 Epoch: 14 Global Step: 141045 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:11:14,193-Speed 130.20 samples/sec Loss 3.0198 Epoch: 14 Global Step: 141050 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:11:33,728-Speed 131.06 samples/sec Loss 3.0814 Epoch: 14 Global Step: 141055 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:11:53,350-Speed 130.47 samples/sec Loss 3.0417 Epoch: 14 Global Step: 141060 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:12:12,810-Speed 131.56 samples/sec Loss 3.0465 Epoch: 14 Global Step: 141065 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:12:32,422-Speed 130.54 samples/sec Loss 3.2379 Epoch: 14 Global Step: 141070 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:12:52,022-Speed 130.62 samples/sec Loss 3.2239 Epoch: 14 Global Step: 141075 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:13:11,510-Speed 131.37 samples/sec Loss 3.2326 Epoch: 14 Global Step: 141080 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:13:31,085-Speed 130.78 samples/sec Loss 3.1170 Epoch: 14 Global Step: 141085 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:13:51,008-Speed 128.50 samples/sec Loss 2.9977 Epoch: 14 Global Step: 141090 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:14:10,452-Speed 131.67 samples/sec Loss 3.1658 Epoch: 14 Global Step: 141095 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:14:30,096-Speed 130.33 samples/sec Loss 3.1213 Epoch: 14 Global Step: 141100 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:14:49,678-Speed 130.74 samples/sec Loss 3.1677 Epoch: 14 Global Step: 141105 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:15:09,230-Speed 130.94 samples/sec Loss 3.0722 Epoch: 14 Global Step: 141110 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:15:28,891-Speed 130.21 samples/sec Loss 3.2153 Epoch: 14 Global Step: 141115 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:15:48,513-Speed 130.48 samples/sec Loss 3.1661 Epoch: 14 Global Step: 141120 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:16:08,038-Speed 131.12 samples/sec Loss 3.1027 Epoch: 14 Global Step: 141125 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:16:27,599-Speed 130.89 samples/sec Loss 2.9091 Epoch: 14 Global Step: 141130 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:16:47,526-Speed 128.47 samples/sec Loss 3.2970 Epoch: 14 Global Step: 141135 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:17:06,963-Speed 131.72 samples/sec Loss 3.1265 Epoch: 14 Global Step: 141140 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:17:26,685-Speed 129.81 samples/sec Loss 3.1303 Epoch: 14 Global Step: 141145 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:17:46,313-Speed 130.43 samples/sec Loss 3.1760 Epoch: 14 Global Step: 141150 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:18:05,847-Speed 131.06 samples/sec Loss 2.9645 Epoch: 14 Global Step: 141155 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:18:25,447-Speed 130.66 samples/sec Loss 3.0377 Epoch: 14 Global Step: 141160 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:18:44,901-Speed 131.60 samples/sec Loss 3.0902 Epoch: 14 Global Step: 141165 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:19:04,693-Speed 129.35 samples/sec Loss 3.0606 Epoch: 14 Global Step: 141170 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:19:24,243-Speed 130.96 samples/sec Loss 3.1335 Epoch: 14 Global Step: 141175 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:19:44,120-Speed 128.80 samples/sec Loss 3.2032 Epoch: 14 Global Step: 141180 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:20:03,579-Speed 131.57 samples/sec Loss 3.1448 Epoch: 14 Global Step: 141185 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:20:23,226-Speed 130.31 samples/sec Loss 3.0207 Epoch: 14 Global Step: 141190 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:20:42,700-Speed 131.47 samples/sec Loss 2.9380 Epoch: 14 Global Step: 141195 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:21:02,244-Speed 131.00 samples/sec Loss 3.0606 Epoch: 14 Global Step: 141200 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:21:21,859-Speed 130.52 samples/sec Loss 3.2655 Epoch: 14 Global Step: 141205 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:21:41,323-Speed 131.53 samples/sec Loss 3.2006 Epoch: 14 Global Step: 141210 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:22:00,936-Speed 130.54 samples/sec Loss 3.0857 Epoch: 14 Global Step: 141215 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:22:20,483-Speed 130.97 samples/sec Loss 3.0645 Epoch: 14 Global Step: 141220 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:22:40,347-Speed 128.89 samples/sec Loss 3.2815 Epoch: 14 Global Step: 141225 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:22:59,805-Speed 131.58 samples/sec Loss 3.1685 Epoch: 14 Global Step: 141230 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:23:19,526-Speed 129.81 samples/sec Loss 3.1774 Epoch: 14 Global Step: 141235 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:23:39,070-Speed 130.99 samples/sec Loss 3.0727 Epoch: 14 Global Step: 141240 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:23:58,686-Speed 130.51 samples/sec Loss 3.1684 Epoch: 14 Global Step: 141245 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:24:18,235-Speed 130.96 samples/sec Loss 2.9807 Epoch: 14 Global Step: 141250 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:24:37,774-Speed 131.02 samples/sec Loss 3.0511 Epoch: 14 Global Step: 141255 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:24:57,337-Speed 130.87 samples/sec Loss 2.9898 Epoch: 14 Global Step: 141260 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:25:16,850-Speed 131.20 samples/sec Loss 3.1532 Epoch: 14 Global Step: 141265 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:25:36,652-Speed 129.29 samples/sec Loss 3.0678 Epoch: 14 Global Step: 141270 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:25:56,102-Speed 131.63 samples/sec Loss 3.1252 Epoch: 14 Global Step: 141275 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:26:15,763-Speed 130.21 samples/sec Loss 3.1443 Epoch: 14 Global Step: 141280 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:26:35,310-Speed 130.97 samples/sec Loss 3.0980 Epoch: 14 Global Step: 141285 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:26:54,909-Speed 130.63 samples/sec Loss 2.9665 Epoch: 14 Global Step: 141290 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:27:14,443-Speed 131.06 samples/sec Loss 3.1436 Epoch: 14 Global Step: 141295 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:27:34,002-Speed 130.89 samples/sec Loss 2.9071 Epoch: 14 Global Step: 141300 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:27:53,538-Speed 131.05 samples/sec Loss 3.1785 Epoch: 14 Global Step: 141305 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:28:13,151-Speed 130.53 samples/sec Loss 3.3071 Epoch: 14 Global Step: 141310 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:28:32,990-Speed 129.05 samples/sec Loss 3.1819 Epoch: 14 Global Step: 141315 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:28:52,710-Speed 129.82 samples/sec Loss 3.1691 Epoch: 14 Global Step: 141320 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:29:12,235-Speed 131.13 samples/sec Loss 3.3267 Epoch: 14 Global Step: 141325 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:29:31,847-Speed 130.54 samples/sec Loss 3.1836 Epoch: 14 Global Step: 141330 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:29:51,639-Speed 129.36 samples/sec Loss 2.9623 Epoch: 14 Global Step: 141335 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:30:11,199-Speed 130.88 samples/sec Loss 3.0988 Epoch: 14 Global Step: 141340 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:30:30,793-Speed 130.66 samples/sec Loss 3.0601 Epoch: 14 Global Step: 141345 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:30:50,343-Speed 130.95 samples/sec Loss 3.0744 Epoch: 14 Global Step: 141350 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:31:09,952-Speed 130.56 samples/sec Loss 2.9975 Epoch: 14 Global Step: 141355 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:31:29,909-Speed 128.28 samples/sec Loss 3.2790 Epoch: 14 Global Step: 141360 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:31:49,406-Speed 131.31 samples/sec Loss 2.9024 Epoch: 14 Global Step: 141365 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:32:09,176-Speed 129.49 samples/sec Loss 3.1431 Epoch: 14 Global Step: 141370 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:32:28,788-Speed 130.54 samples/sec Loss 3.2423 Epoch: 14 Global Step: 141375 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:32:48,324-Speed 131.05 samples/sec Loss 3.2746 Epoch: 14 Global Step: 141380 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:33:07,911-Speed 130.71 samples/sec Loss 3.2445 Epoch: 14 Global Step: 141385 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:33:27,472-Speed 130.88 samples/sec Loss 3.2041 Epoch: 14 Global Step: 141390 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:33:47,078-Speed 130.58 samples/sec Loss 3.1370 Epoch: 14 Global Step: 141395 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:34:06,571-Speed 131.33 samples/sec Loss 3.0443 Epoch: 14 Global Step: 141400 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:34:26,416-Speed 129.01 samples/sec Loss 3.0920 Epoch: 14 Global Step: 141405 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:34:45,847-Speed 131.75 samples/sec Loss 2.9793 Epoch: 14 Global Step: 141410 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:35:05,613-Speed 129.52 samples/sec Loss 3.0781 Epoch: 14 Global Step: 141415 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:35:25,050-Speed 131.72 samples/sec Loss 3.2034 Epoch: 14 Global Step: 141420 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:35:45,346-Speed 126.14 samples/sec Loss 3.0959 Epoch: 14 Global Step: 141425 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:36:04,865-Speed 131.17 samples/sec Loss 3.0520 Epoch: 14 Global Step: 141430 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:36:24,414-Speed 130.96 samples/sec Loss 3.1347 Epoch: 14 Global Step: 141435 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:36:44,058-Speed 130.33 samples/sec Loss 3.2075 Epoch: 14 Global Step: 141440 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:37:03,612-Speed 130.93 samples/sec Loss 2.9216 Epoch: 14 Global Step: 141445 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:37:23,417-Speed 129.26 samples/sec Loss 3.0437 Epoch: 14 Global Step: 141450 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:37:43,159-Speed 129.68 samples/sec Loss 3.0525 Epoch: 14 Global Step: 141455 Fp16 Grad Scale: 8192 Required: 128 hours Training: 2021-06-13 18:38:02,752-Speed 130.68 samples/sec Loss 3.3383 Epoch: 14 Global Step: 141460 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:38:22,320-Speed 130.84 samples/sec Loss 3.0692 Epoch: 14 Global Step: 141465 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:38:42,109-Speed 129.37 samples/sec Loss 3.0019 Epoch: 14 Global Step: 141470 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:39:01,646-Speed 131.04 samples/sec Loss 3.1337 Epoch: 14 Global Step: 141475 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:39:21,246-Speed 130.62 samples/sec Loss 3.2394 Epoch: 14 Global Step: 141480 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:39:40,867-Speed 130.48 samples/sec Loss 3.2881 Epoch: 14 Global Step: 141485 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:40:00,417-Speed 130.95 samples/sec Loss 2.9891 Epoch: 14 Global Step: 141490 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:40:20,167-Speed 129.63 samples/sec Loss 3.0701 Epoch: 14 Global Step: 141495 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:40:39,884-Speed 129.85 samples/sec Loss 3.1284 Epoch: 14 Global Step: 141500 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:40:59,476-Speed 130.67 samples/sec Loss 3.2637 Epoch: 14 Global Step: 141505 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:41:19,011-Speed 131.06 samples/sec Loss 3.0342 Epoch: 14 Global Step: 141510 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:41:38,571-Speed 130.89 samples/sec Loss 3.2218 Epoch: 14 Global Step: 141515 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:41:58,089-Speed 131.17 samples/sec Loss 3.3776 Epoch: 14 Global Step: 141520 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:42:17,679-Speed 130.68 samples/sec Loss 3.1463 Epoch: 14 Global Step: 141525 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:42:37,214-Speed 131.06 samples/sec Loss 3.0471 Epoch: 14 Global Step: 141530 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:42:57,165-Speed 128.32 samples/sec Loss 3.1204 Epoch: 14 Global Step: 141535 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:43:16,932-Speed 129.52 samples/sec Loss 3.0445 Epoch: 14 Global Step: 141540 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:43:36,649-Speed 129.84 samples/sec Loss 3.1123 Epoch: 14 Global Step: 141545 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:43:56,186-Speed 131.04 samples/sec Loss 3.2852 Epoch: 14 Global Step: 141550 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:44:15,787-Speed 130.61 samples/sec Loss 3.1546 Epoch: 14 Global Step: 141555 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:44:35,337-Speed 130.96 samples/sec Loss 3.1645 Epoch: 14 Global Step: 141560 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:44:54,943-Speed 130.58 samples/sec Loss 3.2469 Epoch: 14 Global Step: 141565 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:45:14,467-Speed 131.12 samples/sec Loss 2.9781 Epoch: 14 Global Step: 141570 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:45:34,016-Speed 130.96 samples/sec Loss 2.9347 Epoch: 14 Global Step: 141575 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:45:53,584-Speed 130.84 samples/sec Loss 3.0616 Epoch: 14 Global Step: 141580 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:46:13,514-Speed 128.46 samples/sec Loss 3.0736 Epoch: 14 Global Step: 141585 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:46:32,934-Speed 131.82 samples/sec Loss 3.1544 Epoch: 14 Global Step: 141590 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:46:52,652-Speed 129.84 samples/sec Loss 3.3791 Epoch: 14 Global Step: 141595 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:47:12,194-Speed 131.01 samples/sec Loss 3.2173 Epoch: 14 Global Step: 141600 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:47:31,643-Speed 131.63 samples/sec Loss 3.3501 Epoch: 14 Global Step: 141605 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:47:51,263-Speed 130.49 samples/sec Loss 3.0202 Epoch: 14 Global Step: 141610 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:48:10,881-Speed 130.50 samples/sec Loss 3.0684 Epoch: 14 Global Step: 141615 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:48:30,496-Speed 130.52 samples/sec Loss 3.3467 Epoch: 14 Global Step: 141620 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:48:50,037-Speed 131.01 samples/sec Loss 3.0913 Epoch: 14 Global Step: 141625 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:49:09,857-Speed 129.17 samples/sec Loss 3.2705 Epoch: 14 Global Step: 141630 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:49:29,282-Speed 131.80 samples/sec Loss 3.2221 Epoch: 14 Global Step: 141635 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:49:49,065-Speed 129.41 samples/sec Loss 3.2103 Epoch: 14 Global Step: 141640 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:50:08,593-Speed 131.10 samples/sec Loss 3.1451 Epoch: 14 Global Step: 141645 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:50:28,141-Speed 130.96 samples/sec Loss 3.2946 Epoch: 14 Global Step: 141650 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:50:47,683-Speed 131.01 samples/sec Loss 3.1178 Epoch: 14 Global Step: 141655 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:51:07,277-Speed 130.66 samples/sec Loss 3.1146 Epoch: 14 Global Step: 141660 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:51:26,845-Speed 130.83 samples/sec Loss 3.2237 Epoch: 14 Global Step: 141665 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:51:46,286-Speed 131.69 samples/sec Loss 3.0854 Epoch: 14 Global Step: 141670 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:52:06,092-Speed 129.27 samples/sec Loss 3.2026 Epoch: 14 Global Step: 141675 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:52:25,733-Speed 130.35 samples/sec Loss 3.0052 Epoch: 14 Global Step: 141680 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:52:45,296-Speed 130.86 samples/sec Loss 3.0539 Epoch: 14 Global Step: 141685 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:53:04,755-Speed 131.57 samples/sec Loss 3.1463 Epoch: 14 Global Step: 141690 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:53:24,330-Speed 130.79 samples/sec Loss 3.3330 Epoch: 14 Global Step: 141695 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:53:43,936-Speed 130.58 samples/sec Loss 3.3332 Epoch: 14 Global Step: 141700 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:54:03,519-Speed 130.74 samples/sec Loss 3.1188 Epoch: 14 Global Step: 141705 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:54:23,014-Speed 131.32 samples/sec Loss 3.3381 Epoch: 14 Global Step: 141710 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:54:42,602-Speed 130.70 samples/sec Loss 3.0604 Epoch: 14 Global Step: 141715 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:55:02,346-Speed 129.67 samples/sec Loss 3.2611 Epoch: 14 Global Step: 141720 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:55:22,132-Speed 129.39 samples/sec Loss 3.4363 Epoch: 14 Global Step: 141725 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:55:41,550-Speed 131.84 samples/sec Loss 3.0831 Epoch: 14 Global Step: 141730 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:56:01,162-Speed 130.54 samples/sec Loss 3.1203 Epoch: 14 Global Step: 141735 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:56:20,806-Speed 130.33 samples/sec Loss 3.1526 Epoch: 14 Global Step: 141740 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:56:40,414-Speed 130.57 samples/sec Loss 3.2998 Epoch: 14 Global Step: 141745 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:57:00,145-Speed 129.75 samples/sec Loss 3.2626 Epoch: 14 Global Step: 141750 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:57:19,718-Speed 130.80 samples/sec Loss 3.0984 Epoch: 14 Global Step: 141755 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:57:39,281-Speed 130.87 samples/sec Loss 3.1525 Epoch: 14 Global Step: 141760 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:57:59,128-Speed 128.99 samples/sec Loss 3.1103 Epoch: 14 Global Step: 141765 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:58:18,576-Speed 131.64 samples/sec Loss 3.2806 Epoch: 14 Global Step: 141770 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:58:38,143-Speed 130.84 samples/sec Loss 3.1964 Epoch: 14 Global Step: 141775 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:58:57,757-Speed 130.53 samples/sec Loss 3.0099 Epoch: 14 Global Step: 141780 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:59:17,288-Speed 131.08 samples/sec Loss 3.1909 Epoch: 14 Global Step: 141785 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:59:36,907-Speed 130.49 samples/sec Loss 3.0532 Epoch: 14 Global Step: 141790 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 18:59:56,349-Speed 131.68 samples/sec Loss 3.3135 Epoch: 14 Global Step: 141795 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:00:15,802-Speed 131.61 samples/sec Loss 3.0945 Epoch: 14 Global Step: 141800 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:00:35,373-Speed 130.81 samples/sec Loss 3.2859 Epoch: 14 Global Step: 141805 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:00:55,291-Speed 128.54 samples/sec Loss 3.3064 Epoch: 14 Global Step: 141810 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:01:14,741-Speed 131.62 samples/sec Loss 3.1571 Epoch: 14 Global Step: 141815 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:01:34,204-Speed 131.54 samples/sec Loss 3.2853 Epoch: 14 Global Step: 141820 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:01:53,765-Speed 130.88 samples/sec Loss 3.0401 Epoch: 14 Global Step: 141825 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:02:13,304-Speed 131.03 samples/sec Loss 3.2294 Epoch: 14 Global Step: 141830 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:02:33,000-Speed 129.98 samples/sec Loss 3.3162 Epoch: 14 Global Step: 141835 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:02:52,432-Speed 131.75 samples/sec Loss 3.0780 Epoch: 14 Global Step: 141840 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:03:11,894-Speed 131.55 samples/sec Loss 3.4183 Epoch: 14 Global Step: 141845 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:03:31,477-Speed 130.74 samples/sec Loss 3.2062 Epoch: 14 Global Step: 141850 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:03:51,485-Speed 127.95 samples/sec Loss 3.3249 Epoch: 14 Global Step: 141855 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:04:10,913-Speed 131.77 samples/sec Loss 3.2122 Epoch: 14 Global Step: 141860 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:04:30,521-Speed 130.57 samples/sec Loss 3.1776 Epoch: 14 Global Step: 141865 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:04:50,039-Speed 131.17 samples/sec Loss 3.3690 Epoch: 14 Global Step: 141870 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:05:09,646-Speed 130.58 samples/sec Loss 3.2513 Epoch: 14 Global Step: 141875 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:05:29,251-Speed 130.58 samples/sec Loss 3.1778 Epoch: 14 Global Step: 141880 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:05:48,691-Speed 131.70 samples/sec Loss 3.0996 Epoch: 14 Global Step: 141885 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:06:08,308-Speed 130.50 samples/sec Loss 3.0232 Epoch: 14 Global Step: 141890 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:06:27,770-Speed 131.54 samples/sec Loss 3.1386 Epoch: 14 Global Step: 141895 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:06:47,673-Speed 128.63 samples/sec Loss 3.3925 Epoch: 14 Global Step: 141900 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:07:07,104-Speed 131.76 samples/sec Loss 3.2479 Epoch: 14 Global Step: 141905 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:07:26,669-Speed 130.85 samples/sec Loss 3.2235 Epoch: 14 Global Step: 141910 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:07:46,272-Speed 130.60 samples/sec Loss 3.2189 Epoch: 14 Global Step: 141915 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:08:05,827-Speed 130.92 samples/sec Loss 3.1143 Epoch: 14 Global Step: 141920 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:08:25,385-Speed 130.90 samples/sec Loss 3.3009 Epoch: 14 Global Step: 141925 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:08:44,805-Speed 131.83 samples/sec Loss 3.1125 Epoch: 14 Global Step: 141930 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:09:04,496-Speed 130.02 samples/sec Loss 3.1295 Epoch: 14 Global Step: 141935 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:09:24,098-Speed 130.61 samples/sec Loss 3.2756 Epoch: 14 Global Step: 141940 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:09:43,995-Speed 128.67 samples/sec Loss 3.2602 Epoch: 14 Global Step: 141945 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:10:03,437-Speed 131.68 samples/sec Loss 3.1484 Epoch: 14 Global Step: 141950 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:10:23,156-Speed 129.83 samples/sec Loss 3.2306 Epoch: 14 Global Step: 141955 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:10:42,748-Speed 130.67 samples/sec Loss 3.4450 Epoch: 14 Global Step: 141960 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:11:02,357-Speed 130.56 samples/sec Loss 3.1261 Epoch: 14 Global Step: 141965 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:11:21,788-Speed 131.76 samples/sec Loss 3.2049 Epoch: 14 Global Step: 141970 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:11:41,323-Speed 131.05 samples/sec Loss 3.3134 Epoch: 14 Global Step: 141975 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:12:00,886-Speed 130.86 samples/sec Loss 3.2039 Epoch: 14 Global Step: 141980 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:12:20,436-Speed 130.96 samples/sec Loss 3.2823 Epoch: 14 Global Step: 141985 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:12:40,236-Speed 129.30 samples/sec Loss 3.2478 Epoch: 14 Global Step: 141990 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:12:59,663-Speed 131.78 samples/sec Loss 3.0387 Epoch: 14 Global Step: 141995 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:13:19,444-Speed 129.42 samples/sec Loss 3.0802 Epoch: 14 Global Step: 142000 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:13:38,979-Speed 131.05 samples/sec Loss 3.0403 Epoch: 14 Global Step: 142005 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:13:58,446-Speed 131.51 samples/sec Loss 3.2166 Epoch: 14 Global Step: 142010 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:14:18,014-Speed 130.83 samples/sec Loss 3.2621 Epoch: 14 Global Step: 142015 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:14:37,539-Speed 131.12 samples/sec Loss 3.1653 Epoch: 14 Global Step: 142020 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:14:57,139-Speed 130.62 samples/sec Loss 3.2071 Epoch: 14 Global Step: 142025 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:15:16,710-Speed 130.81 samples/sec Loss 3.3575 Epoch: 14 Global Step: 142030 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:15:36,517-Speed 129.25 samples/sec Loss 3.1623 Epoch: 14 Global Step: 142035 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:15:55,944-Speed 131.78 samples/sec Loss 3.2616 Epoch: 14 Global Step: 142040 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:16:15,586-Speed 130.34 samples/sec Loss 3.2381 Epoch: 14 Global Step: 142045 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:16:35,320-Speed 129.73 samples/sec Loss 3.3551 Epoch: 14 Global Step: 142050 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:16:54,932-Speed 130.54 samples/sec Loss 3.1092 Epoch: 14 Global Step: 142055 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:17:14,460-Speed 131.10 samples/sec Loss 3.2540 Epoch: 14 Global Step: 142060 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:17:33,941-Speed 131.42 samples/sec Loss 3.2510 Epoch: 14 Global Step: 142065 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:17:53,536-Speed 130.65 samples/sec Loss 3.1990 Epoch: 14 Global Step: 142070 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:18:13,014-Speed 131.44 samples/sec Loss 3.2716 Epoch: 14 Global Step: 142075 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:18:32,823-Speed 129.24 samples/sec Loss 3.2222 Epoch: 14 Global Step: 142080 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:18:52,384-Speed 130.87 samples/sec Loss 3.3206 Epoch: 14 Global Step: 142085 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:19:12,002-Speed 130.50 samples/sec Loss 3.3418 Epoch: 14 Global Step: 142090 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:19:31,539-Speed 131.04 samples/sec Loss 3.3264 Epoch: 14 Global Step: 142095 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:19:51,151-Speed 130.54 samples/sec Loss 3.2824 Epoch: 14 Global Step: 142100 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:20:10,674-Speed 131.14 samples/sec Loss 3.1997 Epoch: 14 Global Step: 142105 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:20:30,287-Speed 130.54 samples/sec Loss 3.1252 Epoch: 14 Global Step: 142110 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:20:49,839-Speed 130.94 samples/sec Loss 2.9640 Epoch: 14 Global Step: 142115 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:21:09,279-Speed 131.70 samples/sec Loss 3.2930 Epoch: 14 Global Step: 142120 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:21:29,151-Speed 128.83 samples/sec Loss 3.4139 Epoch: 14 Global Step: 142125 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:21:48,846-Speed 129.99 samples/sec Loss 3.0595 Epoch: 14 Global Step: 142130 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:22:08,307-Speed 131.55 samples/sec Loss 3.2739 Epoch: 14 Global Step: 142135 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:22:27,779-Speed 131.48 samples/sec Loss 3.0470 Epoch: 14 Global Step: 142140 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:22:47,339-Speed 130.89 samples/sec Loss 3.1631 Epoch: 14 Global Step: 142145 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:23:06,864-Speed 131.12 samples/sec Loss 3.2355 Epoch: 14 Global Step: 142150 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:23:26,497-Speed 130.40 samples/sec Loss 3.2981 Epoch: 14 Global Step: 142155 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:23:46,062-Speed 130.86 samples/sec Loss 3.3151 Epoch: 14 Global Step: 142160 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:24:05,499-Speed 131.71 samples/sec Loss 3.3901 Epoch: 14 Global Step: 142165 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:24:25,296-Speed 129.32 samples/sec Loss 3.2551 Epoch: 14 Global Step: 142170 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:24:45,002-Speed 129.92 samples/sec Loss 3.2330 Epoch: 14 Global Step: 142175 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:25:04,480-Speed 131.44 samples/sec Loss 3.1163 Epoch: 14 Global Step: 142180 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:25:23,944-Speed 131.53 samples/sec Loss 3.0268 Epoch: 14 Global Step: 142185 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:25:43,518-Speed 130.79 samples/sec Loss 3.1972 Epoch: 14 Global Step: 142190 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:26:03,075-Speed 130.91 samples/sec Loss 3.1560 Epoch: 14 Global Step: 142195 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:26:22,648-Speed 130.80 samples/sec Loss 3.1721 Epoch: 14 Global Step: 142200 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:26:42,087-Speed 131.71 samples/sec Loss 3.0060 Epoch: 14 Global Step: 142205 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:27:01,665-Speed 130.76 samples/sec Loss 3.3794 Epoch: 14 Global Step: 142210 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:27:21,443-Speed 129.45 samples/sec Loss 3.3078 Epoch: 14 Global Step: 142215 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:27:41,156-Speed 129.87 samples/sec Loss 3.2102 Epoch: 14 Global Step: 142220 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:28:00,682-Speed 131.11 samples/sec Loss 3.0110 Epoch: 14 Global Step: 142225 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:28:20,288-Speed 130.58 samples/sec Loss 3.1941 Epoch: 14 Global Step: 142230 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:28:39,825-Speed 131.04 samples/sec Loss 3.2095 Epoch: 14 Global Step: 142235 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:28:59,548-Speed 129.80 samples/sec Loss 3.4356 Epoch: 14 Global Step: 142240 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:29:19,092-Speed 130.99 samples/sec Loss 3.0967 Epoch: 14 Global Step: 142245 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:29:38,595-Speed 131.27 samples/sec Loss 3.3945 Epoch: 14 Global Step: 142250 Fp16 Grad Scale: 8192 Required: 127 hours Training: 2021-06-13 19:29:58,161-Speed 130.85 samples/sec Loss 3.2600 Epoch: 14 Global Step: 142255 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:30:18,104-Speed 128.37 samples/sec Loss 3.2364 Epoch: 14 Global Step: 142260 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:30:37,541-Speed 131.72 samples/sec Loss 3.1630 Epoch: 14 Global Step: 142265 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:30:57,190-Speed 130.30 samples/sec Loss 3.3748 Epoch: 14 Global Step: 142270 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:31:16,734-Speed 131.00 samples/sec Loss 3.1563 Epoch: 14 Global Step: 142275 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:31:36,266-Speed 131.07 samples/sec Loss 3.3536 Epoch: 14 Global Step: 142280 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:31:55,863-Speed 130.64 samples/sec Loss 3.2941 Epoch: 14 Global Step: 142285 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:32:15,401-Speed 131.04 samples/sec Loss 3.2430 Epoch: 14 Global Step: 142290 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:32:35,003-Speed 130.61 samples/sec Loss 3.2144 Epoch: 14 Global Step: 142295 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:32:54,463-Speed 131.55 samples/sec Loss 3.3172 Epoch: 14 Global Step: 142300 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:33:14,253-Speed 129.37 samples/sec Loss 3.1941 Epoch: 14 Global Step: 142305 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:33:33,834-Speed 130.74 samples/sec Loss 3.1652 Epoch: 14 Global Step: 142310 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:33:53,459-Speed 130.45 samples/sec Loss 3.5382 Epoch: 14 Global Step: 142315 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:34:12,984-Speed 131.12 samples/sec Loss 3.0377 Epoch: 14 Global Step: 142320 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:34:32,596-Speed 130.54 samples/sec Loss 3.3902 Epoch: 14 Global Step: 142325 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:34:52,104-Speed 131.23 samples/sec Loss 3.1830 Epoch: 14 Global Step: 142330 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:35:11,617-Speed 131.20 samples/sec Loss 3.1800 Epoch: 14 Global Step: 142335 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:35:31,173-Speed 130.91 samples/sec Loss 3.0632 Epoch: 14 Global Step: 142340 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:35:50,737-Speed 130.86 samples/sec Loss 3.2296 Epoch: 14 Global Step: 142345 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:36:10,536-Speed 129.31 samples/sec Loss 2.8908 Epoch: 14 Global Step: 142350 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:36:30,250-Speed 129.86 samples/sec Loss 3.1415 Epoch: 14 Global Step: 142355 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:36:49,791-Speed 131.01 samples/sec Loss 3.2793 Epoch: 14 Global Step: 142360 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:37:09,341-Speed 130.96 samples/sec Loss 3.1662 Epoch: 14 Global Step: 142365 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:37:28,955-Speed 130.53 samples/sec Loss 3.3912 Epoch: 14 Global Step: 142370 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:37:48,497-Speed 131.01 samples/sec Loss 3.3564 Epoch: 14 Global Step: 142375 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:38:08,103-Speed 130.58 samples/sec Loss 3.2607 Epoch: 14 Global Step: 142380 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:38:27,622-Speed 131.16 samples/sec Loss 3.2480 Epoch: 14 Global Step: 142385 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:38:47,157-Speed 131.05 samples/sec Loss 3.2998 Epoch: 14 Global Step: 142390 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:39:06,922-Speed 129.53 samples/sec Loss 3.1070 Epoch: 14 Global Step: 142395 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:39:26,648-Speed 129.79 samples/sec Loss 3.2388 Epoch: 14 Global Step: 142400 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:39:46,176-Speed 131.10 samples/sec Loss 3.3056 Epoch: 14 Global Step: 142405 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:40:05,727-Speed 130.95 samples/sec Loss 3.3393 Epoch: 14 Global Step: 142410 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:40:25,306-Speed 130.76 samples/sec Loss 3.3134 Epoch: 14 Global Step: 142415 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:40:44,876-Speed 130.83 samples/sec Loss 3.1735 Epoch: 14 Global Step: 142420 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:41:04,473-Speed 130.64 samples/sec Loss 3.3902 Epoch: 14 Global Step: 142425 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:41:24,008-Speed 131.05 samples/sec Loss 3.1639 Epoch: 14 Global Step: 142430 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:41:43,662-Speed 130.26 samples/sec Loss 3.2733 Epoch: 14 Global Step: 142435 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:42:03,454-Speed 129.36 samples/sec Loss 3.2014 Epoch: 14 Global Step: 142440 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:42:23,172-Speed 129.84 samples/sec Loss 3.2777 Epoch: 14 Global Step: 142445 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:42:42,702-Speed 131.08 samples/sec Loss 3.2383 Epoch: 14 Global Step: 142450 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:43:02,317-Speed 130.52 samples/sec Loss 3.3130 Epoch: 14 Global Step: 142455 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:43:21,830-Speed 131.21 samples/sec Loss 3.1724 Epoch: 14 Global Step: 142460 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:43:41,383-Speed 130.93 samples/sec Loss 3.2296 Epoch: 14 Global Step: 142465 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:44:01,062-Speed 130.10 samples/sec Loss 3.2906 Epoch: 14 Global Step: 142470 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:44:20,588-Speed 131.11 samples/sec Loss 3.3269 Epoch: 14 Global Step: 142475 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:44:40,198-Speed 130.56 samples/sec Loss 3.1615 Epoch: 14 Global Step: 142480 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:44:59,956-Speed 129.57 samples/sec Loss 3.2945 Epoch: 14 Global Step: 142485 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:45:19,648-Speed 130.01 samples/sec Loss 3.1889 Epoch: 14 Global Step: 142490 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:45:39,191-Speed 131.00 samples/sec Loss 3.1274 Epoch: 14 Global Step: 142495 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:45:58,916-Speed 129.79 samples/sec Loss 3.2049 Epoch: 14 Global Step: 142500 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:46:18,514-Speed 130.63 samples/sec Loss 3.3673 Epoch: 14 Global Step: 142505 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:46:38,046-Speed 131.08 samples/sec Loss 3.2167 Epoch: 14 Global Step: 142510 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:46:57,581-Speed 131.05 samples/sec Loss 3.3242 Epoch: 14 Global Step: 142515 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:47:17,141-Speed 130.89 samples/sec Loss 3.1403 Epoch: 14 Global Step: 142520 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:47:36,649-Speed 131.24 samples/sec Loss 3.2946 Epoch: 14 Global Step: 142525 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:47:56,460-Speed 129.22 samples/sec Loss 3.1815 Epoch: 14 Global Step: 142530 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:48:16,159-Speed 129.97 samples/sec Loss 3.3087 Epoch: 14 Global Step: 142535 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:48:35,762-Speed 130.59 samples/sec Loss 3.4097 Epoch: 14 Global Step: 142540 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:48:55,356-Speed 130.66 samples/sec Loss 3.3098 Epoch: 14 Global Step: 142545 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:49:14,858-Speed 131.28 samples/sec Loss 3.2018 Epoch: 14 Global Step: 142550 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:49:34,790-Speed 128.44 samples/sec Loss 3.1298 Epoch: 14 Global Step: 142555 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:49:54,401-Speed 130.55 samples/sec Loss 3.2534 Epoch: 14 Global Step: 142560 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:50:14,015-Speed 130.52 samples/sec Loss 3.2959 Epoch: 14 Global Step: 142565 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:50:33,618-Speed 130.60 samples/sec Loss 3.3077 Epoch: 14 Global Step: 142570 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:50:53,445-Speed 129.12 samples/sec Loss 3.1635 Epoch: 14 Global Step: 142575 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:51:13,089-Speed 130.33 samples/sec Loss 3.2896 Epoch: 14 Global Step: 142580 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:51:32,651-Speed 130.87 samples/sec Loss 3.3047 Epoch: 14 Global Step: 142585 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:51:52,262-Speed 130.55 samples/sec Loss 3.2986 Epoch: 14 Global Step: 142590 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:52:11,813-Speed 130.95 samples/sec Loss 3.1494 Epoch: 14 Global Step: 142595 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:52:31,376-Speed 130.86 samples/sec Loss 3.2421 Epoch: 14 Global Step: 142600 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:52:51,023-Speed 130.31 samples/sec Loss 3.2605 Epoch: 14 Global Step: 142605 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:53:10,549-Speed 131.12 samples/sec Loss 3.4313 Epoch: 14 Global Step: 142610 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:53:30,156-Speed 130.57 samples/sec Loss 3.2792 Epoch: 14 Global Step: 142615 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:53:50,008-Speed 128.96 samples/sec Loss 3.3391 Epoch: 14 Global Step: 142620 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:54:09,527-Speed 131.16 samples/sec Loss 3.3331 Epoch: 14 Global Step: 142625 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:54:29,125-Speed 130.63 samples/sec Loss 3.1667 Epoch: 14 Global Step: 142630 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:54:48,729-Speed 130.59 samples/sec Loss 3.4189 Epoch: 14 Global Step: 142635 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:55:08,276-Speed 130.98 samples/sec Loss 3.3117 Epoch: 14 Global Step: 142640 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:55:27,939-Speed 130.20 samples/sec Loss 3.1495 Epoch: 14 Global Step: 142645 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:55:47,499-Speed 130.89 samples/sec Loss 3.2530 Epoch: 14 Global Step: 142650 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:56:06,965-Speed 131.52 samples/sec Loss 3.3278 Epoch: 14 Global Step: 142655 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:56:26,532-Speed 130.84 samples/sec Loss 3.2042 Epoch: 14 Global Step: 142660 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:56:46,468-Speed 128.42 samples/sec Loss 3.2880 Epoch: 14 Global Step: 142665 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:57:05,956-Speed 131.37 samples/sec Loss 3.1865 Epoch: 14 Global Step: 142670 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:57:25,517-Speed 130.88 samples/sec Loss 3.2125 Epoch: 14 Global Step: 142675 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:57:45,082-Speed 130.85 samples/sec Loss 3.2628 Epoch: 14 Global Step: 142680 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:58:04,630-Speed 130.97 samples/sec Loss 3.4338 Epoch: 14 Global Step: 142685 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:58:24,220-Speed 130.69 samples/sec Loss 3.2041 Epoch: 14 Global Step: 142690 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:58:43,742-Speed 131.14 samples/sec Loss 3.2145 Epoch: 14 Global Step: 142695 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:59:03,268-Speed 131.12 samples/sec Loss 3.3907 Epoch: 14 Global Step: 142700 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:59:22,835-Speed 130.84 samples/sec Loss 3.2486 Epoch: 14 Global Step: 142705 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 19:59:42,719-Speed 128.76 samples/sec Loss 3.2714 Epoch: 14 Global Step: 142710 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:00:02,160-Speed 131.69 samples/sec Loss 3.2391 Epoch: 14 Global Step: 142715 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:00:21,711-Speed 130.94 samples/sec Loss 3.5056 Epoch: 14 Global Step: 142720 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:00:41,328-Speed 130.51 samples/sec Loss 3.0901 Epoch: 14 Global Step: 142725 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:01:00,870-Speed 131.01 samples/sec Loss 3.3185 Epoch: 14 Global Step: 142730 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:01:20,483-Speed 130.53 samples/sec Loss 3.2107 Epoch: 14 Global Step: 142735 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:01:39,919-Speed 131.72 samples/sec Loss 3.3825 Epoch: 14 Global Step: 142740 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:01:59,458-Speed 131.03 samples/sec Loss 3.3956 Epoch: 14 Global Step: 142745 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:02:19,030-Speed 130.81 samples/sec Loss 3.1355 Epoch: 14 Global Step: 142750 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:02:38,950-Speed 128.52 samples/sec Loss 3.3025 Epoch: 14 Global Step: 142755 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:02:58,471-Speed 131.15 samples/sec Loss 3.3971 Epoch: 14 Global Step: 142760 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:03:17,934-Speed 131.54 samples/sec Loss 3.4022 Epoch: 14 Global Step: 142765 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:03:37,596-Speed 130.20 samples/sec Loss 3.3043 Epoch: 14 Global Step: 142770 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:03:57,060-Speed 131.53 samples/sec Loss 3.3677 Epoch: 14 Global Step: 142775 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:04:16,678-Speed 130.50 samples/sec Loss 3.1586 Epoch: 14 Global Step: 142780 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:04:36,210-Speed 131.09 samples/sec Loss 3.1609 Epoch: 14 Global Step: 142785 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:04:55,665-Speed 131.59 samples/sec Loss 3.2012 Epoch: 14 Global Step: 142790 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:05:15,230-Speed 130.85 samples/sec Loss 3.3846 Epoch: 14 Global Step: 142795 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:05:35,003-Speed 129.48 samples/sec Loss 3.4927 Epoch: 14 Global Step: 142800 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:05:54,632-Speed 130.42 samples/sec Loss 3.0938 Epoch: 14 Global Step: 142805 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:06:14,196-Speed 130.86 samples/sec Loss 3.3466 Epoch: 14 Global Step: 142810 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:06:33,810-Speed 130.53 samples/sec Loss 3.3843 Epoch: 14 Global Step: 142815 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:06:53,344-Speed 131.06 samples/sec Loss 3.4227 Epoch: 14 Global Step: 142820 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:07:13,002-Speed 130.23 samples/sec Loss 3.4002 Epoch: 14 Global Step: 142825 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:07:32,548-Speed 130.98 samples/sec Loss 3.1843 Epoch: 14 Global Step: 142830 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:07:52,110-Speed 130.88 samples/sec Loss 3.1974 Epoch: 14 Global Step: 142835 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:08:11,646-Speed 131.05 samples/sec Loss 3.4138 Epoch: 14 Global Step: 142840 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:08:31,625-Speed 128.14 samples/sec Loss 3.1707 Epoch: 14 Global Step: 142845 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:08:51,051-Speed 131.79 samples/sec Loss 3.3168 Epoch: 14 Global Step: 142850 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:09:10,658-Speed 130.58 samples/sec Loss 3.2002 Epoch: 14 Global Step: 142855 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:09:30,192-Speed 131.06 samples/sec Loss 3.1289 Epoch: 14 Global Step: 142860 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:09:49,784-Speed 130.67 samples/sec Loss 3.3620 Epoch: 14 Global Step: 142865 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:10:09,363-Speed 130.76 samples/sec Loss 3.2134 Epoch: 14 Global Step: 142870 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:10:28,944-Speed 130.75 samples/sec Loss 3.2703 Epoch: 14 Global Step: 142875 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:10:48,648-Speed 129.93 samples/sec Loss 3.3388 Epoch: 14 Global Step: 142880 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:11:08,118-Speed 131.50 samples/sec Loss 3.3624 Epoch: 14 Global Step: 142885 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:11:27,923-Speed 129.27 samples/sec Loss 3.0569 Epoch: 14 Global Step: 142890 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:11:47,531-Speed 130.57 samples/sec Loss 3.2174 Epoch: 14 Global Step: 142895 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:12:07,182-Speed 130.28 samples/sec Loss 3.1667 Epoch: 14 Global Step: 142900 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:12:26,643-Speed 131.55 samples/sec Loss 3.4772 Epoch: 14 Global Step: 142905 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:12:46,260-Speed 130.51 samples/sec Loss 3.1531 Epoch: 14 Global Step: 142910 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:13:05,710-Speed 131.63 samples/sec Loss 3.4283 Epoch: 14 Global Step: 142915 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:13:25,248-Speed 131.03 samples/sec Loss 3.1254 Epoch: 14 Global Step: 142920 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:13:44,842-Speed 130.67 samples/sec Loss 3.2798 Epoch: 14 Global Step: 142925 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:14:04,316-Speed 131.46 samples/sec Loss 3.2634 Epoch: 14 Global Step: 142930 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:14:24,110-Speed 129.34 samples/sec Loss 3.3526 Epoch: 14 Global Step: 142935 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:14:43,733-Speed 130.47 samples/sec Loss 3.2783 Epoch: 14 Global Step: 142940 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:15:03,267-Speed 131.06 samples/sec Loss 3.0693 Epoch: 14 Global Step: 142945 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:15:22,853-Speed 130.71 samples/sec Loss 3.1350 Epoch: 14 Global Step: 142950 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:15:42,450-Speed 130.64 samples/sec Loss 3.4306 Epoch: 14 Global Step: 142955 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:16:01,876-Speed 131.79 samples/sec Loss 3.3210 Epoch: 14 Global Step: 142960 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:16:21,436-Speed 130.89 samples/sec Loss 3.1658 Epoch: 14 Global Step: 142965 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:16:40,936-Speed 131.29 samples/sec Loss 3.3259 Epoch: 14 Global Step: 142970 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:17:00,557-Speed 130.48 samples/sec Loss 3.2384 Epoch: 14 Global Step: 142975 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:17:20,383-Speed 129.13 samples/sec Loss 3.2232 Epoch: 14 Global Step: 142980 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:17:39,996-Speed 130.53 samples/sec Loss 3.2565 Epoch: 14 Global Step: 142985 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:17:59,416-Speed 131.83 samples/sec Loss 3.3980 Epoch: 14 Global Step: 142990 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:18:19,029-Speed 130.54 samples/sec Loss 3.3721 Epoch: 14 Global Step: 142995 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:18:38,615-Speed 130.72 samples/sec Loss 3.2686 Epoch: 14 Global Step: 143000 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:18:58,065-Speed 131.63 samples/sec Loss 3.2693 Epoch: 14 Global Step: 143005 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:19:17,662-Speed 130.64 samples/sec Loss 3.4741 Epoch: 14 Global Step: 143010 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:19:37,147-Speed 131.39 samples/sec Loss 3.2274 Epoch: 14 Global Step: 143015 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:19:56,764-Speed 130.50 samples/sec Loss 3.1555 Epoch: 14 Global Step: 143020 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:20:16,528-Speed 129.54 samples/sec Loss 3.2690 Epoch: 14 Global Step: 143025 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:20:36,319-Speed 129.36 samples/sec Loss 3.1398 Epoch: 14 Global Step: 143030 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:20:55,832-Speed 131.20 samples/sec Loss 3.1829 Epoch: 14 Global Step: 143035 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:21:15,425-Speed 130.66 samples/sec Loss 3.2466 Epoch: 14 Global Step: 143040 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:21:34,920-Speed 131.33 samples/sec Loss 3.2471 Epoch: 14 Global Step: 143045 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:21:54,869-Speed 128.33 samples/sec Loss 3.3544 Epoch: 14 Global Step: 143050 Fp16 Grad Scale: 8192 Required: 126 hours Training: 2021-06-13 20:22:14,483-Speed 130.53 samples/sec Loss 3.4482 Epoch: 14 Global Step: 143055 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:22:34,028-Speed 130.99 samples/sec Loss 3.2743 Epoch: 14 Global Step: 143060 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:22:53,637-Speed 130.56 samples/sec Loss 3.2410 Epoch: 14 Global Step: 143065 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:23:13,260-Speed 130.47 samples/sec Loss 3.1595 Epoch: 14 Global Step: 143070 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:23:32,992-Speed 129.74 samples/sec Loss 3.2086 Epoch: 14 Global Step: 143075 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:23:52,615-Speed 130.47 samples/sec Loss 3.4369 Epoch: 14 Global Step: 143080 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:24:12,319-Speed 129.93 samples/sec Loss 3.3630 Epoch: 14 Global Step: 143085 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:24:31,761-Speed 131.68 samples/sec Loss 3.2408 Epoch: 14 Global Step: 143090 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:24:51,375-Speed 130.53 samples/sec Loss 3.2705 Epoch: 14 Global Step: 143095 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:25:10,953-Speed 130.77 samples/sec Loss 3.2518 Epoch: 14 Global Step: 143100 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:25:30,547-Speed 130.66 samples/sec Loss 3.2797 Epoch: 14 Global Step: 143105 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:25:50,141-Speed 130.69 samples/sec Loss 3.3637 Epoch: 14 Global Step: 143110 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:26:09,921-Speed 129.43 samples/sec Loss 3.2173 Epoch: 14 Global Step: 143115 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:26:29,532-Speed 130.55 samples/sec Loss 3.3010 Epoch: 14 Global Step: 143120 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:26:49,108-Speed 130.78 samples/sec Loss 3.2973 Epoch: 14 Global Step: 143125 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:27:08,698-Speed 130.69 samples/sec Loss 3.3588 Epoch: 14 Global Step: 143130 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:27:28,376-Speed 130.10 samples/sec Loss 3.1333 Epoch: 14 Global Step: 143135 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:27:47,970-Speed 130.66 samples/sec Loss 3.4145 Epoch: 14 Global Step: 143140 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:28:07,628-Speed 130.23 samples/sec Loss 3.4239 Epoch: 14 Global Step: 143145 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:28:27,243-Speed 130.53 samples/sec Loss 3.3047 Epoch: 14 Global Step: 143150 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:28:46,874-Speed 130.41 samples/sec Loss 3.2709 Epoch: 14 Global Step: 143155 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:29:07,304-Speed 125.31 samples/sec Loss 3.2639 Epoch: 14 Global Step: 143160 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:29:26,898-Speed 130.66 samples/sec Loss 3.3535 Epoch: 14 Global Step: 143165 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:29:46,415-Speed 131.18 samples/sec Loss 3.3158 Epoch: 14 Global Step: 143170 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:30:05,986-Speed 130.81 samples/sec Loss 3.1053 Epoch: 14 Global Step: 143175 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:30:25,583-Speed 130.65 samples/sec Loss 3.3283 Epoch: 14 Global Step: 143180 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:30:45,204-Speed 130.48 samples/sec Loss 3.5585 Epoch: 14 Global Step: 143185 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:31:04,994-Speed 129.36 samples/sec Loss 3.1139 Epoch: 14 Global Step: 143190 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:31:24,594-Speed 130.62 samples/sec Loss 3.0835 Epoch: 14 Global Step: 143195 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:31:44,129-Speed 131.06 samples/sec Loss 3.1516 Epoch: 14 Global Step: 143200 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:32:04,041-Speed 128.57 samples/sec Loss 3.2141 Epoch: 14 Global Step: 143205 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:32:23,526-Speed 131.39 samples/sec Loss 3.4005 Epoch: 14 Global Step: 143210 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:32:43,305-Speed 129.44 samples/sec Loss 3.2380 Epoch: 14 Global Step: 143215 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:33:02,848-Speed 131.00 samples/sec Loss 3.1755 Epoch: 14 Global Step: 143220 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:33:22,686-Speed 129.05 samples/sec Loss 3.1715 Epoch: 14 Global Step: 143225 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:33:42,400-Speed 129.86 samples/sec Loss 3.4408 Epoch: 14 Global Step: 143230 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:34:02,099-Speed 129.97 samples/sec Loss 3.3172 Epoch: 14 Global Step: 143235 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:34:21,768-Speed 130.16 samples/sec Loss 3.1282 Epoch: 14 Global Step: 143240 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:34:41,672-Speed 128.63 samples/sec Loss 3.2862 Epoch: 14 Global Step: 143245 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:35:01,487-Speed 129.20 samples/sec Loss 3.1159 Epoch: 14 Global Step: 143250 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:35:20,912-Speed 131.80 samples/sec Loss 3.4541 Epoch: 14 Global Step: 143255 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:35:40,612-Speed 129.96 samples/sec Loss 3.2342 Epoch: 14 Global Step: 143260 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:36:00,299-Speed 130.04 samples/sec Loss 3.2702 Epoch: 14 Global Step: 143265 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:36:19,915-Speed 130.51 samples/sec Loss 3.2312 Epoch: 14 Global Step: 143270 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:36:39,603-Speed 130.04 samples/sec Loss 3.2950 Epoch: 14 Global Step: 143275 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:37:00,042-Speed 125.26 samples/sec Loss 3.1098 Epoch: 14 Global Step: 143280 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:37:19,680-Speed 130.37 samples/sec Loss 3.2804 Epoch: 14 Global Step: 143285 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:37:39,220-Speed 131.03 samples/sec Loss 3.1917 Epoch: 14 Global Step: 143290 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:37:59,032-Speed 129.22 samples/sec Loss 3.2798 Epoch: 14 Global Step: 143295 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:38:18,801-Speed 129.50 samples/sec Loss 3.2446 Epoch: 14 Global Step: 143300 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:38:38,389-Speed 130.70 samples/sec Loss 3.5119 Epoch: 14 Global Step: 143305 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:38:57,861-Speed 131.47 samples/sec Loss 3.4205 Epoch: 14 Global Step: 143310 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:39:17,455-Speed 130.67 samples/sec Loss 3.2509 Epoch: 14 Global Step: 143315 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:39:37,095-Speed 130.36 samples/sec Loss 3.2599 Epoch: 14 Global Step: 143320 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:39:56,715-Speed 130.48 samples/sec Loss 3.3930 Epoch: 14 Global Step: 143325 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:40:16,273-Speed 130.90 samples/sec Loss 3.0954 Epoch: 14 Global Step: 143330 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:40:35,743-Speed 131.49 samples/sec Loss 3.3743 Epoch: 14 Global Step: 143335 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:40:55,633-Speed 128.72 samples/sec Loss 3.1340 Epoch: 14 Global Step: 143340 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:41:15,380-Speed 129.65 samples/sec Loss 3.4747 Epoch: 14 Global Step: 143345 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:41:35,019-Speed 130.36 samples/sec Loss 3.1310 Epoch: 14 Global Step: 143350 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:41:54,571-Speed 130.94 samples/sec Loss 3.3612 Epoch: 14 Global Step: 143355 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:42:14,131-Speed 130.89 samples/sec Loss 3.4085 Epoch: 14 Global Step: 143360 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:42:33,698-Speed 130.85 samples/sec Loss 3.2721 Epoch: 14 Global Step: 143365 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:42:53,293-Speed 130.65 samples/sec Loss 3.3194 Epoch: 14 Global Step: 143370 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:43:12,784-Speed 131.35 samples/sec Loss 3.2204 Epoch: 14 Global Step: 143375 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:43:32,407-Speed 130.47 samples/sec Loss 3.1319 Epoch: 14 Global Step: 143380 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:43:52,200-Speed 129.35 samples/sec Loss 3.4679 Epoch: 14 Global Step: 143385 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:44:11,915-Speed 129.85 samples/sec Loss 3.3307 Epoch: 14 Global Step: 143390 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:44:31,454-Speed 131.03 samples/sec Loss 3.3763 Epoch: 14 Global Step: 143395 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:44:51,052-Speed 130.63 samples/sec Loss 3.3150 Epoch: 14 Global Step: 143400 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:45:10,810-Speed 129.57 samples/sec Loss 3.3753 Epoch: 14 Global Step: 143405 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:45:30,408-Speed 130.63 samples/sec Loss 3.3324 Epoch: 14 Global Step: 143410 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:45:49,992-Speed 130.73 samples/sec Loss 3.2641 Epoch: 14 Global Step: 143415 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:46:09,523-Speed 131.08 samples/sec Loss 3.3840 Epoch: 14 Global Step: 143420 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:46:29,141-Speed 130.51 samples/sec Loss 3.2276 Epoch: 14 Global Step: 143425 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:46:48,964-Speed 129.15 samples/sec Loss 3.3720 Epoch: 14 Global Step: 143430 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:47:08,591-Speed 130.44 samples/sec Loss 3.3400 Epoch: 14 Global Step: 143435 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:47:28,132-Speed 131.01 samples/sec Loss 3.3546 Epoch: 14 Global Step: 143440 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:47:47,741-Speed 130.56 samples/sec Loss 3.3181 Epoch: 14 Global Step: 143445 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:48:07,256-Speed 131.19 samples/sec Loss 3.4149 Epoch: 14 Global Step: 143450 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:48:26,804-Speed 130.97 samples/sec Loss 3.4455 Epoch: 14 Global Step: 143455 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:48:46,377-Speed 130.80 samples/sec Loss 3.2575 Epoch: 14 Global Step: 143460 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:49:05,915-Speed 131.03 samples/sec Loss 3.3983 Epoch: 14 Global Step: 143465 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:49:25,509-Speed 130.66 samples/sec Loss 3.2529 Epoch: 14 Global Step: 143470 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:49:45,307-Speed 129.31 samples/sec Loss 3.3283 Epoch: 14 Global Step: 143475 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:50:04,975-Speed 130.17 samples/sec Loss 3.4388 Epoch: 14 Global Step: 143480 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:50:24,453-Speed 131.43 samples/sec Loss 3.2161 Epoch: 14 Global Step: 143485 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:50:44,062-Speed 130.56 samples/sec Loss 3.3945 Epoch: 14 Global Step: 143490 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:51:03,602-Speed 131.02 samples/sec Loss 3.5822 Epoch: 14 Global Step: 143495 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:51:23,162-Speed 130.89 samples/sec Loss 3.4144 Epoch: 14 Global Step: 143500 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:51:42,746-Speed 130.73 samples/sec Loss 3.1647 Epoch: 14 Global Step: 143505 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:52:02,376-Speed 130.42 samples/sec Loss 3.1802 Epoch: 14 Global Step: 143510 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:52:21,989-Speed 130.53 samples/sec Loss 3.2631 Epoch: 14 Global Step: 143515 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:52:41,819-Speed 129.11 samples/sec Loss 3.3607 Epoch: 14 Global Step: 143520 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:53:01,334-Speed 131.18 samples/sec Loss 3.6259 Epoch: 14 Global Step: 143525 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:53:20,870-Speed 131.05 samples/sec Loss 3.3303 Epoch: 14 Global Step: 143530 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:53:40,519-Speed 130.30 samples/sec Loss 3.1913 Epoch: 14 Global Step: 143535 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:54:00,047-Speed 131.10 samples/sec Loss 3.2024 Epoch: 14 Global Step: 143540 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:54:19,629-Speed 130.74 samples/sec Loss 3.3241 Epoch: 14 Global Step: 143545 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:54:39,069-Speed 131.69 samples/sec Loss 3.2607 Epoch: 14 Global Step: 143550 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:54:58,681-Speed 130.54 samples/sec Loss 3.2564 Epoch: 14 Global Step: 143555 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:55:18,251-Speed 130.82 samples/sec Loss 3.3834 Epoch: 14 Global Step: 143560 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:55:38,186-Speed 128.43 samples/sec Loss 3.2432 Epoch: 14 Global Step: 143565 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:55:57,609-Speed 131.81 samples/sec Loss 3.5053 Epoch: 14 Global Step: 143570 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:56:17,211-Speed 130.60 samples/sec Loss 3.1969 Epoch: 14 Global Step: 143575 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:56:36,783-Speed 130.81 samples/sec Loss 3.2484 Epoch: 14 Global Step: 143580 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:56:56,330-Speed 130.97 samples/sec Loss 3.2167 Epoch: 14 Global Step: 143585 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:57:15,932-Speed 130.60 samples/sec Loss 3.2821 Epoch: 14 Global Step: 143590 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:57:35,621-Speed 130.03 samples/sec Loss 3.2479 Epoch: 14 Global Step: 143595 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:57:55,209-Speed 130.70 samples/sec Loss 3.3595 Epoch: 14 Global Step: 143600 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:58:14,745-Speed 131.05 samples/sec Loss 3.3504 Epoch: 14 Global Step: 143605 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:58:34,578-Speed 129.08 samples/sec Loss 3.0125 Epoch: 14 Global Step: 143610 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:58:54,130-Speed 130.94 samples/sec Loss 3.1688 Epoch: 14 Global Step: 143615 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:59:13,904-Speed 129.47 samples/sec Loss 3.3294 Epoch: 14 Global Step: 143620 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:59:33,433-Speed 131.09 samples/sec Loss 3.5460 Epoch: 14 Global Step: 143625 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 20:59:53,039-Speed 130.58 samples/sec Loss 3.3425 Epoch: 14 Global Step: 143630 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:00:12,479-Speed 131.69 samples/sec Loss 3.2363 Epoch: 14 Global Step: 143635 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:00:32,213-Speed 129.74 samples/sec Loss 3.3842 Epoch: 14 Global Step: 143640 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:00:51,935-Speed 129.81 samples/sec Loss 3.3837 Epoch: 14 Global Step: 143645 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:01:11,476-Speed 131.01 samples/sec Loss 3.2166 Epoch: 14 Global Step: 143650 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:01:31,275-Speed 129.31 samples/sec Loss 3.0506 Epoch: 14 Global Step: 143655 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:01:50,753-Speed 131.43 samples/sec Loss 3.3657 Epoch: 14 Global Step: 143660 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:02:10,462-Speed 129.90 samples/sec Loss 3.0981 Epoch: 14 Global Step: 143665 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:02:29,994-Speed 131.07 samples/sec Loss 3.2651 Epoch: 14 Global Step: 143670 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:02:49,622-Speed 130.43 samples/sec Loss 3.3868 Epoch: 14 Global Step: 143675 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:03:09,123-Speed 131.28 samples/sec Loss 3.3437 Epoch: 14 Global Step: 143680 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:03:28,669-Speed 130.98 samples/sec Loss 3.3390 Epoch: 14 Global Step: 143685 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:03:48,230-Speed 130.88 samples/sec Loss 3.2219 Epoch: 14 Global Step: 143690 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:04:07,759-Speed 131.09 samples/sec Loss 3.3736 Epoch: 14 Global Step: 143695 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:04:27,567-Speed 129.25 samples/sec Loss 3.3823 Epoch: 14 Global Step: 143700 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:04:47,180-Speed 130.53 samples/sec Loss 3.4690 Epoch: 14 Global Step: 143705 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:05:06,722-Speed 131.01 samples/sec Loss 3.1058 Epoch: 14 Global Step: 143710 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:05:26,275-Speed 130.93 samples/sec Loss 3.2688 Epoch: 14 Global Step: 143715 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:05:45,872-Speed 130.64 samples/sec Loss 3.3148 Epoch: 14 Global Step: 143720 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:06:05,461-Speed 130.69 samples/sec Loss 3.5850 Epoch: 14 Global Step: 143725 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:06:25,049-Speed 130.70 samples/sec Loss 3.4313 Epoch: 14 Global Step: 143730 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:06:44,613-Speed 130.86 samples/sec Loss 3.4047 Epoch: 14 Global Step: 143735 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:07:04,517-Speed 128.62 samples/sec Loss 3.2049 Epoch: 14 Global Step: 143740 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:07:24,284-Speed 129.52 samples/sec Loss 3.4018 Epoch: 14 Global Step: 143745 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:07:44,071-Speed 129.38 samples/sec Loss 3.2948 Epoch: 14 Global Step: 143750 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:08:03,787-Speed 129.85 samples/sec Loss 3.3714 Epoch: 14 Global Step: 143755 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:08:23,335-Speed 130.97 samples/sec Loss 3.2214 Epoch: 14 Global Step: 143760 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:08:42,903-Speed 130.83 samples/sec Loss 3.2707 Epoch: 14 Global Step: 143765 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:09:02,513-Speed 130.56 samples/sec Loss 3.3760 Epoch: 14 Global Step: 143770 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:09:22,112-Speed 130.63 samples/sec Loss 3.2572 Epoch: 14 Global Step: 143775 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:09:41,653-Speed 131.02 samples/sec Loss 3.3779 Epoch: 14 Global Step: 143780 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:10:01,259-Speed 130.58 samples/sec Loss 3.1834 Epoch: 14 Global Step: 143785 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:10:21,085-Speed 129.13 samples/sec Loss 3.3307 Epoch: 14 Global Step: 143790 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:10:40,711-Speed 130.45 samples/sec Loss 3.2062 Epoch: 14 Global Step: 143795 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:11:00,461-Speed 129.63 samples/sec Loss 3.5478 Epoch: 14 Global Step: 143800 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:11:20,039-Speed 130.77 samples/sec Loss 3.5283 Epoch: 14 Global Step: 143805 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:11:39,590-Speed 130.95 samples/sec Loss 3.3743 Epoch: 14 Global Step: 143810 Fp16 Grad Scale: 8192 Required: 125 hours Training: 2021-06-13 21:11:58,760-Speed 133.55 samples/sec Loss 3.4300 Epoch: 14 Global Step: 143815 Fp16 Grad Scale: 4096 Required: 125 hours Training: 2021-06-13 21:12:18,301-Speed 131.02 samples/sec Loss 3.1777 Epoch: 14 Global Step: 143820 Fp16 Grad Scale: 4096 Required: 125 hours Training: 2021-06-13 21:12:37,945-Speed 130.32 samples/sec Loss 3.2359 Epoch: 14 Global Step: 143825 Fp16 Grad Scale: 4096 Required: 125 hours Training: 2021-06-13 21:12:57,558-Speed 130.54 samples/sec Loss 3.4085 Epoch: 14 Global Step: 143830 Fp16 Grad Scale: 4096 Required: 125 hours Training: 2021-06-13 21:13:17,021-Speed 131.54 samples/sec Loss 3.1222 Epoch: 14 Global Step: 143835 Fp16 Grad Scale: 4096 Required: 125 hours Training: 2021-06-13 21:13:36,636-Speed 130.52 samples/sec Loss 3.1036 Epoch: 14 Global Step: 143840 Fp16 Grad Scale: 4096 Required: 125 hours Training: 2021-06-13 21:13:56,178-Speed 131.01 samples/sec Loss 3.5871 Epoch: 14 Global Step: 143845 Fp16 Grad Scale: 4096 Required: 125 hours Training: 2021-06-13 21:14:15,737-Speed 130.90 samples/sec Loss 3.3652 Epoch: 14 Global Step: 143850 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:14:35,302-Speed 130.85 samples/sec Loss 3.1112 Epoch: 14 Global Step: 143855 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:14:55,128-Speed 129.13 samples/sec Loss 3.3389 Epoch: 14 Global Step: 143860 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:15:14,753-Speed 130.45 samples/sec Loss 3.3287 Epoch: 14 Global Step: 143865 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:15:34,341-Speed 130.70 samples/sec Loss 3.3319 Epoch: 14 Global Step: 143870 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:15:53,773-Speed 131.75 samples/sec Loss 3.5137 Epoch: 14 Global Step: 143875 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:16:13,367-Speed 130.66 samples/sec Loss 3.5217 Epoch: 14 Global Step: 143880 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:16:32,899-Speed 131.07 samples/sec Loss 3.3437 Epoch: 14 Global Step: 143885 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:16:52,508-Speed 130.56 samples/sec Loss 3.4148 Epoch: 14 Global Step: 143890 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:17:12,109-Speed 130.61 samples/sec Loss 3.3982 Epoch: 14 Global Step: 143895 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:17:31,631-Speed 131.14 samples/sec Loss 3.4210 Epoch: 14 Global Step: 143900 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:17:51,404-Speed 129.48 samples/sec Loss 3.2135 Epoch: 14 Global Step: 143905 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:18:11,122-Speed 129.84 samples/sec Loss 3.3673 Epoch: 14 Global Step: 143910 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:18:30,792-Speed 130.16 samples/sec Loss 3.1715 Epoch: 14 Global Step: 143915 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:18:50,358-Speed 130.85 samples/sec Loss 3.2369 Epoch: 14 Global Step: 143920 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:19:09,904-Speed 130.98 samples/sec Loss 3.1098 Epoch: 14 Global Step: 143925 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:19:29,373-Speed 131.50 samples/sec Loss 3.1421 Epoch: 14 Global Step: 143930 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:19:48,974-Speed 130.61 samples/sec Loss 3.2906 Epoch: 14 Global Step: 143935 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:20:08,662-Speed 130.04 samples/sec Loss 3.3643 Epoch: 14 Global Step: 143940 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:20:28,111-Speed 131.63 samples/sec Loss 3.3419 Epoch: 14 Global Step: 143945 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:20:47,885-Speed 129.47 samples/sec Loss 3.4342 Epoch: 14 Global Step: 143950 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:21:07,498-Speed 130.53 samples/sec Loss 3.1510 Epoch: 14 Global Step: 143955 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:21:27,034-Speed 131.05 samples/sec Loss 3.2023 Epoch: 14 Global Step: 143960 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:21:46,624-Speed 130.69 samples/sec Loss 3.3665 Epoch: 14 Global Step: 143965 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:22:06,162-Speed 131.03 samples/sec Loss 3.1831 Epoch: 14 Global Step: 143970 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:22:25,632-Speed 131.49 samples/sec Loss 3.3868 Epoch: 14 Global Step: 143975 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:22:45,210-Speed 130.76 samples/sec Loss 3.0655 Epoch: 14 Global Step: 143980 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:23:04,751-Speed 131.02 samples/sec Loss 3.3034 Epoch: 14 Global Step: 143985 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:23:24,365-Speed 130.53 samples/sec Loss 3.3631 Epoch: 14 Global Step: 143990 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:23:44,139-Speed 129.47 samples/sec Loss 3.5275 Epoch: 14 Global Step: 143995 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:24:03,703-Speed 130.86 samples/sec Loss 3.2266 Epoch: 14 Global Step: 144000 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:24:23,190-Speed 131.38 samples/sec Loss 3.4846 Epoch: 14 Global Step: 144005 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:24:42,803-Speed 130.53 samples/sec Loss 3.2641 Epoch: 14 Global Step: 144010 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:25:02,293-Speed 131.36 samples/sec Loss 3.3573 Epoch: 14 Global Step: 144015 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:25:21,894-Speed 130.61 samples/sec Loss 3.4364 Epoch: 14 Global Step: 144020 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:25:41,483-Speed 130.70 samples/sec Loss 3.4608 Epoch: 14 Global Step: 144025 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:26:01,006-Speed 131.14 samples/sec Loss 3.4150 Epoch: 14 Global Step: 144030 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:26:20,568-Speed 130.87 samples/sec Loss 3.3226 Epoch: 14 Global Step: 144035 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:26:40,400-Speed 129.09 samples/sec Loss 3.4013 Epoch: 14 Global Step: 144040 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:26:59,825-Speed 131.80 samples/sec Loss 3.4673 Epoch: 14 Global Step: 144045 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:27:19,528-Speed 129.93 samples/sec Loss 3.3688 Epoch: 14 Global Step: 144050 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:27:39,120-Speed 130.67 samples/sec Loss 3.2565 Epoch: 14 Global Step: 144055 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:27:58,548-Speed 131.78 samples/sec Loss 3.6378 Epoch: 14 Global Step: 144060 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:28:18,160-Speed 130.54 samples/sec Loss 3.3626 Epoch: 14 Global Step: 144065 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:28:37,775-Speed 130.51 samples/sec Loss 3.2373 Epoch: 14 Global Step: 144070 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:28:57,492-Speed 129.85 samples/sec Loss 3.3879 Epoch: 14 Global Step: 144075 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:29:17,026-Speed 131.06 samples/sec Loss 3.3564 Epoch: 14 Global Step: 144080 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:29:36,828-Speed 129.29 samples/sec Loss 3.3947 Epoch: 14 Global Step: 144085 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:29:56,363-Speed 131.05 samples/sec Loss 3.2486 Epoch: 14 Global Step: 144090 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:30:16,063-Speed 129.96 samples/sec Loss 3.3552 Epoch: 14 Global Step: 144095 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:30:35,619-Speed 130.91 samples/sec Loss 3.3342 Epoch: 14 Global Step: 144100 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:30:55,220-Speed 130.61 samples/sec Loss 3.3519 Epoch: 14 Global Step: 144105 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:31:14,825-Speed 130.59 samples/sec Loss 3.3391 Epoch: 14 Global Step: 144110 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:31:34,412-Speed 130.70 samples/sec Loss 3.2894 Epoch: 14 Global Step: 144115 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:31:54,019-Speed 130.58 samples/sec Loss 3.3007 Epoch: 14 Global Step: 144120 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:32:13,459-Speed 131.69 samples/sec Loss 3.4269 Epoch: 14 Global Step: 144125 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:32:33,274-Speed 129.21 samples/sec Loss 3.2542 Epoch: 14 Global Step: 144130 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:32:53,250-Speed 128.15 samples/sec Loss 3.5872 Epoch: 14 Global Step: 144135 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:33:12,886-Speed 130.38 samples/sec Loss 3.2664 Epoch: 14 Global Step: 144140 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:33:32,362-Speed 131.46 samples/sec Loss 3.2623 Epoch: 14 Global Step: 144145 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:33:51,937-Speed 130.79 samples/sec Loss 3.3253 Epoch: 14 Global Step: 144150 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:34:11,526-Speed 130.69 samples/sec Loss 3.2485 Epoch: 14 Global Step: 144155 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:34:31,255-Speed 129.77 samples/sec Loss 3.3225 Epoch: 14 Global Step: 144160 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:34:50,797-Speed 131.01 samples/sec Loss 3.3252 Epoch: 14 Global Step: 144165 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:35:10,394-Speed 130.64 samples/sec Loss 3.2698 Epoch: 14 Global Step: 144170 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:35:30,169-Speed 129.46 samples/sec Loss 3.3664 Epoch: 14 Global Step: 144175 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:35:49,847-Speed 130.11 samples/sec Loss 3.2855 Epoch: 14 Global Step: 144180 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:36:09,385-Speed 131.03 samples/sec Loss 3.4205 Epoch: 14 Global Step: 144185 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:36:29,043-Speed 130.23 samples/sec Loss 3.4128 Epoch: 14 Global Step: 144190 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:36:48,552-Speed 131.23 samples/sec Loss 3.5550 Epoch: 14 Global Step: 144195 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:37:08,148-Speed 130.64 samples/sec Loss 3.3494 Epoch: 14 Global Step: 144200 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:37:27,692-Speed 130.99 samples/sec Loss 3.3433 Epoch: 14 Global Step: 144205 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:37:47,472-Speed 129.43 samples/sec Loss 3.3469 Epoch: 14 Global Step: 144210 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:38:07,195-Speed 129.80 samples/sec Loss 3.2270 Epoch: 14 Global Step: 144215 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:38:27,156-Speed 128.25 samples/sec Loss 3.3275 Epoch: 14 Global Step: 144220 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:38:46,668-Speed 131.21 samples/sec Loss 3.2056 Epoch: 14 Global Step: 144225 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:39:06,310-Speed 130.34 samples/sec Loss 3.3015 Epoch: 14 Global Step: 144230 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:39:25,878-Speed 130.83 samples/sec Loss 3.2989 Epoch: 14 Global Step: 144235 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:39:45,555-Speed 130.11 samples/sec Loss 3.2850 Epoch: 14 Global Step: 144240 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:40:05,202-Speed 130.31 samples/sec Loss 3.1888 Epoch: 14 Global Step: 144245 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:40:24,742-Speed 131.02 samples/sec Loss 3.3459 Epoch: 14 Global Step: 144250 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:40:44,295-Speed 130.93 samples/sec Loss 3.3052 Epoch: 14 Global Step: 144255 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:41:03,859-Speed 130.86 samples/sec Loss 3.4769 Epoch: 14 Global Step: 144260 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:41:23,036-Speed 133.51 samples/sec Loss 3.3557 Epoch: 14 Global Step: 144265 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:41:42,863-Speed 129.12 samples/sec Loss 3.3122 Epoch: 14 Global Step: 144270 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:42:02,604-Speed 129.69 samples/sec Loss 3.4250 Epoch: 14 Global Step: 144275 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:42:22,181-Speed 130.77 samples/sec Loss 3.3681 Epoch: 14 Global Step: 144280 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:42:41,720-Speed 131.03 samples/sec Loss 3.1785 Epoch: 14 Global Step: 144285 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:43:01,252-Speed 131.07 samples/sec Loss 3.3890 Epoch: 14 Global Step: 144290 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:43:20,789-Speed 131.04 samples/sec Loss 3.5000 Epoch: 14 Global Step: 144295 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:43:40,406-Speed 130.51 samples/sec Loss 3.3632 Epoch: 14 Global Step: 144300 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:44:00,026-Speed 130.49 samples/sec Loss 3.3126 Epoch: 14 Global Step: 144305 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:44:19,932-Speed 128.62 samples/sec Loss 3.4286 Epoch: 14 Global Step: 144310 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:44:39,652-Speed 129.82 samples/sec Loss 3.3796 Epoch: 14 Global Step: 144315 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:44:59,339-Speed 130.04 samples/sec Loss 3.4102 Epoch: 14 Global Step: 144320 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:45:19,009-Speed 130.15 samples/sec Loss 3.4432 Epoch: 14 Global Step: 144325 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:45:38,619-Speed 130.55 samples/sec Loss 3.4501 Epoch: 14 Global Step: 144330 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:45:58,127-Speed 131.23 samples/sec Loss 3.2038 Epoch: 14 Global Step: 144335 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:46:17,774-Speed 130.31 samples/sec Loss 3.3514 Epoch: 14 Global Step: 144340 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:46:37,293-Speed 131.17 samples/sec Loss 3.4409 Epoch: 14 Global Step: 144345 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:46:56,913-Speed 130.48 samples/sec Loss 3.5554 Epoch: 14 Global Step: 144350 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:47:16,710-Speed 129.33 samples/sec Loss 3.3286 Epoch: 14 Global Step: 144355 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:47:36,390-Speed 130.09 samples/sec Loss 3.3001 Epoch: 14 Global Step: 144360 Fp16 Grad Scale: 4096 Required: 124 hours Training: 2021-06-13 21:47:55,898-Speed 131.23 samples/sec Loss 3.4088 Epoch: 14 Global Step: 144365 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:48:15,544-Speed 130.31 samples/sec Loss 3.2761 Epoch: 14 Global Step: 144370 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:48:35,111-Speed 130.84 samples/sec Loss 3.6499 Epoch: 14 Global Step: 144375 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:48:54,672-Speed 130.88 samples/sec Loss 3.2641 Epoch: 14 Global Step: 144380 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:49:14,274-Speed 130.60 samples/sec Loss 3.2304 Epoch: 14 Global Step: 144385 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:49:33,807-Speed 131.07 samples/sec Loss 3.2221 Epoch: 14 Global Step: 144390 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:49:53,465-Speed 130.23 samples/sec Loss 3.4155 Epoch: 14 Global Step: 144395 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:50:13,203-Speed 129.71 samples/sec Loss 3.1362 Epoch: 14 Global Step: 144400 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:50:32,917-Speed 129.86 samples/sec Loss 3.4128 Epoch: 14 Global Step: 144405 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:50:52,449-Speed 131.07 samples/sec Loss 3.3697 Epoch: 14 Global Step: 144410 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:51:12,083-Speed 130.39 samples/sec Loss 3.3825 Epoch: 14 Global Step: 144415 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:51:31,612-Speed 131.10 samples/sec Loss 3.4832 Epoch: 14 Global Step: 144420 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:51:52,309-Speed 123.69 samples/sec Loss 3.4199 Epoch: 14 Global Step: 144425 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:52:11,903-Speed 130.66 samples/sec Loss 3.6070 Epoch: 14 Global Step: 144430 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:52:31,517-Speed 130.53 samples/sec Loss 3.3434 Epoch: 14 Global Step: 144435 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:52:51,110-Speed 130.67 samples/sec Loss 3.3579 Epoch: 14 Global Step: 144440 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:53:10,798-Speed 130.04 samples/sec Loss 3.5018 Epoch: 14 Global Step: 144445 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:53:30,367-Speed 130.83 samples/sec Loss 3.4551 Epoch: 14 Global Step: 144450 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:53:50,150-Speed 129.41 samples/sec Loss 3.3397 Epoch: 14 Global Step: 144455 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:54:09,753-Speed 130.60 samples/sec Loss 3.3786 Epoch: 14 Global Step: 144460 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:54:29,381-Speed 130.44 samples/sec Loss 3.2125 Epoch: 14 Global Step: 144465 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:54:49,002-Speed 130.48 samples/sec Loss 3.4986 Epoch: 14 Global Step: 144470 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:55:09,143-Speed 127.11 samples/sec Loss 3.3989 Epoch: 14 Global Step: 144475 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:55:28,740-Speed 130.64 samples/sec Loss 3.2800 Epoch: 14 Global Step: 144480 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:55:48,310-Speed 130.82 samples/sec Loss 3.3896 Epoch: 14 Global Step: 144485 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:56:08,296-Speed 128.10 samples/sec Loss 3.4665 Epoch: 14 Global Step: 144490 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:56:27,737-Speed 131.68 samples/sec Loss 3.4052 Epoch: 14 Global Step: 144495 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:56:47,352-Speed 130.52 samples/sec Loss 3.2360 Epoch: 14 Global Step: 144500 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:57:06,929-Speed 130.78 samples/sec Loss 3.3125 Epoch: 14 Global Step: 144505 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:57:26,998-Speed 127.56 samples/sec Loss 3.2419 Epoch: 14 Global Step: 144510 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:57:46,770-Speed 129.49 samples/sec Loss 3.5069 Epoch: 14 Global Step: 144515 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:58:06,362-Speed 130.67 samples/sec Loss 3.4067 Epoch: 14 Global Step: 144520 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:58:25,964-Speed 130.61 samples/sec Loss 3.5470 Epoch: 14 Global Step: 144525 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:58:45,449-Speed 131.39 samples/sec Loss 3.3116 Epoch: 14 Global Step: 144530 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:59:05,350-Speed 128.65 samples/sec Loss 3.2931 Epoch: 14 Global Step: 144535 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:59:24,785-Speed 131.73 samples/sec Loss 3.5241 Epoch: 14 Global Step: 144540 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 21:59:44,558-Speed 129.47 samples/sec Loss 3.4004 Epoch: 14 Global Step: 144545 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:00:04,094-Speed 131.05 samples/sec Loss 3.4694 Epoch: 14 Global Step: 144550 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:00:23,836-Speed 129.68 samples/sec Loss 3.5549 Epoch: 14 Global Step: 144555 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:00:43,407-Speed 130.81 samples/sec Loss 3.3415 Epoch: 14 Global Step: 144560 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:01:03,030-Speed 130.47 samples/sec Loss 3.3679 Epoch: 14 Global Step: 144565 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:01:22,624-Speed 130.66 samples/sec Loss 3.3960 Epoch: 14 Global Step: 144570 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:01:42,161-Speed 131.04 samples/sec Loss 3.1654 Epoch: 14 Global Step: 144575 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:02:01,985-Speed 129.15 samples/sec Loss 3.3069 Epoch: 14 Global Step: 144580 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:02:21,457-Speed 131.48 samples/sec Loss 3.2104 Epoch: 14 Global Step: 144585 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:02:41,131-Speed 130.13 samples/sec Loss 3.4295 Epoch: 14 Global Step: 144590 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:03:00,687-Speed 130.91 samples/sec Loss 3.0278 Epoch: 14 Global Step: 144595 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:03:20,307-Speed 130.49 samples/sec Loss 3.3818 Epoch: 14 Global Step: 144600 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:03:39,898-Speed 130.68 samples/sec Loss 3.4890 Epoch: 14 Global Step: 144605 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:03:59,380-Speed 131.42 samples/sec Loss 3.4044 Epoch: 14 Global Step: 144610 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:04:19,072-Speed 130.00 samples/sec Loss 3.3282 Epoch: 14 Global Step: 144615 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:04:38,762-Speed 130.02 samples/sec Loss 3.5106 Epoch: 14 Global Step: 144620 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:04:58,580-Speed 129.18 samples/sec Loss 3.2332 Epoch: 14 Global Step: 144625 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:05:18,439-Speed 128.92 samples/sec Loss 3.3226 Epoch: 14 Global Step: 144630 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:05:37,986-Speed 130.97 samples/sec Loss 3.0684 Epoch: 14 Global Step: 144635 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:05:57,572-Speed 130.71 samples/sec Loss 3.5629 Epoch: 14 Global Step: 144640 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:06:17,136-Speed 130.86 samples/sec Loss 3.4273 Epoch: 14 Global Step: 144645 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:06:36,767-Speed 130.42 samples/sec Loss 3.3268 Epoch: 14 Global Step: 144650 Fp16 Grad Scale: 8192 Required: 124 hours Training: 2021-06-13 22:06:56,368-Speed 130.61 samples/sec Loss 3.4801 Epoch: 14 Global Step: 144655 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:07:15,902-Speed 131.06 samples/sec Loss 3.2648 Epoch: 14 Global Step: 144660 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:07:35,456-Speed 130.93 samples/sec Loss 3.1233 Epoch: 14 Global Step: 144665 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:07:55,230-Speed 129.47 samples/sec Loss 3.3892 Epoch: 14 Global Step: 144670 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:08:14,899-Speed 130.16 samples/sec Loss 3.4223 Epoch: 14 Global Step: 144675 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:08:34,340-Speed 131.69 samples/sec Loss 3.5312 Epoch: 14 Global Step: 144680 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:08:53,894-Speed 130.93 samples/sec Loss 3.6508 Epoch: 14 Global Step: 144685 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:09:13,508-Speed 130.53 samples/sec Loss 3.3522 Epoch: 14 Global Step: 144690 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:09:32,996-Speed 131.38 samples/sec Loss 3.2577 Epoch: 14 Global Step: 144695 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:09:52,604-Speed 130.57 samples/sec Loss 3.0850 Epoch: 14 Global Step: 144700 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:10:12,132-Speed 131.10 samples/sec Loss 3.5235 Epoch: 14 Global Step: 144705 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:10:31,705-Speed 130.80 samples/sec Loss 3.4285 Epoch: 14 Global Step: 144710 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:10:51,608-Speed 128.63 samples/sec Loss 3.3366 Epoch: 14 Global Step: 144715 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:11:11,225-Speed 130.51 samples/sec Loss 3.2987 Epoch: 14 Global Step: 144720 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:11:30,802-Speed 130.77 samples/sec Loss 3.2431 Epoch: 14 Global Step: 144725 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:11:50,315-Speed 131.20 samples/sec Loss 3.3749 Epoch: 14 Global Step: 144730 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:12:09,963-Speed 130.30 samples/sec Loss 3.4251 Epoch: 14 Global Step: 144735 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:12:29,640-Speed 130.11 samples/sec Loss 3.3754 Epoch: 14 Global Step: 144740 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:12:49,243-Speed 130.60 samples/sec Loss 3.4074 Epoch: 14 Global Step: 144745 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:13:08,892-Speed 130.30 samples/sec Loss 3.5505 Epoch: 14 Global Step: 144750 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:13:28,478-Speed 130.71 samples/sec Loss 3.2934 Epoch: 14 Global Step: 144755 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:13:48,269-Speed 129.36 samples/sec Loss 3.2726 Epoch: 14 Global Step: 144760 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:14:07,979-Speed 129.89 samples/sec Loss 3.3628 Epoch: 14 Global Step: 144765 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:14:27,524-Speed 130.99 samples/sec Loss 3.2463 Epoch: 14 Global Step: 144770 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:14:47,032-Speed 131.23 samples/sec Loss 3.3612 Epoch: 14 Global Step: 144775 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:15:06,557-Speed 131.13 samples/sec Loss 3.3079 Epoch: 14 Global Step: 144780 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:15:26,081-Speed 131.12 samples/sec Loss 3.1786 Epoch: 14 Global Step: 144785 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:15:45,717-Speed 130.38 samples/sec Loss 3.5942 Epoch: 14 Global Step: 144790 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:16:05,270-Speed 130.93 samples/sec Loss 3.3274 Epoch: 14 Global Step: 144795 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:16:24,864-Speed 130.66 samples/sec Loss 3.4821 Epoch: 14 Global Step: 144800 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:16:44,664-Speed 129.30 samples/sec Loss 3.3050 Epoch: 14 Global Step: 144805 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:17:04,208-Speed 130.99 samples/sec Loss 3.4448 Epoch: 14 Global Step: 144810 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:17:23,756-Speed 130.97 samples/sec Loss 3.2162 Epoch: 14 Global Step: 144815 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:17:43,603-Speed 128.99 samples/sec Loss 3.4549 Epoch: 14 Global Step: 144820 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:18:03,050-Speed 131.65 samples/sec Loss 3.4871 Epoch: 14 Global Step: 144825 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:18:22,639-Speed 130.69 samples/sec Loss 3.4290 Epoch: 14 Global Step: 144830 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:18:42,219-Speed 130.75 samples/sec Loss 3.4893 Epoch: 14 Global Step: 144835 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:19:01,743-Speed 131.12 samples/sec Loss 3.3015 Epoch: 14 Global Step: 144840 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:19:21,344-Speed 130.62 samples/sec Loss 3.3011 Epoch: 14 Global Step: 144845 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:19:41,233-Speed 128.72 samples/sec Loss 3.3731 Epoch: 14 Global Step: 144850 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:20:00,680-Speed 131.64 samples/sec Loss 3.5462 Epoch: 14 Global Step: 144855 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:20:20,288-Speed 130.57 samples/sec Loss 3.5504 Epoch: 14 Global Step: 144860 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:20:39,884-Speed 130.65 samples/sec Loss 3.3776 Epoch: 14 Global Step: 144865 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:20:59,427-Speed 131.00 samples/sec Loss 3.3065 Epoch: 14 Global Step: 144870 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:21:19,027-Speed 130.63 samples/sec Loss 3.3921 Epoch: 14 Global Step: 144875 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:21:38,703-Speed 130.11 samples/sec Loss 3.4207 Epoch: 14 Global Step: 144880 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:21:58,311-Speed 130.57 samples/sec Loss 3.3845 Epoch: 14 Global Step: 144885 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:22:17,805-Speed 131.33 samples/sec Loss 3.4221 Epoch: 14 Global Step: 144890 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:22:37,605-Speed 129.30 samples/sec Loss 3.1779 Epoch: 14 Global Step: 144895 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:22:57,101-Speed 131.32 samples/sec Loss 3.3770 Epoch: 14 Global Step: 144900 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:23:16,838-Speed 129.71 samples/sec Loss 3.5366 Epoch: 14 Global Step: 144905 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:23:36,347-Speed 131.23 samples/sec Loss 3.3062 Epoch: 14 Global Step: 144910 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:23:55,949-Speed 130.61 samples/sec Loss 3.3864 Epoch: 14 Global Step: 144915 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:24:15,441-Speed 131.34 samples/sec Loss 3.4616 Epoch: 14 Global Step: 144920 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:24:34,978-Speed 131.04 samples/sec Loss 3.4311 Epoch: 14 Global Step: 144925 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:24:54,586-Speed 130.57 samples/sec Loss 3.3917 Epoch: 14 Global Step: 144930 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:25:14,123-Speed 131.04 samples/sec Loss 3.5295 Epoch: 14 Global Step: 144935 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:25:33,910-Speed 129.38 samples/sec Loss 3.3967 Epoch: 14 Global Step: 144940 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:25:53,478-Speed 130.83 samples/sec Loss 3.4763 Epoch: 14 Global Step: 144945 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:26:13,101-Speed 130.47 samples/sec Loss 3.4995 Epoch: 14 Global Step: 144950 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:26:32,680-Speed 130.76 samples/sec Loss 3.3808 Epoch: 14 Global Step: 144955 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:26:52,262-Speed 130.74 samples/sec Loss 3.3164 Epoch: 14 Global Step: 144960 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:27:11,903-Speed 130.36 samples/sec Loss 3.4810 Epoch: 14 Global Step: 144965 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:27:31,709-Speed 129.26 samples/sec Loss 3.3778 Epoch: 14 Global Step: 144970 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:27:51,305-Speed 130.65 samples/sec Loss 3.4772 Epoch: 14 Global Step: 144975 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:28:10,852-Speed 130.97 samples/sec Loss 3.2866 Epoch: 14 Global Step: 144980 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:28:30,655-Speed 129.28 samples/sec Loss 3.4436 Epoch: 14 Global Step: 144985 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:28:50,369-Speed 129.86 samples/sec Loss 3.4636 Epoch: 14 Global Step: 144990 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:29:09,811-Speed 131.68 samples/sec Loss 3.4017 Epoch: 14 Global Step: 144995 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:29:29,347-Speed 131.04 samples/sec Loss 3.4179 Epoch: 14 Global Step: 145000 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:29:48,917-Speed 130.82 samples/sec Loss 3.1685 Epoch: 14 Global Step: 145005 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:30:08,668-Speed 129.62 samples/sec Loss 3.3129 Epoch: 14 Global Step: 145010 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:30:28,260-Speed 130.67 samples/sec Loss 3.2922 Epoch: 14 Global Step: 145015 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:30:47,800-Speed 131.02 samples/sec Loss 3.5352 Epoch: 14 Global Step: 145020 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:31:07,339-Speed 131.03 samples/sec Loss 3.2811 Epoch: 14 Global Step: 145025 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:31:27,154-Speed 129.20 samples/sec Loss 3.5953 Epoch: 14 Global Step: 145030 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:31:46,707-Speed 130.94 samples/sec Loss 3.2178 Epoch: 14 Global Step: 145035 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:32:06,333-Speed 130.44 samples/sec Loss 3.4297 Epoch: 14 Global Step: 145040 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:32:25,890-Speed 130.91 samples/sec Loss 3.3304 Epoch: 14 Global Step: 145045 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:32:45,486-Speed 130.65 samples/sec Loss 3.2899 Epoch: 14 Global Step: 145050 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:33:05,013-Speed 131.11 samples/sec Loss 3.3428 Epoch: 14 Global Step: 145055 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:33:24,608-Speed 130.65 samples/sec Loss 2.9631 Epoch: 14 Global Step: 145060 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:33:44,136-Speed 131.11 samples/sec Loss 3.3996 Epoch: 14 Global Step: 145065 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:34:03,688-Speed 130.94 samples/sec Loss 3.6619 Epoch: 14 Global Step: 145070 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:34:23,536-Speed 128.99 samples/sec Loss 3.1952 Epoch: 14 Global Step: 145075 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:34:43,295-Speed 129.57 samples/sec Loss 3.4208 Epoch: 14 Global Step: 145080 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:35:02,743-Speed 131.64 samples/sec Loss 3.3333 Epoch: 14 Global Step: 145085 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:35:22,286-Speed 131.00 samples/sec Loss 3.2821 Epoch: 14 Global Step: 145090 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:35:41,864-Speed 130.76 samples/sec Loss 3.4064 Epoch: 14 Global Step: 145095 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:36:01,452-Speed 130.70 samples/sec Loss 3.4434 Epoch: 14 Global Step: 145100 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:36:21,101-Speed 130.30 samples/sec Loss 3.3928 Epoch: 14 Global Step: 145105 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:36:41,013-Speed 128.57 samples/sec Loss 3.3247 Epoch: 14 Global Step: 145110 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:37:00,617-Speed 130.59 samples/sec Loss 3.3247 Epoch: 14 Global Step: 145115 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:37:20,456-Speed 129.05 samples/sec Loss 3.2801 Epoch: 14 Global Step: 145120 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:37:40,243-Speed 129.38 samples/sec Loss 3.4777 Epoch: 14 Global Step: 145125 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:37:59,775-Speed 131.07 samples/sec Loss 3.6867 Epoch: 14 Global Step: 145130 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:38:19,410-Speed 130.39 samples/sec Loss 3.4029 Epoch: 14 Global Step: 145135 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:38:38,932-Speed 131.14 samples/sec Loss 3.5204 Epoch: 14 Global Step: 145140 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:38:58,539-Speed 130.57 samples/sec Loss 3.3171 Epoch: 14 Global Step: 145145 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:39:17,994-Speed 131.59 samples/sec Loss 3.4418 Epoch: 14 Global Step: 145150 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:39:37,558-Speed 130.86 samples/sec Loss 3.3920 Epoch: 14 Global Step: 145155 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:39:57,350-Speed 129.36 samples/sec Loss 3.4640 Epoch: 14 Global Step: 145160 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:40:17,254-Speed 128.63 samples/sec Loss 3.1748 Epoch: 14 Global Step: 145165 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:40:36,874-Speed 130.49 samples/sec Loss 3.3885 Epoch: 14 Global Step: 145170 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:40:56,967-Speed 127.41 samples/sec Loss 3.2737 Epoch: 14 Global Step: 145175 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:41:16,647-Speed 130.09 samples/sec Loss 3.4348 Epoch: 14 Global Step: 145180 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:41:36,304-Speed 130.24 samples/sec Loss 3.3906 Epoch: 14 Global Step: 145185 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:41:55,905-Speed 130.61 samples/sec Loss 3.5853 Epoch: 14 Global Step: 145190 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:42:15,462-Speed 130.91 samples/sec Loss 3.2555 Epoch: 14 Global Step: 145195 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:42:35,000-Speed 131.04 samples/sec Loss 3.3942 Epoch: 14 Global Step: 145200 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:42:54,611-Speed 130.55 samples/sec Loss 3.4125 Epoch: 14 Global Step: 145205 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:43:14,578-Speed 128.22 samples/sec Loss 3.2667 Epoch: 14 Global Step: 145210 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:43:34,017-Speed 131.71 samples/sec Loss 3.3900 Epoch: 14 Global Step: 145215 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:43:53,616-Speed 130.63 samples/sec Loss 3.2952 Epoch: 14 Global Step: 145220 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:44:13,184-Speed 130.83 samples/sec Loss 3.4884 Epoch: 14 Global Step: 145225 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:44:32,915-Speed 129.75 samples/sec Loss 3.4581 Epoch: 14 Global Step: 145230 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:44:52,518-Speed 130.59 samples/sec Loss 3.2854 Epoch: 14 Global Step: 145235 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:45:12,065-Speed 130.98 samples/sec Loss 3.5240 Epoch: 14 Global Step: 145240 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:45:31,592-Speed 131.11 samples/sec Loss 3.3877 Epoch: 14 Global Step: 145245 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:45:51,162-Speed 130.82 samples/sec Loss 3.5519 Epoch: 14 Global Step: 145250 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:46:11,066-Speed 128.63 samples/sec Loss 3.5216 Epoch: 14 Global Step: 145255 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:46:30,498-Speed 131.75 samples/sec Loss 3.4361 Epoch: 14 Global Step: 145260 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:46:50,079-Speed 130.74 samples/sec Loss 3.4596 Epoch: 14 Global Step: 145265 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:47:09,676-Speed 130.64 samples/sec Loss 3.4084 Epoch: 14 Global Step: 145270 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:47:29,339-Speed 130.20 samples/sec Loss 3.5075 Epoch: 14 Global Step: 145275 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:47:48,989-Speed 130.28 samples/sec Loss 3.2732 Epoch: 14 Global Step: 145280 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:48:08,528-Speed 131.03 samples/sec Loss 3.2376 Epoch: 14 Global Step: 145285 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:48:28,148-Speed 130.49 samples/sec Loss 3.5747 Epoch: 14 Global Step: 145290 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:48:47,751-Speed 130.60 samples/sec Loss 3.4197 Epoch: 14 Global Step: 145295 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:49:07,656-Speed 128.62 samples/sec Loss 3.3684 Epoch: 14 Global Step: 145300 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:49:27,077-Speed 131.82 samples/sec Loss 3.3229 Epoch: 14 Global Step: 145305 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:49:46,723-Speed 130.31 samples/sec Loss 3.6518 Epoch: 14 Global Step: 145310 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:50:06,333-Speed 130.56 samples/sec Loss 3.3398 Epoch: 14 Global Step: 145315 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:50:25,806-Speed 131.47 samples/sec Loss 3.3735 Epoch: 14 Global Step: 145320 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:50:45,478-Speed 130.15 samples/sec Loss 3.3826 Epoch: 14 Global Step: 145325 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:51:04,942-Speed 131.53 samples/sec Loss 3.3111 Epoch: 14 Global Step: 145330 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:51:24,548-Speed 130.58 samples/sec Loss 3.4127 Epoch: 14 Global Step: 145335 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:51:44,107-Speed 130.90 samples/sec Loss 3.2270 Epoch: 14 Global Step: 145340 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:52:03,979-Speed 128.83 samples/sec Loss 3.3415 Epoch: 14 Global Step: 145345 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:52:23,415-Speed 131.72 samples/sec Loss 3.4398 Epoch: 14 Global Step: 145350 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:52:43,021-Speed 130.58 samples/sec Loss 3.4382 Epoch: 14 Global Step: 145355 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:53:02,581-Speed 130.88 samples/sec Loss 3.3791 Epoch: 14 Global Step: 145360 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:53:22,126-Speed 130.99 samples/sec Loss 3.3345 Epoch: 14 Global Step: 145365 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:53:41,734-Speed 130.57 samples/sec Loss 3.5740 Epoch: 14 Global Step: 145370 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:54:01,363-Speed 130.42 samples/sec Loss 3.2477 Epoch: 14 Global Step: 145375 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:54:20,964-Speed 130.61 samples/sec Loss 3.4990 Epoch: 14 Global Step: 145380 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:54:40,510-Speed 130.98 samples/sec Loss 3.4482 Epoch: 14 Global Step: 145385 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:55:00,310-Speed 129.30 samples/sec Loss 3.5037 Epoch: 14 Global Step: 145390 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:55:19,878-Speed 130.83 samples/sec Loss 3.3913 Epoch: 14 Global Step: 145395 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:55:39,620-Speed 129.68 samples/sec Loss 3.3510 Epoch: 14 Global Step: 145400 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:55:59,135-Speed 131.19 samples/sec Loss 3.4573 Epoch: 14 Global Step: 145405 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:56:18,900-Speed 129.53 samples/sec Loss 3.4180 Epoch: 14 Global Step: 145410 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:56:38,457-Speed 130.90 samples/sec Loss 3.1672 Epoch: 14 Global Step: 145415 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:56:57,991-Speed 131.06 samples/sec Loss 3.2098 Epoch: 14 Global Step: 145420 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:57:17,593-Speed 130.61 samples/sec Loss 3.4439 Epoch: 14 Global Step: 145425 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:57:37,147-Speed 130.93 samples/sec Loss 3.5850 Epoch: 14 Global Step: 145430 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:57:56,938-Speed 129.36 samples/sec Loss 3.1859 Epoch: 14 Global Step: 145435 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:58:16,383-Speed 131.66 samples/sec Loss 3.4716 Epoch: 14 Global Step: 145440 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:58:36,037-Speed 130.26 samples/sec Loss 3.2484 Epoch: 14 Global Step: 145445 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:58:55,560-Speed 131.13 samples/sec Loss 3.6355 Epoch: 14 Global Step: 145450 Fp16 Grad Scale: 8192 Required: 123 hours Training: 2021-06-13 22:59:15,150-Speed 130.69 samples/sec Loss 3.5005 Epoch: 14 Global Step: 145455 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 22:59:34,586-Speed 131.72 samples/sec Loss 3.3783 Epoch: 14 Global Step: 145460 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 22:59:54,110-Speed 131.13 samples/sec Loss 3.3827 Epoch: 14 Global Step: 145465 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:00:13,690-Speed 130.75 samples/sec Loss 3.6120 Epoch: 14 Global Step: 145470 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:00:33,237-Speed 130.98 samples/sec Loss 3.2054 Epoch: 14 Global Step: 145475 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:00:53,167-Speed 128.46 samples/sec Loss 3.4484 Epoch: 14 Global Step: 145480 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:01:12,752-Speed 130.72 samples/sec Loss 3.4200 Epoch: 14 Global Step: 145485 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:01:32,489-Speed 129.71 samples/sec Loss 3.3440 Epoch: 14 Global Step: 145490 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:01:51,990-Speed 131.29 samples/sec Loss 3.2816 Epoch: 14 Global Step: 145495 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:02:11,589-Speed 130.62 samples/sec Loss 3.4658 Epoch: 14 Global Step: 145500 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:02:31,141-Speed 130.95 samples/sec Loss 3.5040 Epoch: 14 Global Step: 145505 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:02:50,895-Speed 129.60 samples/sec Loss 3.4263 Epoch: 14 Global Step: 145510 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:03:10,453-Speed 130.90 samples/sec Loss 3.4193 Epoch: 14 Global Step: 145515 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:03:30,063-Speed 130.55 samples/sec Loss 3.5004 Epoch: 14 Global Step: 145520 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:03:49,854-Speed 129.36 samples/sec Loss 3.2923 Epoch: 14 Global Step: 145525 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:04:09,617-Speed 129.55 samples/sec Loss 3.4413 Epoch: 14 Global Step: 145530 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:04:29,153-Speed 131.04 samples/sec Loss 3.4483 Epoch: 14 Global Step: 145535 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:04:48,686-Speed 131.07 samples/sec Loss 3.6864 Epoch: 14 Global Step: 145540 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:05:08,254-Speed 130.83 samples/sec Loss 3.2260 Epoch: 14 Global Step: 145545 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:05:27,841-Speed 130.71 samples/sec Loss 3.4825 Epoch: 14 Global Step: 145550 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:05:47,445-Speed 130.59 samples/sec Loss 3.5857 Epoch: 14 Global Step: 145555 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:06:06,961-Speed 131.18 samples/sec Loss 3.3253 Epoch: 14 Global Step: 145560 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:06:26,580-Speed 130.49 samples/sec Loss 3.5718 Epoch: 14 Global Step: 145565 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:06:46,337-Speed 129.59 samples/sec Loss 3.4410 Epoch: 14 Global Step: 145570 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:07:06,109-Speed 129.48 samples/sec Loss 3.2556 Epoch: 14 Global Step: 145575 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:07:25,735-Speed 130.45 samples/sec Loss 3.2974 Epoch: 14 Global Step: 145580 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:07:45,337-Speed 130.60 samples/sec Loss 3.3559 Epoch: 14 Global Step: 145585 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:08:04,872-Speed 131.05 samples/sec Loss 3.3919 Epoch: 14 Global Step: 145590 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:08:24,480-Speed 130.57 samples/sec Loss 3.2011 Epoch: 14 Global Step: 145595 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:08:44,056-Speed 130.78 samples/sec Loss 3.4805 Epoch: 14 Global Step: 145600 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:09:03,513-Speed 131.58 samples/sec Loss 3.3378 Epoch: 14 Global Step: 145605 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:09:23,119-Speed 130.58 samples/sec Loss 3.4691 Epoch: 14 Global Step: 145610 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:09:43,005-Speed 128.74 samples/sec Loss 3.2904 Epoch: 14 Global Step: 145615 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:10:02,490-Speed 131.39 samples/sec Loss 3.6771 Epoch: 14 Global Step: 145620 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:10:22,027-Speed 131.04 samples/sec Loss 3.3410 Epoch: 14 Global Step: 145625 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:10:41,629-Speed 130.61 samples/sec Loss 3.3658 Epoch: 14 Global Step: 145630 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:11:01,225-Speed 130.65 samples/sec Loss 3.4495 Epoch: 14 Global Step: 145635 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:11:20,845-Speed 130.49 samples/sec Loss 3.3055 Epoch: 14 Global Step: 145640 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:11:40,288-Speed 131.67 samples/sec Loss 3.3781 Epoch: 14 Global Step: 145645 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:11:59,833-Speed 130.99 samples/sec Loss 3.3665 Epoch: 14 Global Step: 145650 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:12:19,431-Speed 130.63 samples/sec Loss 3.6210 Epoch: 14 Global Step: 145655 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:12:39,402-Speed 128.19 samples/sec Loss 3.5873 Epoch: 14 Global Step: 145660 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:12:58,868-Speed 131.52 samples/sec Loss 3.3900 Epoch: 14 Global Step: 145665 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:13:18,397-Speed 131.10 samples/sec Loss 3.6611 Epoch: 14 Global Step: 145670 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:13:37,949-Speed 130.94 samples/sec Loss 3.3670 Epoch: 14 Global Step: 145675 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:13:57,548-Speed 130.62 samples/sec Loss 3.4253 Epoch: 14 Global Step: 145680 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:14:17,151-Speed 130.60 samples/sec Loss 3.3452 Epoch: 14 Global Step: 145685 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:14:36,591-Speed 131.70 samples/sec Loss 3.4337 Epoch: 14 Global Step: 145690 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:14:56,180-Speed 130.69 samples/sec Loss 3.3312 Epoch: 14 Global Step: 145695 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:15:15,718-Speed 131.03 samples/sec Loss 3.3652 Epoch: 14 Global Step: 145700 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:15:35,628-Speed 128.58 samples/sec Loss 3.2926 Epoch: 14 Global Step: 145705 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:15:55,036-Speed 131.91 samples/sec Loss 3.4736 Epoch: 14 Global Step: 145710 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:16:14,847-Speed 129.23 samples/sec Loss 3.6651 Epoch: 14 Global Step: 145715 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:16:34,468-Speed 130.48 samples/sec Loss 3.3873 Epoch: 14 Global Step: 145720 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:16:54,000-Speed 131.08 samples/sec Loss 3.3191 Epoch: 14 Global Step: 145725 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:17:13,636-Speed 130.38 samples/sec Loss 3.4909 Epoch: 14 Global Step: 145730 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:17:33,210-Speed 130.79 samples/sec Loss 3.4522 Epoch: 14 Global Step: 145735 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:17:52,819-Speed 130.56 samples/sec Loss 3.3978 Epoch: 14 Global Step: 145740 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:18:12,433-Speed 130.52 samples/sec Loss 3.2103 Epoch: 14 Global Step: 145745 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:18:32,325-Speed 128.70 samples/sec Loss 3.5539 Epoch: 14 Global Step: 145750 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:18:51,758-Speed 131.74 samples/sec Loss 3.2457 Epoch: 14 Global Step: 145755 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:19:11,474-Speed 129.85 samples/sec Loss 3.3724 Epoch: 14 Global Step: 145760 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:19:31,004-Speed 131.09 samples/sec Loss 3.3122 Epoch: 14 Global Step: 145765 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:19:50,527-Speed 131.13 samples/sec Loss 3.3596 Epoch: 14 Global Step: 145770 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:20:10,080-Speed 130.94 samples/sec Loss 3.3758 Epoch: 14 Global Step: 145775 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:20:29,525-Speed 131.66 samples/sec Loss 3.3925 Epoch: 14 Global Step: 145780 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:20:49,132-Speed 130.57 samples/sec Loss 3.3555 Epoch: 14 Global Step: 145785 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:21:08,680-Speed 130.97 samples/sec Loss 3.4330 Epoch: 14 Global Step: 145790 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:21:28,559-Speed 128.78 samples/sec Loss 3.3643 Epoch: 14 Global Step: 145795 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:21:48,174-Speed 130.52 samples/sec Loss 3.3346 Epoch: 14 Global Step: 145800 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:22:07,774-Speed 130.62 samples/sec Loss 3.3099 Epoch: 14 Global Step: 145805 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:22:27,305-Speed 131.08 samples/sec Loss 3.2840 Epoch: 14 Global Step: 145810 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:22:46,912-Speed 130.57 samples/sec Loss 3.1593 Epoch: 14 Global Step: 145815 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:23:06,447-Speed 131.06 samples/sec Loss 3.5020 Epoch: 14 Global Step: 145820 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:23:25,921-Speed 131.46 samples/sec Loss 3.2221 Epoch: 14 Global Step: 145825 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:23:45,525-Speed 130.59 samples/sec Loss 3.5214 Epoch: 14 Global Step: 145830 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:24:05,076-Speed 130.95 samples/sec Loss 3.5316 Epoch: 14 Global Step: 145835 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:24:24,880-Speed 129.27 samples/sec Loss 3.5248 Epoch: 14 Global Step: 145840 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:24:44,476-Speed 130.65 samples/sec Loss 3.4807 Epoch: 14 Global Step: 145845 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:25:04,356-Speed 128.78 samples/sec Loss 3.4606 Epoch: 14 Global Step: 145850 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:25:23,990-Speed 130.39 samples/sec Loss 3.1999 Epoch: 14 Global Step: 145855 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:25:43,601-Speed 130.55 samples/sec Loss 3.3121 Epoch: 14 Global Step: 145860 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:26:03,091-Speed 131.36 samples/sec Loss 3.4560 Epoch: 14 Global Step: 145865 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:26:22,689-Speed 130.63 samples/sec Loss 3.3244 Epoch: 14 Global Step: 145870 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:26:42,310-Speed 130.48 samples/sec Loss 3.4929 Epoch: 14 Global Step: 145875 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:27:01,810-Speed 131.29 samples/sec Loss 3.2716 Epoch: 14 Global Step: 145880 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:27:21,629-Speed 129.18 samples/sec Loss 3.5089 Epoch: 14 Global Step: 145885 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:27:41,262-Speed 130.40 samples/sec Loss 3.3711 Epoch: 14 Global Step: 145890 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:28:00,789-Speed 131.11 samples/sec Loss 3.4063 Epoch: 14 Global Step: 145895 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:28:20,265-Speed 131.45 samples/sec Loss 3.3505 Epoch: 14 Global Step: 145900 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:28:39,890-Speed 130.46 samples/sec Loss 3.4007 Epoch: 14 Global Step: 145905 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:28:59,440-Speed 130.95 samples/sec Loss 3.3702 Epoch: 14 Global Step: 145910 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:29:19,174-Speed 129.73 samples/sec Loss 3.4128 Epoch: 14 Global Step: 145915 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:29:38,723-Speed 130.97 samples/sec Loss 3.5389 Epoch: 14 Global Step: 145920 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:29:58,153-Speed 131.77 samples/sec Loss 3.4098 Epoch: 14 Global Step: 145925 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:30:17,965-Speed 129.22 samples/sec Loss 3.6448 Epoch: 14 Global Step: 145930 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:30:37,599-Speed 130.40 samples/sec Loss 3.5358 Epoch: 14 Global Step: 145935 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:30:57,223-Speed 130.46 samples/sec Loss 3.6628 Epoch: 14 Global Step: 145940 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:31:16,674-Speed 131.62 samples/sec Loss 3.4350 Epoch: 14 Global Step: 145945 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:31:36,282-Speed 130.57 samples/sec Loss 3.4457 Epoch: 14 Global Step: 145950 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:31:55,803-Speed 131.15 samples/sec Loss 3.3689 Epoch: 14 Global Step: 145955 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:32:15,442-Speed 130.36 samples/sec Loss 3.4665 Epoch: 14 Global Step: 145960 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:32:34,968-Speed 131.12 samples/sec Loss 3.5336 Epoch: 14 Global Step: 145965 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:32:54,603-Speed 130.39 samples/sec Loss 3.3735 Epoch: 14 Global Step: 145970 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:33:14,353-Speed 129.63 samples/sec Loss 3.4045 Epoch: 14 Global Step: 145975 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:33:34,084-Speed 129.75 samples/sec Loss 3.6586 Epoch: 14 Global Step: 145980 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:33:53,612-Speed 131.10 samples/sec Loss 3.5403 Epoch: 14 Global Step: 145985 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:34:13,189-Speed 130.77 samples/sec Loss 3.3595 Epoch: 14 Global Step: 145990 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:34:32,646-Speed 131.58 samples/sec Loss 3.4522 Epoch: 14 Global Step: 145995 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:34:52,271-Speed 130.45 samples/sec Loss 3.3876 Epoch: 14 Global Step: 146000 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:35:11,924-Speed 130.27 samples/sec Loss 3.4989 Epoch: 14 Global Step: 146005 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:35:31,473-Speed 130.96 samples/sec Loss 3.2912 Epoch: 14 Global Step: 146010 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:35:51,073-Speed 130.62 samples/sec Loss 3.1492 Epoch: 14 Global Step: 146015 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:36:10,762-Speed 130.03 samples/sec Loss 3.1237 Epoch: 14 Global Step: 146020 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:36:30,484-Speed 129.81 samples/sec Loss 3.2287 Epoch: 14 Global Step: 146025 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:36:50,020-Speed 131.05 samples/sec Loss 3.4092 Epoch: 14 Global Step: 146030 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:37:09,616-Speed 130.64 samples/sec Loss 3.3304 Epoch: 14 Global Step: 146035 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:37:29,148-Speed 131.07 samples/sec Loss 3.3627 Epoch: 14 Global Step: 146040 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:37:48,769-Speed 130.48 samples/sec Loss 3.4706 Epoch: 14 Global Step: 146045 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:38:08,189-Speed 131.83 samples/sec Loss 3.3487 Epoch: 14 Global Step: 146050 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:38:27,716-Speed 131.11 samples/sec Loss 3.3534 Epoch: 14 Global Step: 146055 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:38:47,339-Speed 130.46 samples/sec Loss 3.6434 Epoch: 14 Global Step: 146060 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:39:07,188-Speed 128.99 samples/sec Loss 3.4472 Epoch: 14 Global Step: 146065 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:39:26,697-Speed 131.23 samples/sec Loss 3.4971 Epoch: 14 Global Step: 146070 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:39:46,214-Speed 131.18 samples/sec Loss 3.4876 Epoch: 14 Global Step: 146075 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:40:05,828-Speed 130.53 samples/sec Loss 3.5381 Epoch: 14 Global Step: 146080 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:40:25,428-Speed 130.62 samples/sec Loss 3.4717 Epoch: 14 Global Step: 146085 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:40:45,042-Speed 130.53 samples/sec Loss 3.3786 Epoch: 14 Global Step: 146090 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:41:04,708-Speed 130.18 samples/sec Loss 3.5893 Epoch: 14 Global Step: 146095 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:41:24,252-Speed 131.00 samples/sec Loss 3.4257 Epoch: 14 Global Step: 146100 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:41:43,830-Speed 130.77 samples/sec Loss 3.3624 Epoch: 14 Global Step: 146105 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:42:03,815-Speed 128.10 samples/sec Loss 3.4160 Epoch: 14 Global Step: 146110 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:42:23,431-Speed 130.52 samples/sec Loss 3.3951 Epoch: 14 Global Step: 146115 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:42:43,189-Speed 129.57 samples/sec Loss 3.4110 Epoch: 14 Global Step: 146120 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:43:02,742-Speed 130.94 samples/sec Loss 3.3477 Epoch: 14 Global Step: 146125 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:43:22,356-Speed 130.52 samples/sec Loss 3.3508 Epoch: 14 Global Step: 146130 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:43:41,953-Speed 130.64 samples/sec Loss 3.4507 Epoch: 14 Global Step: 146135 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:44:01,487-Speed 131.06 samples/sec Loss 3.4674 Epoch: 14 Global Step: 146140 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:44:21,346-Speed 128.91 samples/sec Loss 3.6285 Epoch: 14 Global Step: 146145 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:44:40,868-Speed 131.15 samples/sec Loss 3.2701 Epoch: 14 Global Step: 146150 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:45:01,030-Speed 126.98 samples/sec Loss 3.2000 Epoch: 14 Global Step: 146155 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:45:20,454-Speed 131.80 samples/sec Loss 3.2915 Epoch: 14 Global Step: 146160 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:45:40,005-Speed 130.95 samples/sec Loss 3.2370 Epoch: 14 Global Step: 146165 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:45:59,570-Speed 130.85 samples/sec Loss 3.4470 Epoch: 14 Global Step: 146170 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:46:19,088-Speed 131.16 samples/sec Loss 3.6277 Epoch: 14 Global Step: 146175 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:46:38,875-Speed 129.39 samples/sec Loss 3.5010 Epoch: 14 Global Step: 146180 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:46:58,418-Speed 131.00 samples/sec Loss 3.4677 Epoch: 14 Global Step: 146185 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:47:18,015-Speed 130.64 samples/sec Loss 3.5017 Epoch: 14 Global Step: 146190 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:47:37,620-Speed 130.59 samples/sec Loss 3.5042 Epoch: 14 Global Step: 146195 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:47:57,433-Speed 129.22 samples/sec Loss 3.4516 Epoch: 14 Global Step: 146200 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:48:16,856-Speed 131.81 samples/sec Loss 3.3309 Epoch: 14 Global Step: 146205 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:48:36,646-Speed 129.37 samples/sec Loss 3.3694 Epoch: 14 Global Step: 146210 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:48:56,539-Speed 128.69 samples/sec Loss 3.3209 Epoch: 14 Global Step: 146215 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:49:16,084-Speed 130.98 samples/sec Loss 3.5553 Epoch: 14 Global Step: 146220 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:49:35,657-Speed 130.80 samples/sec Loss 3.5249 Epoch: 14 Global Step: 146225 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:49:55,162-Speed 131.26 samples/sec Loss 3.3966 Epoch: 14 Global Step: 146230 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:50:14,759-Speed 130.64 samples/sec Loss 3.2165 Epoch: 14 Global Step: 146235 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:50:34,387-Speed 130.43 samples/sec Loss 3.4105 Epoch: 14 Global Step: 146240 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:50:54,218-Speed 129.09 samples/sec Loss 3.3265 Epoch: 14 Global Step: 146245 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:51:13,736-Speed 131.17 samples/sec Loss 3.2011 Epoch: 14 Global Step: 146250 Fp16 Grad Scale: 8192 Required: 122 hours Training: 2021-06-13 23:51:33,352-Speed 130.51 samples/sec Loss 3.5834 Epoch: 14 Global Step: 146255 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:51:52,956-Speed 130.60 samples/sec Loss 3.4712 Epoch: 14 Global Step: 146260 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:52:12,561-Speed 130.58 samples/sec Loss 3.3489 Epoch: 14 Global Step: 146265 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:52:32,202-Speed 130.35 samples/sec Loss 3.4617 Epoch: 14 Global Step: 146270 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:52:51,656-Speed 131.60 samples/sec Loss 3.5237 Epoch: 14 Global Step: 146275 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:53:11,251-Speed 130.65 samples/sec Loss 3.5088 Epoch: 14 Global Step: 146280 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:53:30,858-Speed 130.57 samples/sec Loss 3.3898 Epoch: 14 Global Step: 146285 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:53:50,697-Speed 129.04 samples/sec Loss 3.4826 Epoch: 14 Global Step: 146290 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:54:10,239-Speed 131.01 samples/sec Loss 3.4840 Epoch: 14 Global Step: 146295 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:54:29,805-Speed 130.85 samples/sec Loss 3.2179 Epoch: 14 Global Step: 146300 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:54:49,405-Speed 130.62 samples/sec Loss 3.4176 Epoch: 14 Global Step: 146305 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:55:09,023-Speed 130.50 samples/sec Loss 3.2680 Epoch: 14 Global Step: 146310 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:55:28,527-Speed 131.26 samples/sec Loss 3.3151 Epoch: 14 Global Step: 146315 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:55:48,043-Speed 131.18 samples/sec Loss 3.4796 Epoch: 14 Global Step: 146320 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:56:07,684-Speed 130.35 samples/sec Loss 3.3538 Epoch: 14 Global Step: 146325 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:56:27,342-Speed 130.23 samples/sec Loss 3.2764 Epoch: 14 Global Step: 146330 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:56:47,150-Speed 129.24 samples/sec Loss 3.4581 Epoch: 14 Global Step: 146335 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:57:06,693-Speed 131.01 samples/sec Loss 3.5133 Epoch: 14 Global Step: 146340 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:57:26,310-Speed 130.51 samples/sec Loss 3.3745 Epoch: 14 Global Step: 146345 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:57:45,866-Speed 130.92 samples/sec Loss 3.2659 Epoch: 14 Global Step: 146350 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:58:05,477-Speed 130.54 samples/sec Loss 3.3495 Epoch: 14 Global Step: 146355 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:58:24,925-Speed 131.64 samples/sec Loss 3.3379 Epoch: 14 Global Step: 146360 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:58:44,533-Speed 130.57 samples/sec Loss 3.1906 Epoch: 14 Global Step: 146365 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:59:04,126-Speed 130.67 samples/sec Loss 3.5293 Epoch: 14 Global Step: 146370 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:59:23,671-Speed 130.99 samples/sec Loss 3.4967 Epoch: 14 Global Step: 146375 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-13 23:59:43,428-Speed 129.58 samples/sec Loss 3.4858 Epoch: 14 Global Step: 146380 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:00:03,033-Speed 130.58 samples/sec Loss 3.5206 Epoch: 14 Global Step: 146385 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:00:22,633-Speed 130.62 samples/sec Loss 3.5523 Epoch: 14 Global Step: 146390 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:00:42,308-Speed 130.12 samples/sec Loss 3.3381 Epoch: 14 Global Step: 146395 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:01:01,754-Speed 131.65 samples/sec Loss 3.4954 Epoch: 14 Global Step: 146400 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:01:21,287-Speed 131.07 samples/sec Loss 3.2668 Epoch: 14 Global Step: 146405 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:01:41,016-Speed 129.76 samples/sec Loss 3.3106 Epoch: 14 Global Step: 146410 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:02:00,621-Speed 130.59 samples/sec Loss 3.4812 Epoch: 14 Global Step: 146415 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:02:20,222-Speed 130.61 samples/sec Loss 3.5467 Epoch: 14 Global Step: 146420 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:02:40,071-Speed 128.99 samples/sec Loss 3.4589 Epoch: 14 Global Step: 146425 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:02:59,525-Speed 131.59 samples/sec Loss 3.4269 Epoch: 14 Global Step: 146430 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:03:19,708-Speed 126.85 samples/sec Loss 3.2973 Epoch: 14 Global Step: 146435 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:03:39,287-Speed 130.76 samples/sec Loss 3.4318 Epoch: 14 Global Step: 146440 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:03:58,755-Speed 131.50 samples/sec Loss 3.4106 Epoch: 14 Global Step: 146445 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:04:18,365-Speed 130.56 samples/sec Loss 3.4197 Epoch: 14 Global Step: 146450 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:04:37,919-Speed 130.92 samples/sec Loss 3.3565 Epoch: 14 Global Step: 146455 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:04:57,512-Speed 130.67 samples/sec Loss 3.4935 Epoch: 14 Global Step: 146460 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:05:17,048-Speed 131.05 samples/sec Loss 3.3798 Epoch: 14 Global Step: 146465 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:05:36,951-Speed 128.63 samples/sec Loss 3.2664 Epoch: 14 Global Step: 146470 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:05:56,369-Speed 131.84 samples/sec Loss 3.4437 Epoch: 14 Global Step: 146475 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:06:16,135-Speed 129.52 samples/sec Loss 3.6049 Epoch: 14 Global Step: 146480 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:06:35,671-Speed 131.05 samples/sec Loss 3.2344 Epoch: 14 Global Step: 146485 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:06:55,337-Speed 130.18 samples/sec Loss 3.2423 Epoch: 14 Global Step: 146490 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:07:14,879-Speed 131.01 samples/sec Loss 3.6126 Epoch: 14 Global Step: 146495 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:07:34,493-Speed 130.53 samples/sec Loss 3.6321 Epoch: 14 Global Step: 146500 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:07:54,118-Speed 130.45 samples/sec Loss 3.3383 Epoch: 14 Global Step: 146505 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:08:13,650-Speed 131.08 samples/sec Loss 3.4353 Epoch: 14 Global Step: 146510 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:08:33,449-Speed 129.31 samples/sec Loss 3.4502 Epoch: 14 Global Step: 146515 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:08:53,011-Speed 130.87 samples/sec Loss 3.4491 Epoch: 14 Global Step: 146520 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:09:12,657-Speed 130.36 samples/sec Loss 3.6166 Epoch: 14 Global Step: 146525 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:09:32,186-Speed 131.09 samples/sec Loss 3.4805 Epoch: 14 Global Step: 146530 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:09:51,832-Speed 130.31 samples/sec Loss 3.5309 Epoch: 14 Global Step: 146535 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:10:11,427-Speed 130.65 samples/sec Loss 3.3881 Epoch: 14 Global Step: 146540 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:10:30,989-Speed 130.87 samples/sec Loss 3.5873 Epoch: 14 Global Step: 146545 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:10:50,560-Speed 130.82 samples/sec Loss 3.5259 Epoch: 14 Global Step: 146550 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:11:10,097-Speed 131.04 samples/sec Loss 3.3211 Epoch: 14 Global Step: 146555 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:11:29,903-Speed 129.26 samples/sec Loss 3.3309 Epoch: 14 Global Step: 146560 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:11:49,683-Speed 129.43 samples/sec Loss 3.4529 Epoch: 14 Global Step: 146565 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:12:09,205-Speed 131.14 samples/sec Loss 3.2995 Epoch: 14 Global Step: 146570 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:12:28,808-Speed 130.60 samples/sec Loss 3.1424 Epoch: 14 Global Step: 146575 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:12:48,381-Speed 130.79 samples/sec Loss 3.4531 Epoch: 14 Global Step: 146580 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:13:07,928-Speed 130.98 samples/sec Loss 3.4362 Epoch: 14 Global Step: 146585 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:13:27,631-Speed 129.94 samples/sec Loss 3.4792 Epoch: 14 Global Step: 146590 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:13:47,173-Speed 131.01 samples/sec Loss 3.3194 Epoch: 14 Global Step: 146595 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:14:06,880-Speed 129.91 samples/sec Loss 3.5494 Epoch: 14 Global Step: 146600 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:14:26,658-Speed 129.45 samples/sec Loss 3.3731 Epoch: 14 Global Step: 146605 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:14:46,503-Speed 129.00 samples/sec Loss 3.1968 Epoch: 14 Global Step: 146610 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:15:06,056-Speed 130.93 samples/sec Loss 3.4643 Epoch: 14 Global Step: 146615 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:15:25,694-Speed 130.37 samples/sec Loss 3.3192 Epoch: 14 Global Step: 146620 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:15:45,301-Speed 130.57 samples/sec Loss 3.2846 Epoch: 14 Global Step: 146625 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:16:04,924-Speed 130.47 samples/sec Loss 3.3129 Epoch: 14 Global Step: 146630 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:16:24,459-Speed 131.06 samples/sec Loss 3.4988 Epoch: 14 Global Step: 146635 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:16:43,983-Speed 131.13 samples/sec Loss 3.3678 Epoch: 14 Global Step: 146640 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:17:03,564-Speed 130.75 samples/sec Loss 3.5760 Epoch: 14 Global Step: 146645 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:17:23,536-Speed 128.19 samples/sec Loss 3.4592 Epoch: 14 Global Step: 146650 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:17:43,069-Speed 131.06 samples/sec Loss 3.4946 Epoch: 14 Global Step: 146655 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:18:02,687-Speed 130.50 samples/sec Loss 3.5209 Epoch: 14 Global Step: 146660 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:18:22,292-Speed 130.59 samples/sec Loss 3.3699 Epoch: 14 Global Step: 146665 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:18:41,787-Speed 131.32 samples/sec Loss 3.4343 Epoch: 14 Global Step: 146670 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:19:01,388-Speed 130.61 samples/sec Loss 3.3678 Epoch: 14 Global Step: 146675 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:19:21,045-Speed 130.24 samples/sec Loss 3.2809 Epoch: 14 Global Step: 146680 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:19:40,784-Speed 129.70 samples/sec Loss 3.3798 Epoch: 14 Global Step: 146685 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:20:00,842-Speed 127.63 samples/sec Loss 3.3849 Epoch: 14 Global Step: 146690 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:20:20,879-Speed 127.77 samples/sec Loss 3.2195 Epoch: 14 Global Step: 146695 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:20:40,387-Speed 131.24 samples/sec Loss 3.3827 Epoch: 14 Global Step: 146700 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:21:00,174-Speed 129.39 samples/sec Loss 3.5416 Epoch: 14 Global Step: 146705 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:21:19,863-Speed 130.03 samples/sec Loss 3.4290 Epoch: 14 Global Step: 146710 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:21:39,469-Speed 130.57 samples/sec Loss 3.4760 Epoch: 14 Global Step: 146715 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:21:58,937-Speed 131.51 samples/sec Loss 3.4141 Epoch: 14 Global Step: 146720 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:22:18,416-Speed 131.43 samples/sec Loss 3.5245 Epoch: 14 Global Step: 146725 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:22:37,967-Speed 130.95 samples/sec Loss 3.4836 Epoch: 14 Global Step: 146730 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:22:57,505-Speed 131.04 samples/sec Loss 3.5152 Epoch: 14 Global Step: 146735 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:23:17,319-Speed 129.21 samples/sec Loss 3.3446 Epoch: 14 Global Step: 146740 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:23:37,027-Speed 129.91 samples/sec Loss 3.4266 Epoch: 14 Global Step: 146745 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:23:56,554-Speed 131.11 samples/sec Loss 3.4492 Epoch: 14 Global Step: 146750 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:24:16,281-Speed 129.78 samples/sec Loss 3.2847 Epoch: 14 Global Step: 146755 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:24:35,840-Speed 130.89 samples/sec Loss 3.4300 Epoch: 14 Global Step: 146760 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:24:55,428-Speed 130.70 samples/sec Loss 3.5033 Epoch: 14 Global Step: 146765 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:25:14,972-Speed 130.99 samples/sec Loss 3.5309 Epoch: 14 Global Step: 146770 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:25:34,563-Speed 130.68 samples/sec Loss 3.4213 Epoch: 14 Global Step: 146775 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:25:54,112-Speed 130.96 samples/sec Loss 3.5969 Epoch: 14 Global Step: 146780 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:26:13,945-Speed 129.08 samples/sec Loss 3.7480 Epoch: 14 Global Step: 146785 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:26:33,885-Speed 128.39 samples/sec Loss 3.7026 Epoch: 14 Global Step: 146790 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:26:53,553-Speed 130.17 samples/sec Loss 3.3010 Epoch: 14 Global Step: 146795 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:27:13,098-Speed 130.99 samples/sec Loss 3.2492 Epoch: 14 Global Step: 146800 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:27:32,666-Speed 130.83 samples/sec Loss 3.4458 Epoch: 14 Global Step: 146805 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:27:52,194-Speed 131.10 samples/sec Loss 3.6456 Epoch: 14 Global Step: 146810 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:28:11,801-Speed 130.57 samples/sec Loss 3.1793 Epoch: 14 Global Step: 146815 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:28:31,301-Speed 131.29 samples/sec Loss 3.2629 Epoch: 14 Global Step: 146820 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:28:50,840-Speed 131.03 samples/sec Loss 3.4943 Epoch: 14 Global Step: 146825 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:29:10,589-Speed 129.64 samples/sec Loss 3.6781 Epoch: 14 Global Step: 146830 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:29:30,307-Speed 129.84 samples/sec Loss 3.4086 Epoch: 14 Global Step: 146835 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:29:49,748-Speed 131.69 samples/sec Loss 3.2286 Epoch: 14 Global Step: 146840 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:30:09,329-Speed 130.74 samples/sec Loss 3.4960 Epoch: 14 Global Step: 146845 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:30:28,893-Speed 130.86 samples/sec Loss 3.6441 Epoch: 14 Global Step: 146850 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:30:48,503-Speed 130.56 samples/sec Loss 3.3310 Epoch: 14 Global Step: 146855 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:31:08,120-Speed 130.51 samples/sec Loss 3.4529 Epoch: 14 Global Step: 146860 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:31:27,561-Speed 131.69 samples/sec Loss 3.5908 Epoch: 14 Global Step: 146865 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:31:47,174-Speed 130.53 samples/sec Loss 3.3183 Epoch: 14 Global Step: 146870 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:32:06,856-Speed 130.07 samples/sec Loss 3.4354 Epoch: 14 Global Step: 146875 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:32:26,504-Speed 130.30 samples/sec Loss 3.5157 Epoch: 14 Global Step: 146880 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:32:46,010-Speed 131.25 samples/sec Loss 3.2893 Epoch: 14 Global Step: 146885 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:33:06,155-Speed 127.09 samples/sec Loss 3.4485 Epoch: 14 Global Step: 146890 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:33:25,625-Speed 131.49 samples/sec Loss 3.4892 Epoch: 14 Global Step: 146895 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:33:45,228-Speed 130.60 samples/sec Loss 3.3552 Epoch: 14 Global Step: 146900 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:34:04,727-Speed 131.29 samples/sec Loss 3.3387 Epoch: 14 Global Step: 146905 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:34:24,275-Speed 130.97 samples/sec Loss 3.3897 Epoch: 14 Global Step: 146910 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:34:43,854-Speed 130.76 samples/sec Loss 3.4336 Epoch: 14 Global Step: 146915 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:35:03,760-Speed 128.61 samples/sec Loss 3.4848 Epoch: 14 Global Step: 146920 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:35:23,309-Speed 130.96 samples/sec Loss 3.3658 Epoch: 14 Global Step: 146925 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:35:42,882-Speed 130.80 samples/sec Loss 3.5747 Epoch: 14 Global Step: 146930 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:36:02,568-Speed 130.05 samples/sec Loss 3.5331 Epoch: 14 Global Step: 146935 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:36:22,154-Speed 130.71 samples/sec Loss 3.5658 Epoch: 14 Global Step: 146940 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:36:41,764-Speed 130.55 samples/sec Loss 3.3651 Epoch: 14 Global Step: 146945 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:37:01,206-Speed 131.69 samples/sec Loss 3.5253 Epoch: 14 Global Step: 146950 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:37:20,734-Speed 131.10 samples/sec Loss 3.4855 Epoch: 14 Global Step: 146955 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:37:40,291-Speed 130.91 samples/sec Loss 3.5134 Epoch: 14 Global Step: 146960 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:38:00,359-Speed 127.58 samples/sec Loss 3.3318 Epoch: 14 Global Step: 146965 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:38:19,909-Speed 130.95 samples/sec Loss 3.2154 Epoch: 14 Global Step: 146970 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:38:39,453-Speed 130.99 samples/sec Loss 3.4078 Epoch: 14 Global Step: 146975 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:38:59,050-Speed 130.64 samples/sec Loss 3.4701 Epoch: 14 Global Step: 146980 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:39:18,573-Speed 131.13 samples/sec Loss 3.2992 Epoch: 14 Global Step: 146985 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:39:38,182-Speed 130.56 samples/sec Loss 3.5173 Epoch: 14 Global Step: 146990 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:39:57,693-Speed 131.22 samples/sec Loss 3.2869 Epoch: 14 Global Step: 146995 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:40:17,231-Speed 131.04 samples/sec Loss 3.3081 Epoch: 14 Global Step: 147000 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:40:36,844-Speed 130.54 samples/sec Loss 3.4152 Epoch: 14 Global Step: 147005 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:40:56,809-Speed 128.23 samples/sec Loss 3.4326 Epoch: 14 Global Step: 147010 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:41:16,246-Speed 131.72 samples/sec Loss 3.5195 Epoch: 14 Global Step: 147015 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:41:35,890-Speed 130.32 samples/sec Loss 3.3334 Epoch: 14 Global Step: 147020 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:41:55,495-Speed 130.58 samples/sec Loss 3.4836 Epoch: 14 Global Step: 147025 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:42:15,125-Speed 130.42 samples/sec Loss 3.6040 Epoch: 14 Global Step: 147030 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:42:34,747-Speed 130.47 samples/sec Loss 3.4263 Epoch: 14 Global Step: 147035 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:42:54,297-Speed 130.96 samples/sec Loss 3.5275 Epoch: 14 Global Step: 147040 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:43:13,841-Speed 130.99 samples/sec Loss 3.4067 Epoch: 14 Global Step: 147045 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:43:33,513-Speed 130.14 samples/sec Loss 3.4616 Epoch: 14 Global Step: 147050 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:43:53,407-Speed 128.69 samples/sec Loss 3.4035 Epoch: 14 Global Step: 147055 Fp16 Grad Scale: 8192 Required: 121 hours Training: 2021-06-14 00:44:12,835-Speed 131.77 samples/sec Loss 3.5084 Epoch: 14 Global Step: 147060 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:44:32,663-Speed 129.12 samples/sec Loss 3.3836 Epoch: 14 Global Step: 147065 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:44:52,273-Speed 130.55 samples/sec Loss 3.4449 Epoch: 14 Global Step: 147070 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:45:11,750-Speed 131.44 samples/sec Loss 3.3660 Epoch: 14 Global Step: 147075 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:45:31,331-Speed 130.75 samples/sec Loss 3.5272 Epoch: 14 Global Step: 147080 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:45:50,857-Speed 131.11 samples/sec Loss 3.5911 Epoch: 14 Global Step: 147085 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:46:10,453-Speed 130.65 samples/sec Loss 3.5010 Epoch: 14 Global Step: 147090 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:46:30,058-Speed 130.59 samples/sec Loss 3.4536 Epoch: 14 Global Step: 147095 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:46:49,875-Speed 129.18 samples/sec Loss 3.4592 Epoch: 14 Global Step: 147100 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:47:09,301-Speed 131.79 samples/sec Loss 3.4991 Epoch: 14 Global Step: 147105 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:47:29,055-Speed 129.60 samples/sec Loss 3.3427 Epoch: 14 Global Step: 147110 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:47:48,698-Speed 130.33 samples/sec Loss 3.4891 Epoch: 14 Global Step: 147115 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:48:08,176-Speed 131.44 samples/sec Loss 3.4786 Epoch: 14 Global Step: 147120 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:48:27,751-Speed 130.79 samples/sec Loss 3.2807 Epoch: 14 Global Step: 147125 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:48:47,289-Speed 131.03 samples/sec Loss 3.4924 Epoch: 14 Global Step: 147130 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:49:06,873-Speed 130.73 samples/sec Loss 3.4224 Epoch: 14 Global Step: 147135 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:49:26,436-Speed 130.86 samples/sec Loss 3.4726 Epoch: 14 Global Step: 147140 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:49:46,227-Speed 129.36 samples/sec Loss 3.4356 Epoch: 14 Global Step: 147145 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:50:05,705-Speed 131.44 samples/sec Loss 3.3258 Epoch: 14 Global Step: 147150 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:50:25,595-Speed 128.72 samples/sec Loss 3.7109 Epoch: 14 Global Step: 147155 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:50:45,084-Speed 131.36 samples/sec Loss 3.5374 Epoch: 14 Global Step: 147160 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:51:04,650-Speed 130.85 samples/sec Loss 3.5704 Epoch: 14 Global Step: 147165 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:51:24,222-Speed 130.81 samples/sec Loss 3.3541 Epoch: 14 Global Step: 147170 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:51:43,674-Speed 131.62 samples/sec Loss 3.3647 Epoch: 14 Global Step: 147175 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:52:03,278-Speed 130.59 samples/sec Loss 3.4081 Epoch: 14 Global Step: 147180 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:52:22,758-Speed 131.43 samples/sec Loss 3.3477 Epoch: 14 Global Step: 147185 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:52:42,672-Speed 128.56 samples/sec Loss 3.4320 Epoch: 14 Global Step: 147190 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:53:02,131-Speed 131.56 samples/sec Loss 3.4203 Epoch: 14 Global Step: 147195 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:53:21,887-Speed 129.59 samples/sec Loss 3.4024 Epoch: 14 Global Step: 147200 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:53:41,347-Speed 131.56 samples/sec Loss 3.2243 Epoch: 14 Global Step: 147205 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:54:00,959-Speed 130.54 samples/sec Loss 3.6874 Epoch: 14 Global Step: 147210 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:54:20,530-Speed 130.81 samples/sec Loss 3.3647 Epoch: 14 Global Step: 147215 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:54:40,077-Speed 130.97 samples/sec Loss 3.4483 Epoch: 14 Global Step: 147220 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:54:59,772-Speed 129.99 samples/sec Loss 3.2637 Epoch: 14 Global Step: 147225 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:55:19,416-Speed 130.33 samples/sec Loss 3.3447 Epoch: 14 Global Step: 147230 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:55:39,220-Speed 129.27 samples/sec Loss 3.4671 Epoch: 14 Global Step: 147235 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:55:59,214-Speed 128.05 samples/sec Loss 3.3841 Epoch: 14 Global Step: 147240 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:56:18,732-Speed 131.17 samples/sec Loss 3.4991 Epoch: 14 Global Step: 147245 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:56:38,273-Speed 131.01 samples/sec Loss 3.5289 Epoch: 14 Global Step: 147250 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:56:57,880-Speed 130.58 samples/sec Loss 3.3596 Epoch: 14 Global Step: 147255 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:57:17,420-Speed 131.02 samples/sec Loss 3.2945 Epoch: 14 Global Step: 147260 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:57:36,924-Speed 131.26 samples/sec Loss 3.5205 Epoch: 14 Global Step: 147265 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:57:56,559-Speed 130.39 samples/sec Loss 3.4050 Epoch: 14 Global Step: 147270 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:58:16,116-Speed 130.91 samples/sec Loss 3.5900 Epoch: 14 Global Step: 147275 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:58:36,021-Speed 128.62 samples/sec Loss 3.3506 Epoch: 14 Global Step: 147280 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:58:55,562-Speed 131.02 samples/sec Loss 3.5428 Epoch: 14 Global Step: 147285 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:59:15,349-Speed 129.39 samples/sec Loss 3.2231 Epoch: 14 Global Step: 147290 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:59:34,914-Speed 130.85 samples/sec Loss 3.2864 Epoch: 14 Global Step: 147295 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 00:59:54,679-Speed 129.53 samples/sec Loss 3.3085 Epoch: 14 Global Step: 147300 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:00:14,166-Speed 131.38 samples/sec Loss 3.3075 Epoch: 14 Global Step: 147305 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:00:33,795-Speed 130.43 samples/sec Loss 3.5990 Epoch: 14 Global Step: 147310 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:00:53,341-Speed 130.98 samples/sec Loss 3.4419 Epoch: 14 Global Step: 147315 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:01:12,863-Speed 131.14 samples/sec Loss 3.3895 Epoch: 14 Global Step: 147320 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:01:32,636-Speed 129.48 samples/sec Loss 3.3746 Epoch: 14 Global Step: 147325 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:01:52,336-Speed 129.96 samples/sec Loss 3.4298 Epoch: 14 Global Step: 147330 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:02:11,863-Speed 131.10 samples/sec Loss 3.3625 Epoch: 14 Global Step: 147335 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:02:31,385-Speed 131.14 samples/sec Loss 3.5207 Epoch: 14 Global Step: 147340 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:02:51,040-Speed 130.25 samples/sec Loss 3.3970 Epoch: 14 Global Step: 147345 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:03:10,668-Speed 130.44 samples/sec Loss 3.4126 Epoch: 14 Global Step: 147350 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:03:30,325-Speed 130.24 samples/sec Loss 3.1584 Epoch: 14 Global Step: 147355 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:03:49,870-Speed 130.99 samples/sec Loss 3.3650 Epoch: 14 Global Step: 147360 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:04:09,407-Speed 131.04 samples/sec Loss 3.4078 Epoch: 14 Global Step: 147365 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:04:29,170-Speed 129.54 samples/sec Loss 3.4220 Epoch: 14 Global Step: 147370 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:04:48,948-Speed 129.45 samples/sec Loss 3.3819 Epoch: 14 Global Step: 147375 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:05:08,525-Speed 130.78 samples/sec Loss 3.5218 Epoch: 14 Global Step: 147380 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:05:28,137-Speed 130.54 samples/sec Loss 3.4208 Epoch: 14 Global Step: 147385 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:05:47,593-Speed 131.59 samples/sec Loss 3.4171 Epoch: 14 Global Step: 147390 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:06:07,194-Speed 130.61 samples/sec Loss 3.6305 Epoch: 14 Global Step: 147395 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:06:26,624-Speed 131.76 samples/sec Loss 3.6288 Epoch: 14 Global Step: 147400 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:06:46,173-Speed 130.96 samples/sec Loss 3.4795 Epoch: 14 Global Step: 147405 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:07:05,851-Speed 130.10 samples/sec Loss 3.4676 Epoch: 14 Global Step: 147410 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:07:25,803-Speed 128.32 samples/sec Loss 3.3552 Epoch: 14 Global Step: 147415 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:07:45,233-Speed 131.76 samples/sec Loss 3.5101 Epoch: 14 Global Step: 147420 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:08:04,793-Speed 130.89 samples/sec Loss 3.5690 Epoch: 14 Global Step: 147425 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:08:24,388-Speed 130.65 samples/sec Loss 3.5044 Epoch: 14 Global Step: 147430 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:08:44,059-Speed 130.15 samples/sec Loss 3.3245 Epoch: 14 Global Step: 147435 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:09:03,643-Speed 130.72 samples/sec Loss 3.4110 Epoch: 14 Global Step: 147440 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:09:23,082-Speed 131.70 samples/sec Loss 3.4970 Epoch: 14 Global Step: 147445 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:09:42,730-Speed 130.30 samples/sec Loss 3.3657 Epoch: 14 Global Step: 147450 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:10:02,219-Speed 131.36 samples/sec Loss 3.7016 Epoch: 14 Global Step: 147455 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:10:22,018-Speed 129.31 samples/sec Loss 3.3776 Epoch: 14 Global Step: 147460 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:10:41,519-Speed 131.28 samples/sec Loss 3.3583 Epoch: 14 Global Step: 147465 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:11:01,103-Speed 130.73 samples/sec Loss 3.3004 Epoch: 14 Global Step: 147470 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:11:20,569-Speed 131.52 samples/sec Loss 3.3471 Epoch: 14 Global Step: 147475 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:11:40,178-Speed 130.56 samples/sec Loss 3.4363 Epoch: 14 Global Step: 147480 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:11:59,600-Speed 131.82 samples/sec Loss 3.1917 Epoch: 14 Global Step: 147485 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:12:19,135-Speed 131.05 samples/sec Loss 3.3940 Epoch: 14 Global Step: 147490 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:12:38,699-Speed 130.87 samples/sec Loss 3.3879 Epoch: 14 Global Step: 147495 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:12:58,210-Speed 131.21 samples/sec Loss 3.4540 Epoch: 14 Global Step: 147500 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:13:18,016-Speed 129.27 samples/sec Loss 3.5150 Epoch: 14 Global Step: 147505 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:13:37,492-Speed 131.45 samples/sec Loss 3.3088 Epoch: 14 Global Step: 147510 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:13:57,127-Speed 130.39 samples/sec Loss 3.4615 Epoch: 14 Global Step: 147515 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:14:16,730-Speed 130.60 samples/sec Loss 3.5862 Epoch: 14 Global Step: 147520 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:14:36,321-Speed 130.68 samples/sec Loss 3.4504 Epoch: 14 Global Step: 147525 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:14:55,845-Speed 131.13 samples/sec Loss 3.4841 Epoch: 14 Global Step: 147530 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:15:15,402-Speed 130.91 samples/sec Loss 3.3931 Epoch: 14 Global Step: 147535 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:15:35,012-Speed 130.55 samples/sec Loss 3.3739 Epoch: 14 Global Step: 147540 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:15:54,640-Speed 130.43 samples/sec Loss 3.5536 Epoch: 14 Global Step: 147545 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:16:14,437-Speed 129.33 samples/sec Loss 3.3003 Epoch: 14 Global Step: 147550 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:16:34,136-Speed 129.96 samples/sec Loss 3.4227 Epoch: 14 Global Step: 147555 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:16:53,640-Speed 131.27 samples/sec Loss 3.2478 Epoch: 14 Global Step: 147560 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:17:13,226-Speed 130.71 samples/sec Loss 3.6198 Epoch: 14 Global Step: 147565 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:17:32,803-Speed 130.77 samples/sec Loss 3.3488 Epoch: 14 Global Step: 147570 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:17:52,305-Speed 131.28 samples/sec Loss 3.3547 Epoch: 14 Global Step: 147575 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:18:11,913-Speed 130.57 samples/sec Loss 3.4274 Epoch: 14 Global Step: 147580 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:18:31,450-Speed 131.04 samples/sec Loss 3.2498 Epoch: 14 Global Step: 147585 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:18:50,943-Speed 131.34 samples/sec Loss 3.6136 Epoch: 14 Global Step: 147590 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:19:10,696-Speed 129.61 samples/sec Loss 3.4453 Epoch: 14 Global Step: 147595 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:19:30,338-Speed 130.33 samples/sec Loss 3.4463 Epoch: 14 Global Step: 147600 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:19:49,888-Speed 130.96 samples/sec Loss 3.5365 Epoch: 14 Global Step: 147605 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:20:09,521-Speed 130.40 samples/sec Loss 3.3002 Epoch: 14 Global Step: 147610 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:20:29,048-Speed 131.10 samples/sec Loss 3.3836 Epoch: 14 Global Step: 147615 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:20:48,854-Speed 129.26 samples/sec Loss 3.2658 Epoch: 14 Global Step: 147620 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:21:08,388-Speed 131.06 samples/sec Loss 3.4859 Epoch: 14 Global Step: 147625 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:21:27,985-Speed 130.64 samples/sec Loss 3.2994 Epoch: 14 Global Step: 147630 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:21:47,588-Speed 130.60 samples/sec Loss 3.3745 Epoch: 14 Global Step: 147635 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:22:07,505-Speed 128.54 samples/sec Loss 3.3072 Epoch: 14 Global Step: 147640 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:22:26,921-Speed 131.85 samples/sec Loss 3.4900 Epoch: 14 Global Step: 147645 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:22:46,619-Speed 129.97 samples/sec Loss 3.4682 Epoch: 14 Global Step: 147650 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:23:06,237-Speed 130.50 samples/sec Loss 3.4108 Epoch: 14 Global Step: 147655 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:23:25,785-Speed 130.97 samples/sec Loss 3.2380 Epoch: 14 Global Step: 147660 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:23:45,386-Speed 130.61 samples/sec Loss 3.3999 Epoch: 14 Global Step: 147665 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:24:04,934-Speed 130.97 samples/sec Loss 3.3149 Epoch: 14 Global Step: 147670 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:24:24,556-Speed 130.48 samples/sec Loss 3.3565 Epoch: 14 Global Step: 147675 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:24:44,073-Speed 131.18 samples/sec Loss 3.4625 Epoch: 14 Global Step: 147680 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:25:03,972-Speed 128.66 samples/sec Loss 3.6094 Epoch: 14 Global Step: 147685 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:25:23,489-Speed 131.17 samples/sec Loss 3.4853 Epoch: 14 Global Step: 147690 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:25:43,097-Speed 130.57 samples/sec Loss 3.3351 Epoch: 14 Global Step: 147695 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:26:02,627-Speed 131.09 samples/sec Loss 3.3723 Epoch: 14 Global Step: 147700 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:26:22,087-Speed 131.56 samples/sec Loss 3.5719 Epoch: 14 Global Step: 147705 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:26:41,900-Speed 129.21 samples/sec Loss 3.4973 Epoch: 14 Global Step: 147710 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:27:01,444-Speed 131.00 samples/sec Loss 3.2850 Epoch: 14 Global Step: 147715 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:27:21,052-Speed 130.56 samples/sec Loss 3.2018 Epoch: 14 Global Step: 147720 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:27:40,580-Speed 131.10 samples/sec Loss 3.3469 Epoch: 14 Global Step: 147725 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:28:00,448-Speed 128.86 samples/sec Loss 3.4559 Epoch: 14 Global Step: 147730 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:28:19,879-Speed 131.75 samples/sec Loss 3.7100 Epoch: 14 Global Step: 147735 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:28:39,664-Speed 129.40 samples/sec Loss 3.4408 Epoch: 14 Global Step: 147740 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:28:59,206-Speed 131.01 samples/sec Loss 3.3093 Epoch: 14 Global Step: 147745 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:29:18,755-Speed 130.96 samples/sec Loss 3.4644 Epoch: 14 Global Step: 147750 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:29:38,027-Speed 132.84 samples/sec Loss 3.5634 Epoch: 14 Global Step: 147755 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:29:57,620-Speed 130.67 samples/sec Loss 3.5284 Epoch: 14 Global Step: 147760 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:30:17,225-Speed 130.58 samples/sec Loss 3.5153 Epoch: 14 Global Step: 147765 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:30:36,737-Speed 131.21 samples/sec Loss 3.5133 Epoch: 14 Global Step: 147770 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:30:56,570-Speed 129.08 samples/sec Loss 3.5548 Epoch: 14 Global Step: 147775 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:31:15,995-Speed 131.80 samples/sec Loss 3.5727 Epoch: 14 Global Step: 147780 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:31:35,759-Speed 129.54 samples/sec Loss 3.3576 Epoch: 14 Global Step: 147785 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:31:55,280-Speed 131.15 samples/sec Loss 3.2754 Epoch: 14 Global Step: 147790 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:32:14,813-Speed 131.07 samples/sec Loss 3.6105 Epoch: 14 Global Step: 147795 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:32:34,531-Speed 129.84 samples/sec Loss 3.4735 Epoch: 14 Global Step: 147800 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:32:54,075-Speed 130.99 samples/sec Loss 3.4680 Epoch: 14 Global Step: 147805 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:33:13,686-Speed 130.55 samples/sec Loss 3.4900 Epoch: 14 Global Step: 147810 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:33:33,290-Speed 130.59 samples/sec Loss 3.3363 Epoch: 14 Global Step: 147815 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:33:53,170-Speed 128.78 samples/sec Loss 3.2342 Epoch: 14 Global Step: 147820 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:34:12,675-Speed 131.25 samples/sec Loss 3.2833 Epoch: 14 Global Step: 147825 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:34:32,365-Speed 130.03 samples/sec Loss 3.5387 Epoch: 14 Global Step: 147830 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:34:51,905-Speed 131.02 samples/sec Loss 3.5250 Epoch: 14 Global Step: 147835 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:35:11,488-Speed 130.74 samples/sec Loss 3.3970 Epoch: 14 Global Step: 147840 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:35:31,083-Speed 130.65 samples/sec Loss 3.3061 Epoch: 14 Global Step: 147845 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:35:50,660-Speed 130.77 samples/sec Loss 3.1453 Epoch: 14 Global Step: 147850 Fp16 Grad Scale: 4096 Required: 120 hours Training: 2021-06-14 01:36:10,269-Speed 130.56 samples/sec Loss 3.3743 Epoch: 14 Global Step: 147855 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:36:29,813-Speed 130.99 samples/sec Loss 3.4026 Epoch: 14 Global Step: 147860 Fp16 Grad Scale: 8192 Required: 120 hours Training: 2021-06-14 01:36:49,613-Speed 129.31 samples/sec Loss 3.5577 Epoch: 14 Global Step: 147865 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:37:09,236-Speed 130.47 samples/sec Loss 3.3055 Epoch: 14 Global Step: 147870 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:37:28,872-Speed 130.38 samples/sec Loss 3.5386 Epoch: 14 Global Step: 147875 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:37:48,405-Speed 131.07 samples/sec Loss 3.3083 Epoch: 14 Global Step: 147880 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:38:08,042-Speed 130.37 samples/sec Loss 3.2700 Epoch: 14 Global Step: 147885 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:38:27,747-Speed 129.92 samples/sec Loss 3.4907 Epoch: 14 Global Step: 147890 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:38:47,275-Speed 131.11 samples/sec Loss 3.5093 Epoch: 14 Global Step: 147895 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:39:06,834-Speed 130.89 samples/sec Loss 3.5543 Epoch: 14 Global Step: 147900 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:39:26,357-Speed 131.13 samples/sec Loss 3.4626 Epoch: 14 Global Step: 147905 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:39:46,160-Speed 129.29 samples/sec Loss 3.5022 Epoch: 14 Global Step: 147910 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:40:05,753-Speed 130.67 samples/sec Loss 3.3177 Epoch: 14 Global Step: 147915 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:40:25,400-Speed 130.30 samples/sec Loss 3.1713 Epoch: 14 Global Step: 147920 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:40:44,926-Speed 131.11 samples/sec Loss 3.2856 Epoch: 14 Global Step: 147925 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:41:04,625-Speed 129.96 samples/sec Loss 3.6576 Epoch: 14 Global Step: 147930 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:41:24,189-Speed 130.86 samples/sec Loss 3.4002 Epoch: 14 Global Step: 147935 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:41:43,669-Speed 131.42 samples/sec Loss 3.7300 Epoch: 14 Global Step: 147940 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:42:03,274-Speed 130.59 samples/sec Loss 3.3939 Epoch: 14 Global Step: 147945 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:42:22,826-Speed 130.94 samples/sec Loss 3.5198 Epoch: 14 Global Step: 147950 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:42:42,654-Speed 129.12 samples/sec Loss 3.4802 Epoch: 14 Global Step: 147955 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:43:02,381-Speed 129.78 samples/sec Loss 3.4224 Epoch: 14 Global Step: 147960 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:43:21,915-Speed 131.06 samples/sec Loss 3.6569 Epoch: 14 Global Step: 147965 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:43:41,472-Speed 130.91 samples/sec Loss 3.3896 Epoch: 14 Global Step: 147970 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:44:01,082-Speed 130.55 samples/sec Loss 3.5184 Epoch: 14 Global Step: 147975 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:44:20,621-Speed 131.03 samples/sec Loss 3.6793 Epoch: 14 Global Step: 147980 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:44:40,156-Speed 131.05 samples/sec Loss 3.2068 Epoch: 14 Global Step: 147985 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:44:59,677-Speed 131.15 samples/sec Loss 3.5044 Epoch: 14 Global Step: 147990 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:45:19,217-Speed 131.02 samples/sec Loss 3.2234 Epoch: 14 Global Step: 147995 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:45:38,973-Speed 129.59 samples/sec Loss 3.5334 Epoch: 14 Global Step: 148000 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:45:58,717-Speed 129.67 samples/sec Loss 3.4728 Epoch: 14 Global Step: 148005 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:46:18,812-Speed 127.40 samples/sec Loss 3.4552 Epoch: 14 Global Step: 148010 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:46:38,507-Speed 129.99 samples/sec Loss 3.4487 Epoch: 14 Global Step: 148015 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:46:57,964-Speed 131.58 samples/sec Loss 3.3085 Epoch: 14 Global Step: 148020 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:47:17,443-Speed 131.43 samples/sec Loss 3.6510 Epoch: 14 Global Step: 148025 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:47:37,083-Speed 130.36 samples/sec Loss 3.3356 Epoch: 14 Global Step: 148030 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:47:56,642-Speed 130.89 samples/sec Loss 3.3896 Epoch: 14 Global Step: 148035 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:48:16,231-Speed 130.69 samples/sec Loss 3.5115 Epoch: 14 Global Step: 148040 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:48:35,955-Speed 129.80 samples/sec Loss 3.3920 Epoch: 14 Global Step: 148045 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:48:55,568-Speed 130.54 samples/sec Loss 3.4673 Epoch: 14 Global Step: 148050 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:49:15,110-Speed 131.00 samples/sec Loss 3.5245 Epoch: 14 Global Step: 148055 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:49:34,728-Speed 130.50 samples/sec Loss 3.3911 Epoch: 14 Global Step: 148060 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:49:54,281-Speed 130.93 samples/sec Loss 3.4838 Epoch: 14 Global Step: 148065 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:50:13,826-Speed 130.99 samples/sec Loss 3.2691 Epoch: 14 Global Step: 148070 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:50:33,443-Speed 130.51 samples/sec Loss 3.2140 Epoch: 14 Global Step: 148075 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:50:52,986-Speed 131.00 samples/sec Loss 3.5692 Epoch: 14 Global Step: 148080 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:51:12,552-Speed 130.85 samples/sec Loss 3.5551 Epoch: 14 Global Step: 148085 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:51:32,772-Speed 126.61 samples/sec Loss 3.4931 Epoch: 14 Global Step: 148090 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:51:52,492-Speed 129.82 samples/sec Loss 3.6019 Epoch: 14 Global Step: 148095 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:52:12,022-Speed 131.09 samples/sec Loss 3.6679 Epoch: 14 Global Step: 148100 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:52:31,559-Speed 131.04 samples/sec Loss 3.4118 Epoch: 14 Global Step: 148105 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:52:51,192-Speed 130.40 samples/sec Loss 3.2366 Epoch: 14 Global Step: 148110 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:53:10,896-Speed 129.94 samples/sec Loss 3.4046 Epoch: 14 Global Step: 148115 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:53:30,566-Speed 130.15 samples/sec Loss 3.1791 Epoch: 14 Global Step: 148120 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:53:50,168-Speed 130.61 samples/sec Loss 3.5628 Epoch: 14 Global Step: 148125 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:54:09,781-Speed 130.53 samples/sec Loss 3.3888 Epoch: 14 Global Step: 148130 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:54:29,563-Speed 129.42 samples/sec Loss 3.3674 Epoch: 14 Global Step: 148135 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:54:49,135-Speed 130.80 samples/sec Loss 3.4434 Epoch: 14 Global Step: 148140 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:55:08,770-Speed 130.38 samples/sec Loss 3.2749 Epoch: 14 Global Step: 148145 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:55:28,374-Speed 130.60 samples/sec Loss 3.6405 Epoch: 14 Global Step: 148150 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:55:47,920-Speed 130.98 samples/sec Loss 3.4466 Epoch: 14 Global Step: 148155 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:56:07,513-Speed 130.67 samples/sec Loss 3.4810 Epoch: 14 Global Step: 148160 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:56:27,068-Speed 130.92 samples/sec Loss 3.4246 Epoch: 14 Global Step: 148165 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:56:46,624-Speed 130.91 samples/sec Loss 3.6423 Epoch: 14 Global Step: 148170 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:57:06,206-Speed 130.75 samples/sec Loss 3.4723 Epoch: 14 Global Step: 148175 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:57:26,123-Speed 128.54 samples/sec Loss 3.4493 Epoch: 14 Global Step: 148180 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:57:45,611-Speed 131.37 samples/sec Loss 3.4921 Epoch: 14 Global Step: 148185 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:58:05,157-Speed 130.99 samples/sec Loss 3.3187 Epoch: 14 Global Step: 148190 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:58:24,714-Speed 130.90 samples/sec Loss 3.5232 Epoch: 14 Global Step: 148195 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:58:44,379-Speed 130.19 samples/sec Loss 3.3789 Epoch: 14 Global Step: 148200 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:59:03,979-Speed 130.62 samples/sec Loss 3.4729 Epoch: 14 Global Step: 148205 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:59:23,559-Speed 130.75 samples/sec Loss 3.8120 Epoch: 14 Global Step: 148210 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 01:59:43,157-Speed 130.64 samples/sec Loss 3.4713 Epoch: 14 Global Step: 148215 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:00:02,684-Speed 131.10 samples/sec Loss 3.5063 Epoch: 14 Global Step: 148220 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:00:22,578-Speed 128.69 samples/sec Loss 3.5503 Epoch: 14 Global Step: 148225 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:00:42,000-Speed 131.82 samples/sec Loss 3.1550 Epoch: 14 Global Step: 148230 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:01:01,788-Speed 129.38 samples/sec Loss 3.4714 Epoch: 14 Global Step: 148235 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:01:21,319-Speed 131.08 samples/sec Loss 3.5064 Epoch: 14 Global Step: 148240 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:01:40,853-Speed 131.06 samples/sec Loss 3.6142 Epoch: 14 Global Step: 148245 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:02:00,410-Speed 130.91 samples/sec Loss 3.3576 Epoch: 14 Global Step: 148250 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:02:19,978-Speed 130.84 samples/sec Loss 3.3445 Epoch: 14 Global Step: 148255 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:02:39,577-Speed 130.62 samples/sec Loss 3.3763 Epoch: 14 Global Step: 148260 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:02:59,096-Speed 131.16 samples/sec Loss 3.6081 Epoch: 14 Global Step: 148265 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:03:18,977-Speed 128.78 samples/sec Loss 3.6650 Epoch: 14 Global Step: 148270 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:03:38,571-Speed 130.66 samples/sec Loss 3.5844 Epoch: 14 Global Step: 148275 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:03:58,221-Speed 130.29 samples/sec Loss 3.2997 Epoch: 14 Global Step: 148280 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:04:17,779-Speed 130.90 samples/sec Loss 3.5587 Epoch: 14 Global Step: 148285 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:04:37,372-Speed 130.67 samples/sec Loss 3.3102 Epoch: 14 Global Step: 148290 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:04:56,970-Speed 130.63 samples/sec Loss 3.4669 Epoch: 14 Global Step: 148295 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:05:16,513-Speed 131.00 samples/sec Loss 3.2740 Epoch: 14 Global Step: 148300 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:05:36,076-Speed 130.87 samples/sec Loss 3.5422 Epoch: 14 Global Step: 148305 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:05:55,577-Speed 131.28 samples/sec Loss 3.2164 Epoch: 14 Global Step: 148310 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:06:15,505-Speed 128.47 samples/sec Loss 3.3911 Epoch: 14 Global Step: 148315 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:06:35,270-Speed 129.53 samples/sec Loss 3.3515 Epoch: 14 Global Step: 148320 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:06:54,796-Speed 131.11 samples/sec Loss 3.4154 Epoch: 14 Global Step: 148325 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:07:14,436-Speed 130.35 samples/sec Loss 3.5252 Epoch: 14 Global Step: 148330 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:07:34,028-Speed 130.67 samples/sec Loss 3.4092 Epoch: 14 Global Step: 148335 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:07:53,647-Speed 130.50 samples/sec Loss 3.5352 Epoch: 14 Global Step: 148340 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:08:13,261-Speed 130.52 samples/sec Loss 3.5781 Epoch: 14 Global Step: 148345 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:08:32,809-Speed 130.97 samples/sec Loss 3.3260 Epoch: 14 Global Step: 148350 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:08:52,315-Speed 131.25 samples/sec Loss 3.7643 Epoch: 14 Global Step: 148355 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:09:12,111-Speed 129.32 samples/sec Loss 3.4579 Epoch: 14 Global Step: 148360 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:09:31,851-Speed 129.70 samples/sec Loss 3.2053 Epoch: 14 Global Step: 148365 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:09:51,388-Speed 131.04 samples/sec Loss 3.4100 Epoch: 14 Global Step: 148370 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:10:10,932-Speed 130.99 samples/sec Loss 3.5396 Epoch: 14 Global Step: 148375 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:10:30,532-Speed 130.62 samples/sec Loss 3.6535 Epoch: 14 Global Step: 148380 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:10:50,030-Speed 131.31 samples/sec Loss 3.3412 Epoch: 14 Global Step: 148385 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:11:09,601-Speed 130.81 samples/sec Loss 3.3890 Epoch: 14 Global Step: 148390 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:11:29,135-Speed 131.06 samples/sec Loss 3.5102 Epoch: 14 Global Step: 148395 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:11:48,605-Speed 131.50 samples/sec Loss 3.4633 Epoch: 14 Global Step: 148400 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:12:08,407-Speed 129.29 samples/sec Loss 3.4731 Epoch: 14 Global Step: 148405 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:12:28,016-Speed 130.56 samples/sec Loss 3.4841 Epoch: 14 Global Step: 148410 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:12:47,558-Speed 131.01 samples/sec Loss 3.4353 Epoch: 14 Global Step: 148415 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:13:07,076-Speed 131.17 samples/sec Loss 3.5663 Epoch: 14 Global Step: 148420 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:13:26,673-Speed 130.64 samples/sec Loss 3.3993 Epoch: 14 Global Step: 148425 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:13:46,105-Speed 131.75 samples/sec Loss 3.4580 Epoch: 14 Global Step: 148430 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:14:05,716-Speed 130.55 samples/sec Loss 3.5216 Epoch: 14 Global Step: 148435 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:14:25,265-Speed 130.96 samples/sec Loss 3.4693 Epoch: 14 Global Step: 148440 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:14:44,869-Speed 130.59 samples/sec Loss 3.5954 Epoch: 14 Global Step: 148445 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:15:04,712-Speed 129.02 samples/sec Loss 3.4125 Epoch: 14 Global Step: 148450 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:15:24,285-Speed 130.79 samples/sec Loss 3.4162 Epoch: 14 Global Step: 148455 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:15:43,716-Speed 131.76 samples/sec Loss 3.2963 Epoch: 14 Global Step: 148460 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:16:03,317-Speed 130.61 samples/sec Loss 3.4748 Epoch: 14 Global Step: 148465 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:16:22,818-Speed 131.29 samples/sec Loss 3.5465 Epoch: 14 Global Step: 148470 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:16:42,363-Speed 130.99 samples/sec Loss 3.4783 Epoch: 14 Global Step: 148475 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:17:01,908-Speed 130.99 samples/sec Loss 3.3867 Epoch: 14 Global Step: 148480 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:17:21,331-Speed 131.81 samples/sec Loss 3.3720 Epoch: 14 Global Step: 148485 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:17:40,945-Speed 130.53 samples/sec Loss 3.5713 Epoch: 14 Global Step: 148490 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:18:00,714-Speed 129.50 samples/sec Loss 3.3991 Epoch: 14 Global Step: 148495 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:18:20,265-Speed 130.95 samples/sec Loss 3.5335 Epoch: 14 Global Step: 148500 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:18:39,708-Speed 131.67 samples/sec Loss 3.5603 Epoch: 14 Global Step: 148505 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:18:59,314-Speed 130.58 samples/sec Loss 3.4471 Epoch: 14 Global Step: 148510 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:19:18,733-Speed 131.84 samples/sec Loss 3.4521 Epoch: 14 Global Step: 148515 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:19:38,266-Speed 131.07 samples/sec Loss 3.4573 Epoch: 14 Global Step: 148520 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:19:57,803-Speed 131.04 samples/sec Loss 3.3937 Epoch: 14 Global Step: 148525 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:20:17,325-Speed 131.14 samples/sec Loss 3.3633 Epoch: 14 Global Step: 148530 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:20:36,932-Speed 130.57 samples/sec Loss 3.2120 Epoch: 14 Global Step: 148535 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:20:56,768-Speed 129.07 samples/sec Loss 3.4584 Epoch: 14 Global Step: 148540 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:21:16,196-Speed 131.77 samples/sec Loss 3.2244 Epoch: 14 Global Step: 148545 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:21:35,908-Speed 129.88 samples/sec Loss 3.4197 Epoch: 14 Global Step: 148550 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:21:55,512-Speed 130.59 samples/sec Loss 3.2411 Epoch: 14 Global Step: 148555 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:22:15,046-Speed 131.06 samples/sec Loss 3.4731 Epoch: 14 Global Step: 148560 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:22:34,637-Speed 130.68 samples/sec Loss 3.5945 Epoch: 14 Global Step: 148565 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:22:54,061-Speed 131.80 samples/sec Loss 3.5306 Epoch: 14 Global Step: 148570 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:23:13,597-Speed 131.05 samples/sec Loss 3.4324 Epoch: 14 Global Step: 148575 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:23:33,164-Speed 130.84 samples/sec Loss 3.5708 Epoch: 14 Global Step: 148580 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:23:53,087-Speed 128.50 samples/sec Loss 3.3604 Epoch: 14 Global Step: 148585 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:24:12,516-Speed 131.77 samples/sec Loss 3.3118 Epoch: 14 Global Step: 148590 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:24:32,075-Speed 130.89 samples/sec Loss 3.3850 Epoch: 14 Global Step: 148595 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:24:51,644-Speed 130.83 samples/sec Loss 3.4827 Epoch: 14 Global Step: 148600 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:25:11,121-Speed 131.44 samples/sec Loss 3.5596 Epoch: 14 Global Step: 148605 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:25:30,694-Speed 130.80 samples/sec Loss 3.5285 Epoch: 14 Global Step: 148610 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:25:50,157-Speed 131.54 samples/sec Loss 3.7719 Epoch: 14 Global Step: 148615 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:26:09,754-Speed 130.64 samples/sec Loss 3.7325 Epoch: 14 Global Step: 148620 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:26:29,345-Speed 130.68 samples/sec Loss 3.3558 Epoch: 14 Global Step: 148625 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:26:49,142-Speed 129.32 samples/sec Loss 3.3890 Epoch: 14 Global Step: 148630 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:27:08,887-Speed 129.66 samples/sec Loss 3.3818 Epoch: 14 Global Step: 148635 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:27:28,629-Speed 129.68 samples/sec Loss 3.5637 Epoch: 14 Global Step: 148640 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:27:48,202-Speed 130.80 samples/sec Loss 3.6086 Epoch: 14 Global Step: 148645 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:28:07,742-Speed 131.02 samples/sec Loss 3.4951 Epoch: 14 Global Step: 148650 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:28:27,235-Speed 131.34 samples/sec Loss 3.3876 Epoch: 14 Global Step: 148655 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:28:46,794-Speed 130.90 samples/sec Loss 3.6119 Epoch: 14 Global Step: 148660 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:29:06,499-Speed 129.92 samples/sec Loss 3.5692 Epoch: 14 Global Step: 148665 Fp16 Grad Scale: 8192 Required: 119 hours Training: 2021-06-14 02:29:26,108-Speed 130.56 samples/sec Loss 3.5724 Epoch: 14 Global Step: 148670 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:29:45,947-Speed 129.04 samples/sec Loss 3.5552 Epoch: 14 Global Step: 148675 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:30:05,371-Speed 131.80 samples/sec Loss 3.5344 Epoch: 14 Global Step: 148680 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:30:25,142-Speed 129.49 samples/sec Loss 3.3352 Epoch: 14 Global Step: 148685 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:30:44,687-Speed 130.99 samples/sec Loss 3.4042 Epoch: 14 Global Step: 148690 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:31:04,282-Speed 130.65 samples/sec Loss 3.2380 Epoch: 14 Global Step: 148695 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:31:23,817-Speed 131.06 samples/sec Loss 3.4827 Epoch: 14 Global Step: 148700 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:31:43,428-Speed 130.55 samples/sec Loss 3.4824 Epoch: 14 Global Step: 148705 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:32:02,999-Speed 130.82 samples/sec Loss 3.5729 Epoch: 14 Global Step: 148710 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:32:22,546-Speed 130.97 samples/sec Loss 3.6687 Epoch: 14 Global Step: 148715 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:32:42,648-Speed 127.36 samples/sec Loss 3.2384 Epoch: 14 Global Step: 148720 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:33:02,371-Speed 129.81 samples/sec Loss 3.4365 Epoch: 14 Global Step: 148725 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:33:21,909-Speed 131.03 samples/sec Loss 3.5611 Epoch: 14 Global Step: 148730 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:33:41,471-Speed 130.87 samples/sec Loss 3.3031 Epoch: 14 Global Step: 148735 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:34:00,966-Speed 131.33 samples/sec Loss 3.4816 Epoch: 14 Global Step: 148740 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:34:20,590-Speed 130.46 samples/sec Loss 3.4113 Epoch: 14 Global Step: 148745 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:34:40,206-Speed 130.52 samples/sec Loss 3.2304 Epoch: 14 Global Step: 148750 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:35:00,308-Speed 127.36 samples/sec Loss 3.2613 Epoch: 14 Global Step: 148755 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:35:19,850-Speed 131.01 samples/sec Loss 3.3856 Epoch: 14 Global Step: 148760 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:35:39,588-Speed 129.71 samples/sec Loss 3.5492 Epoch: 14 Global Step: 148765 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:35:59,281-Speed 130.00 samples/sec Loss 3.4444 Epoch: 14 Global Step: 148770 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:36:18,814-Speed 131.07 samples/sec Loss 3.0832 Epoch: 14 Global Step: 148775 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:36:38,423-Speed 130.56 samples/sec Loss 3.3985 Epoch: 14 Global Step: 148780 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:36:58,085-Speed 130.21 samples/sec Loss 3.4714 Epoch: 14 Global Step: 148785 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:37:17,641-Speed 130.91 samples/sec Loss 3.4968 Epoch: 14 Global Step: 148790 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:37:37,230-Speed 130.70 samples/sec Loss 3.5384 Epoch: 14 Global Step: 148795 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:37:56,857-Speed 130.44 samples/sec Loss 3.5684 Epoch: 14 Global Step: 148800 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:38:16,311-Speed 131.60 samples/sec Loss 3.4924 Epoch: 14 Global Step: 148805 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:38:36,082-Speed 129.49 samples/sec Loss 3.4666 Epoch: 14 Global Step: 148810 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:38:55,856-Speed 129.47 samples/sec Loss 3.3549 Epoch: 14 Global Step: 148815 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:39:15,408-Speed 130.94 samples/sec Loss 3.5166 Epoch: 14 Global Step: 148820 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:39:35,013-Speed 130.59 samples/sec Loss 3.6376 Epoch: 14 Global Step: 148825 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:39:54,546-Speed 131.07 samples/sec Loss 3.6078 Epoch: 14 Global Step: 148830 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:40:14,285-Speed 129.70 samples/sec Loss 3.5370 Epoch: 14 Global Step: 148835 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:40:33,970-Speed 130.06 samples/sec Loss 3.4600 Epoch: 14 Global Step: 148840 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:40:53,454-Speed 131.40 samples/sec Loss 3.5173 Epoch: 14 Global Step: 148845 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:41:12,999-Speed 130.98 samples/sec Loss 3.7276 Epoch: 14 Global Step: 148850 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:41:32,761-Speed 129.55 samples/sec Loss 3.3804 Epoch: 14 Global Step: 148855 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:41:52,504-Speed 129.67 samples/sec Loss 3.4573 Epoch: 14 Global Step: 148860 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:42:11,938-Speed 131.73 samples/sec Loss 3.3790 Epoch: 14 Global Step: 148865 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:42:31,546-Speed 130.57 samples/sec Loss 3.2742 Epoch: 14 Global Step: 148870 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:42:51,111-Speed 130.86 samples/sec Loss 3.5887 Epoch: 14 Global Step: 148875 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:43:10,721-Speed 130.55 samples/sec Loss 3.4466 Epoch: 14 Global Step: 148880 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:43:30,247-Speed 131.12 samples/sec Loss 3.5860 Epoch: 14 Global Step: 148885 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:43:49,774-Speed 131.11 samples/sec Loss 3.4197 Epoch: 14 Global Step: 148890 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:44:09,377-Speed 130.60 samples/sec Loss 3.4885 Epoch: 14 Global Step: 148895 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:44:29,305-Speed 128.47 samples/sec Loss 3.5772 Epoch: 14 Global Step: 148900 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:44:48,749-Speed 131.67 samples/sec Loss 3.4027 Epoch: 14 Global Step: 148905 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:45:08,395-Speed 130.31 samples/sec Loss 3.2607 Epoch: 14 Global Step: 148910 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:45:27,959-Speed 130.86 samples/sec Loss 3.2960 Epoch: 14 Global Step: 148915 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:45:47,506-Speed 130.97 samples/sec Loss 3.5992 Epoch: 14 Global Step: 148920 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:46:07,114-Speed 130.57 samples/sec Loss 3.2879 Epoch: 14 Global Step: 148925 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:46:27,030-Speed 128.55 samples/sec Loss 3.3725 Epoch: 14 Global Step: 148930 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:46:46,626-Speed 130.64 samples/sec Loss 3.2871 Epoch: 14 Global Step: 148935 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:47:06,157-Speed 131.08 samples/sec Loss 3.4139 Epoch: 14 Global Step: 148940 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:47:25,966-Speed 129.24 samples/sec Loss 3.5790 Epoch: 14 Global Step: 148945 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:47:45,438-Speed 131.48 samples/sec Loss 3.2334 Epoch: 14 Global Step: 148950 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:48:05,125-Speed 130.04 samples/sec Loss 3.3892 Epoch: 14 Global Step: 148955 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:48:24,663-Speed 131.03 samples/sec Loss 3.4537 Epoch: 14 Global Step: 148960 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:48:44,345-Speed 130.08 samples/sec Loss 3.2151 Epoch: 14 Global Step: 148965 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:49:03,867-Speed 131.14 samples/sec Loss 3.5501 Epoch: 14 Global Step: 148970 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:49:23,341-Speed 131.47 samples/sec Loss 3.5167 Epoch: 14 Global Step: 148975 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:49:42,943-Speed 130.61 samples/sec Loss 3.2358 Epoch: 14 Global Step: 148980 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:50:02,500-Speed 130.91 samples/sec Loss 3.3350 Epoch: 14 Global Step: 148985 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:50:22,306-Speed 129.26 samples/sec Loss 3.5318 Epoch: 14 Global Step: 148990 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:50:41,938-Speed 130.40 samples/sec Loss 3.4246 Epoch: 14 Global Step: 148995 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:51:01,376-Speed 131.70 samples/sec Loss 3.5864 Epoch: 14 Global Step: 149000 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:51:20,433-Speed 134.35 samples/sec Loss 3.5391 Epoch: 14 Global Step: 149005 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:51:39,995-Speed 130.87 samples/sec Loss 3.4172 Epoch: 14 Global Step: 149010 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:51:59,530-Speed 131.05 samples/sec Loss 3.2741 Epoch: 14 Global Step: 149015 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:52:19,147-Speed 130.51 samples/sec Loss 3.3005 Epoch: 14 Global Step: 149020 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:52:38,665-Speed 131.17 samples/sec Loss 3.3405 Epoch: 14 Global Step: 149025 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:52:58,257-Speed 130.67 samples/sec Loss 3.4866 Epoch: 14 Global Step: 149030 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:53:17,828-Speed 130.81 samples/sec Loss 3.4812 Epoch: 14 Global Step: 149035 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:53:37,381-Speed 130.94 samples/sec Loss 3.5940 Epoch: 14 Global Step: 149040 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:53:57,172-Speed 129.36 samples/sec Loss 3.5392 Epoch: 14 Global Step: 149045 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:54:16,695-Speed 131.13 samples/sec Loss 3.4024 Epoch: 14 Global Step: 149050 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:54:36,606-Speed 128.58 samples/sec Loss 3.5645 Epoch: 14 Global Step: 149055 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:54:56,174-Speed 130.83 samples/sec Loss 3.2523 Epoch: 14 Global Step: 149060 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:55:15,786-Speed 130.54 samples/sec Loss 3.3034 Epoch: 14 Global Step: 149065 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:55:35,317-Speed 131.08 samples/sec Loss 3.4726 Epoch: 14 Global Step: 149070 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:55:54,893-Speed 130.78 samples/sec Loss 3.4140 Epoch: 14 Global Step: 149075 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:56:14,469-Speed 130.78 samples/sec Loss 3.5098 Epoch: 14 Global Step: 149080 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:56:34,011-Speed 131.01 samples/sec Loss 3.3430 Epoch: 14 Global Step: 149085 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:56:53,817-Speed 129.28 samples/sec Loss 3.5093 Epoch: 14 Global Step: 149090 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:57:13,574-Speed 129.59 samples/sec Loss 3.3560 Epoch: 14 Global Step: 149095 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:57:33,034-Speed 131.55 samples/sec Loss 3.3887 Epoch: 14 Global Step: 149100 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 02:57:52,550-Speed 131.19 samples/sec Loss 3.4561 Epoch: 14 Global Step: 149105 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:58:12,131-Speed 130.74 samples/sec Loss 3.4030 Epoch: 14 Global Step: 149110 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:58:31,737-Speed 130.58 samples/sec Loss 3.3351 Epoch: 14 Global Step: 149115 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:58:51,341-Speed 130.59 samples/sec Loss 3.3811 Epoch: 14 Global Step: 149120 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:59:10,869-Speed 131.10 samples/sec Loss 3.3229 Epoch: 14 Global Step: 149125 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:59:30,370-Speed 131.29 samples/sec Loss 3.4347 Epoch: 14 Global Step: 149130 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 02:59:50,150-Speed 129.44 samples/sec Loss 3.3926 Epoch: 14 Global Step: 149135 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:00:09,929-Speed 129.44 samples/sec Loss 3.2802 Epoch: 14 Global Step: 149140 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:00:29,460-Speed 131.08 samples/sec Loss 3.4980 Epoch: 14 Global Step: 149145 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:00:48,943-Speed 131.41 samples/sec Loss 3.3597 Epoch: 14 Global Step: 149150 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:01:08,519-Speed 130.78 samples/sec Loss 3.3732 Epoch: 14 Global Step: 149155 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:01:28,494-Speed 128.16 samples/sec Loss 3.2359 Epoch: 14 Global Step: 149160 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:01:48,111-Speed 130.51 samples/sec Loss 3.4090 Epoch: 14 Global Step: 149165 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:02:07,655-Speed 130.99 samples/sec Loss 3.6163 Epoch: 14 Global Step: 149170 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:02:27,119-Speed 131.53 samples/sec Loss 3.3507 Epoch: 14 Global Step: 149175 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:02:46,894-Speed 129.47 samples/sec Loss 3.4249 Epoch: 14 Global Step: 149180 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:03:06,672-Speed 129.44 samples/sec Loss 3.5012 Epoch: 14 Global Step: 149185 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:03:26,195-Speed 131.14 samples/sec Loss 3.4159 Epoch: 14 Global Step: 149190 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:03:45,717-Speed 131.14 samples/sec Loss 3.4789 Epoch: 14 Global Step: 149195 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:04:05,375-Speed 130.24 samples/sec Loss 3.3046 Epoch: 14 Global Step: 149200 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:04:24,950-Speed 130.78 samples/sec Loss 3.3636 Epoch: 14 Global Step: 149205 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:04:44,550-Speed 130.62 samples/sec Loss 3.2912 Epoch: 14 Global Step: 149210 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:05:03,990-Speed 131.69 samples/sec Loss 3.5308 Epoch: 14 Global Step: 149215 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:05:23,530-Speed 131.03 samples/sec Loss 3.4884 Epoch: 14 Global Step: 149220 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:05:43,331-Speed 129.29 samples/sec Loss 3.5709 Epoch: 14 Global Step: 149225 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:06:02,986-Speed 130.26 samples/sec Loss 3.3434 Epoch: 14 Global Step: 149230 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:06:22,426-Speed 131.69 samples/sec Loss 3.4722 Epoch: 14 Global Step: 149235 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:06:42,138-Speed 129.88 samples/sec Loss 3.4184 Epoch: 14 Global Step: 149240 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:07:01,735-Speed 130.64 samples/sec Loss 3.2493 Epoch: 14 Global Step: 149245 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:07:21,200-Speed 131.53 samples/sec Loss 3.4010 Epoch: 14 Global Step: 149250 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:07:40,780-Speed 130.75 samples/sec Loss 3.3094 Epoch: 14 Global Step: 149255 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:08:00,207-Speed 131.78 samples/sec Loss 3.5925 Epoch: 14 Global Step: 149260 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:08:19,804-Speed 130.64 samples/sec Loss 3.5345 Epoch: 14 Global Step: 149265 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:08:39,603-Speed 129.31 samples/sec Loss 3.4385 Epoch: 14 Global Step: 149270 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:08:59,294-Speed 130.02 samples/sec Loss 3.3653 Epoch: 14 Global Step: 149275 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:09:18,734-Speed 131.70 samples/sec Loss 3.4786 Epoch: 14 Global Step: 149280 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:09:38,332-Speed 130.63 samples/sec Loss 3.3562 Epoch: 14 Global Step: 149285 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:09:57,912-Speed 130.75 samples/sec Loss 3.4836 Epoch: 14 Global Step: 149290 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:10:17,460-Speed 130.97 samples/sec Loss 3.5548 Epoch: 14 Global Step: 149295 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:10:37,053-Speed 130.67 samples/sec Loss 3.8590 Epoch: 14 Global Step: 149300 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:10:56,592-Speed 131.03 samples/sec Loss 3.3526 Epoch: 14 Global Step: 149305 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:11:16,222-Speed 130.42 samples/sec Loss 3.4896 Epoch: 14 Global Step: 149310 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:11:36,029-Speed 129.25 samples/sec Loss 3.3848 Epoch: 14 Global Step: 149315 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:11:55,661-Speed 130.41 samples/sec Loss 3.4058 Epoch: 14 Global Step: 149320 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:12:15,197-Speed 131.05 samples/sec Loss 3.6568 Epoch: 14 Global Step: 149325 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:12:34,362-Speed 133.58 samples/sec Loss 3.5328 Epoch: 14 Global Step: 149330 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:12:53,888-Speed 131.11 samples/sec Loss 3.3399 Epoch: 14 Global Step: 149335 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:13:13,576-Speed 130.04 samples/sec Loss 3.6703 Epoch: 14 Global Step: 149340 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:13:32,998-Speed 131.81 samples/sec Loss 3.3513 Epoch: 14 Global Step: 149345 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:13:52,547-Speed 130.96 samples/sec Loss 3.5922 Epoch: 14 Global Step: 149350 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:14:12,147-Speed 130.62 samples/sec Loss 3.3911 Epoch: 14 Global Step: 149355 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:14:31,997-Speed 128.97 samples/sec Loss 3.3868 Epoch: 14 Global Step: 149360 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:14:51,502-Speed 131.26 samples/sec Loss 3.4899 Epoch: 14 Global Step: 149365 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:15:10,989-Speed 131.37 samples/sec Loss 3.3870 Epoch: 14 Global Step: 149370 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:15:30,663-Speed 130.13 samples/sec Loss 3.4084 Epoch: 14 Global Step: 149375 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:15:50,258-Speed 130.65 samples/sec Loss 3.3482 Epoch: 14 Global Step: 149380 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:16:09,870-Speed 130.54 samples/sec Loss 3.4978 Epoch: 14 Global Step: 149385 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:16:29,400-Speed 131.09 samples/sec Loss 3.3012 Epoch: 14 Global Step: 149390 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:16:49,145-Speed 129.66 samples/sec Loss 3.4101 Epoch: 14 Global Step: 149395 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:17:08,645-Speed 131.29 samples/sec Loss 3.3913 Epoch: 14 Global Step: 149400 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:17:28,554-Speed 128.59 samples/sec Loss 3.5283 Epoch: 14 Global Step: 149405 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:17:47,998-Speed 131.67 samples/sec Loss 3.5063 Epoch: 14 Global Step: 149410 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:18:07,722-Speed 129.80 samples/sec Loss 3.5046 Epoch: 14 Global Step: 149415 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:18:27,283-Speed 130.88 samples/sec Loss 3.5154 Epoch: 14 Global Step: 149420 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:18:46,731-Speed 131.64 samples/sec Loss 3.5772 Epoch: 14 Global Step: 149425 Fp16 Grad Scale: 4096 Required: 118 hours Training: 2021-06-14 03:19:06,332-Speed 130.61 samples/sec Loss 3.6831 Epoch: 14 Global Step: 149430 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:19:25,999-Speed 130.17 samples/sec Loss 3.2342 Epoch: 14 Global Step: 149435 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:19:45,620-Speed 130.48 samples/sec Loss 3.6670 Epoch: 14 Global Step: 149440 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:20:05,154-Speed 131.06 samples/sec Loss 3.3627 Epoch: 14 Global Step: 149445 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:20:24,960-Speed 129.26 samples/sec Loss 3.6239 Epoch: 14 Global Step: 149450 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:20:44,365-Speed 131.93 samples/sec Loss 3.4401 Epoch: 14 Global Step: 149455 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:21:04,073-Speed 129.90 samples/sec Loss 3.4494 Epoch: 14 Global Step: 149460 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:21:39,717-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-14 03:22:06,914-Speed 40.74 samples/sec Loss 2.7195 Epoch: 15 Global Step: 149465 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:22:26,746-Speed 129.09 samples/sec Loss 2.7261 Epoch: 15 Global Step: 149470 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:22:46,548-Speed 129.28 samples/sec Loss 2.6666 Epoch: 15 Global Step: 149475 Fp16 Grad Scale: 8192 Required: 118 hours Training: 2021-06-14 03:23:06,448-Speed 128.65 samples/sec Loss 2.6937 Epoch: 15 Global Step: 149480 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:23:26,179-Speed 129.75 samples/sec Loss 2.8510 Epoch: 15 Global Step: 149485 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:23:45,981-Speed 129.29 samples/sec Loss 3.1047 Epoch: 15 Global Step: 149490 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:24:05,703-Speed 129.81 samples/sec Loss 2.6619 Epoch: 15 Global Step: 149495 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:24:25,579-Speed 128.80 samples/sec Loss 2.7926 Epoch: 15 Global Step: 149500 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:24:45,808-Speed 126.56 samples/sec Loss 2.6058 Epoch: 15 Global Step: 149505 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:25:05,573-Speed 129.52 samples/sec Loss 2.7966 Epoch: 15 Global Step: 149510 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:25:25,310-Speed 129.71 samples/sec Loss 2.7062 Epoch: 15 Global Step: 149515 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:25:45,273-Speed 128.25 samples/sec Loss 2.7165 Epoch: 15 Global Step: 149520 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:26:04,965-Speed 130.01 samples/sec Loss 2.7751 Epoch: 15 Global Step: 149525 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:26:24,763-Speed 129.31 samples/sec Loss 2.8148 Epoch: 15 Global Step: 149530 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:26:44,621-Speed 128.92 samples/sec Loss 2.7499 Epoch: 15 Global Step: 149535 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:27:04,435-Speed 129.21 samples/sec Loss 2.6964 Epoch: 15 Global Step: 149540 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:27:24,238-Speed 129.28 samples/sec Loss 2.9817 Epoch: 15 Global Step: 149545 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:27:44,712-Speed 125.04 samples/sec Loss 2.5871 Epoch: 15 Global Step: 149550 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:28:04,431-Speed 129.83 samples/sec Loss 2.6870 Epoch: 15 Global Step: 149555 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:28:24,256-Speed 129.13 samples/sec Loss 2.7915 Epoch: 15 Global Step: 149560 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:28:44,075-Speed 129.18 samples/sec Loss 2.7376 Epoch: 15 Global Step: 149565 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:29:03,795-Speed 129.82 samples/sec Loss 2.7997 Epoch: 15 Global Step: 149570 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:29:23,747-Speed 128.32 samples/sec Loss 2.6508 Epoch: 15 Global Step: 149575 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:29:43,524-Speed 129.45 samples/sec Loss 2.8319 Epoch: 15 Global Step: 149580 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:30:03,324-Speed 129.30 samples/sec Loss 2.8801 Epoch: 15 Global Step: 149585 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:30:23,029-Speed 129.92 samples/sec Loss 2.7619 Epoch: 15 Global Step: 149590 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:30:43,230-Speed 126.73 samples/sec Loss 2.9385 Epoch: 15 Global Step: 149595 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:31:03,026-Speed 129.32 samples/sec Loss 2.8007 Epoch: 15 Global Step: 149600 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:31:22,819-Speed 129.35 samples/sec Loss 2.7738 Epoch: 15 Global Step: 149605 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:31:42,551-Speed 129.74 samples/sec Loss 2.7758 Epoch: 15 Global Step: 149610 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:32:02,353-Speed 129.29 samples/sec Loss 2.7145 Epoch: 15 Global Step: 149615 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:32:22,190-Speed 129.07 samples/sec Loss 2.8944 Epoch: 15 Global Step: 149620 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:32:41,914-Speed 129.80 samples/sec Loss 2.7471 Epoch: 15 Global Step: 149625 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:33:01,774-Speed 128.90 samples/sec Loss 2.9219 Epoch: 15 Global Step: 149630 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:33:21,487-Speed 129.87 samples/sec Loss 2.7464 Epoch: 15 Global Step: 149635 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:33:41,808-Speed 125.98 samples/sec Loss 2.9044 Epoch: 15 Global Step: 149640 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:34:01,689-Speed 128.77 samples/sec Loss 2.5007 Epoch: 15 Global Step: 149645 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:34:21,392-Speed 129.94 samples/sec Loss 2.9503 Epoch: 15 Global Step: 149650 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:34:41,155-Speed 129.55 samples/sec Loss 2.7069 Epoch: 15 Global Step: 149655 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:35:00,919-Speed 129.53 samples/sec Loss 2.8224 Epoch: 15 Global Step: 149660 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:35:20,705-Speed 129.39 samples/sec Loss 2.7196 Epoch: 15 Global Step: 149665 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:35:40,499-Speed 129.34 samples/sec Loss 2.6116 Epoch: 15 Global Step: 149670 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:36:00,236-Speed 129.71 samples/sec Loss 2.8738 Epoch: 15 Global Step: 149675 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:36:20,022-Speed 129.39 samples/sec Loss 2.8644 Epoch: 15 Global Step: 149680 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:36:40,310-Speed 126.19 samples/sec Loss 2.9170 Epoch: 15 Global Step: 149685 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:37:00,159-Speed 128.98 samples/sec Loss 3.0199 Epoch: 15 Global Step: 149690 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:37:19,884-Speed 129.79 samples/sec Loss 2.8152 Epoch: 15 Global Step: 149695 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:37:39,689-Speed 129.27 samples/sec Loss 2.8109 Epoch: 15 Global Step: 149700 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:37:59,471-Speed 129.42 samples/sec Loss 2.8186 Epoch: 15 Global Step: 149705 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:38:19,382-Speed 128.58 samples/sec Loss 2.8680 Epoch: 15 Global Step: 149710 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:38:39,052-Speed 130.15 samples/sec Loss 2.8595 Epoch: 15 Global Step: 149715 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:38:58,775-Speed 129.81 samples/sec Loss 2.8150 Epoch: 15 Global Step: 149720 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:39:18,632-Speed 128.93 samples/sec Loss 2.9469 Epoch: 15 Global Step: 149725 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:39:38,855-Speed 126.59 samples/sec Loss 2.8184 Epoch: 15 Global Step: 149730 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:39:59,206-Speed 125.80 samples/sec Loss 3.0274 Epoch: 15 Global Step: 149735 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:40:18,943-Speed 129.71 samples/sec Loss 2.9316 Epoch: 15 Global Step: 149740 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:40:38,841-Speed 128.67 samples/sec Loss 2.7743 Epoch: 15 Global Step: 149745 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:40:58,550-Speed 129.89 samples/sec Loss 2.7959 Epoch: 15 Global Step: 149750 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:41:18,576-Speed 127.85 samples/sec Loss 2.9312 Epoch: 15 Global Step: 149755 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:41:38,325-Speed 129.63 samples/sec Loss 2.7334 Epoch: 15 Global Step: 149760 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:41:58,152-Speed 129.13 samples/sec Loss 2.8305 Epoch: 15 Global Step: 149765 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:42:17,961-Speed 129.24 samples/sec Loss 2.8145 Epoch: 15 Global Step: 149770 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:42:38,206-Speed 126.46 samples/sec Loss 2.8328 Epoch: 15 Global Step: 149775 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:42:57,911-Speed 129.92 samples/sec Loss 2.8226 Epoch: 15 Global Step: 149780 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:43:17,796-Speed 128.74 samples/sec Loss 2.7107 Epoch: 15 Global Step: 149785 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:43:38,333-Speed 124.66 samples/sec Loss 2.8112 Epoch: 15 Global Step: 149790 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:43:58,206-Speed 128.82 samples/sec Loss 2.8428 Epoch: 15 Global Step: 149795 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:44:17,920-Speed 129.87 samples/sec Loss 2.9352 Epoch: 15 Global Step: 149800 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:44:37,577-Speed 130.24 samples/sec Loss 2.9941 Epoch: 15 Global Step: 149805 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:44:57,525-Speed 128.34 samples/sec Loss 2.5833 Epoch: 15 Global Step: 149810 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:45:17,262-Speed 129.72 samples/sec Loss 2.8421 Epoch: 15 Global Step: 149815 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:45:37,510-Speed 126.44 samples/sec Loss 2.8445 Epoch: 15 Global Step: 149820 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:45:57,248-Speed 129.70 samples/sec Loss 2.8957 Epoch: 15 Global Step: 149825 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:46:17,178-Speed 128.45 samples/sec Loss 2.7217 Epoch: 15 Global Step: 149830 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:46:36,874-Speed 129.99 samples/sec Loss 2.9495 Epoch: 15 Global Step: 149835 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:46:56,779-Speed 128.62 samples/sec Loss 2.8997 Epoch: 15 Global Step: 149840 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:47:16,507-Speed 129.77 samples/sec Loss 2.8818 Epoch: 15 Global Step: 149845 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:47:36,313-Speed 129.26 samples/sec Loss 2.7448 Epoch: 15 Global Step: 149850 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:47:56,123-Speed 129.24 samples/sec Loss 2.7803 Epoch: 15 Global Step: 149855 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:48:15,830-Speed 129.90 samples/sec Loss 2.8622 Epoch: 15 Global Step: 149860 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:48:36,068-Speed 126.50 samples/sec Loss 2.9853 Epoch: 15 Global Step: 149865 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:48:56,026-Speed 128.28 samples/sec Loss 2.7796 Epoch: 15 Global Step: 149870 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:49:15,742-Speed 129.85 samples/sec Loss 2.7429 Epoch: 15 Global Step: 149875 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:49:35,631-Speed 128.72 samples/sec Loss 2.8682 Epoch: 15 Global Step: 149880 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:49:55,245-Speed 130.53 samples/sec Loss 2.8805 Epoch: 15 Global Step: 149885 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:50:15,085-Speed 129.04 samples/sec Loss 2.9263 Epoch: 15 Global Step: 149890 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:50:34,958-Speed 128.82 samples/sec Loss 3.0671 Epoch: 15 Global Step: 149895 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:50:54,807-Speed 128.99 samples/sec Loss 2.7658 Epoch: 15 Global Step: 149900 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:51:14,776-Speed 128.21 samples/sec Loss 2.7705 Epoch: 15 Global Step: 149905 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:51:35,055-Speed 126.24 samples/sec Loss 2.9373 Epoch: 15 Global Step: 149910 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:51:54,928-Speed 128.82 samples/sec Loss 2.9396 Epoch: 15 Global Step: 149915 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:52:14,590-Speed 130.21 samples/sec Loss 2.8993 Epoch: 15 Global Step: 149920 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:52:34,754-Speed 126.97 samples/sec Loss 2.9436 Epoch: 15 Global Step: 149925 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:52:54,453-Speed 129.97 samples/sec Loss 2.8575 Epoch: 15 Global Step: 149930 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:53:14,416-Speed 128.24 samples/sec Loss 2.8814 Epoch: 15 Global Step: 149935 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:53:34,133-Speed 129.85 samples/sec Loss 3.0386 Epoch: 15 Global Step: 149940 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:53:53,915-Speed 129.42 samples/sec Loss 2.8797 Epoch: 15 Global Step: 149945 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:54:13,637-Speed 129.81 samples/sec Loss 2.6556 Epoch: 15 Global Step: 149950 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:54:34,095-Speed 125.14 samples/sec Loss 3.1130 Epoch: 15 Global Step: 149955 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:54:53,707-Speed 130.53 samples/sec Loss 2.9452 Epoch: 15 Global Step: 149960 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:55:13,458-Speed 129.62 samples/sec Loss 2.7752 Epoch: 15 Global Step: 149965 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:55:33,218-Speed 129.56 samples/sec Loss 2.8603 Epoch: 15 Global Step: 149970 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:55:52,930-Speed 129.88 samples/sec Loss 2.7758 Epoch: 15 Global Step: 149975 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:56:12,831-Speed 128.64 samples/sec Loss 2.9001 Epoch: 15 Global Step: 149980 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:56:32,436-Speed 130.59 samples/sec Loss 3.0184 Epoch: 15 Global Step: 149985 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:56:52,332-Speed 128.67 samples/sec Loss 2.7849 Epoch: 15 Global Step: 149990 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:57:12,098-Speed 129.52 samples/sec Loss 2.8894 Epoch: 15 Global Step: 149995 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 03:57:32,497-Speed 125.50 samples/sec Loss 2.6712 Epoch: 15 Global Step: 150000 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:12:19,895-[lfw][150000]XNorm: 21.083983 Training: 2021-06-14 04:12:19,895-[lfw][150000]Accuracy-Flip: 0.99817+-0.00229 Training: 2021-06-14 04:12:19,895-[lfw][150000]Accuracy-Highest: 0.99833 Training: 2021-06-14 04:29:37,217-[cfp_fp][150000]XNorm: 21.025839 Training: 2021-06-14 04:29:37,217-[cfp_fp][150000]Accuracy-Flip: 0.98886+-0.00291 Training: 2021-06-14 04:29:37,217-[cfp_fp][150000]Accuracy-Highest: 0.99071 Training: 2021-06-14 04:44:25,198-[agedb_30][150000]XNorm: 21.818736 Training: 2021-06-14 04:44:25,199-[agedb_30][150000]Accuracy-Flip: 0.98217+-0.00723 Training: 2021-06-14 04:44:25,199-[agedb_30][150000]Accuracy-Highest: 0.98417 Training: 2021-06-14 04:44:44,850-Speed 0.90 samples/sec Loss 2.9541 Epoch: 15 Global Step: 150005 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:45:04,588-Speed 129.71 samples/sec Loss 2.6857 Epoch: 15 Global Step: 150010 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:45:24,025-Speed 131.71 samples/sec Loss 2.9191 Epoch: 15 Global Step: 150015 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:45:43,642-Speed 130.51 samples/sec Loss 2.7596 Epoch: 15 Global Step: 150020 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:46:03,218-Speed 130.78 samples/sec Loss 2.6872 Epoch: 15 Global Step: 150025 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:46:22,670-Speed 131.62 samples/sec Loss 2.9734 Epoch: 15 Global Step: 150030 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:46:42,465-Speed 129.33 samples/sec Loss 2.7814 Epoch: 15 Global Step: 150035 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:47:02,125-Speed 130.22 samples/sec Loss 2.9014 Epoch: 15 Global Step: 150040 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:47:21,568-Speed 131.68 samples/sec Loss 2.8720 Epoch: 15 Global Step: 150045 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:47:41,033-Speed 131.53 samples/sec Loss 2.8372 Epoch: 15 Global Step: 150050 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:48:00,612-Speed 130.76 samples/sec Loss 2.9748 Epoch: 15 Global Step: 150055 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:48:20,036-Speed 131.80 samples/sec Loss 2.8485 Epoch: 15 Global Step: 150060 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:48:39,669-Speed 130.40 samples/sec Loss 2.8025 Epoch: 15 Global Step: 150065 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:48:59,138-Speed 131.50 samples/sec Loss 2.8254 Epoch: 15 Global Step: 150070 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:49:18,711-Speed 130.80 samples/sec Loss 2.8629 Epoch: 15 Global Step: 150075 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:49:38,475-Speed 129.54 samples/sec Loss 2.9545 Epoch: 15 Global Step: 150080 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:49:58,074-Speed 130.63 samples/sec Loss 2.9146 Epoch: 15 Global Step: 150085 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:50:17,508-Speed 131.74 samples/sec Loss 2.9728 Epoch: 15 Global Step: 150090 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:50:37,104-Speed 130.65 samples/sec Loss 2.9266 Epoch: 15 Global Step: 150095 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:50:56,554-Speed 131.63 samples/sec Loss 2.9231 Epoch: 15 Global Step: 150100 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:51:16,045-Speed 131.35 samples/sec Loss 2.9815 Epoch: 15 Global Step: 150105 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:51:35,661-Speed 130.51 samples/sec Loss 2.8144 Epoch: 15 Global Step: 150110 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:51:55,195-Speed 131.06 samples/sec Loss 2.9608 Epoch: 15 Global Step: 150115 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:52:14,764-Speed 130.82 samples/sec Loss 2.8921 Epoch: 15 Global Step: 150120 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:52:34,567-Speed 129.28 samples/sec Loss 3.0319 Epoch: 15 Global Step: 150125 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:52:54,253-Speed 130.05 samples/sec Loss 2.8306 Epoch: 15 Global Step: 150130 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:53:13,851-Speed 130.63 samples/sec Loss 2.9547 Epoch: 15 Global Step: 150135 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:53:33,455-Speed 130.59 samples/sec Loss 3.0050 Epoch: 15 Global Step: 150140 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:53:53,007-Speed 130.94 samples/sec Loss 2.8648 Epoch: 15 Global Step: 150145 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:54:12,619-Speed 130.54 samples/sec Loss 3.0814 Epoch: 15 Global Step: 150150 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:54:32,061-Speed 131.68 samples/sec Loss 3.0947 Epoch: 15 Global Step: 150155 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:54:51,673-Speed 130.54 samples/sec Loss 3.0299 Epoch: 15 Global Step: 150160 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:55:11,232-Speed 130.89 samples/sec Loss 3.0194 Epoch: 15 Global Step: 150165 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:55:31,170-Speed 128.41 samples/sec Loss 2.8162 Epoch: 15 Global Step: 150170 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:55:50,592-Speed 131.81 samples/sec Loss 2.8449 Epoch: 15 Global Step: 150175 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:56:10,300-Speed 129.90 samples/sec Loss 2.7921 Epoch: 15 Global Step: 150180 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:56:29,838-Speed 131.03 samples/sec Loss 2.8259 Epoch: 15 Global Step: 150185 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:56:49,552-Speed 129.87 samples/sec Loss 2.8716 Epoch: 15 Global Step: 150190 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:57:09,157-Speed 130.59 samples/sec Loss 3.1051 Epoch: 15 Global Step: 150195 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:57:28,769-Speed 130.54 samples/sec Loss 2.8577 Epoch: 15 Global Step: 150200 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:57:48,367-Speed 130.63 samples/sec Loss 3.1239 Epoch: 15 Global Step: 150205 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:58:08,109-Speed 129.68 samples/sec Loss 2.8242 Epoch: 15 Global Step: 150210 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:58:27,914-Speed 129.27 samples/sec Loss 2.9120 Epoch: 15 Global Step: 150215 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:58:47,480-Speed 130.84 samples/sec Loss 2.9563 Epoch: 15 Global Step: 150220 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:59:07,489-Speed 127.95 samples/sec Loss 2.8978 Epoch: 15 Global Step: 150225 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:59:27,060-Speed 130.82 samples/sec Loss 3.0635 Epoch: 15 Global Step: 150230 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 04:59:46,645-Speed 130.72 samples/sec Loss 3.0390 Epoch: 15 Global Step: 150235 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:00:06,190-Speed 130.99 samples/sec Loss 2.9981 Epoch: 15 Global Step: 150240 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:00:25,719-Speed 131.09 samples/sec Loss 2.8910 Epoch: 15 Global Step: 150245 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:00:45,294-Speed 130.78 samples/sec Loss 2.9520 Epoch: 15 Global Step: 150250 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:01:05,002-Speed 129.91 samples/sec Loss 2.9281 Epoch: 15 Global Step: 150255 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:01:24,807-Speed 129.27 samples/sec Loss 2.8566 Epoch: 15 Global Step: 150260 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:01:44,550-Speed 129.68 samples/sec Loss 2.9793 Epoch: 15 Global Step: 150265 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:02:04,074-Speed 131.13 samples/sec Loss 3.0582 Epoch: 15 Global Step: 150270 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:02:23,601-Speed 131.10 samples/sec Loss 2.9192 Epoch: 15 Global Step: 150275 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:02:43,199-Speed 130.64 samples/sec Loss 3.0342 Epoch: 15 Global Step: 150280 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:03:02,764-Speed 130.86 samples/sec Loss 2.9205 Epoch: 15 Global Step: 150285 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:03:22,242-Speed 131.44 samples/sec Loss 3.0222 Epoch: 15 Global Step: 150290 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:03:41,800-Speed 130.91 samples/sec Loss 2.9380 Epoch: 15 Global Step: 150295 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:04:01,328-Speed 131.10 samples/sec Loss 2.8360 Epoch: 15 Global Step: 150300 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:04:21,132-Speed 129.28 samples/sec Loss 2.8830 Epoch: 15 Global Step: 150305 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:04:40,767-Speed 130.38 samples/sec Loss 3.0523 Epoch: 15 Global Step: 150310 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:05:00,301-Speed 131.06 samples/sec Loss 3.1846 Epoch: 15 Global Step: 150315 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:05:19,802-Speed 131.28 samples/sec Loss 2.9476 Epoch: 15 Global Step: 150320 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:05:39,414-Speed 130.54 samples/sec Loss 2.8558 Epoch: 15 Global Step: 150325 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:05:58,947-Speed 131.07 samples/sec Loss 2.7593 Epoch: 15 Global Step: 150330 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:06:18,566-Speed 130.49 samples/sec Loss 3.1772 Epoch: 15 Global Step: 150335 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:06:38,115-Speed 130.96 samples/sec Loss 2.9506 Epoch: 15 Global Step: 150340 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:06:57,651-Speed 131.05 samples/sec Loss 2.7266 Epoch: 15 Global Step: 150345 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:07:17,450-Speed 129.31 samples/sec Loss 3.1004 Epoch: 15 Global Step: 150350 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:07:37,064-Speed 130.52 samples/sec Loss 3.1178 Epoch: 15 Global Step: 150355 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:07:56,643-Speed 130.75 samples/sec Loss 2.8382 Epoch: 15 Global Step: 150360 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:08:16,243-Speed 130.63 samples/sec Loss 2.8335 Epoch: 15 Global Step: 150365 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:08:35,868-Speed 130.45 samples/sec Loss 2.7553 Epoch: 15 Global Step: 150370 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:08:55,397-Speed 131.09 samples/sec Loss 2.9348 Epoch: 15 Global Step: 150375 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:09:14,970-Speed 130.80 samples/sec Loss 3.1135 Epoch: 15 Global Step: 150380 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:09:34,513-Speed 131.00 samples/sec Loss 3.1518 Epoch: 15 Global Step: 150385 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:09:54,194-Speed 130.08 samples/sec Loss 2.8071 Epoch: 15 Global Step: 150390 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:10:14,170-Speed 128.16 samples/sec Loss 2.9406 Epoch: 15 Global Step: 150395 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:10:33,618-Speed 131.64 samples/sec Loss 3.0054 Epoch: 15 Global Step: 150400 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:10:53,156-Speed 131.03 samples/sec Loss 2.9177 Epoch: 15 Global Step: 150405 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:11:12,763-Speed 130.57 samples/sec Loss 2.8499 Epoch: 15 Global Step: 150410 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:11:32,212-Speed 131.63 samples/sec Loss 2.9760 Epoch: 15 Global Step: 150415 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:11:52,659-Speed 125.21 samples/sec Loss 2.7510 Epoch: 15 Global Step: 150420 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:12:12,190-Speed 131.08 samples/sec Loss 2.8376 Epoch: 15 Global Step: 150425 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:12:31,687-Speed 131.31 samples/sec Loss 2.7544 Epoch: 15 Global Step: 150430 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:12:51,296-Speed 130.56 samples/sec Loss 3.0252 Epoch: 15 Global Step: 150435 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:13:11,176-Speed 128.78 samples/sec Loss 2.8531 Epoch: 15 Global Step: 150440 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:13:30,608-Speed 131.75 samples/sec Loss 2.9137 Epoch: 15 Global Step: 150445 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:13:50,318-Speed 129.89 samples/sec Loss 3.0025 Epoch: 15 Global Step: 150450 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:14:09,933-Speed 130.52 samples/sec Loss 3.0569 Epoch: 15 Global Step: 150455 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:14:29,490-Speed 130.91 samples/sec Loss 3.2039 Epoch: 15 Global Step: 150460 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:14:49,046-Speed 130.91 samples/sec Loss 2.9428 Epoch: 15 Global Step: 150465 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:15:08,554-Speed 131.23 samples/sec Loss 2.9969 Epoch: 15 Global Step: 150470 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:15:28,156-Speed 130.61 samples/sec Loss 2.7905 Epoch: 15 Global Step: 150475 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:15:47,771-Speed 130.52 samples/sec Loss 2.9451 Epoch: 15 Global Step: 150480 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:16:07,652-Speed 128.78 samples/sec Loss 3.0743 Epoch: 15 Global Step: 150485 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:16:27,212-Speed 130.89 samples/sec Loss 2.8756 Epoch: 15 Global Step: 150490 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:16:46,867-Speed 130.26 samples/sec Loss 2.9712 Epoch: 15 Global Step: 150495 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:17:06,397-Speed 131.09 samples/sec Loss 2.8750 Epoch: 15 Global Step: 150500 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:17:25,998-Speed 130.61 samples/sec Loss 2.9136 Epoch: 15 Global Step: 150505 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:17:45,510-Speed 131.21 samples/sec Loss 3.1042 Epoch: 15 Global Step: 150510 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:18:04,986-Speed 131.45 samples/sec Loss 2.9084 Epoch: 15 Global Step: 150515 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:18:24,583-Speed 130.64 samples/sec Loss 2.9364 Epoch: 15 Global Step: 150520 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:18:44,132-Speed 130.96 samples/sec Loss 3.1539 Epoch: 15 Global Step: 150525 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:19:03,941-Speed 129.24 samples/sec Loss 2.8614 Epoch: 15 Global Step: 150530 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:19:23,654-Speed 129.87 samples/sec Loss 3.1793 Epoch: 15 Global Step: 150535 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:19:43,195-Speed 131.01 samples/sec Loss 2.8813 Epoch: 15 Global Step: 150540 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:20:02,796-Speed 130.62 samples/sec Loss 2.7983 Epoch: 15 Global Step: 150545 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:20:22,338-Speed 131.00 samples/sec Loss 2.8859 Epoch: 15 Global Step: 150550 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:20:41,900-Speed 130.87 samples/sec Loss 3.3284 Epoch: 15 Global Step: 150555 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:21:01,548-Speed 130.31 samples/sec Loss 3.0194 Epoch: 15 Global Step: 150560 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:21:21,090-Speed 131.01 samples/sec Loss 3.1455 Epoch: 15 Global Step: 150565 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:21:40,685-Speed 130.65 samples/sec Loss 2.9558 Epoch: 15 Global Step: 150570 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:22:00,348-Speed 130.20 samples/sec Loss 3.0502 Epoch: 15 Global Step: 150575 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:22:19,926-Speed 130.77 samples/sec Loss 3.0766 Epoch: 15 Global Step: 150580 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:22:39,495-Speed 130.82 samples/sec Loss 3.0640 Epoch: 15 Global Step: 150585 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:22:59,090-Speed 130.65 samples/sec Loss 2.8883 Epoch: 15 Global Step: 150590 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:23:18,630-Speed 131.02 samples/sec Loss 2.9322 Epoch: 15 Global Step: 150595 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:23:38,232-Speed 130.61 samples/sec Loss 3.0864 Epoch: 15 Global Step: 150600 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:23:57,801-Speed 130.82 samples/sec Loss 3.0672 Epoch: 15 Global Step: 150605 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:24:17,409-Speed 130.57 samples/sec Loss 2.8450 Epoch: 15 Global Step: 150610 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:24:37,023-Speed 130.53 samples/sec Loss 3.1177 Epoch: 15 Global Step: 150615 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:24:56,935-Speed 128.57 samples/sec Loss 3.0748 Epoch: 15 Global Step: 150620 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:25:16,376-Speed 131.68 samples/sec Loss 3.0989 Epoch: 15 Global Step: 150625 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:25:36,033-Speed 130.24 samples/sec Loss 3.1866 Epoch: 15 Global Step: 150630 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:25:55,609-Speed 130.78 samples/sec Loss 2.9889 Epoch: 15 Global Step: 150635 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:26:15,131-Speed 131.14 samples/sec Loss 2.9561 Epoch: 15 Global Step: 150640 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:26:34,727-Speed 130.65 samples/sec Loss 3.0344 Epoch: 15 Global Step: 150645 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:26:54,273-Speed 130.98 samples/sec Loss 2.8988 Epoch: 15 Global Step: 150650 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:27:13,878-Speed 130.59 samples/sec Loss 2.9258 Epoch: 15 Global Step: 150655 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:27:33,321-Speed 131.67 samples/sec Loss 2.9025 Epoch: 15 Global Step: 150660 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:27:53,208-Speed 128.74 samples/sec Loss 3.2075 Epoch: 15 Global Step: 150665 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:28:12,772-Speed 130.86 samples/sec Loss 2.9867 Epoch: 15 Global Step: 150670 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:28:32,718-Speed 128.35 samples/sec Loss 3.0487 Epoch: 15 Global Step: 150675 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:28:52,251-Speed 131.07 samples/sec Loss 2.8298 Epoch: 15 Global Step: 150680 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:29:11,872-Speed 130.48 samples/sec Loss 2.9511 Epoch: 15 Global Step: 150685 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:29:31,439-Speed 130.84 samples/sec Loss 3.0044 Epoch: 15 Global Step: 150690 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:29:50,976-Speed 131.04 samples/sec Loss 3.1586 Epoch: 15 Global Step: 150695 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:30:10,534-Speed 130.90 samples/sec Loss 2.9762 Epoch: 15 Global Step: 150700 Fp16 Grad Scale: 8192 Required: 117 hours Training: 2021-06-14 05:30:30,127-Speed 130.66 samples/sec Loss 2.8860 Epoch: 15 Global Step: 150705 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:30:49,953-Speed 129.13 samples/sec Loss 2.9283 Epoch: 15 Global Step: 150710 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:31:09,695-Speed 129.68 samples/sec Loss 3.1549 Epoch: 15 Global Step: 150715 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:31:29,174-Speed 131.43 samples/sec Loss 2.9959 Epoch: 15 Global Step: 150720 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:31:48,702-Speed 131.11 samples/sec Loss 2.9272 Epoch: 15 Global Step: 150725 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:32:08,304-Speed 130.61 samples/sec Loss 3.0264 Epoch: 15 Global Step: 150730 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:32:27,824-Speed 131.15 samples/sec Loss 3.0051 Epoch: 15 Global Step: 150735 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:32:47,421-Speed 130.64 samples/sec Loss 2.9766 Epoch: 15 Global Step: 150740 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:33:06,880-Speed 131.57 samples/sec Loss 2.9874 Epoch: 15 Global Step: 150745 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:33:26,433-Speed 130.94 samples/sec Loss 3.0101 Epoch: 15 Global Step: 150750 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:33:46,248-Speed 129.21 samples/sec Loss 2.8484 Epoch: 15 Global Step: 150755 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:34:06,000-Speed 129.61 samples/sec Loss 3.1163 Epoch: 15 Global Step: 150760 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:34:25,478-Speed 131.44 samples/sec Loss 3.1599 Epoch: 15 Global Step: 150765 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:34:45,084-Speed 130.58 samples/sec Loss 2.9073 Epoch: 15 Global Step: 150770 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:35:05,577-Speed 124.93 samples/sec Loss 2.9003 Epoch: 15 Global Step: 150775 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:35:25,118-Speed 131.02 samples/sec Loss 3.1677 Epoch: 15 Global Step: 150780 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:35:44,694-Speed 130.78 samples/sec Loss 3.0740 Epoch: 15 Global Step: 150785 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:36:04,239-Speed 130.99 samples/sec Loss 3.0834 Epoch: 15 Global Step: 150790 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:36:23,825-Speed 130.71 samples/sec Loss 3.1252 Epoch: 15 Global Step: 150795 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:36:43,742-Speed 128.54 samples/sec Loss 3.0405 Epoch: 15 Global Step: 150800 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:37:03,177-Speed 131.73 samples/sec Loss 2.8470 Epoch: 15 Global Step: 150805 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:37:22,732-Speed 130.92 samples/sec Loss 3.0068 Epoch: 15 Global Step: 150810 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:37:42,382-Speed 130.29 samples/sec Loss 3.0186 Epoch: 15 Global Step: 150815 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:38:01,932-Speed 130.95 samples/sec Loss 2.9081 Epoch: 15 Global Step: 150820 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:38:21,536-Speed 130.59 samples/sec Loss 3.0450 Epoch: 15 Global Step: 150825 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:38:40,993-Speed 131.58 samples/sec Loss 3.0175 Epoch: 15 Global Step: 150830 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:39:00,609-Speed 130.52 samples/sec Loss 2.9420 Epoch: 15 Global Step: 150835 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:39:20,185-Speed 130.78 samples/sec Loss 2.9987 Epoch: 15 Global Step: 150840 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:39:40,169-Speed 128.11 samples/sec Loss 2.8860 Epoch: 15 Global Step: 150845 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:39:59,594-Speed 131.80 samples/sec Loss 3.1776 Epoch: 15 Global Step: 150850 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:40:19,321-Speed 129.78 samples/sec Loss 3.0850 Epoch: 15 Global Step: 150855 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:40:38,853-Speed 131.08 samples/sec Loss 2.8937 Epoch: 15 Global Step: 150860 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:40:58,460-Speed 130.57 samples/sec Loss 3.0298 Epoch: 15 Global Step: 150865 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:41:17,911-Speed 131.62 samples/sec Loss 2.9897 Epoch: 15 Global Step: 150870 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:41:37,450-Speed 131.02 samples/sec Loss 3.0499 Epoch: 15 Global Step: 150875 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:41:57,047-Speed 130.64 samples/sec Loss 2.8743 Epoch: 15 Global Step: 150880 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:42:16,577-Speed 131.09 samples/sec Loss 3.0152 Epoch: 15 Global Step: 150885 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:42:36,428-Speed 128.97 samples/sec Loss 3.1451 Epoch: 15 Global Step: 150890 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:42:55,897-Speed 131.50 samples/sec Loss 3.1514 Epoch: 15 Global Step: 150895 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:43:15,561-Speed 130.19 samples/sec Loss 3.0204 Epoch: 15 Global Step: 150900 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:43:35,088-Speed 131.11 samples/sec Loss 3.0755 Epoch: 15 Global Step: 150905 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:43:54,711-Speed 130.47 samples/sec Loss 3.0383 Epoch: 15 Global Step: 150910 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:44:14,204-Speed 131.34 samples/sec Loss 3.2838 Epoch: 15 Global Step: 150915 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:44:33,742-Speed 131.04 samples/sec Loss 3.1195 Epoch: 15 Global Step: 150920 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:44:53,333-Speed 130.67 samples/sec Loss 3.0945 Epoch: 15 Global Step: 150925 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:45:12,827-Speed 131.33 samples/sec Loss 2.8895 Epoch: 15 Global Step: 150930 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:45:32,597-Speed 129.50 samples/sec Loss 3.0478 Epoch: 15 Global Step: 150935 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:45:52,290-Speed 130.00 samples/sec Loss 3.0610 Epoch: 15 Global Step: 150940 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:46:11,848-Speed 130.90 samples/sec Loss 3.0011 Epoch: 15 Global Step: 150945 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:46:31,440-Speed 130.67 samples/sec Loss 2.9092 Epoch: 15 Global Step: 150950 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:46:50,877-Speed 131.71 samples/sec Loss 3.2005 Epoch: 15 Global Step: 150955 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:47:10,416-Speed 131.03 samples/sec Loss 2.9964 Epoch: 15 Global Step: 150960 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:47:30,028-Speed 130.54 samples/sec Loss 2.9923 Epoch: 15 Global Step: 150965 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:47:49,559-Speed 131.08 samples/sec Loss 3.0204 Epoch: 15 Global Step: 150970 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:48:09,171-Speed 130.54 samples/sec Loss 3.1011 Epoch: 15 Global Step: 150975 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:48:28,848-Speed 130.11 samples/sec Loss 3.0968 Epoch: 15 Global Step: 150980 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:48:48,468-Speed 130.49 samples/sec Loss 2.9563 Epoch: 15 Global Step: 150985 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:49:08,032-Speed 130.85 samples/sec Loss 3.0101 Epoch: 15 Global Step: 150990 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:49:27,665-Speed 130.40 samples/sec Loss 3.2557 Epoch: 15 Global Step: 150995 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:49:47,182-Speed 131.17 samples/sec Loss 3.0509 Epoch: 15 Global Step: 151000 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:50:06,792-Speed 130.56 samples/sec Loss 3.1192 Epoch: 15 Global Step: 151005 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:50:26,406-Speed 130.52 samples/sec Loss 3.2996 Epoch: 15 Global Step: 151010 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:50:45,979-Speed 130.80 samples/sec Loss 3.1554 Epoch: 15 Global Step: 151015 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:51:05,556-Speed 130.77 samples/sec Loss 3.0743 Epoch: 15 Global Step: 151020 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:51:25,448-Speed 128.70 samples/sec Loss 3.0179 Epoch: 15 Global Step: 151025 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:51:44,880-Speed 131.75 samples/sec Loss 2.7910 Epoch: 15 Global Step: 151030 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:52:04,626-Speed 129.65 samples/sec Loss 3.2038 Epoch: 15 Global Step: 151035 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:52:24,218-Speed 130.68 samples/sec Loss 3.0691 Epoch: 15 Global Step: 151040 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:52:43,753-Speed 131.05 samples/sec Loss 2.9319 Epoch: 15 Global Step: 151045 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:53:03,781-Speed 127.83 samples/sec Loss 3.2432 Epoch: 15 Global Step: 151050 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:53:23,305-Speed 131.13 samples/sec Loss 3.1647 Epoch: 15 Global Step: 151055 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:53:42,852-Speed 130.97 samples/sec Loss 2.9130 Epoch: 15 Global Step: 151060 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:54:02,433-Speed 130.75 samples/sec Loss 3.2520 Epoch: 15 Global Step: 151065 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:54:22,346-Speed 128.57 samples/sec Loss 3.1192 Epoch: 15 Global Step: 151070 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:54:41,815-Speed 131.49 samples/sec Loss 3.1545 Epoch: 15 Global Step: 151075 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:55:01,396-Speed 130.75 samples/sec Loss 2.9393 Epoch: 15 Global Step: 151080 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:55:21,078-Speed 130.08 samples/sec Loss 3.1288 Epoch: 15 Global Step: 151085 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:55:40,608-Speed 131.09 samples/sec Loss 3.0105 Epoch: 15 Global Step: 151090 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:56:00,317-Speed 129.90 samples/sec Loss 2.9762 Epoch: 15 Global Step: 151095 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:56:19,872-Speed 130.92 samples/sec Loss 3.0567 Epoch: 15 Global Step: 151100 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:56:39,474-Speed 130.60 samples/sec Loss 3.1446 Epoch: 15 Global Step: 151105 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:56:58,918-Speed 131.67 samples/sec Loss 3.1129 Epoch: 15 Global Step: 151110 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:57:18,844-Speed 128.49 samples/sec Loss 3.2405 Epoch: 15 Global Step: 151115 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:57:38,301-Speed 131.57 samples/sec Loss 3.1765 Epoch: 15 Global Step: 151120 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:57:58,031-Speed 129.76 samples/sec Loss 2.9378 Epoch: 15 Global Step: 151125 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:58:17,561-Speed 131.09 samples/sec Loss 2.9959 Epoch: 15 Global Step: 151130 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:58:37,179-Speed 130.50 samples/sec Loss 3.1953 Epoch: 15 Global Step: 151135 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:58:56,740-Speed 130.88 samples/sec Loss 3.2521 Epoch: 15 Global Step: 151140 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:59:16,301-Speed 130.88 samples/sec Loss 2.9673 Epoch: 15 Global Step: 151145 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:59:36,004-Speed 129.94 samples/sec Loss 3.1812 Epoch: 15 Global Step: 151150 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 05:59:55,649-Speed 130.32 samples/sec Loss 2.9078 Epoch: 15 Global Step: 151155 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:00:15,446-Speed 129.32 samples/sec Loss 3.1862 Epoch: 15 Global Step: 151160 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:00:34,953-Speed 131.24 samples/sec Loss 2.9564 Epoch: 15 Global Step: 151165 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:00:54,645-Speed 130.01 samples/sec Loss 2.9256 Epoch: 15 Global Step: 151170 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:01:14,171-Speed 131.12 samples/sec Loss 3.1150 Epoch: 15 Global Step: 151175 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:01:33,773-Speed 130.60 samples/sec Loss 3.1802 Epoch: 15 Global Step: 151180 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:01:53,328-Speed 130.92 samples/sec Loss 3.0735 Epoch: 15 Global Step: 151185 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:02:12,824-Speed 131.32 samples/sec Loss 3.1389 Epoch: 15 Global Step: 151190 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:02:32,374-Speed 130.96 samples/sec Loss 3.0244 Epoch: 15 Global Step: 151195 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:02:52,121-Speed 129.65 samples/sec Loss 3.1326 Epoch: 15 Global Step: 151200 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:03:11,864-Speed 129.67 samples/sec Loss 3.0238 Epoch: 15 Global Step: 151205 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:03:31,613-Speed 129.64 samples/sec Loss 2.9621 Epoch: 15 Global Step: 151210 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:03:51,035-Speed 131.81 samples/sec Loss 3.1128 Epoch: 15 Global Step: 151215 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:04:10,660-Speed 130.46 samples/sec Loss 3.3387 Epoch: 15 Global Step: 151220 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:04:30,208-Speed 130.97 samples/sec Loss 3.0194 Epoch: 15 Global Step: 151225 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:04:49,742-Speed 131.06 samples/sec Loss 3.0748 Epoch: 15 Global Step: 151230 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:05:09,338-Speed 130.64 samples/sec Loss 3.2942 Epoch: 15 Global Step: 151235 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:05:28,808-Speed 131.49 samples/sec Loss 3.0708 Epoch: 15 Global Step: 151240 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:05:48,550-Speed 129.68 samples/sec Loss 3.1083 Epoch: 15 Global Step: 151245 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:06:08,332-Speed 129.42 samples/sec Loss 3.0728 Epoch: 15 Global Step: 151250 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:06:28,031-Speed 129.96 samples/sec Loss 3.1279 Epoch: 15 Global Step: 151255 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:06:47,554-Speed 131.14 samples/sec Loss 3.0714 Epoch: 15 Global Step: 151260 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:07:07,098-Speed 131.00 samples/sec Loss 3.1008 Epoch: 15 Global Step: 151265 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:07:26,688-Speed 130.68 samples/sec Loss 2.9746 Epoch: 15 Global Step: 151270 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:07:46,279-Speed 130.68 samples/sec Loss 3.1542 Epoch: 15 Global Step: 151275 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:08:05,926-Speed 130.31 samples/sec Loss 3.0944 Epoch: 15 Global Step: 151280 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:08:25,547-Speed 130.48 samples/sec Loss 2.9655 Epoch: 15 Global Step: 151285 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:08:45,064-Speed 131.17 samples/sec Loss 3.2981 Epoch: 15 Global Step: 151290 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:09:04,854-Speed 129.37 samples/sec Loss 3.2880 Epoch: 15 Global Step: 151295 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:09:24,516-Speed 130.21 samples/sec Loss 3.1667 Epoch: 15 Global Step: 151300 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:09:44,124-Speed 130.57 samples/sec Loss 3.1642 Epoch: 15 Global Step: 151305 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:10:03,737-Speed 130.53 samples/sec Loss 3.0778 Epoch: 15 Global Step: 151310 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:10:23,273-Speed 131.05 samples/sec Loss 3.1199 Epoch: 15 Global Step: 151315 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:10:42,727-Speed 131.60 samples/sec Loss 3.2027 Epoch: 15 Global Step: 151320 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:11:02,312-Speed 130.72 samples/sec Loss 3.3368 Epoch: 15 Global Step: 151325 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:11:21,891-Speed 130.76 samples/sec Loss 3.2054 Epoch: 15 Global Step: 151330 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:11:41,502-Speed 130.55 samples/sec Loss 3.2810 Epoch: 15 Global Step: 151335 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:12:01,267-Speed 129.53 samples/sec Loss 3.1032 Epoch: 15 Global Step: 151340 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:12:20,876-Speed 130.56 samples/sec Loss 3.0922 Epoch: 15 Global Step: 151345 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:12:40,418-Speed 131.01 samples/sec Loss 3.1927 Epoch: 15 Global Step: 151350 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:13:00,030-Speed 130.54 samples/sec Loss 3.1840 Epoch: 15 Global Step: 151355 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:13:19,485-Speed 131.59 samples/sec Loss 3.0468 Epoch: 15 Global Step: 151360 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:13:39,031-Speed 130.98 samples/sec Loss 3.2473 Epoch: 15 Global Step: 151365 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:13:58,632-Speed 130.61 samples/sec Loss 3.1077 Epoch: 15 Global Step: 151370 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:14:18,180-Speed 130.97 samples/sec Loss 3.1000 Epoch: 15 Global Step: 151375 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:14:37,785-Speed 130.59 samples/sec Loss 3.1522 Epoch: 15 Global Step: 151380 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:14:57,580-Speed 129.33 samples/sec Loss 3.1251 Epoch: 15 Global Step: 151385 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:15:17,008-Speed 131.78 samples/sec Loss 2.9528 Epoch: 15 Global Step: 151390 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:15:36,706-Speed 129.96 samples/sec Loss 3.2590 Epoch: 15 Global Step: 151395 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:15:56,338-Speed 130.41 samples/sec Loss 2.8830 Epoch: 15 Global Step: 151400 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:16:15,782-Speed 131.67 samples/sec Loss 3.0626 Epoch: 15 Global Step: 151405 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:16:35,379-Speed 130.64 samples/sec Loss 3.1872 Epoch: 15 Global Step: 151410 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:16:54,907-Speed 131.10 samples/sec Loss 3.3124 Epoch: 15 Global Step: 151415 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:17:14,455-Speed 130.96 samples/sec Loss 2.9911 Epoch: 15 Global Step: 151420 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:17:34,032-Speed 130.78 samples/sec Loss 2.9609 Epoch: 15 Global Step: 151425 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:17:53,840-Speed 129.25 samples/sec Loss 3.1034 Epoch: 15 Global Step: 151430 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:18:13,287-Speed 131.64 samples/sec Loss 3.0401 Epoch: 15 Global Step: 151435 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:18:33,044-Speed 129.58 samples/sec Loss 3.0401 Epoch: 15 Global Step: 151440 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:18:52,620-Speed 130.78 samples/sec Loss 3.0509 Epoch: 15 Global Step: 151445 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:19:12,058-Speed 131.71 samples/sec Loss 3.0389 Epoch: 15 Global Step: 151450 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:19:31,677-Speed 130.49 samples/sec Loss 3.1002 Epoch: 15 Global Step: 151455 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:19:51,208-Speed 131.08 samples/sec Loss 3.0766 Epoch: 15 Global Step: 151460 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:20:10,881-Speed 130.14 samples/sec Loss 3.0594 Epoch: 15 Global Step: 151465 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:20:30,438-Speed 130.91 samples/sec Loss 3.2133 Epoch: 15 Global Step: 151470 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:20:50,633-Speed 126.78 samples/sec Loss 3.1184 Epoch: 15 Global Step: 151475 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:21:10,064-Speed 131.75 samples/sec Loss 3.2479 Epoch: 15 Global Step: 151480 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:21:29,736-Speed 130.14 samples/sec Loss 3.0516 Epoch: 15 Global Step: 151485 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:21:49,296-Speed 130.89 samples/sec Loss 3.2890 Epoch: 15 Global Step: 151490 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:22:08,733-Speed 131.72 samples/sec Loss 3.2240 Epoch: 15 Global Step: 151495 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:22:28,333-Speed 130.62 samples/sec Loss 3.1326 Epoch: 15 Global Step: 151500 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:22:47,878-Speed 130.99 samples/sec Loss 3.2945 Epoch: 15 Global Step: 151505 Fp16 Grad Scale: 8192 Required: 116 hours Training: 2021-06-14 06:23:07,499-Speed 130.48 samples/sec Loss 3.1562 Epoch: 15 Global Step: 151510 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:23:26,928-Speed 131.77 samples/sec Loss 3.0048 Epoch: 15 Global Step: 151515 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:23:46,866-Speed 128.40 samples/sec Loss 3.1758 Epoch: 15 Global Step: 151520 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:24:06,319-Speed 131.61 samples/sec Loss 3.1451 Epoch: 15 Global Step: 151525 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:24:26,155-Speed 129.07 samples/sec Loss 3.1102 Epoch: 15 Global Step: 151530 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:24:45,696-Speed 131.02 samples/sec Loss 3.0596 Epoch: 15 Global Step: 151535 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:25:05,223-Speed 131.11 samples/sec Loss 3.0502 Epoch: 15 Global Step: 151540 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:25:24,770-Speed 130.97 samples/sec Loss 3.1492 Epoch: 15 Global Step: 151545 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:25:44,293-Speed 131.14 samples/sec Loss 3.0430 Epoch: 15 Global Step: 151550 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:26:03,879-Speed 130.71 samples/sec Loss 3.1972 Epoch: 15 Global Step: 151555 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:26:23,559-Speed 130.09 samples/sec Loss 3.0872 Epoch: 15 Global Step: 151560 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:26:43,663-Speed 127.34 samples/sec Loss 3.3007 Epoch: 15 Global Step: 151565 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:27:03,093-Speed 131.76 samples/sec Loss 3.0929 Epoch: 15 Global Step: 151570 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:27:22,821-Speed 129.77 samples/sec Loss 3.2444 Epoch: 15 Global Step: 151575 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:27:42,353-Speed 131.07 samples/sec Loss 3.2105 Epoch: 15 Global Step: 151580 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:28:02,026-Speed 130.14 samples/sec Loss 3.1177 Epoch: 15 Global Step: 151585 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:28:21,492-Speed 131.51 samples/sec Loss 3.2196 Epoch: 15 Global Step: 151590 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:28:41,309-Speed 129.19 samples/sec Loss 3.0532 Epoch: 15 Global Step: 151595 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:29:00,975-Speed 130.18 samples/sec Loss 3.2731 Epoch: 15 Global Step: 151600 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:29:20,500-Speed 131.12 samples/sec Loss 3.1867 Epoch: 15 Global Step: 151605 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:29:40,288-Speed 129.38 samples/sec Loss 3.2180 Epoch: 15 Global Step: 151610 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:29:59,931-Speed 130.33 samples/sec Loss 2.9528 Epoch: 15 Global Step: 151615 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:30:19,603-Speed 130.14 samples/sec Loss 3.2342 Epoch: 15 Global Step: 151620 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:30:39,367-Speed 129.54 samples/sec Loss 3.0979 Epoch: 15 Global Step: 151625 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:30:58,982-Speed 130.52 samples/sec Loss 3.0153 Epoch: 15 Global Step: 151630 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:31:18,568-Speed 130.71 samples/sec Loss 3.3555 Epoch: 15 Global Step: 151635 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:31:38,176-Speed 130.56 samples/sec Loss 3.2609 Epoch: 15 Global Step: 151640 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:31:57,718-Speed 131.01 samples/sec Loss 3.1022 Epoch: 15 Global Step: 151645 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:32:17,334-Speed 130.51 samples/sec Loss 3.3961 Epoch: 15 Global Step: 151650 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:32:37,112-Speed 129.45 samples/sec Loss 3.0550 Epoch: 15 Global Step: 151655 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:32:56,855-Speed 129.67 samples/sec Loss 3.2548 Epoch: 15 Global Step: 151660 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:33:16,384-Speed 131.09 samples/sec Loss 3.0811 Epoch: 15 Global Step: 151665 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:33:36,006-Speed 130.47 samples/sec Loss 3.1620 Epoch: 15 Global Step: 151670 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:33:55,485-Speed 131.43 samples/sec Loss 3.2391 Epoch: 15 Global Step: 151675 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:34:15,039-Speed 130.93 samples/sec Loss 3.1231 Epoch: 15 Global Step: 151680 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:34:34,621-Speed 130.74 samples/sec Loss 3.0531 Epoch: 15 Global Step: 151685 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:34:54,153-Speed 131.08 samples/sec Loss 2.9925 Epoch: 15 Global Step: 151690 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:35:13,745-Speed 130.67 samples/sec Loss 3.1731 Epoch: 15 Global Step: 151695 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:35:33,401-Speed 130.24 samples/sec Loss 3.2410 Epoch: 15 Global Step: 151700 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:35:53,157-Speed 129.59 samples/sec Loss 3.1042 Epoch: 15 Global Step: 151705 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:36:12,744-Speed 130.71 samples/sec Loss 3.3080 Epoch: 15 Global Step: 151710 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:36:32,347-Speed 130.60 samples/sec Loss 3.1742 Epoch: 15 Global Step: 151715 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:36:51,863-Speed 131.18 samples/sec Loss 3.2113 Epoch: 15 Global Step: 151720 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:37:11,526-Speed 130.20 samples/sec Loss 3.2195 Epoch: 15 Global Step: 151725 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:37:31,049-Speed 131.14 samples/sec Loss 3.1137 Epoch: 15 Global Step: 151730 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:37:50,656-Speed 130.57 samples/sec Loss 3.1173 Epoch: 15 Global Step: 151735 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:38:10,264-Speed 130.57 samples/sec Loss 3.2604 Epoch: 15 Global Step: 151740 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:38:30,160-Speed 128.67 samples/sec Loss 3.1151 Epoch: 15 Global Step: 151745 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:38:49,666-Speed 131.25 samples/sec Loss 3.1303 Epoch: 15 Global Step: 151750 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:39:09,321-Speed 130.25 samples/sec Loss 3.0459 Epoch: 15 Global Step: 151755 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:39:28,940-Speed 130.49 samples/sec Loss 3.1803 Epoch: 15 Global Step: 151760 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:39:48,479-Speed 131.03 samples/sec Loss 3.1115 Epoch: 15 Global Step: 151765 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:40:08,072-Speed 130.66 samples/sec Loss 3.2153 Epoch: 15 Global Step: 151770 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:40:27,498-Speed 131.79 samples/sec Loss 3.0958 Epoch: 15 Global Step: 151775 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:40:47,107-Speed 130.56 samples/sec Loss 3.0234 Epoch: 15 Global Step: 151780 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:41:06,687-Speed 130.76 samples/sec Loss 3.2839 Epoch: 15 Global Step: 151785 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:41:26,508-Speed 129.17 samples/sec Loss 3.1807 Epoch: 15 Global Step: 151790 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:41:45,941-Speed 131.74 samples/sec Loss 3.0952 Epoch: 15 Global Step: 151795 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:42:05,563-Speed 130.48 samples/sec Loss 3.1953 Epoch: 15 Global Step: 151800 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:42:25,112-Speed 130.96 samples/sec Loss 3.2109 Epoch: 15 Global Step: 151805 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:42:44,579-Speed 131.51 samples/sec Loss 3.2710 Epoch: 15 Global Step: 151810 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:43:04,138-Speed 130.90 samples/sec Loss 3.1515 Epoch: 15 Global Step: 151815 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:43:23,677-Speed 131.03 samples/sec Loss 3.1102 Epoch: 15 Global Step: 151820 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:43:43,288-Speed 130.55 samples/sec Loss 3.2033 Epoch: 15 Global Step: 151825 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:44:02,819-Speed 131.08 samples/sec Loss 3.0716 Epoch: 15 Global Step: 151830 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:44:22,616-Speed 129.32 samples/sec Loss 3.0457 Epoch: 15 Global Step: 151835 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:44:42,090-Speed 131.46 samples/sec Loss 3.1137 Epoch: 15 Global Step: 151840 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:45:01,806-Speed 129.85 samples/sec Loss 3.1958 Epoch: 15 Global Step: 151845 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:45:21,366-Speed 130.89 samples/sec Loss 2.9887 Epoch: 15 Global Step: 151850 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:45:40,959-Speed 130.67 samples/sec Loss 3.2943 Epoch: 15 Global Step: 151855 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:46:00,386-Speed 131.78 samples/sec Loss 3.2815 Epoch: 15 Global Step: 151860 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:46:20,016-Speed 130.42 samples/sec Loss 3.0896 Epoch: 15 Global Step: 151865 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:46:39,624-Speed 130.57 samples/sec Loss 3.2582 Epoch: 15 Global Step: 151870 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:46:59,064-Speed 131.69 samples/sec Loss 3.2837 Epoch: 15 Global Step: 151875 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:47:18,843-Speed 129.44 samples/sec Loss 3.1966 Epoch: 15 Global Step: 151880 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:47:38,340-Speed 131.31 samples/sec Loss 3.2600 Epoch: 15 Global Step: 151885 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:47:58,026-Speed 130.05 samples/sec Loss 3.3456 Epoch: 15 Global Step: 151890 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:48:17,465-Speed 131.70 samples/sec Loss 3.1886 Epoch: 15 Global Step: 151895 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:48:37,053-Speed 130.70 samples/sec Loss 3.1838 Epoch: 15 Global Step: 151900 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:48:56,646-Speed 130.67 samples/sec Loss 3.0566 Epoch: 15 Global Step: 151905 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:49:16,176-Speed 131.08 samples/sec Loss 3.1081 Epoch: 15 Global Step: 151910 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:49:35,871-Speed 129.99 samples/sec Loss 3.1462 Epoch: 15 Global Step: 151915 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:49:55,401-Speed 131.08 samples/sec Loss 3.0681 Epoch: 15 Global Step: 151920 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:50:15,216-Speed 129.20 samples/sec Loss 3.1024 Epoch: 15 Global Step: 151925 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:50:34,712-Speed 131.32 samples/sec Loss 3.3580 Epoch: 15 Global Step: 151930 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:50:54,433-Speed 129.82 samples/sec Loss 2.9970 Epoch: 15 Global Step: 151935 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:51:13,987-Speed 130.93 samples/sec Loss 3.1504 Epoch: 15 Global Step: 151940 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:51:33,479-Speed 131.34 samples/sec Loss 3.2322 Epoch: 15 Global Step: 151945 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:51:53,110-Speed 130.41 samples/sec Loss 3.1831 Epoch: 15 Global Step: 151950 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:52:12,642-Speed 131.08 samples/sec Loss 3.0899 Epoch: 15 Global Step: 151955 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:52:32,223-Speed 130.74 samples/sec Loss 3.2339 Epoch: 15 Global Step: 151960 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:52:51,724-Speed 131.28 samples/sec Loss 3.2375 Epoch: 15 Global Step: 151965 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:53:11,516-Speed 129.35 samples/sec Loss 3.2731 Epoch: 15 Global Step: 151970 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:53:31,098-Speed 130.74 samples/sec Loss 3.2514 Epoch: 15 Global Step: 151975 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:53:50,761-Speed 130.20 samples/sec Loss 2.9977 Epoch: 15 Global Step: 151980 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:54:10,185-Speed 131.80 samples/sec Loss 3.0269 Epoch: 15 Global Step: 151985 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:54:29,775-Speed 130.69 samples/sec Loss 3.0664 Epoch: 15 Global Step: 151990 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:54:49,318-Speed 131.00 samples/sec Loss 3.1761 Epoch: 15 Global Step: 151995 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:55:08,884-Speed 130.85 samples/sec Loss 3.2923 Epoch: 15 Global Step: 152000 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:55:28,485-Speed 130.61 samples/sec Loss 3.1689 Epoch: 15 Global Step: 152005 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:55:47,935-Speed 131.63 samples/sec Loss 3.1466 Epoch: 15 Global Step: 152010 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:56:07,733-Speed 129.31 samples/sec Loss 3.1453 Epoch: 15 Global Step: 152015 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:56:27,309-Speed 130.78 samples/sec Loss 3.2939 Epoch: 15 Global Step: 152020 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:56:46,967-Speed 130.24 samples/sec Loss 3.2332 Epoch: 15 Global Step: 152025 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:57:06,409-Speed 131.68 samples/sec Loss 3.1634 Epoch: 15 Global Step: 152030 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:57:26,006-Speed 130.64 samples/sec Loss 3.1803 Epoch: 15 Global Step: 152035 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:57:45,532-Speed 131.11 samples/sec Loss 3.3171 Epoch: 15 Global Step: 152040 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:58:05,074-Speed 131.01 samples/sec Loss 3.1879 Epoch: 15 Global Step: 152045 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:58:24,682-Speed 130.57 samples/sec Loss 3.1827 Epoch: 15 Global Step: 152050 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:58:44,213-Speed 131.08 samples/sec Loss 3.2037 Epoch: 15 Global Step: 152055 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:59:04,401-Speed 126.82 samples/sec Loss 3.0614 Epoch: 15 Global Step: 152060 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:59:24,102-Speed 129.95 samples/sec Loss 2.9703 Epoch: 15 Global Step: 152065 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 06:59:43,619-Speed 131.17 samples/sec Loss 3.2570 Epoch: 15 Global Step: 152070 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:00:03,241-Speed 130.47 samples/sec Loss 3.2416 Epoch: 15 Global Step: 152075 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:00:22,846-Speed 130.59 samples/sec Loss 3.3271 Epoch: 15 Global Step: 152080 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:00:42,381-Speed 131.05 samples/sec Loss 3.1745 Epoch: 15 Global Step: 152085 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:01:01,959-Speed 130.76 samples/sec Loss 3.1980 Epoch: 15 Global Step: 152090 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:01:21,624-Speed 130.19 samples/sec Loss 3.0081 Epoch: 15 Global Step: 152095 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:01:41,225-Speed 130.61 samples/sec Loss 3.2173 Epoch: 15 Global Step: 152100 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:02:00,993-Speed 129.51 samples/sec Loss 3.2757 Epoch: 15 Global Step: 152105 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:02:20,686-Speed 130.00 samples/sec Loss 3.0708 Epoch: 15 Global Step: 152110 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:02:40,284-Speed 130.63 samples/sec Loss 3.3297 Epoch: 15 Global Step: 152115 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:02:59,895-Speed 130.55 samples/sec Loss 3.2586 Epoch: 15 Global Step: 152120 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:03:19,466-Speed 130.82 samples/sec Loss 3.2507 Epoch: 15 Global Step: 152125 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:03:39,017-Speed 130.94 samples/sec Loss 3.2409 Epoch: 15 Global Step: 152130 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:03:58,614-Speed 130.64 samples/sec Loss 3.2231 Epoch: 15 Global Step: 152135 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:04:18,200-Speed 130.71 samples/sec Loss 3.2606 Epoch: 15 Global Step: 152140 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:04:37,716-Speed 131.18 samples/sec Loss 3.0753 Epoch: 15 Global Step: 152145 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:04:57,520-Speed 129.28 samples/sec Loss 3.1780 Epoch: 15 Global Step: 152150 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:05:17,127-Speed 130.57 samples/sec Loss 3.2073 Epoch: 15 Global Step: 152155 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:05:37,239-Speed 127.30 samples/sec Loss 3.1840 Epoch: 15 Global Step: 152160 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:05:56,695-Speed 131.59 samples/sec Loss 3.1411 Epoch: 15 Global Step: 152165 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:06:16,268-Speed 130.80 samples/sec Loss 3.0964 Epoch: 15 Global Step: 152170 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:06:35,726-Speed 131.57 samples/sec Loss 3.1524 Epoch: 15 Global Step: 152175 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:06:55,335-Speed 130.56 samples/sec Loss 3.2438 Epoch: 15 Global Step: 152180 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:07:14,975-Speed 130.35 samples/sec Loss 3.1095 Epoch: 15 Global Step: 152185 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:07:34,439-Speed 131.53 samples/sec Loss 3.2682 Epoch: 15 Global Step: 152190 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:07:54,203-Speed 129.54 samples/sec Loss 3.3021 Epoch: 15 Global Step: 152195 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:08:13,824-Speed 130.47 samples/sec Loss 3.1595 Epoch: 15 Global Step: 152200 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:08:33,358-Speed 131.06 samples/sec Loss 3.1108 Epoch: 15 Global Step: 152205 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:08:52,965-Speed 130.57 samples/sec Loss 3.4493 Epoch: 15 Global Step: 152210 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:09:12,405-Speed 131.70 samples/sec Loss 3.1445 Epoch: 15 Global Step: 152215 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:09:31,855-Speed 131.63 samples/sec Loss 3.1951 Epoch: 15 Global Step: 152220 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:09:51,418-Speed 130.87 samples/sec Loss 3.2613 Epoch: 15 Global Step: 152225 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:10:10,944-Speed 131.11 samples/sec Loss 3.3859 Epoch: 15 Global Step: 152230 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:10:30,600-Speed 130.25 samples/sec Loss 3.3139 Epoch: 15 Global Step: 152235 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:10:50,569-Speed 128.21 samples/sec Loss 3.1039 Epoch: 15 Global Step: 152240 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:11:10,271-Speed 129.94 samples/sec Loss 3.2484 Epoch: 15 Global Step: 152245 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:11:29,720-Speed 131.63 samples/sec Loss 3.0988 Epoch: 15 Global Step: 152250 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:11:49,344-Speed 130.46 samples/sec Loss 3.1240 Epoch: 15 Global Step: 152255 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:12:08,791-Speed 131.65 samples/sec Loss 3.1567 Epoch: 15 Global Step: 152260 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:12:28,390-Speed 130.62 samples/sec Loss 3.2805 Epoch: 15 Global Step: 152265 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:12:48,253-Speed 128.89 samples/sec Loss 2.9746 Epoch: 15 Global Step: 152270 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:13:07,681-Speed 131.78 samples/sec Loss 3.2124 Epoch: 15 Global Step: 152275 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:13:27,303-Speed 130.48 samples/sec Loss 3.3601 Epoch: 15 Global Step: 152280 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:13:47,014-Speed 129.88 samples/sec Loss 3.1112 Epoch: 15 Global Step: 152285 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:14:06,620-Speed 130.58 samples/sec Loss 3.3631 Epoch: 15 Global Step: 152290 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:14:26,206-Speed 130.71 samples/sec Loss 3.2236 Epoch: 15 Global Step: 152295 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:14:45,846-Speed 130.36 samples/sec Loss 3.3412 Epoch: 15 Global Step: 152300 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:15:05,483-Speed 130.37 samples/sec Loss 3.2108 Epoch: 15 Global Step: 152305 Fp16 Grad Scale: 8192 Required: 115 hours Training: 2021-06-14 07:15:25,069-Speed 130.72 samples/sec Loss 3.1427 Epoch: 15 Global Step: 152310 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:15:44,651-Speed 130.74 samples/sec Loss 3.3318 Epoch: 15 Global Step: 152315 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:16:04,259-Speed 130.57 samples/sec Loss 3.0937 Epoch: 15 Global Step: 152320 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:16:23,702-Speed 131.68 samples/sec Loss 3.1410 Epoch: 15 Global Step: 152325 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:16:43,584-Speed 128.77 samples/sec Loss 3.2259 Epoch: 15 Global Step: 152330 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:17:03,201-Speed 130.50 samples/sec Loss 3.2365 Epoch: 15 Global Step: 152335 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:17:22,812-Speed 130.55 samples/sec Loss 3.2977 Epoch: 15 Global Step: 152340 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:17:42,352-Speed 131.02 samples/sec Loss 3.2306 Epoch: 15 Global Step: 152345 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:18:01,790-Speed 131.71 samples/sec Loss 2.9461 Epoch: 15 Global Step: 152350 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:18:21,382-Speed 130.67 samples/sec Loss 3.1581 Epoch: 15 Global Step: 152355 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:18:40,943-Speed 130.89 samples/sec Loss 3.0887 Epoch: 15 Global Step: 152360 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:19:00,545-Speed 130.60 samples/sec Loss 3.2791 Epoch: 15 Global Step: 152365 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:19:20,040-Speed 131.33 samples/sec Loss 3.1540 Epoch: 15 Global Step: 152370 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:19:39,846-Speed 129.26 samples/sec Loss 3.1215 Epoch: 15 Global Step: 152375 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:19:59,305-Speed 131.56 samples/sec Loss 3.1263 Epoch: 15 Global Step: 152380 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:20:19,181-Speed 128.81 samples/sec Loss 3.2172 Epoch: 15 Global Step: 152385 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:20:38,878-Speed 129.97 samples/sec Loss 3.2634 Epoch: 15 Global Step: 152390 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:20:58,335-Speed 131.58 samples/sec Loss 3.2183 Epoch: 15 Global Step: 152395 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:21:17,945-Speed 130.55 samples/sec Loss 3.1954 Epoch: 15 Global Step: 152400 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:21:37,409-Speed 131.53 samples/sec Loss 3.2143 Epoch: 15 Global Step: 152405 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:21:56,999-Speed 130.69 samples/sec Loss 3.1908 Epoch: 15 Global Step: 152410 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:22:16,433-Speed 131.74 samples/sec Loss 3.2602 Epoch: 15 Global Step: 152415 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:22:36,253-Speed 129.17 samples/sec Loss 3.2484 Epoch: 15 Global Step: 152420 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:22:55,696-Speed 131.68 samples/sec Loss 3.3650 Epoch: 15 Global Step: 152425 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:23:15,420-Speed 129.79 samples/sec Loss 3.1935 Epoch: 15 Global Step: 152430 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:23:34,877-Speed 131.58 samples/sec Loss 3.1640 Epoch: 15 Global Step: 152435 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:23:54,464-Speed 130.71 samples/sec Loss 3.2011 Epoch: 15 Global Step: 152440 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:24:14,025-Speed 130.88 samples/sec Loss 3.2065 Epoch: 15 Global Step: 152445 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:24:33,587-Speed 130.88 samples/sec Loss 3.4861 Epoch: 15 Global Step: 152450 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:24:53,183-Speed 130.64 samples/sec Loss 3.1487 Epoch: 15 Global Step: 152455 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:25:12,625-Speed 131.68 samples/sec Loss 3.3269 Epoch: 15 Global Step: 152460 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:25:32,428-Speed 129.28 samples/sec Loss 3.1181 Epoch: 15 Global Step: 152465 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:25:52,149-Speed 129.82 samples/sec Loss 3.2091 Epoch: 15 Global Step: 152470 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:26:11,579-Speed 131.76 samples/sec Loss 3.0404 Epoch: 15 Global Step: 152475 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:26:31,085-Speed 131.25 samples/sec Loss 3.1558 Epoch: 15 Global Step: 152480 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:26:50,668-Speed 130.73 samples/sec Loss 3.3793 Epoch: 15 Global Step: 152485 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:27:09,769-Speed 134.03 samples/sec Loss 3.3920 Epoch: 15 Global Step: 152490 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:27:29,389-Speed 130.49 samples/sec Loss 3.3280 Epoch: 15 Global Step: 152495 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:27:48,822-Speed 131.74 samples/sec Loss 3.2720 Epoch: 15 Global Step: 152500 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:28:08,369-Speed 130.98 samples/sec Loss 3.1798 Epoch: 15 Global Step: 152505 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:28:28,133-Speed 129.53 samples/sec Loss 3.1458 Epoch: 15 Global Step: 152510 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:28:47,837-Speed 129.93 samples/sec Loss 3.1310 Epoch: 15 Global Step: 152515 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:29:07,314-Speed 131.44 samples/sec Loss 3.4807 Epoch: 15 Global Step: 152520 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:29:26,955-Speed 130.35 samples/sec Loss 3.2376 Epoch: 15 Global Step: 152525 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:29:46,507-Speed 130.94 samples/sec Loss 3.4772 Epoch: 15 Global Step: 152530 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:30:06,047-Speed 131.02 samples/sec Loss 3.1715 Epoch: 15 Global Step: 152535 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:30:25,627-Speed 130.76 samples/sec Loss 3.1150 Epoch: 15 Global Step: 152540 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:30:45,047-Speed 131.83 samples/sec Loss 3.2406 Epoch: 15 Global Step: 152545 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:31:04,657-Speed 130.56 samples/sec Loss 3.3186 Epoch: 15 Global Step: 152550 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:31:24,404-Speed 129.65 samples/sec Loss 3.3321 Epoch: 15 Global Step: 152555 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:31:44,121-Speed 129.85 samples/sec Loss 3.3381 Epoch: 15 Global Step: 152560 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:32:03,540-Speed 131.83 samples/sec Loss 3.1275 Epoch: 15 Global Step: 152565 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:32:23,359-Speed 129.18 samples/sec Loss 3.1989 Epoch: 15 Global Step: 152570 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:32:42,876-Speed 131.17 samples/sec Loss 3.1984 Epoch: 15 Global Step: 152575 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:33:02,423-Speed 130.98 samples/sec Loss 3.1911 Epoch: 15 Global Step: 152580 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:33:22,032-Speed 130.56 samples/sec Loss 3.1510 Epoch: 15 Global Step: 152585 Fp16 Grad Scale: 4096 Required: 114 hours Training: 2021-06-14 07:33:41,636-Speed 130.59 samples/sec Loss 3.1787 Epoch: 15 Global Step: 152590 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:34:01,251-Speed 130.52 samples/sec Loss 3.2651 Epoch: 15 Global Step: 152595 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:34:21,175-Speed 128.50 samples/sec Loss 3.3503 Epoch: 15 Global Step: 152600 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:34:40,607-Speed 131.75 samples/sec Loss 3.1963 Epoch: 15 Global Step: 152605 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:35:00,220-Speed 130.53 samples/sec Loss 3.1821 Epoch: 15 Global Step: 152610 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:35:19,794-Speed 130.79 samples/sec Loss 3.2979 Epoch: 15 Global Step: 152615 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:35:39,437-Speed 130.33 samples/sec Loss 3.1815 Epoch: 15 Global Step: 152620 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:35:59,041-Speed 130.59 samples/sec Loss 3.2786 Epoch: 15 Global Step: 152625 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:36:18,492-Speed 131.62 samples/sec Loss 3.3494 Epoch: 15 Global Step: 152630 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:36:38,097-Speed 130.59 samples/sec Loss 3.3608 Epoch: 15 Global Step: 152635 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:36:57,651-Speed 130.93 samples/sec Loss 3.0836 Epoch: 15 Global Step: 152640 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:37:17,527-Speed 128.80 samples/sec Loss 3.2426 Epoch: 15 Global Step: 152645 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:37:36,986-Speed 131.57 samples/sec Loss 3.2231 Epoch: 15 Global Step: 152650 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:37:56,678-Speed 130.01 samples/sec Loss 3.0767 Epoch: 15 Global Step: 152655 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:38:16,249-Speed 130.81 samples/sec Loss 3.1170 Epoch: 15 Global Step: 152660 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:38:35,794-Speed 130.99 samples/sec Loss 3.2713 Epoch: 15 Global Step: 152665 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:38:55,299-Speed 131.26 samples/sec Loss 3.2198 Epoch: 15 Global Step: 152670 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:39:14,909-Speed 130.55 samples/sec Loss 3.0272 Epoch: 15 Global Step: 152675 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:39:34,503-Speed 130.66 samples/sec Loss 3.1184 Epoch: 15 Global Step: 152680 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:39:54,041-Speed 131.03 samples/sec Loss 3.1710 Epoch: 15 Global Step: 152685 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:40:13,885-Speed 129.01 samples/sec Loss 3.1896 Epoch: 15 Global Step: 152690 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:40:33,303-Speed 131.85 samples/sec Loss 3.3729 Epoch: 15 Global Step: 152695 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:40:53,281-Speed 128.15 samples/sec Loss 3.2875 Epoch: 15 Global Step: 152700 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:41:12,912-Speed 130.41 samples/sec Loss 3.3373 Epoch: 15 Global Step: 152705 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:41:32,377-Speed 131.53 samples/sec Loss 3.1936 Epoch: 15 Global Step: 152710 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:41:51,933-Speed 130.91 samples/sec Loss 3.2555 Epoch: 15 Global Step: 152715 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:42:11,545-Speed 130.54 samples/sec Loss 3.1505 Epoch: 15 Global Step: 152720 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:42:31,159-Speed 130.52 samples/sec Loss 3.1637 Epoch: 15 Global Step: 152725 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:42:50,713-Speed 130.93 samples/sec Loss 3.2501 Epoch: 15 Global Step: 152730 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:43:10,492-Speed 129.44 samples/sec Loss 3.3072 Epoch: 15 Global Step: 152735 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:43:29,978-Speed 131.39 samples/sec Loss 3.2844 Epoch: 15 Global Step: 152740 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:43:49,684-Speed 129.91 samples/sec Loss 3.2336 Epoch: 15 Global Step: 152745 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:44:09,219-Speed 131.05 samples/sec Loss 3.2492 Epoch: 15 Global Step: 152750 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:44:28,875-Speed 130.25 samples/sec Loss 3.2409 Epoch: 15 Global Step: 152755 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:44:48,386-Speed 131.22 samples/sec Loss 3.1553 Epoch: 15 Global Step: 152760 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:45:07,970-Speed 130.73 samples/sec Loss 3.3385 Epoch: 15 Global Step: 152765 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:45:27,580-Speed 130.55 samples/sec Loss 3.1808 Epoch: 15 Global Step: 152770 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:45:47,113-Speed 131.07 samples/sec Loss 3.2018 Epoch: 15 Global Step: 152775 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:46:06,910-Speed 129.32 samples/sec Loss 3.4410 Epoch: 15 Global Step: 152780 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:46:26,498-Speed 130.70 samples/sec Loss 3.1222 Epoch: 15 Global Step: 152785 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:46:46,156-Speed 130.24 samples/sec Loss 3.3012 Epoch: 15 Global Step: 152790 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:47:05,613-Speed 131.58 samples/sec Loss 3.1207 Epoch: 15 Global Step: 152795 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:47:25,202-Speed 130.70 samples/sec Loss 3.0971 Epoch: 15 Global Step: 152800 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:47:44,805-Speed 130.60 samples/sec Loss 3.3228 Epoch: 15 Global Step: 152805 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:48:04,365-Speed 130.89 samples/sec Loss 3.2292 Epoch: 15 Global Step: 152810 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:48:23,898-Speed 131.07 samples/sec Loss 3.5519 Epoch: 15 Global Step: 152815 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:48:43,364-Speed 131.52 samples/sec Loss 3.1991 Epoch: 15 Global Step: 152820 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:49:03,118-Speed 129.60 samples/sec Loss 3.1816 Epoch: 15 Global Step: 152825 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:49:22,896-Speed 129.45 samples/sec Loss 3.1021 Epoch: 15 Global Step: 152830 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:49:42,410-Speed 131.19 samples/sec Loss 3.2790 Epoch: 15 Global Step: 152835 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:50:02,116-Speed 129.92 samples/sec Loss 3.3519 Epoch: 15 Global Step: 152840 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:50:21,659-Speed 131.00 samples/sec Loss 3.3806 Epoch: 15 Global Step: 152845 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:50:41,235-Speed 130.78 samples/sec Loss 3.0320 Epoch: 15 Global Step: 152850 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:51:00,854-Speed 130.50 samples/sec Loss 3.4156 Epoch: 15 Global Step: 152855 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:51:20,307-Speed 131.60 samples/sec Loss 3.4225 Epoch: 15 Global Step: 152860 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:51:39,899-Speed 130.68 samples/sec Loss 3.0089 Epoch: 15 Global Step: 152865 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:52:00,373-Speed 125.04 samples/sec Loss 3.1115 Epoch: 15 Global Step: 152870 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:52:20,004-Speed 130.41 samples/sec Loss 3.4360 Epoch: 15 Global Step: 152875 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:52:39,427-Speed 131.81 samples/sec Loss 3.2812 Epoch: 15 Global Step: 152880 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:52:59,012-Speed 130.72 samples/sec Loss 3.2724 Epoch: 15 Global Step: 152885 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:53:18,626-Speed 130.52 samples/sec Loss 3.3387 Epoch: 15 Global Step: 152890 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:53:38,154-Speed 131.10 samples/sec Loss 3.4635 Epoch: 15 Global Step: 152895 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:53:57,718-Speed 130.86 samples/sec Loss 3.2069 Epoch: 15 Global Step: 152900 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:54:17,284-Speed 130.85 samples/sec Loss 3.3528 Epoch: 15 Global Step: 152905 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:54:36,902-Speed 130.50 samples/sec Loss 3.2238 Epoch: 15 Global Step: 152910 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:54:56,826-Speed 128.49 samples/sec Loss 3.1879 Epoch: 15 Global Step: 152915 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:55:16,261-Speed 131.73 samples/sec Loss 3.0758 Epoch: 15 Global Step: 152920 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:55:35,886-Speed 130.45 samples/sec Loss 3.2448 Epoch: 15 Global Step: 152925 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:55:55,744-Speed 128.93 samples/sec Loss 3.3338 Epoch: 15 Global Step: 152930 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:56:15,191-Speed 131.65 samples/sec Loss 3.2271 Epoch: 15 Global Step: 152935 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:56:34,787-Speed 130.65 samples/sec Loss 3.4272 Epoch: 15 Global Step: 152940 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:56:54,216-Speed 131.77 samples/sec Loss 3.1897 Epoch: 15 Global Step: 152945 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:57:13,732-Speed 131.18 samples/sec Loss 3.3138 Epoch: 15 Global Step: 152950 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:57:33,346-Speed 130.52 samples/sec Loss 3.3880 Epoch: 15 Global Step: 152955 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:57:53,122-Speed 129.46 samples/sec Loss 3.0950 Epoch: 15 Global Step: 152960 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:58:12,630-Speed 131.24 samples/sec Loss 3.1060 Epoch: 15 Global Step: 152965 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:58:32,145-Speed 131.18 samples/sec Loss 3.3879 Epoch: 15 Global Step: 152970 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:58:51,740-Speed 130.65 samples/sec Loss 3.4119 Epoch: 15 Global Step: 152975 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:59:11,268-Speed 131.10 samples/sec Loss 3.1630 Epoch: 15 Global Step: 152980 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:59:30,973-Speed 129.93 samples/sec Loss 3.2348 Epoch: 15 Global Step: 152985 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 07:59:50,503-Speed 131.09 samples/sec Loss 3.2764 Epoch: 15 Global Step: 152990 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:00:09,986-Speed 131.41 samples/sec Loss 3.0778 Epoch: 15 Global Step: 152995 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:00:29,646-Speed 130.23 samples/sec Loss 3.1882 Epoch: 15 Global Step: 153000 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:00:49,576-Speed 128.46 samples/sec Loss 3.3686 Epoch: 15 Global Step: 153005 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:01:09,015-Speed 131.70 samples/sec Loss 3.2547 Epoch: 15 Global Step: 153010 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:01:28,617-Speed 130.61 samples/sec Loss 3.3672 Epoch: 15 Global Step: 153015 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:01:48,210-Speed 130.67 samples/sec Loss 3.0634 Epoch: 15 Global Step: 153020 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:02:07,773-Speed 130.86 samples/sec Loss 3.4341 Epoch: 15 Global Step: 153025 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:02:27,371-Speed 130.63 samples/sec Loss 3.0761 Epoch: 15 Global Step: 153030 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:02:46,811-Speed 131.69 samples/sec Loss 3.3398 Epoch: 15 Global Step: 153035 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:03:06,422-Speed 130.55 samples/sec Loss 3.1798 Epoch: 15 Global Step: 153040 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:03:25,889-Speed 131.51 samples/sec Loss 3.2615 Epoch: 15 Global Step: 153045 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:03:45,732-Speed 129.02 samples/sec Loss 3.3687 Epoch: 15 Global Step: 153050 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:04:05,163-Speed 131.76 samples/sec Loss 3.4301 Epoch: 15 Global Step: 153055 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:04:24,863-Speed 129.96 samples/sec Loss 3.1410 Epoch: 15 Global Step: 153060 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:04:44,370-Speed 131.24 samples/sec Loss 3.3493 Epoch: 15 Global Step: 153065 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:05:03,970-Speed 130.62 samples/sec Loss 3.1908 Epoch: 15 Global Step: 153070 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:05:23,440-Speed 131.49 samples/sec Loss 3.1476 Epoch: 15 Global Step: 153075 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:05:42,994-Speed 130.93 samples/sec Loss 3.2254 Epoch: 15 Global Step: 153080 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:06:02,595-Speed 130.61 samples/sec Loss 3.3550 Epoch: 15 Global Step: 153085 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:06:22,188-Speed 130.67 samples/sec Loss 3.2076 Epoch: 15 Global Step: 153090 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:06:42,452-Speed 126.34 samples/sec Loss 3.1705 Epoch: 15 Global Step: 153095 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:07:02,059-Speed 130.58 samples/sec Loss 3.2769 Epoch: 15 Global Step: 153100 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:07:21,649-Speed 130.68 samples/sec Loss 3.2637 Epoch: 15 Global Step: 153105 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:07:41,193-Speed 130.99 samples/sec Loss 3.2521 Epoch: 15 Global Step: 153110 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:08:00,808-Speed 130.52 samples/sec Loss 3.2355 Epoch: 15 Global Step: 153115 Fp16 Grad Scale: 8192 Required: 114 hours Training: 2021-06-14 08:08:20,346-Speed 131.03 samples/sec Loss 3.2610 Epoch: 15 Global Step: 153120 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:08:39,830-Speed 131.40 samples/sec Loss 3.3656 Epoch: 15 Global Step: 153125 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:08:59,420-Speed 130.69 samples/sec Loss 3.3344 Epoch: 15 Global Step: 153130 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:09:19,869-Speed 125.20 samples/sec Loss 3.2986 Epoch: 15 Global Step: 153135 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:09:39,623-Speed 129.61 samples/sec Loss 3.2357 Epoch: 15 Global Step: 153140 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:09:59,332-Speed 129.90 samples/sec Loss 3.3713 Epoch: 15 Global Step: 153145 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:10:18,944-Speed 130.54 samples/sec Loss 3.3664 Epoch: 15 Global Step: 153150 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:10:38,485-Speed 131.01 samples/sec Loss 3.2752 Epoch: 15 Global Step: 153155 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:10:58,045-Speed 130.89 samples/sec Loss 3.1056 Epoch: 15 Global Step: 153160 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:11:17,464-Speed 131.84 samples/sec Loss 3.3319 Epoch: 15 Global Step: 153165 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:11:37,088-Speed 130.46 samples/sec Loss 3.3358 Epoch: 15 Global Step: 153170 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:11:56,659-Speed 130.81 samples/sec Loss 3.1995 Epoch: 15 Global Step: 153175 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:12:16,123-Speed 131.53 samples/sec Loss 3.2447 Epoch: 15 Global Step: 153180 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:12:35,913-Speed 129.36 samples/sec Loss 3.2449 Epoch: 15 Global Step: 153185 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:12:55,597-Speed 130.06 samples/sec Loss 3.4071 Epoch: 15 Global Step: 153190 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:13:15,175-Speed 130.77 samples/sec Loss 3.3674 Epoch: 15 Global Step: 153195 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:13:34,821-Speed 130.31 samples/sec Loss 3.2648 Epoch: 15 Global Step: 153200 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:13:54,256-Speed 131.73 samples/sec Loss 3.3960 Epoch: 15 Global Step: 153205 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:14:13,821-Speed 130.85 samples/sec Loss 3.2198 Epoch: 15 Global Step: 153210 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:14:33,438-Speed 130.50 samples/sec Loss 3.1821 Epoch: 15 Global Step: 153215 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:14:53,000-Speed 130.87 samples/sec Loss 3.1381 Epoch: 15 Global Step: 153220 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:15:12,604-Speed 130.59 samples/sec Loss 3.2832 Epoch: 15 Global Step: 153225 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:15:32,275-Speed 130.15 samples/sec Loss 3.0992 Epoch: 15 Global Step: 153230 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:15:51,957-Speed 130.08 samples/sec Loss 2.9727 Epoch: 15 Global Step: 153235 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:16:11,509-Speed 130.93 samples/sec Loss 3.2625 Epoch: 15 Global Step: 153240 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:16:31,240-Speed 129.75 samples/sec Loss 3.3537 Epoch: 15 Global Step: 153245 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:16:50,782-Speed 131.01 samples/sec Loss 3.2967 Epoch: 15 Global Step: 153250 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:17:10,400-Speed 130.50 samples/sec Loss 3.1150 Epoch: 15 Global Step: 153255 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:17:29,975-Speed 130.79 samples/sec Loss 3.2722 Epoch: 15 Global Step: 153260 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:17:49,439-Speed 131.53 samples/sec Loss 3.1751 Epoch: 15 Global Step: 153265 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:18:09,044-Speed 130.59 samples/sec Loss 3.4505 Epoch: 15 Global Step: 153270 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:18:28,848-Speed 129.27 samples/sec Loss 3.0938 Epoch: 15 Global Step: 153275 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:18:48,546-Speed 129.97 samples/sec Loss 3.3765 Epoch: 15 Global Step: 153280 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:19:08,412-Speed 128.87 samples/sec Loss 3.0772 Epoch: 15 Global Step: 153285 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:19:28,126-Speed 129.86 samples/sec Loss 3.1792 Epoch: 15 Global Step: 153290 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:19:47,658-Speed 131.07 samples/sec Loss 3.4297 Epoch: 15 Global Step: 153295 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:20:07,260-Speed 130.61 samples/sec Loss 3.4398 Epoch: 15 Global Step: 153300 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:20:26,796-Speed 131.05 samples/sec Loss 3.0924 Epoch: 15 Global Step: 153305 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:20:46,272-Speed 131.45 samples/sec Loss 3.3675 Epoch: 15 Global Step: 153310 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:21:05,843-Speed 130.81 samples/sec Loss 3.2741 Epoch: 15 Global Step: 153315 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:21:25,732-Speed 128.73 samples/sec Loss 3.3004 Epoch: 15 Global Step: 153320 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:21:45,148-Speed 131.85 samples/sec Loss 3.1010 Epoch: 15 Global Step: 153325 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:22:04,734-Speed 130.72 samples/sec Loss 3.3865 Epoch: 15 Global Step: 153330 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:22:24,333-Speed 130.63 samples/sec Loss 3.1675 Epoch: 15 Global Step: 153335 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:22:43,926-Speed 130.66 samples/sec Loss 3.1712 Epoch: 15 Global Step: 153340 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:23:03,537-Speed 130.55 samples/sec Loss 3.3310 Epoch: 15 Global Step: 153345 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:23:23,059-Speed 131.15 samples/sec Loss 3.3960 Epoch: 15 Global Step: 153350 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:23:42,817-Speed 129.57 samples/sec Loss 3.1786 Epoch: 15 Global Step: 153355 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:24:02,375-Speed 130.91 samples/sec Loss 3.3715 Epoch: 15 Global Step: 153360 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:24:22,365-Speed 128.07 samples/sec Loss 3.4198 Epoch: 15 Global Step: 153365 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:24:41,797-Speed 131.74 samples/sec Loss 3.2587 Epoch: 15 Global Step: 153370 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:25:01,452-Speed 130.26 samples/sec Loss 3.5147 Epoch: 15 Global Step: 153375 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:25:21,082-Speed 130.42 samples/sec Loss 3.2949 Epoch: 15 Global Step: 153380 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:25:40,710-Speed 130.43 samples/sec Loss 3.3455 Epoch: 15 Global Step: 153385 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:26:00,308-Speed 130.64 samples/sec Loss 3.4714 Epoch: 15 Global Step: 153390 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:26:19,743-Speed 131.73 samples/sec Loss 3.2363 Epoch: 15 Global Step: 153395 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:26:39,398-Speed 130.26 samples/sec Loss 3.1347 Epoch: 15 Global Step: 153400 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:26:58,906-Speed 131.23 samples/sec Loss 3.3403 Epoch: 15 Global Step: 153405 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:27:18,754-Speed 128.99 samples/sec Loss 3.2187 Epoch: 15 Global Step: 153410 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:27:38,289-Speed 131.06 samples/sec Loss 3.3214 Epoch: 15 Global Step: 153415 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:27:57,900-Speed 130.54 samples/sec Loss 3.2662 Epoch: 15 Global Step: 153420 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:28:17,524-Speed 130.46 samples/sec Loss 3.4060 Epoch: 15 Global Step: 153425 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:28:37,138-Speed 130.52 samples/sec Loss 3.1737 Epoch: 15 Global Step: 153430 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:28:57,372-Speed 126.53 samples/sec Loss 3.3981 Epoch: 15 Global Step: 153435 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:29:16,811-Speed 131.70 samples/sec Loss 3.3136 Epoch: 15 Global Step: 153440 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:29:36,414-Speed 130.61 samples/sec Loss 3.1701 Epoch: 15 Global Step: 153445 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:29:55,964-Speed 130.95 samples/sec Loss 3.3885 Epoch: 15 Global Step: 153450 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:30:15,819-Speed 128.94 samples/sec Loss 3.3507 Epoch: 15 Global Step: 153455 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:30:35,289-Speed 131.50 samples/sec Loss 3.3032 Epoch: 15 Global Step: 153460 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:30:54,915-Speed 130.45 samples/sec Loss 3.2239 Epoch: 15 Global Step: 153465 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:31:14,449-Speed 131.06 samples/sec Loss 3.2495 Epoch: 15 Global Step: 153470 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:31:34,065-Speed 130.51 samples/sec Loss 3.2924 Epoch: 15 Global Step: 153475 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:31:53,481-Speed 131.86 samples/sec Loss 3.2841 Epoch: 15 Global Step: 153480 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:32:13,038-Speed 130.90 samples/sec Loss 3.3197 Epoch: 15 Global Step: 153485 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:32:32,653-Speed 130.52 samples/sec Loss 3.2475 Epoch: 15 Global Step: 153490 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:32:52,122-Speed 131.50 samples/sec Loss 3.3525 Epoch: 15 Global Step: 153495 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:33:11,912-Speed 129.37 samples/sec Loss 3.3864 Epoch: 15 Global Step: 153500 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:33:31,382-Speed 131.49 samples/sec Loss 3.2493 Epoch: 15 Global Step: 153505 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:33:51,096-Speed 129.86 samples/sec Loss 3.3430 Epoch: 15 Global Step: 153510 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:34:10,629-Speed 131.07 samples/sec Loss 3.2515 Epoch: 15 Global Step: 153515 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:34:30,244-Speed 130.52 samples/sec Loss 3.2961 Epoch: 15 Global Step: 153520 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:34:49,694-Speed 131.63 samples/sec Loss 3.1230 Epoch: 15 Global Step: 153525 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:35:09,285-Speed 130.68 samples/sec Loss 3.2550 Epoch: 15 Global Step: 153530 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:35:28,900-Speed 130.52 samples/sec Loss 3.5135 Epoch: 15 Global Step: 153535 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:35:48,456-Speed 130.91 samples/sec Loss 3.3477 Epoch: 15 Global Step: 153540 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:36:08,256-Speed 129.30 samples/sec Loss 3.3512 Epoch: 15 Global Step: 153545 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:36:27,827-Speed 130.81 samples/sec Loss 3.3022 Epoch: 15 Global Step: 153550 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:36:47,392-Speed 130.85 samples/sec Loss 3.3709 Epoch: 15 Global Step: 153555 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:37:06,951-Speed 130.89 samples/sec Loss 3.4301 Epoch: 15 Global Step: 153560 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:37:26,507-Speed 130.92 samples/sec Loss 3.4796 Epoch: 15 Global Step: 153565 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:37:46,071-Speed 130.86 samples/sec Loss 3.2159 Epoch: 15 Global Step: 153570 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:38:05,749-Speed 130.10 samples/sec Loss 3.3630 Epoch: 15 Global Step: 153575 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:38:25,298-Speed 130.96 samples/sec Loss 3.5210 Epoch: 15 Global Step: 153580 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:38:44,891-Speed 130.67 samples/sec Loss 3.2528 Epoch: 15 Global Step: 153585 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:39:04,581-Speed 130.02 samples/sec Loss 3.3098 Epoch: 15 Global Step: 153590 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:39:24,327-Speed 129.66 samples/sec Loss 3.5788 Epoch: 15 Global Step: 153595 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:39:43,882-Speed 130.91 samples/sec Loss 3.2747 Epoch: 15 Global Step: 153600 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:40:03,495-Speed 130.54 samples/sec Loss 3.2982 Epoch: 15 Global Step: 153605 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:40:23,624-Speed 127.19 samples/sec Loss 3.2501 Epoch: 15 Global Step: 153610 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:40:43,153-Speed 131.09 samples/sec Loss 3.4187 Epoch: 15 Global Step: 153615 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:41:02,742-Speed 130.69 samples/sec Loss 3.3322 Epoch: 15 Global Step: 153620 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:41:22,268-Speed 131.12 samples/sec Loss 3.1169 Epoch: 15 Global Step: 153625 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:41:41,889-Speed 130.48 samples/sec Loss 3.3263 Epoch: 15 Global Step: 153630 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:42:01,791-Speed 128.63 samples/sec Loss 3.2402 Epoch: 15 Global Step: 153635 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:42:21,463-Speed 130.14 samples/sec Loss 3.1691 Epoch: 15 Global Step: 153640 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:42:40,990-Speed 131.11 samples/sec Loss 3.2719 Epoch: 15 Global Step: 153645 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:43:00,598-Speed 130.57 samples/sec Loss 3.2964 Epoch: 15 Global Step: 153650 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:43:20,152-Speed 130.93 samples/sec Loss 3.3476 Epoch: 15 Global Step: 153655 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:43:39,756-Speed 130.59 samples/sec Loss 3.4436 Epoch: 15 Global Step: 153660 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:43:59,319-Speed 130.87 samples/sec Loss 3.5152 Epoch: 15 Global Step: 153665 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:44:18,869-Speed 130.95 samples/sec Loss 3.1694 Epoch: 15 Global Step: 153670 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:44:38,473-Speed 130.59 samples/sec Loss 3.0590 Epoch: 15 Global Step: 153675 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:44:58,255-Speed 129.42 samples/sec Loss 3.1673 Epoch: 15 Global Step: 153680 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:45:17,960-Speed 129.92 samples/sec Loss 3.3118 Epoch: 15 Global Step: 153685 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:45:37,492-Speed 131.07 samples/sec Loss 3.1498 Epoch: 15 Global Step: 153690 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:45:57,123-Speed 130.42 samples/sec Loss 3.3612 Epoch: 15 Global Step: 153695 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:46:16,764-Speed 130.34 samples/sec Loss 3.3496 Epoch: 15 Global Step: 153700 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:46:36,374-Speed 130.55 samples/sec Loss 3.3916 Epoch: 15 Global Step: 153705 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:46:55,959-Speed 130.72 samples/sec Loss 3.3553 Epoch: 15 Global Step: 153710 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:47:15,496-Speed 131.04 samples/sec Loss 3.1643 Epoch: 15 Global Step: 153715 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:47:35,137-Speed 130.35 samples/sec Loss 3.1608 Epoch: 15 Global Step: 153720 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:47:55,032-Speed 128.68 samples/sec Loss 3.2240 Epoch: 15 Global Step: 153725 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:48:14,505-Speed 131.47 samples/sec Loss 3.3587 Epoch: 15 Global Step: 153730 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:48:34,164-Speed 130.23 samples/sec Loss 3.4223 Epoch: 15 Global Step: 153735 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:48:53,744-Speed 130.75 samples/sec Loss 3.2302 Epoch: 15 Global Step: 153740 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:49:13,270-Speed 131.11 samples/sec Loss 3.5030 Epoch: 15 Global Step: 153745 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:49:32,886-Speed 130.51 samples/sec Loss 3.3129 Epoch: 15 Global Step: 153750 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:49:52,437-Speed 130.95 samples/sec Loss 3.0499 Epoch: 15 Global Step: 153755 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:50:11,984-Speed 130.97 samples/sec Loss 3.3915 Epoch: 15 Global Step: 153760 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:50:31,547-Speed 130.87 samples/sec Loss 3.5484 Epoch: 15 Global Step: 153765 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:50:51,526-Speed 128.14 samples/sec Loss 3.3370 Epoch: 15 Global Step: 153770 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:51:10,959-Speed 131.74 samples/sec Loss 3.0536 Epoch: 15 Global Step: 153775 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:51:30,714-Speed 129.59 samples/sec Loss 3.3007 Epoch: 15 Global Step: 153780 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:51:50,274-Speed 130.89 samples/sec Loss 3.3149 Epoch: 15 Global Step: 153785 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:52:09,832-Speed 130.90 samples/sec Loss 3.5044 Epoch: 15 Global Step: 153790 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:52:29,458-Speed 130.45 samples/sec Loss 3.4781 Epoch: 15 Global Step: 153795 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:52:48,931-Speed 131.47 samples/sec Loss 3.2965 Epoch: 15 Global Step: 153800 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:53:08,472-Speed 131.01 samples/sec Loss 3.3451 Epoch: 15 Global Step: 153805 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:53:28,031-Speed 130.90 samples/sec Loss 3.2319 Epoch: 15 Global Step: 153810 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:53:47,913-Speed 128.76 samples/sec Loss 3.5017 Epoch: 15 Global Step: 153815 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:54:07,409-Speed 131.31 samples/sec Loss 3.0920 Epoch: 15 Global Step: 153820 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:54:27,064-Speed 130.25 samples/sec Loss 3.2894 Epoch: 15 Global Step: 153825 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:54:46,684-Speed 130.49 samples/sec Loss 3.4490 Epoch: 15 Global Step: 153830 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:55:06,115-Speed 131.76 samples/sec Loss 3.3881 Epoch: 15 Global Step: 153835 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:55:25,765-Speed 130.28 samples/sec Loss 3.6449 Epoch: 15 Global Step: 153840 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:55:45,295-Speed 131.09 samples/sec Loss 3.4306 Epoch: 15 Global Step: 153845 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:56:04,788-Speed 131.34 samples/sec Loss 3.1963 Epoch: 15 Global Step: 153850 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:56:24,400-Speed 130.54 samples/sec Loss 3.3600 Epoch: 15 Global Step: 153855 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:56:44,216-Speed 129.20 samples/sec Loss 3.2347 Epoch: 15 Global Step: 153860 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:57:03,643-Speed 131.78 samples/sec Loss 3.2039 Epoch: 15 Global Step: 153865 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:57:23,374-Speed 129.75 samples/sec Loss 3.2729 Epoch: 15 Global Step: 153870 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:57:43,098-Speed 129.80 samples/sec Loss 3.2048 Epoch: 15 Global Step: 153875 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:58:02,528-Speed 131.77 samples/sec Loss 3.3182 Epoch: 15 Global Step: 153880 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:58:22,243-Speed 129.85 samples/sec Loss 3.2196 Epoch: 15 Global Step: 153885 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:58:42,488-Speed 126.46 samples/sec Loss 3.0622 Epoch: 15 Global Step: 153890 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:59:02,090-Speed 130.61 samples/sec Loss 3.3238 Epoch: 15 Global Step: 153895 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:59:21,604-Speed 131.20 samples/sec Loss 3.2426 Epoch: 15 Global Step: 153900 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 08:59:41,404-Speed 129.30 samples/sec Loss 3.2841 Epoch: 15 Global Step: 153905 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 09:00:00,957-Speed 130.93 samples/sec Loss 3.2215 Epoch: 15 Global Step: 153910 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 09:00:20,648-Speed 130.01 samples/sec Loss 3.0956 Epoch: 15 Global Step: 153915 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 09:00:40,082-Speed 131.74 samples/sec Loss 3.4289 Epoch: 15 Global Step: 153920 Fp16 Grad Scale: 8192 Required: 113 hours Training: 2021-06-14 09:00:59,683-Speed 130.61 samples/sec Loss 3.5136 Epoch: 15 Global Step: 153925 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:01:19,252-Speed 130.83 samples/sec Loss 3.3506 Epoch: 15 Global Step: 153930 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:01:38,798-Speed 130.98 samples/sec Loss 3.1402 Epoch: 15 Global Step: 153935 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:01:58,401-Speed 130.60 samples/sec Loss 3.3757 Epoch: 15 Global Step: 153940 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:02:17,828-Speed 131.78 samples/sec Loss 3.4582 Epoch: 15 Global Step: 153945 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:02:37,622-Speed 129.34 samples/sec Loss 3.1813 Epoch: 15 Global Step: 153950 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:02:57,369-Speed 129.64 samples/sec Loss 3.1566 Epoch: 15 Global Step: 153955 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:03:16,889-Speed 131.15 samples/sec Loss 3.4660 Epoch: 15 Global Step: 153960 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:03:36,427-Speed 131.04 samples/sec Loss 3.1796 Epoch: 15 Global Step: 153965 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:03:55,963-Speed 131.05 samples/sec Loss 3.3731 Epoch: 15 Global Step: 153970 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:04:15,764-Speed 129.30 samples/sec Loss 3.3583 Epoch: 15 Global Step: 153975 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:04:35,381-Speed 130.51 samples/sec Loss 3.3433 Epoch: 15 Global Step: 153980 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:04:54,831-Speed 131.62 samples/sec Loss 3.4573 Epoch: 15 Global Step: 153985 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:05:14,424-Speed 130.67 samples/sec Loss 3.1901 Epoch: 15 Global Step: 153990 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:05:34,214-Speed 129.37 samples/sec Loss 3.3395 Epoch: 15 Global Step: 153995 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:05:53,857-Speed 130.33 samples/sec Loss 3.3069 Epoch: 15 Global Step: 154000 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:06:13,319-Speed 131.54 samples/sec Loss 3.2510 Epoch: 15 Global Step: 154005 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:06:32,915-Speed 130.65 samples/sec Loss 3.3403 Epoch: 15 Global Step: 154010 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:06:52,471-Speed 130.91 samples/sec Loss 3.4131 Epoch: 15 Global Step: 154015 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:07:12,126-Speed 130.25 samples/sec Loss 3.3581 Epoch: 15 Global Step: 154020 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:07:31,676-Speed 130.96 samples/sec Loss 3.3943 Epoch: 15 Global Step: 154025 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:07:51,092-Speed 131.86 samples/sec Loss 3.4709 Epoch: 15 Global Step: 154030 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:08:10,807-Speed 129.86 samples/sec Loss 3.1567 Epoch: 15 Global Step: 154035 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:08:30,680-Speed 128.82 samples/sec Loss 3.3982 Epoch: 15 Global Step: 154040 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:08:50,251-Speed 130.81 samples/sec Loss 3.4418 Epoch: 15 Global Step: 154045 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:09:09,691-Speed 131.70 samples/sec Loss 3.3343 Epoch: 15 Global Step: 154050 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:09:29,308-Speed 130.56 samples/sec Loss 3.4227 Epoch: 15 Global Step: 154055 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:09:48,858-Speed 130.95 samples/sec Loss 3.2738 Epoch: 15 Global Step: 154060 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:10:08,456-Speed 130.63 samples/sec Loss 3.1654 Epoch: 15 Global Step: 154065 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:10:27,897-Speed 131.69 samples/sec Loss 3.2954 Epoch: 15 Global Step: 154070 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:10:47,446-Speed 130.96 samples/sec Loss 3.1854 Epoch: 15 Global Step: 154075 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:11:07,020-Speed 130.80 samples/sec Loss 3.2850 Epoch: 15 Global Step: 154080 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:11:26,927-Speed 128.60 samples/sec Loss 3.4147 Epoch: 15 Global Step: 154085 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:11:46,350-Speed 131.81 samples/sec Loss 3.3157 Epoch: 15 Global Step: 154090 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:12:06,158-Speed 129.25 samples/sec Loss 3.3388 Epoch: 15 Global Step: 154095 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:12:25,691-Speed 131.06 samples/sec Loss 3.3123 Epoch: 15 Global Step: 154100 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:12:45,203-Speed 131.21 samples/sec Loss 3.2474 Epoch: 15 Global Step: 154105 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:13:04,811-Speed 130.57 samples/sec Loss 3.2668 Epoch: 15 Global Step: 154110 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:13:24,355-Speed 130.99 samples/sec Loss 3.3582 Epoch: 15 Global Step: 154115 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:13:43,977-Speed 130.48 samples/sec Loss 3.5315 Epoch: 15 Global Step: 154120 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:14:03,092-Speed 133.94 samples/sec Loss 3.3934 Epoch: 15 Global Step: 154125 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:14:22,581-Speed 131.36 samples/sec Loss 3.2960 Epoch: 15 Global Step: 154130 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:14:42,173-Speed 130.68 samples/sec Loss 3.1601 Epoch: 15 Global Step: 154135 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:15:02,579-Speed 125.46 samples/sec Loss 3.4078 Epoch: 15 Global Step: 154140 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:15:22,008-Speed 131.77 samples/sec Loss 3.4544 Epoch: 15 Global Step: 154145 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:15:41,844-Speed 129.07 samples/sec Loss 3.5418 Epoch: 15 Global Step: 154150 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:16:01,504-Speed 130.22 samples/sec Loss 3.1997 Epoch: 15 Global Step: 154155 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:16:21,038-Speed 131.06 samples/sec Loss 3.2492 Epoch: 15 Global Step: 154160 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:16:40,614-Speed 130.78 samples/sec Loss 3.3040 Epoch: 15 Global Step: 154165 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:17:00,902-Speed 126.19 samples/sec Loss 3.3540 Epoch: 15 Global Step: 154170 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:17:20,513-Speed 130.55 samples/sec Loss 3.4098 Epoch: 15 Global Step: 154175 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:17:40,202-Speed 130.03 samples/sec Loss 3.2916 Epoch: 15 Global Step: 154180 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:17:59,982-Speed 129.43 samples/sec Loss 3.3775 Epoch: 15 Global Step: 154185 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:18:19,590-Speed 130.56 samples/sec Loss 3.2761 Epoch: 15 Global Step: 154190 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:18:39,189-Speed 130.63 samples/sec Loss 3.2965 Epoch: 15 Global Step: 154195 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:18:58,729-Speed 131.02 samples/sec Loss 3.1822 Epoch: 15 Global Step: 154200 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:19:18,337-Speed 130.57 samples/sec Loss 3.3415 Epoch: 15 Global Step: 154205 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:19:37,879-Speed 131.01 samples/sec Loss 3.3230 Epoch: 15 Global Step: 154210 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:19:57,397-Speed 131.16 samples/sec Loss 3.4154 Epoch: 15 Global Step: 154215 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:20:17,011-Speed 130.53 samples/sec Loss 3.3566 Epoch: 15 Global Step: 154220 Fp16 Grad Scale: 4096 Required: 112 hours Training: 2021-06-14 09:20:36,530-Speed 131.16 samples/sec Loss 3.2614 Epoch: 15 Global Step: 154225 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:20:56,364-Speed 129.08 samples/sec Loss 3.2862 Epoch: 15 Global Step: 154230 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:21:16,001-Speed 130.37 samples/sec Loss 3.2498 Epoch: 15 Global Step: 154235 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:21:35,580-Speed 130.76 samples/sec Loss 3.3529 Epoch: 15 Global Step: 154240 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:21:55,111-Speed 131.08 samples/sec Loss 3.3666 Epoch: 15 Global Step: 154245 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:22:14,718-Speed 130.57 samples/sec Loss 3.4803 Epoch: 15 Global Step: 154250 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:22:34,244-Speed 131.12 samples/sec Loss 3.5477 Epoch: 15 Global Step: 154255 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:22:53,717-Speed 131.47 samples/sec Loss 3.3431 Epoch: 15 Global Step: 154260 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:23:13,319-Speed 130.60 samples/sec Loss 3.3076 Epoch: 15 Global Step: 154265 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:23:32,861-Speed 131.01 samples/sec Loss 3.3198 Epoch: 15 Global Step: 154270 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:23:52,663-Speed 129.29 samples/sec Loss 3.3680 Epoch: 15 Global Step: 154275 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:24:12,340-Speed 130.11 samples/sec Loss 3.3447 Epoch: 15 Global Step: 154280 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:24:32,088-Speed 129.64 samples/sec Loss 3.3887 Epoch: 15 Global Step: 154285 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:24:51,629-Speed 131.02 samples/sec Loss 3.4578 Epoch: 15 Global Step: 154290 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:25:11,249-Speed 130.48 samples/sec Loss 3.2312 Epoch: 15 Global Step: 154295 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:25:30,692-Speed 131.67 samples/sec Loss 3.4000 Epoch: 15 Global Step: 154300 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:25:50,331-Speed 130.36 samples/sec Loss 3.3931 Epoch: 15 Global Step: 154305 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:26:09,975-Speed 130.33 samples/sec Loss 3.2062 Epoch: 15 Global Step: 154310 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:26:29,495-Speed 131.15 samples/sec Loss 3.2543 Epoch: 15 Global Step: 154315 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:26:49,271-Speed 129.46 samples/sec Loss 3.4623 Epoch: 15 Global Step: 154320 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:27:08,994-Speed 129.81 samples/sec Loss 3.3849 Epoch: 15 Global Step: 154325 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:27:28,489-Speed 131.32 samples/sec Loss 3.2919 Epoch: 15 Global Step: 154330 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:27:48,045-Speed 130.91 samples/sec Loss 3.1947 Epoch: 15 Global Step: 154335 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:28:07,621-Speed 130.78 samples/sec Loss 3.4669 Epoch: 15 Global Step: 154340 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:28:27,224-Speed 130.60 samples/sec Loss 3.3299 Epoch: 15 Global Step: 154345 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:28:46,828-Speed 130.60 samples/sec Loss 3.3775 Epoch: 15 Global Step: 154350 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:29:06,368-Speed 131.02 samples/sec Loss 3.1860 Epoch: 15 Global Step: 154355 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:29:25,903-Speed 131.05 samples/sec Loss 3.2670 Epoch: 15 Global Step: 154360 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:29:45,646-Speed 129.68 samples/sec Loss 3.1887 Epoch: 15 Global Step: 154365 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:30:05,459-Speed 129.21 samples/sec Loss 3.4037 Epoch: 15 Global Step: 154370 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:30:24,946-Speed 131.38 samples/sec Loss 3.1325 Epoch: 15 Global Step: 154375 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:30:44,545-Speed 130.62 samples/sec Loss 3.2225 Epoch: 15 Global Step: 154380 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:31:04,074-Speed 131.10 samples/sec Loss 3.1125 Epoch: 15 Global Step: 154385 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:31:23,691-Speed 130.51 samples/sec Loss 3.3088 Epoch: 15 Global Step: 154390 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:31:43,227-Speed 131.04 samples/sec Loss 3.2622 Epoch: 15 Global Step: 154395 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:32:02,784-Speed 130.91 samples/sec Loss 3.2749 Epoch: 15 Global Step: 154400 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:32:22,356-Speed 130.81 samples/sec Loss 3.2381 Epoch: 15 Global Step: 154405 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:32:42,293-Speed 128.41 samples/sec Loss 3.5055 Epoch: 15 Global Step: 154410 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:33:01,900-Speed 130.57 samples/sec Loss 3.3276 Epoch: 15 Global Step: 154415 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:33:21,519-Speed 130.50 samples/sec Loss 3.2088 Epoch: 15 Global Step: 154420 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:33:41,091-Speed 130.80 samples/sec Loss 3.3488 Epoch: 15 Global Step: 154425 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:34:00,723-Speed 130.41 samples/sec Loss 3.3993 Epoch: 15 Global Step: 154430 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:34:20,319-Speed 130.65 samples/sec Loss 3.3498 Epoch: 15 Global Step: 154435 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:34:39,887-Speed 130.83 samples/sec Loss 3.0823 Epoch: 15 Global Step: 154440 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:34:59,379-Speed 131.35 samples/sec Loss 3.5120 Epoch: 15 Global Step: 154445 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:35:18,952-Speed 130.80 samples/sec Loss 3.3047 Epoch: 15 Global Step: 154450 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:35:38,948-Speed 128.03 samples/sec Loss 3.2968 Epoch: 15 Global Step: 154455 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:35:58,376-Speed 131.77 samples/sec Loss 3.2260 Epoch: 15 Global Step: 154460 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:36:18,095-Speed 129.84 samples/sec Loss 3.4114 Epoch: 15 Global Step: 154465 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:36:37,624-Speed 131.10 samples/sec Loss 3.3775 Epoch: 15 Global Step: 154470 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:36:58,421-Speed 123.10 samples/sec Loss 3.3655 Epoch: 15 Global Step: 154475 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:37:18,046-Speed 130.45 samples/sec Loss 3.2249 Epoch: 15 Global Step: 154480 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:37:37,616-Speed 130.82 samples/sec Loss 3.4805 Epoch: 15 Global Step: 154485 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:37:57,212-Speed 130.65 samples/sec Loss 3.1564 Epoch: 15 Global Step: 154490 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:38:16,752-Speed 131.02 samples/sec Loss 3.2550 Epoch: 15 Global Step: 154495 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:38:36,604-Speed 128.96 samples/sec Loss 3.4422 Epoch: 15 Global Step: 154500 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:38:56,029-Speed 131.80 samples/sec Loss 3.4608 Epoch: 15 Global Step: 154505 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:39:15,741-Speed 129.88 samples/sec Loss 3.3050 Epoch: 15 Global Step: 154510 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:39:35,357-Speed 130.51 samples/sec Loss 3.2861 Epoch: 15 Global Step: 154515 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:39:54,908-Speed 130.94 samples/sec Loss 3.1965 Epoch: 15 Global Step: 154520 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:40:14,495-Speed 130.71 samples/sec Loss 3.4667 Epoch: 15 Global Step: 154525 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:40:34,292-Speed 129.33 samples/sec Loss 3.2963 Epoch: 15 Global Step: 154530 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:40:53,896-Speed 130.59 samples/sec Loss 3.3591 Epoch: 15 Global Step: 154535 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:41:13,507-Speed 130.55 samples/sec Loss 3.2820 Epoch: 15 Global Step: 154540 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:41:33,304-Speed 129.32 samples/sec Loss 3.3350 Epoch: 15 Global Step: 154545 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:41:52,994-Speed 130.02 samples/sec Loss 3.1825 Epoch: 15 Global Step: 154550 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:42:12,645-Speed 130.28 samples/sec Loss 3.3455 Epoch: 15 Global Step: 154555 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:42:32,246-Speed 130.61 samples/sec Loss 3.3340 Epoch: 15 Global Step: 154560 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:42:51,840-Speed 130.66 samples/sec Loss 3.3391 Epoch: 15 Global Step: 154565 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:43:11,396-Speed 130.92 samples/sec Loss 3.4442 Epoch: 15 Global Step: 154570 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:43:30,910-Speed 131.19 samples/sec Loss 3.2757 Epoch: 15 Global Step: 154575 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:43:50,480-Speed 130.82 samples/sec Loss 3.4087 Epoch: 15 Global Step: 154580 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:44:10,101-Speed 130.48 samples/sec Loss 3.2939 Epoch: 15 Global Step: 154585 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:44:29,912-Speed 129.23 samples/sec Loss 3.4858 Epoch: 15 Global Step: 154590 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:44:49,466-Speed 130.93 samples/sec Loss 3.2956 Epoch: 15 Global Step: 154595 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:45:09,085-Speed 130.49 samples/sec Loss 3.4661 Epoch: 15 Global Step: 154600 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:45:28,686-Speed 130.61 samples/sec Loss 3.3504 Epoch: 15 Global Step: 154605 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:45:48,380-Speed 130.00 samples/sec Loss 3.2943 Epoch: 15 Global Step: 154610 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:46:07,954-Speed 130.79 samples/sec Loss 3.2274 Epoch: 15 Global Step: 154615 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:46:27,491-Speed 131.05 samples/sec Loss 3.5172 Epoch: 15 Global Step: 154620 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:46:47,067-Speed 130.78 samples/sec Loss 3.3501 Epoch: 15 Global Step: 154625 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:47:06,581-Speed 131.20 samples/sec Loss 3.5236 Epoch: 15 Global Step: 154630 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:47:26,572-Speed 128.07 samples/sec Loss 3.3558 Epoch: 15 Global Step: 154635 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:47:46,282-Speed 129.89 samples/sec Loss 3.0728 Epoch: 15 Global Step: 154640 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:48:05,881-Speed 130.63 samples/sec Loss 3.3316 Epoch: 15 Global Step: 154645 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:48:25,506-Speed 130.46 samples/sec Loss 3.4481 Epoch: 15 Global Step: 154650 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:48:45,155-Speed 130.29 samples/sec Loss 3.2663 Epoch: 15 Global Step: 154655 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:49:04,744-Speed 130.70 samples/sec Loss 3.3217 Epoch: 15 Global Step: 154660 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:49:24,397-Speed 130.26 samples/sec Loss 3.3229 Epoch: 15 Global Step: 154665 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:49:44,008-Speed 130.55 samples/sec Loss 3.3257 Epoch: 15 Global Step: 154670 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:50:03,585-Speed 130.77 samples/sec Loss 3.4834 Epoch: 15 Global Step: 154675 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:50:23,367-Speed 129.41 samples/sec Loss 3.3684 Epoch: 15 Global Step: 154680 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:50:43,084-Speed 129.85 samples/sec Loss 3.3901 Epoch: 15 Global Step: 154685 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:51:02,635-Speed 130.94 samples/sec Loss 3.3958 Epoch: 15 Global Step: 154690 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:51:22,233-Speed 130.63 samples/sec Loss 3.5503 Epoch: 15 Global Step: 154695 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:51:41,801-Speed 130.84 samples/sec Loss 3.3937 Epoch: 15 Global Step: 154700 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:52:01,350-Speed 130.95 samples/sec Loss 3.2971 Epoch: 15 Global Step: 154705 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:52:20,966-Speed 130.52 samples/sec Loss 3.3409 Epoch: 15 Global Step: 154710 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:52:40,469-Speed 131.27 samples/sec Loss 3.4145 Epoch: 15 Global Step: 154715 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:53:00,017-Speed 130.97 samples/sec Loss 3.2858 Epoch: 15 Global Step: 154720 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:53:20,153-Speed 127.14 samples/sec Loss 3.4055 Epoch: 15 Global Step: 154725 Fp16 Grad Scale: 8192 Required: 112 hours Training: 2021-06-14 09:53:39,842-Speed 130.03 samples/sec Loss 3.4215 Epoch: 15 Global Step: 154730 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:53:59,492-Speed 130.28 samples/sec Loss 3.3999 Epoch: 15 Global Step: 154735 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:54:19,106-Speed 130.53 samples/sec Loss 3.3006 Epoch: 15 Global Step: 154740 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:54:38,648-Speed 131.01 samples/sec Loss 3.3005 Epoch: 15 Global Step: 154745 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:54:58,186-Speed 131.03 samples/sec Loss 3.4009 Epoch: 15 Global Step: 154750 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:55:17,787-Speed 130.61 samples/sec Loss 3.2674 Epoch: 15 Global Step: 154755 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:55:37,333-Speed 130.98 samples/sec Loss 3.5494 Epoch: 15 Global Step: 154760 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:55:56,938-Speed 130.59 samples/sec Loss 3.3846 Epoch: 15 Global Step: 154765 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:56:16,827-Speed 128.72 samples/sec Loss 3.2908 Epoch: 15 Global Step: 154770 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:56:36,288-Speed 131.55 samples/sec Loss 3.1158 Epoch: 15 Global Step: 154775 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:56:55,988-Speed 129.96 samples/sec Loss 3.3491 Epoch: 15 Global Step: 154780 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:57:15,576-Speed 130.70 samples/sec Loss 3.5415 Epoch: 15 Global Step: 154785 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:57:35,119-Speed 131.00 samples/sec Loss 3.4080 Epoch: 15 Global Step: 154790 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:57:54,733-Speed 130.52 samples/sec Loss 3.4679 Epoch: 15 Global Step: 154795 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:58:14,341-Speed 130.57 samples/sec Loss 3.3597 Epoch: 15 Global Step: 154800 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:58:33,911-Speed 130.82 samples/sec Loss 3.3549 Epoch: 15 Global Step: 154805 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:58:53,511-Speed 130.64 samples/sec Loss 3.1534 Epoch: 15 Global Step: 154810 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:59:13,266-Speed 129.60 samples/sec Loss 3.4022 Epoch: 15 Global Step: 154815 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:59:32,832-Speed 130.85 samples/sec Loss 3.3472 Epoch: 15 Global Step: 154820 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 09:59:52,395-Speed 130.87 samples/sec Loss 3.3115 Epoch: 15 Global Step: 154825 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:00:12,018-Speed 130.47 samples/sec Loss 3.2751 Epoch: 15 Global Step: 154830 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:00:31,452-Speed 131.73 samples/sec Loss 3.2202 Epoch: 15 Global Step: 154835 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:00:50,971-Speed 131.16 samples/sec Loss 3.3755 Epoch: 15 Global Step: 154840 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:01:10,497-Speed 131.12 samples/sec Loss 3.4455 Epoch: 15 Global Step: 154845 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:01:30,068-Speed 130.82 samples/sec Loss 3.2470 Epoch: 15 Global Step: 154850 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:01:49,655-Speed 130.71 samples/sec Loss 3.3569 Epoch: 15 Global Step: 154855 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:02:09,452-Speed 129.32 samples/sec Loss 3.2139 Epoch: 15 Global Step: 154860 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:02:28,889-Speed 131.72 samples/sec Loss 3.3849 Epoch: 15 Global Step: 154865 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:02:48,540-Speed 130.28 samples/sec Loss 3.4174 Epoch: 15 Global Step: 154870 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:03:08,144-Speed 130.59 samples/sec Loss 3.2819 Epoch: 15 Global Step: 154875 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:03:27,584-Speed 131.69 samples/sec Loss 3.4602 Epoch: 15 Global Step: 154880 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:03:47,301-Speed 129.85 samples/sec Loss 3.3591 Epoch: 15 Global Step: 154885 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:04:06,850-Speed 130.96 samples/sec Loss 3.3041 Epoch: 15 Global Step: 154890 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:04:26,394-Speed 131.00 samples/sec Loss 3.1958 Epoch: 15 Global Step: 154895 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:04:46,564-Speed 126.93 samples/sec Loss 3.4462 Epoch: 15 Global Step: 154900 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:05:06,369-Speed 129.27 samples/sec Loss 3.3732 Epoch: 15 Global Step: 154905 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:05:25,810-Speed 131.68 samples/sec Loss 3.1633 Epoch: 15 Global Step: 154910 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:05:45,468-Speed 130.24 samples/sec Loss 3.3548 Epoch: 15 Global Step: 154915 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:06:05,028-Speed 130.89 samples/sec Loss 3.4675 Epoch: 15 Global Step: 154920 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:06:24,450-Speed 131.81 samples/sec Loss 3.2965 Epoch: 15 Global Step: 154925 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:06:44,061-Speed 130.55 samples/sec Loss 3.2691 Epoch: 15 Global Step: 154930 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:07:03,612-Speed 130.94 samples/sec Loss 3.3169 Epoch: 15 Global Step: 154935 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:07:23,208-Speed 130.65 samples/sec Loss 3.2253 Epoch: 15 Global Step: 154940 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:07:42,789-Speed 130.74 samples/sec Loss 3.2553 Epoch: 15 Global Step: 154945 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:08:02,668-Speed 128.79 samples/sec Loss 3.4572 Epoch: 15 Global Step: 154950 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:08:22,097-Speed 131.76 samples/sec Loss 3.5319 Epoch: 15 Global Step: 154955 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:08:41,749-Speed 130.27 samples/sec Loss 3.4977 Epoch: 15 Global Step: 154960 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:09:01,245-Speed 131.32 samples/sec Loss 3.2890 Epoch: 15 Global Step: 154965 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:09:20,781-Speed 131.05 samples/sec Loss 3.6270 Epoch: 15 Global Step: 154970 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:09:40,354-Speed 130.80 samples/sec Loss 3.3333 Epoch: 15 Global Step: 154975 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:09:59,866-Speed 131.21 samples/sec Loss 3.4296 Epoch: 15 Global Step: 154980 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:10:19,592-Speed 129.78 samples/sec Loss 3.2690 Epoch: 15 Global Step: 154985 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:10:39,027-Speed 131.73 samples/sec Loss 3.4544 Epoch: 15 Global Step: 154990 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:10:58,981-Speed 128.31 samples/sec Loss 3.4864 Epoch: 15 Global Step: 154995 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:11:18,415-Speed 131.74 samples/sec Loss 3.5870 Epoch: 15 Global Step: 155000 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:11:38,125-Speed 129.88 samples/sec Loss 3.3076 Epoch: 15 Global Step: 155005 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:11:57,545-Speed 131.84 samples/sec Loss 3.2131 Epoch: 15 Global Step: 155010 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:12:17,175-Speed 130.42 samples/sec Loss 3.4156 Epoch: 15 Global Step: 155015 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:12:36,714-Speed 131.03 samples/sec Loss 3.4031 Epoch: 15 Global Step: 155020 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:12:56,199-Speed 131.39 samples/sec Loss 3.3981 Epoch: 15 Global Step: 155025 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:13:15,790-Speed 130.68 samples/sec Loss 3.1670 Epoch: 15 Global Step: 155030 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:13:35,255-Speed 131.52 samples/sec Loss 3.2391 Epoch: 15 Global Step: 155035 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:13:55,060-Speed 129.27 samples/sec Loss 3.3937 Epoch: 15 Global Step: 155040 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:14:14,586-Speed 131.11 samples/sec Loss 3.3247 Epoch: 15 Global Step: 155045 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:14:34,243-Speed 130.24 samples/sec Loss 3.3589 Epoch: 15 Global Step: 155050 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:14:53,832-Speed 130.69 samples/sec Loss 3.3523 Epoch: 15 Global Step: 155055 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:15:13,553-Speed 129.82 samples/sec Loss 3.3781 Epoch: 15 Global Step: 155060 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:15:33,096-Speed 131.00 samples/sec Loss 3.4425 Epoch: 15 Global Step: 155065 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:15:52,553-Speed 131.58 samples/sec Loss 3.6004 Epoch: 15 Global Step: 155070 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:16:12,110-Speed 130.91 samples/sec Loss 3.3960 Epoch: 15 Global Step: 155075 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:16:31,853-Speed 129.67 samples/sec Loss 3.3667 Epoch: 15 Global Step: 155080 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:16:51,665-Speed 129.23 samples/sec Loss 3.5026 Epoch: 15 Global Step: 155085 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:17:11,174-Speed 131.23 samples/sec Loss 3.4964 Epoch: 15 Global Step: 155090 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:17:30,746-Speed 130.80 samples/sec Loss 3.5420 Epoch: 15 Global Step: 155095 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:17:50,278-Speed 131.08 samples/sec Loss 3.2998 Epoch: 15 Global Step: 155100 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:18:09,901-Speed 130.47 samples/sec Loss 3.3580 Epoch: 15 Global Step: 155105 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:18:29,466-Speed 130.85 samples/sec Loss 3.3349 Epoch: 15 Global Step: 155110 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:18:48,939-Speed 131.47 samples/sec Loss 3.3963 Epoch: 15 Global Step: 155115 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:19:08,539-Speed 130.62 samples/sec Loss 3.2741 Epoch: 15 Global Step: 155120 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:19:28,085-Speed 130.98 samples/sec Loss 3.0558 Epoch: 15 Global Step: 155125 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:19:47,913-Speed 129.12 samples/sec Loss 3.3912 Epoch: 15 Global Step: 155130 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:20:07,589-Speed 130.11 samples/sec Loss 3.4455 Epoch: 15 Global Step: 155135 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:20:27,335-Speed 129.66 samples/sec Loss 3.5253 Epoch: 15 Global Step: 155140 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:20:47,005-Speed 130.16 samples/sec Loss 3.3179 Epoch: 15 Global Step: 155145 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:21:06,574-Speed 130.82 samples/sec Loss 3.3393 Epoch: 15 Global Step: 155150 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:21:26,112-Speed 131.03 samples/sec Loss 3.3792 Epoch: 15 Global Step: 155155 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:21:45,747-Speed 130.39 samples/sec Loss 3.3393 Epoch: 15 Global Step: 155160 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:22:05,395-Speed 130.30 samples/sec Loss 3.5483 Epoch: 15 Global Step: 155165 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:22:24,935-Speed 131.02 samples/sec Loss 3.6018 Epoch: 15 Global Step: 155170 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:22:44,712-Speed 129.45 samples/sec Loss 3.2641 Epoch: 15 Global Step: 155175 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:23:04,346-Speed 130.39 samples/sec Loss 3.3423 Epoch: 15 Global Step: 155180 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:23:23,933-Speed 130.70 samples/sec Loss 3.2853 Epoch: 15 Global Step: 155185 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:23:43,560-Speed 130.44 samples/sec Loss 3.4845 Epoch: 15 Global Step: 155190 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:24:03,010-Speed 131.63 samples/sec Loss 3.2825 Epoch: 15 Global Step: 155195 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:24:22,478-Speed 131.51 samples/sec Loss 3.2356 Epoch: 15 Global Step: 155200 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:24:42,074-Speed 130.65 samples/sec Loss 3.3092 Epoch: 15 Global Step: 155205 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:25:01,648-Speed 130.79 samples/sec Loss 3.3340 Epoch: 15 Global Step: 155210 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:25:21,231-Speed 130.73 samples/sec Loss 3.3126 Epoch: 15 Global Step: 155215 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:25:40,992-Speed 129.55 samples/sec Loss 3.3392 Epoch: 15 Global Step: 155220 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:26:00,588-Speed 130.65 samples/sec Loss 3.4152 Epoch: 15 Global Step: 155225 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:26:20,181-Speed 130.66 samples/sec Loss 3.0829 Epoch: 15 Global Step: 155230 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:26:39,804-Speed 130.47 samples/sec Loss 3.1777 Epoch: 15 Global Step: 155235 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:26:59,327-Speed 131.13 samples/sec Loss 3.6428 Epoch: 15 Global Step: 155240 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:27:18,944-Speed 130.51 samples/sec Loss 3.4952 Epoch: 15 Global Step: 155245 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:27:38,567-Speed 130.46 samples/sec Loss 3.5275 Epoch: 15 Global Step: 155250 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:27:58,110-Speed 131.01 samples/sec Loss 3.3700 Epoch: 15 Global Step: 155255 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:28:17,671-Speed 130.88 samples/sec Loss 3.3564 Epoch: 15 Global Step: 155260 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:28:37,583-Speed 128.57 samples/sec Loss 3.3084 Epoch: 15 Global Step: 155265 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:28:57,029-Speed 131.65 samples/sec Loss 3.2081 Epoch: 15 Global Step: 155270 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:29:16,631-Speed 130.61 samples/sec Loss 3.3783 Epoch: 15 Global Step: 155275 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:29:36,240-Speed 130.56 samples/sec Loss 3.3485 Epoch: 15 Global Step: 155280 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:29:55,772-Speed 131.08 samples/sec Loss 3.0554 Epoch: 15 Global Step: 155285 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:30:15,379-Speed 130.58 samples/sec Loss 3.4116 Epoch: 15 Global Step: 155290 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:30:34,954-Speed 130.79 samples/sec Loss 3.4002 Epoch: 15 Global Step: 155295 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:30:54,494-Speed 131.02 samples/sec Loss 3.3293 Epoch: 15 Global Step: 155300 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:31:14,098-Speed 130.59 samples/sec Loss 3.3831 Epoch: 15 Global Step: 155305 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:31:34,001-Speed 128.63 samples/sec Loss 3.0238 Epoch: 15 Global Step: 155310 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:31:53,435-Speed 131.74 samples/sec Loss 3.2745 Epoch: 15 Global Step: 155315 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:32:13,106-Speed 130.15 samples/sec Loss 3.4108 Epoch: 15 Global Step: 155320 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:32:32,642-Speed 131.05 samples/sec Loss 3.3051 Epoch: 15 Global Step: 155325 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:32:52,192-Speed 130.95 samples/sec Loss 3.2295 Epoch: 15 Global Step: 155330 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:33:11,805-Speed 130.53 samples/sec Loss 3.3275 Epoch: 15 Global Step: 155335 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:33:31,338-Speed 131.07 samples/sec Loss 3.4981 Epoch: 15 Global Step: 155340 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:33:50,954-Speed 130.51 samples/sec Loss 3.2431 Epoch: 15 Global Step: 155345 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:34:10,410-Speed 131.59 samples/sec Loss 3.1969 Epoch: 15 Global Step: 155350 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:34:30,335-Speed 128.49 samples/sec Loss 3.2450 Epoch: 15 Global Step: 155355 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:34:49,868-Speed 131.07 samples/sec Loss 3.4430 Epoch: 15 Global Step: 155360 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:35:09,577-Speed 129.90 samples/sec Loss 3.3529 Epoch: 15 Global Step: 155365 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:35:29,250-Speed 130.13 samples/sec Loss 3.3688 Epoch: 15 Global Step: 155370 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:35:48,781-Speed 131.08 samples/sec Loss 3.4253 Epoch: 15 Global Step: 155375 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:36:08,380-Speed 130.62 samples/sec Loss 3.2635 Epoch: 15 Global Step: 155380 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:36:27,588-Speed 133.29 samples/sec Loss 3.3049 Epoch: 15 Global Step: 155385 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:36:47,202-Speed 130.53 samples/sec Loss 3.3844 Epoch: 15 Global Step: 155390 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:37:06,755-Speed 130.93 samples/sec Loss 3.4823 Epoch: 15 Global Step: 155395 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:37:26,693-Speed 128.40 samples/sec Loss 3.2242 Epoch: 15 Global Step: 155400 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:37:46,133-Speed 131.70 samples/sec Loss 3.3547 Epoch: 15 Global Step: 155405 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:38:05,848-Speed 129.86 samples/sec Loss 3.3688 Epoch: 15 Global Step: 155410 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:38:25,370-Speed 131.14 samples/sec Loss 3.3045 Epoch: 15 Global Step: 155415 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:38:44,989-Speed 130.50 samples/sec Loss 3.3094 Epoch: 15 Global Step: 155420 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:39:04,551-Speed 130.88 samples/sec Loss 3.5357 Epoch: 15 Global Step: 155425 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:39:24,036-Speed 131.39 samples/sec Loss 3.3092 Epoch: 15 Global Step: 155430 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:39:43,638-Speed 130.61 samples/sec Loss 3.5751 Epoch: 15 Global Step: 155435 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:40:03,189-Speed 130.95 samples/sec Loss 3.3566 Epoch: 15 Global Step: 155440 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:40:23,006-Speed 129.19 samples/sec Loss 3.4711 Epoch: 15 Global Step: 155445 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:40:42,545-Speed 131.03 samples/sec Loss 3.3060 Epoch: 15 Global Step: 155450 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:41:02,180-Speed 130.38 samples/sec Loss 3.6419 Epoch: 15 Global Step: 155455 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:41:21,740-Speed 130.89 samples/sec Loss 3.2632 Epoch: 15 Global Step: 155460 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:41:41,290-Speed 130.96 samples/sec Loss 3.3478 Epoch: 15 Global Step: 155465 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:42:00,855-Speed 130.85 samples/sec Loss 3.2158 Epoch: 15 Global Step: 155470 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:42:20,389-Speed 131.06 samples/sec Loss 3.4735 Epoch: 15 Global Step: 155475 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:42:40,109-Speed 129.83 samples/sec Loss 3.4646 Epoch: 15 Global Step: 155480 Fp16 Grad Scale: 4096 Required: 111 hours Training: 2021-06-14 10:42:59,815-Speed 129.92 samples/sec Loss 3.2227 Epoch: 15 Global Step: 155485 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:43:19,599-Speed 129.40 samples/sec Loss 3.5301 Epoch: 15 Global Step: 155490 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:43:39,123-Speed 131.13 samples/sec Loss 3.5032 Epoch: 15 Global Step: 155495 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:43:58,821-Speed 129.97 samples/sec Loss 3.4071 Epoch: 15 Global Step: 155500 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:44:18,414-Speed 130.66 samples/sec Loss 3.5589 Epoch: 15 Global Step: 155505 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:44:38,013-Speed 130.63 samples/sec Loss 3.2493 Epoch: 15 Global Step: 155510 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:44:57,450-Speed 131.72 samples/sec Loss 3.3751 Epoch: 15 Global Step: 155515 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:45:16,983-Speed 131.07 samples/sec Loss 3.4812 Epoch: 15 Global Step: 155520 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:45:36,558-Speed 130.79 samples/sec Loss 3.5421 Epoch: 15 Global Step: 155525 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:45:56,196-Speed 130.37 samples/sec Loss 3.2522 Epoch: 15 Global Step: 155530 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:46:15,994-Speed 129.31 samples/sec Loss 3.6371 Epoch: 15 Global Step: 155535 Fp16 Grad Scale: 8192 Required: 111 hours Training: 2021-06-14 10:46:35,656-Speed 130.21 samples/sec Loss 3.3969 Epoch: 15 Global Step: 155540 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:46:55,268-Speed 130.54 samples/sec Loss 3.4656 Epoch: 15 Global Step: 155545 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:47:14,757-Speed 131.36 samples/sec Loss 3.3724 Epoch: 15 Global Step: 155550 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:47:34,344-Speed 130.70 samples/sec Loss 3.4586 Epoch: 15 Global Step: 155555 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:47:53,874-Speed 131.09 samples/sec Loss 3.5390 Epoch: 15 Global Step: 155560 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:48:13,354-Speed 131.42 samples/sec Loss 3.5152 Epoch: 15 Global Step: 155565 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:48:32,950-Speed 130.65 samples/sec Loss 3.2298 Epoch: 15 Global Step: 155570 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:48:52,400-Speed 131.63 samples/sec Loss 3.4274 Epoch: 15 Global Step: 155575 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:49:12,194-Speed 129.34 samples/sec Loss 3.2748 Epoch: 15 Global Step: 155580 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:49:31,858-Speed 130.20 samples/sec Loss 3.3393 Epoch: 15 Global Step: 155585 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:49:51,464-Speed 130.58 samples/sec Loss 3.4498 Epoch: 15 Global Step: 155590 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:50:11,041-Speed 130.77 samples/sec Loss 3.3927 Epoch: 15 Global Step: 155595 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:50:30,615-Speed 130.79 samples/sec Loss 3.4081 Epoch: 15 Global Step: 155600 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:50:50,102-Speed 131.38 samples/sec Loss 3.3804 Epoch: 15 Global Step: 155605 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:51:09,708-Speed 130.58 samples/sec Loss 3.2526 Epoch: 15 Global Step: 155610 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:51:29,259-Speed 130.95 samples/sec Loss 3.3611 Epoch: 15 Global Step: 155615 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:51:48,808-Speed 130.96 samples/sec Loss 3.2399 Epoch: 15 Global Step: 155620 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:52:08,609-Speed 129.29 samples/sec Loss 3.1336 Epoch: 15 Global Step: 155625 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:52:28,234-Speed 130.45 samples/sec Loss 3.1296 Epoch: 15 Global Step: 155630 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:52:47,700-Speed 131.52 samples/sec Loss 3.2423 Epoch: 15 Global Step: 155635 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:53:07,265-Speed 130.85 samples/sec Loss 3.2761 Epoch: 15 Global Step: 155640 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:53:26,853-Speed 130.70 samples/sec Loss 3.3220 Epoch: 15 Global Step: 155645 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:53:46,287-Speed 131.73 samples/sec Loss 3.3976 Epoch: 15 Global Step: 155650 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:54:05,882-Speed 130.65 samples/sec Loss 3.3299 Epoch: 15 Global Step: 155655 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:54:25,450-Speed 130.83 samples/sec Loss 3.5016 Epoch: 15 Global Step: 155660 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:54:44,986-Speed 131.05 samples/sec Loss 3.6678 Epoch: 15 Global Step: 155665 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:55:04,745-Speed 129.57 samples/sec Loss 3.5327 Epoch: 15 Global Step: 155670 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:55:24,486-Speed 129.68 samples/sec Loss 3.3645 Epoch: 15 Global Step: 155675 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:55:43,935-Speed 131.63 samples/sec Loss 3.4773 Epoch: 15 Global Step: 155680 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:56:03,545-Speed 130.56 samples/sec Loss 3.4408 Epoch: 15 Global Step: 155685 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:56:22,980-Speed 131.73 samples/sec Loss 3.3390 Epoch: 15 Global Step: 155690 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:56:42,556-Speed 130.77 samples/sec Loss 3.4671 Epoch: 15 Global Step: 155695 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:57:02,191-Speed 130.39 samples/sec Loss 3.3865 Epoch: 15 Global Step: 155700 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:57:21,722-Speed 131.08 samples/sec Loss 3.4044 Epoch: 15 Global Step: 155705 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:57:41,282-Speed 130.88 samples/sec Loss 3.3736 Epoch: 15 Global Step: 155710 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:58:00,972-Speed 130.03 samples/sec Loss 3.3341 Epoch: 15 Global Step: 155715 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:58:20,634-Speed 130.20 samples/sec Loss 3.3959 Epoch: 15 Global Step: 155720 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:58:40,172-Speed 131.03 samples/sec Loss 3.3948 Epoch: 15 Global Step: 155725 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:58:59,758-Speed 130.71 samples/sec Loss 3.4506 Epoch: 15 Global Step: 155730 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:59:19,285-Speed 131.13 samples/sec Loss 3.3325 Epoch: 15 Global Step: 155735 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:59:38,816-Speed 131.08 samples/sec Loss 3.5425 Epoch: 15 Global Step: 155740 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 10:59:58,386-Speed 130.82 samples/sec Loss 3.3407 Epoch: 15 Global Step: 155745 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:00:17,985-Speed 130.63 samples/sec Loss 3.4418 Epoch: 15 Global Step: 155750 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:00:37,568-Speed 130.74 samples/sec Loss 3.4322 Epoch: 15 Global Step: 155755 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:00:57,326-Speed 129.57 samples/sec Loss 3.4108 Epoch: 15 Global Step: 155760 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:01:17,048-Speed 129.81 samples/sec Loss 3.3504 Epoch: 15 Global Step: 155765 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:01:36,963-Speed 128.55 samples/sec Loss 3.4391 Epoch: 15 Global Step: 155770 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:01:56,562-Speed 130.63 samples/sec Loss 3.2703 Epoch: 15 Global Step: 155775 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:02:16,122-Speed 130.88 samples/sec Loss 3.4581 Epoch: 15 Global Step: 155780 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:02:35,724-Speed 130.61 samples/sec Loss 3.2272 Epoch: 15 Global Step: 155785 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:02:55,307-Speed 130.73 samples/sec Loss 3.4773 Epoch: 15 Global Step: 155790 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:03:14,856-Speed 130.96 samples/sec Loss 3.4466 Epoch: 15 Global Step: 155795 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:03:34,454-Speed 130.63 samples/sec Loss 3.2642 Epoch: 15 Global Step: 155800 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:03:54,306-Speed 128.96 samples/sec Loss 3.4806 Epoch: 15 Global Step: 155805 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:04:13,766-Speed 131.56 samples/sec Loss 3.2105 Epoch: 15 Global Step: 155810 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:04:33,411-Speed 130.32 samples/sec Loss 3.3439 Epoch: 15 Global Step: 155815 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:04:52,973-Speed 130.87 samples/sec Loss 3.4672 Epoch: 15 Global Step: 155820 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:05:12,488-Speed 131.19 samples/sec Loss 3.3082 Epoch: 15 Global Step: 155825 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:05:32,105-Speed 130.50 samples/sec Loss 3.3966 Epoch: 15 Global Step: 155830 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:05:51,874-Speed 129.50 samples/sec Loss 3.3245 Epoch: 15 Global Step: 155835 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:06:11,414-Speed 131.02 samples/sec Loss 3.3783 Epoch: 15 Global Step: 155840 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:06:31,002-Speed 130.70 samples/sec Loss 3.5232 Epoch: 15 Global Step: 155845 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:06:50,972-Speed 128.20 samples/sec Loss 3.2293 Epoch: 15 Global Step: 155850 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:07:10,402-Speed 131.76 samples/sec Loss 3.2966 Epoch: 15 Global Step: 155855 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:07:30,225-Speed 129.15 samples/sec Loss 3.3398 Epoch: 15 Global Step: 155860 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:07:49,764-Speed 131.03 samples/sec Loss 3.3455 Epoch: 15 Global Step: 155865 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:08:09,388-Speed 130.46 samples/sec Loss 3.3319 Epoch: 15 Global Step: 155870 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:08:28,997-Speed 130.56 samples/sec Loss 3.3096 Epoch: 15 Global Step: 155875 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:08:48,518-Speed 131.15 samples/sec Loss 3.4081 Epoch: 15 Global Step: 155880 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:09:08,113-Speed 130.65 samples/sec Loss 3.3944 Epoch: 15 Global Step: 155885 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:09:27,600-Speed 131.38 samples/sec Loss 3.4809 Epoch: 15 Global Step: 155890 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:09:47,567-Speed 128.22 samples/sec Loss 3.3974 Epoch: 15 Global Step: 155895 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:10:06,997-Speed 131.77 samples/sec Loss 3.3954 Epoch: 15 Global Step: 155900 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:10:26,625-Speed 130.43 samples/sec Loss 3.2912 Epoch: 15 Global Step: 155905 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:10:46,144-Speed 131.16 samples/sec Loss 3.6008 Epoch: 15 Global Step: 155910 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:11:05,693-Speed 130.96 samples/sec Loss 3.6414 Epoch: 15 Global Step: 155915 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:11:25,345-Speed 130.27 samples/sec Loss 3.5893 Epoch: 15 Global Step: 155920 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:11:44,782-Speed 131.72 samples/sec Loss 3.5562 Epoch: 15 Global Step: 155925 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:12:04,374-Speed 130.68 samples/sec Loss 3.3965 Epoch: 15 Global Step: 155930 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:12:23,966-Speed 130.67 samples/sec Loss 3.5757 Epoch: 15 Global Step: 155935 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:12:43,778-Speed 129.22 samples/sec Loss 3.3944 Epoch: 15 Global Step: 155940 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:13:03,374-Speed 130.65 samples/sec Loss 3.3116 Epoch: 15 Global Step: 155945 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:13:22,942-Speed 130.83 samples/sec Loss 3.2565 Epoch: 15 Global Step: 155950 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:13:42,538-Speed 130.64 samples/sec Loss 3.3877 Epoch: 15 Global Step: 155955 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:14:02,140-Speed 130.61 samples/sec Loss 3.2419 Epoch: 15 Global Step: 155960 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:14:21,674-Speed 131.07 samples/sec Loss 3.5467 Epoch: 15 Global Step: 155965 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:14:41,233-Speed 130.89 samples/sec Loss 3.4662 Epoch: 15 Global Step: 155970 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:15:00,795-Speed 130.87 samples/sec Loss 3.4560 Epoch: 15 Global Step: 155975 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:15:20,338-Speed 131.00 samples/sec Loss 3.1686 Epoch: 15 Global Step: 155980 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:15:40,147-Speed 129.24 samples/sec Loss 3.3250 Epoch: 15 Global Step: 155985 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:15:59,866-Speed 129.83 samples/sec Loss 3.4939 Epoch: 15 Global Step: 155990 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:16:19,409-Speed 131.00 samples/sec Loss 3.2341 Epoch: 15 Global Step: 155995 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:16:38,953-Speed 130.99 samples/sec Loss 3.3750 Epoch: 15 Global Step: 156000 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:16:58,511-Speed 130.90 samples/sec Loss 3.3496 Epoch: 15 Global Step: 156005 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:17:18,166-Speed 130.25 samples/sec Loss 3.2156 Epoch: 15 Global Step: 156010 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:17:37,898-Speed 129.74 samples/sec Loss 3.4221 Epoch: 15 Global Step: 156015 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:17:57,500-Speed 130.61 samples/sec Loss 3.4496 Epoch: 15 Global Step: 156020 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:18:17,022-Speed 131.14 samples/sec Loss 3.2166 Epoch: 15 Global Step: 156025 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:18:36,750-Speed 129.77 samples/sec Loss 3.3911 Epoch: 15 Global Step: 156030 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:18:56,554-Speed 129.27 samples/sec Loss 3.3733 Epoch: 15 Global Step: 156035 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:19:16,156-Speed 130.60 samples/sec Loss 3.4372 Epoch: 15 Global Step: 156040 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:19:35,757-Speed 130.61 samples/sec Loss 3.4498 Epoch: 15 Global Step: 156045 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:19:55,211-Speed 131.60 samples/sec Loss 3.3691 Epoch: 15 Global Step: 156050 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:20:14,754-Speed 131.00 samples/sec Loss 3.2183 Epoch: 15 Global Step: 156055 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:20:34,343-Speed 130.70 samples/sec Loss 3.3334 Epoch: 15 Global Step: 156060 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:20:53,950-Speed 130.57 samples/sec Loss 3.6668 Epoch: 15 Global Step: 156065 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:21:13,475-Speed 131.13 samples/sec Loss 3.2837 Epoch: 15 Global Step: 156070 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:21:33,245-Speed 129.50 samples/sec Loss 3.4966 Epoch: 15 Global Step: 156075 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:21:53,086-Speed 129.03 samples/sec Loss 3.3894 Epoch: 15 Global Step: 156080 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:22:12,644-Speed 130.90 samples/sec Loss 3.4060 Epoch: 15 Global Step: 156085 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:22:32,255-Speed 130.55 samples/sec Loss 3.3484 Epoch: 15 Global Step: 156090 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:22:51,795-Speed 131.02 samples/sec Loss 3.3288 Epoch: 15 Global Step: 156095 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:23:11,247-Speed 131.61 samples/sec Loss 3.3049 Epoch: 15 Global Step: 156100 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:23:31,009-Speed 129.55 samples/sec Loss 3.3729 Epoch: 15 Global Step: 156105 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:23:50,525-Speed 131.19 samples/sec Loss 3.5119 Epoch: 15 Global Step: 156110 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:24:10,160-Speed 130.39 samples/sec Loss 3.4719 Epoch: 15 Global Step: 156115 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:24:29,945-Speed 129.39 samples/sec Loss 3.5241 Epoch: 15 Global Step: 156120 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:24:49,516-Speed 130.81 samples/sec Loss 3.7160 Epoch: 15 Global Step: 156125 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:25:09,065-Speed 130.96 samples/sec Loss 3.4162 Epoch: 15 Global Step: 156130 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:25:28,651-Speed 130.71 samples/sec Loss 3.4342 Epoch: 15 Global Step: 156135 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:25:48,170-Speed 131.16 samples/sec Loss 3.1651 Epoch: 15 Global Step: 156140 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:26:07,760-Speed 130.69 samples/sec Loss 3.5249 Epoch: 15 Global Step: 156145 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:26:27,374-Speed 130.53 samples/sec Loss 3.3981 Epoch: 15 Global Step: 156150 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:26:46,906-Speed 131.07 samples/sec Loss 3.2354 Epoch: 15 Global Step: 156155 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:27:06,478-Speed 130.81 samples/sec Loss 3.3705 Epoch: 15 Global Step: 156160 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:27:26,300-Speed 129.16 samples/sec Loss 3.5775 Epoch: 15 Global Step: 156165 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:27:45,730-Speed 131.76 samples/sec Loss 3.2357 Epoch: 15 Global Step: 156170 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:28:05,433-Speed 129.94 samples/sec Loss 3.5630 Epoch: 15 Global Step: 156175 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:28:25,018-Speed 130.72 samples/sec Loss 3.3445 Epoch: 15 Global Step: 156180 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:28:44,459-Speed 131.68 samples/sec Loss 3.4216 Epoch: 15 Global Step: 156185 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:29:04,072-Speed 130.53 samples/sec Loss 3.4201 Epoch: 15 Global Step: 156190 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:29:23,677-Speed 130.59 samples/sec Loss 3.4511 Epoch: 15 Global Step: 156195 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:29:43,217-Speed 131.02 samples/sec Loss 3.4706 Epoch: 15 Global Step: 156200 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:30:02,791-Speed 130.79 samples/sec Loss 3.4324 Epoch: 15 Global Step: 156205 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:30:22,651-Speed 128.92 samples/sec Loss 3.4279 Epoch: 15 Global Step: 156210 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:30:42,100-Speed 131.63 samples/sec Loss 3.6326 Epoch: 15 Global Step: 156215 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:31:01,857-Speed 129.58 samples/sec Loss 3.3954 Epoch: 15 Global Step: 156220 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:31:21,409-Speed 130.94 samples/sec Loss 3.3535 Epoch: 15 Global Step: 156225 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:31:40,962-Speed 130.93 samples/sec Loss 3.4270 Epoch: 15 Global Step: 156230 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:32:00,552-Speed 130.68 samples/sec Loss 3.4367 Epoch: 15 Global Step: 156235 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:32:20,086-Speed 131.07 samples/sec Loss 3.4512 Epoch: 15 Global Step: 156240 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:32:39,706-Speed 130.48 samples/sec Loss 3.2958 Epoch: 15 Global Step: 156245 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:32:59,153-Speed 131.64 samples/sec Loss 3.2592 Epoch: 15 Global Step: 156250 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:33:19,036-Speed 128.76 samples/sec Loss 3.3183 Epoch: 15 Global Step: 156255 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:33:38,497-Speed 131.55 samples/sec Loss 3.3804 Epoch: 15 Global Step: 156260 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:33:58,199-Speed 129.94 samples/sec Loss 3.3938 Epoch: 15 Global Step: 156265 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:34:17,744-Speed 130.99 samples/sec Loss 3.5640 Epoch: 15 Global Step: 156270 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:34:37,278-Speed 131.06 samples/sec Loss 3.3814 Epoch: 15 Global Step: 156275 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:34:56,923-Speed 130.32 samples/sec Loss 3.3742 Epoch: 15 Global Step: 156280 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:35:16,455-Speed 131.08 samples/sec Loss 3.2837 Epoch: 15 Global Step: 156285 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:35:36,057-Speed 130.61 samples/sec Loss 3.2907 Epoch: 15 Global Step: 156290 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:35:55,601-Speed 130.99 samples/sec Loss 3.2827 Epoch: 15 Global Step: 156295 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:36:15,411-Speed 129.24 samples/sec Loss 3.6723 Epoch: 15 Global Step: 156300 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:36:35,032-Speed 130.48 samples/sec Loss 3.4177 Epoch: 15 Global Step: 156305 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:36:54,687-Speed 130.26 samples/sec Loss 3.6135 Epoch: 15 Global Step: 156310 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:37:14,222-Speed 131.05 samples/sec Loss 3.3960 Epoch: 15 Global Step: 156315 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:37:33,839-Speed 130.51 samples/sec Loss 3.6229 Epoch: 15 Global Step: 156320 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:37:53,430-Speed 130.68 samples/sec Loss 3.3605 Epoch: 15 Global Step: 156325 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:38:12,991-Speed 130.88 samples/sec Loss 3.4435 Epoch: 15 Global Step: 156330 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:38:32,554-Speed 130.87 samples/sec Loss 3.0628 Epoch: 15 Global Step: 156335 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:38:52,060-Speed 131.24 samples/sec Loss 3.4842 Epoch: 15 Global Step: 156340 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:39:11,851-Speed 129.36 samples/sec Loss 3.4527 Epoch: 15 Global Step: 156345 Fp16 Grad Scale: 8192 Required: 110 hours Training: 2021-06-14 11:39:31,417-Speed 130.85 samples/sec Loss 3.4580 Epoch: 15 Global Step: 156350 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:39:51,064-Speed 130.31 samples/sec Loss 3.2781 Epoch: 15 Global Step: 156355 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:40:10,497-Speed 131.74 samples/sec Loss 3.3744 Epoch: 15 Global Step: 156360 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:40:30,298-Speed 129.29 samples/sec Loss 3.5518 Epoch: 15 Global Step: 156365 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:40:49,861-Speed 130.87 samples/sec Loss 3.4743 Epoch: 15 Global Step: 156370 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:41:09,475-Speed 130.52 samples/sec Loss 3.3666 Epoch: 15 Global Step: 156375 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:41:29,126-Speed 130.28 samples/sec Loss 3.5208 Epoch: 15 Global Step: 156380 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:41:48,650-Speed 131.13 samples/sec Loss 3.3563 Epoch: 15 Global Step: 156385 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:42:08,436-Speed 129.39 samples/sec Loss 3.4263 Epoch: 15 Global Step: 156390 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:42:28,166-Speed 129.76 samples/sec Loss 3.3855 Epoch: 15 Global Step: 156395 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:42:47,586-Speed 131.83 samples/sec Loss 3.5038 Epoch: 15 Global Step: 156400 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:43:07,119-Speed 131.07 samples/sec Loss 3.3115 Epoch: 15 Global Step: 156405 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:43:26,649-Speed 131.08 samples/sec Loss 3.4483 Epoch: 15 Global Step: 156410 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:43:46,207-Speed 130.90 samples/sec Loss 3.4363 Epoch: 15 Global Step: 156415 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:44:05,814-Speed 130.58 samples/sec Loss 3.1973 Epoch: 15 Global Step: 156420 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:44:25,236-Speed 131.81 samples/sec Loss 3.4141 Epoch: 15 Global Step: 156425 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:44:44,771-Speed 131.06 samples/sec Loss 3.4357 Epoch: 15 Global Step: 156430 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:45:04,544-Speed 129.48 samples/sec Loss 3.1750 Epoch: 15 Global Step: 156435 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:45:24,222-Speed 130.10 samples/sec Loss 3.3797 Epoch: 15 Global Step: 156440 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:45:43,645-Speed 131.81 samples/sec Loss 3.1554 Epoch: 15 Global Step: 156445 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:46:03,248-Speed 130.60 samples/sec Loss 3.5796 Epoch: 15 Global Step: 156450 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:46:22,806-Speed 130.90 samples/sec Loss 3.3721 Epoch: 15 Global Step: 156455 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:46:42,346-Speed 131.03 samples/sec Loss 3.5556 Epoch: 15 Global Step: 156460 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:47:01,951-Speed 130.58 samples/sec Loss 3.5095 Epoch: 15 Global Step: 156465 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:47:21,366-Speed 131.87 samples/sec Loss 3.4010 Epoch: 15 Global Step: 156470 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:47:40,998-Speed 130.41 samples/sec Loss 3.4194 Epoch: 15 Global Step: 156475 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:48:00,777-Speed 129.44 samples/sec Loss 3.2622 Epoch: 15 Global Step: 156480 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:48:20,488-Speed 129.88 samples/sec Loss 3.2392 Epoch: 15 Global Step: 156485 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:48:39,921-Speed 131.74 samples/sec Loss 3.3973 Epoch: 15 Global Step: 156490 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:48:59,508-Speed 130.71 samples/sec Loss 3.3352 Epoch: 15 Global Step: 156495 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:49:19,097-Speed 130.70 samples/sec Loss 3.3122 Epoch: 15 Global Step: 156500 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:49:38,730-Speed 130.40 samples/sec Loss 3.4135 Epoch: 15 Global Step: 156505 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:49:58,332-Speed 130.61 samples/sec Loss 3.3891 Epoch: 15 Global Step: 156510 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:50:17,937-Speed 130.59 samples/sec Loss 3.2815 Epoch: 15 Global Step: 156515 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:50:37,493-Speed 130.91 samples/sec Loss 3.5168 Epoch: 15 Global Step: 156520 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:50:57,254-Speed 129.55 samples/sec Loss 3.3811 Epoch: 15 Global Step: 156525 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:51:16,824-Speed 130.82 samples/sec Loss 3.2830 Epoch: 15 Global Step: 156530 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:51:36,430-Speed 130.58 samples/sec Loss 3.2478 Epoch: 15 Global Step: 156535 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:51:56,055-Speed 130.45 samples/sec Loss 3.4255 Epoch: 15 Global Step: 156540 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:52:15,589-Speed 131.06 samples/sec Loss 3.5156 Epoch: 15 Global Step: 156545 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:52:35,059-Speed 131.50 samples/sec Loss 3.2625 Epoch: 15 Global Step: 156550 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:52:54,627-Speed 130.83 samples/sec Loss 3.3501 Epoch: 15 Global Step: 156555 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:53:14,248-Speed 130.48 samples/sec Loss 3.3289 Epoch: 15 Global Step: 156560 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:53:34,040-Speed 129.36 samples/sec Loss 3.6295 Epoch: 15 Global Step: 156565 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:53:53,821-Speed 129.42 samples/sec Loss 3.6252 Epoch: 15 Global Step: 156570 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:54:13,500-Speed 130.10 samples/sec Loss 3.5449 Epoch: 15 Global Step: 156575 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:54:33,073-Speed 130.80 samples/sec Loss 3.4979 Epoch: 15 Global Step: 156580 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:54:52,650-Speed 130.77 samples/sec Loss 3.2648 Epoch: 15 Global Step: 156585 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:55:12,175-Speed 131.13 samples/sec Loss 3.3230 Epoch: 15 Global Step: 156590 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:55:31,762-Speed 130.71 samples/sec Loss 3.1546 Epoch: 15 Global Step: 156595 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:55:51,289-Speed 131.10 samples/sec Loss 3.2840 Epoch: 15 Global Step: 156600 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:56:10,899-Speed 130.56 samples/sec Loss 3.3956 Epoch: 15 Global Step: 156605 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:56:30,455-Speed 130.91 samples/sec Loss 3.3745 Epoch: 15 Global Step: 156610 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:56:50,242-Speed 129.39 samples/sec Loss 3.6216 Epoch: 15 Global Step: 156615 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:57:09,668-Speed 131.79 samples/sec Loss 3.4193 Epoch: 15 Global Step: 156620 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:57:29,484-Speed 129.19 samples/sec Loss 3.4014 Epoch: 15 Global Step: 156625 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:57:49,042-Speed 130.90 samples/sec Loss 3.3228 Epoch: 15 Global Step: 156630 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:58:08,651-Speed 130.56 samples/sec Loss 3.4886 Epoch: 15 Global Step: 156635 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:58:28,129-Speed 131.44 samples/sec Loss 3.4134 Epoch: 15 Global Step: 156640 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:58:47,743-Speed 130.53 samples/sec Loss 3.5727 Epoch: 15 Global Step: 156645 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:59:07,324-Speed 130.74 samples/sec Loss 3.4604 Epoch: 15 Global Step: 156650 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:59:26,872-Speed 130.97 samples/sec Loss 3.3802 Epoch: 15 Global Step: 156655 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 11:59:46,655-Speed 129.41 samples/sec Loss 3.5484 Epoch: 15 Global Step: 156660 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:00:06,317-Speed 130.21 samples/sec Loss 3.3120 Epoch: 15 Global Step: 156665 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:00:26,017-Speed 129.95 samples/sec Loss 3.3349 Epoch: 15 Global Step: 156670 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:00:45,562-Speed 130.99 samples/sec Loss 3.4279 Epoch: 15 Global Step: 156675 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:01:05,171-Speed 130.56 samples/sec Loss 3.4772 Epoch: 15 Global Step: 156680 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:01:24,747-Speed 130.78 samples/sec Loss 3.3109 Epoch: 15 Global Step: 156685 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:01:44,211-Speed 131.53 samples/sec Loss 3.3941 Epoch: 15 Global Step: 156690 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:02:03,781-Speed 130.82 samples/sec Loss 3.1532 Epoch: 15 Global Step: 156695 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:02:23,326-Speed 130.98 samples/sec Loss 3.2435 Epoch: 15 Global Step: 156700 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:02:43,118-Speed 129.35 samples/sec Loss 3.3390 Epoch: 15 Global Step: 156705 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:03:03,004-Speed 128.74 samples/sec Loss 3.2595 Epoch: 15 Global Step: 156710 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:03:22,497-Speed 131.34 samples/sec Loss 3.3372 Epoch: 15 Global Step: 156715 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:03:41,992-Speed 131.32 samples/sec Loss 3.3484 Epoch: 15 Global Step: 156720 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:04:01,557-Speed 130.86 samples/sec Loss 3.4837 Epoch: 15 Global Step: 156725 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:04:21,153-Speed 130.64 samples/sec Loss 3.3912 Epoch: 15 Global Step: 156730 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:04:40,881-Speed 129.77 samples/sec Loss 3.2171 Epoch: 15 Global Step: 156735 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:05:00,406-Speed 131.13 samples/sec Loss 3.4271 Epoch: 15 Global Step: 156740 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:05:19,981-Speed 130.79 samples/sec Loss 3.4962 Epoch: 15 Global Step: 156745 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:05:39,834-Speed 128.96 samples/sec Loss 3.3081 Epoch: 15 Global Step: 156750 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:05:59,456-Speed 130.47 samples/sec Loss 3.5093 Epoch: 15 Global Step: 156755 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:06:18,915-Speed 131.57 samples/sec Loss 3.2662 Epoch: 15 Global Step: 156760 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:06:38,509-Speed 130.66 samples/sec Loss 3.2858 Epoch: 15 Global Step: 156765 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:06:58,091-Speed 130.74 samples/sec Loss 3.7094 Epoch: 15 Global Step: 156770 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:07:17,707-Speed 130.51 samples/sec Loss 3.3170 Epoch: 15 Global Step: 156775 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:07:37,260-Speed 130.93 samples/sec Loss 3.3026 Epoch: 15 Global Step: 156780 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:07:56,784-Speed 131.13 samples/sec Loss 3.3808 Epoch: 15 Global Step: 156785 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:08:16,363-Speed 130.76 samples/sec Loss 3.4532 Epoch: 15 Global Step: 156790 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:08:36,674-Speed 126.05 samples/sec Loss 3.4457 Epoch: 15 Global Step: 156795 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:08:56,110-Speed 131.72 samples/sec Loss 3.5746 Epoch: 15 Global Step: 156800 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:09:15,750-Speed 130.35 samples/sec Loss 3.6720 Epoch: 15 Global Step: 156805 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:09:35,309-Speed 130.89 samples/sec Loss 3.2911 Epoch: 15 Global Step: 156810 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:09:54,916-Speed 130.58 samples/sec Loss 3.3814 Epoch: 15 Global Step: 156815 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:10:14,505-Speed 130.69 samples/sec Loss 3.5402 Epoch: 15 Global Step: 156820 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:10:34,054-Speed 130.96 samples/sec Loss 3.4481 Epoch: 15 Global Step: 156825 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:10:53,555-Speed 131.28 samples/sec Loss 3.3488 Epoch: 15 Global Step: 156830 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:11:13,179-Speed 130.46 samples/sec Loss 3.3362 Epoch: 15 Global Step: 156835 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:11:33,134-Speed 128.29 samples/sec Loss 3.3133 Epoch: 15 Global Step: 156840 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:11:52,568-Speed 131.74 samples/sec Loss 3.2356 Epoch: 15 Global Step: 156845 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:12:12,270-Speed 129.94 samples/sec Loss 3.4307 Epoch: 15 Global Step: 156850 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:12:31,861-Speed 130.68 samples/sec Loss 3.4801 Epoch: 15 Global Step: 156855 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:12:51,609-Speed 129.64 samples/sec Loss 3.4520 Epoch: 15 Global Step: 156860 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:13:11,146-Speed 131.04 samples/sec Loss 3.3633 Epoch: 15 Global Step: 156865 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:13:30,799-Speed 130.27 samples/sec Loss 3.4656 Epoch: 15 Global Step: 156870 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:13:50,360-Speed 130.88 samples/sec Loss 3.4618 Epoch: 15 Global Step: 156875 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:14:09,956-Speed 130.65 samples/sec Loss 3.3576 Epoch: 15 Global Step: 156880 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:14:29,727-Speed 129.50 samples/sec Loss 3.3259 Epoch: 15 Global Step: 156885 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:14:49,445-Speed 129.84 samples/sec Loss 3.4388 Epoch: 15 Global Step: 156890 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:15:08,882-Speed 131.71 samples/sec Loss 3.4404 Epoch: 15 Global Step: 156895 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:15:28,503-Speed 130.48 samples/sec Loss 3.3024 Epoch: 15 Global Step: 156900 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:15:48,330-Speed 129.12 samples/sec Loss 3.2641 Epoch: 15 Global Step: 156905 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:16:07,857-Speed 131.11 samples/sec Loss 3.2827 Epoch: 15 Global Step: 156910 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:16:27,417-Speed 130.89 samples/sec Loss 3.2579 Epoch: 15 Global Step: 156915 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:16:46,857-Speed 131.69 samples/sec Loss 3.4086 Epoch: 15 Global Step: 156920 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:17:06,465-Speed 130.57 samples/sec Loss 3.2131 Epoch: 15 Global Step: 156925 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:17:26,280-Speed 129.21 samples/sec Loss 3.4043 Epoch: 15 Global Step: 156930 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:17:45,794-Speed 131.19 samples/sec Loss 3.4255 Epoch: 15 Global Step: 156935 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:18:05,343-Speed 130.96 samples/sec Loss 3.5070 Epoch: 15 Global Step: 156940 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:18:24,952-Speed 130.55 samples/sec Loss 3.3815 Epoch: 15 Global Step: 156945 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:18:44,532-Speed 130.75 samples/sec Loss 3.4983 Epoch: 15 Global Step: 156950 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:19:04,132-Speed 130.62 samples/sec Loss 3.4238 Epoch: 15 Global Step: 156955 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:19:23,588-Speed 131.59 samples/sec Loss 3.3688 Epoch: 15 Global Step: 156960 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:19:43,205-Speed 130.51 samples/sec Loss 3.5431 Epoch: 15 Global Step: 156965 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:20:02,773-Speed 130.83 samples/sec Loss 3.4540 Epoch: 15 Global Step: 156970 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:20:22,242-Speed 131.50 samples/sec Loss 3.3525 Epoch: 15 Global Step: 156975 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:20:41,672-Speed 131.76 samples/sec Loss 3.2312 Epoch: 15 Global Step: 156980 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:21:01,422-Speed 129.63 samples/sec Loss 3.2449 Epoch: 15 Global Step: 156985 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:21:21,033-Speed 130.55 samples/sec Loss 3.4041 Epoch: 15 Global Step: 156990 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:21:40,571-Speed 131.03 samples/sec Loss 3.7719 Epoch: 15 Global Step: 156995 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:22:00,167-Speed 130.65 samples/sec Loss 3.5435 Epoch: 15 Global Step: 157000 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:22:19,703-Speed 131.05 samples/sec Loss 3.5608 Epoch: 15 Global Step: 157005 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:22:39,324-Speed 130.48 samples/sec Loss 3.4027 Epoch: 15 Global Step: 157010 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:22:58,878-Speed 130.93 samples/sec Loss 3.5865 Epoch: 15 Global Step: 157015 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:23:18,776-Speed 128.66 samples/sec Loss 3.0809 Epoch: 15 Global Step: 157020 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:23:38,195-Speed 131.83 samples/sec Loss 3.3589 Epoch: 15 Global Step: 157025 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:23:57,918-Speed 129.81 samples/sec Loss 3.4966 Epoch: 15 Global Step: 157030 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:24:17,483-Speed 130.85 samples/sec Loss 3.3459 Epoch: 15 Global Step: 157035 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:24:37,311-Speed 129.12 samples/sec Loss 3.5414 Epoch: 15 Global Step: 157040 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:24:56,899-Speed 130.70 samples/sec Loss 3.3424 Epoch: 15 Global Step: 157045 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:25:16,344-Speed 131.66 samples/sec Loss 3.4771 Epoch: 15 Global Step: 157050 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:25:35,959-Speed 130.52 samples/sec Loss 3.4874 Epoch: 15 Global Step: 157055 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:25:55,443-Speed 131.40 samples/sec Loss 3.3491 Epoch: 15 Global Step: 157060 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:26:15,264-Speed 129.16 samples/sec Loss 3.4679 Epoch: 15 Global Step: 157065 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:26:34,695-Speed 131.75 samples/sec Loss 3.5673 Epoch: 15 Global Step: 157070 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:26:54,407-Speed 129.88 samples/sec Loss 3.3731 Epoch: 15 Global Step: 157075 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:27:13,894-Speed 131.37 samples/sec Loss 3.4434 Epoch: 15 Global Step: 157080 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:27:33,367-Speed 131.47 samples/sec Loss 3.5327 Epoch: 15 Global Step: 157085 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:27:52,930-Speed 130.87 samples/sec Loss 3.2311 Epoch: 15 Global Step: 157090 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:28:12,540-Speed 130.55 samples/sec Loss 3.4509 Epoch: 15 Global Step: 157095 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:28:32,145-Speed 130.59 samples/sec Loss 3.4501 Epoch: 15 Global Step: 157100 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:28:51,612-Speed 131.51 samples/sec Loss 3.3083 Epoch: 15 Global Step: 157105 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:29:11,437-Speed 129.14 samples/sec Loss 3.3507 Epoch: 15 Global Step: 157110 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:29:30,854-Speed 131.85 samples/sec Loss 3.5286 Epoch: 15 Global Step: 157115 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:29:50,642-Speed 129.38 samples/sec Loss 3.5528 Epoch: 15 Global Step: 157120 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:30:10,178-Speed 131.05 samples/sec Loss 3.5948 Epoch: 15 Global Step: 157125 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:30:29,710-Speed 131.07 samples/sec Loss 3.2952 Epoch: 15 Global Step: 157130 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:30:49,276-Speed 130.85 samples/sec Loss 3.6254 Epoch: 15 Global Step: 157135 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:31:08,832-Speed 130.92 samples/sec Loss 3.5885 Epoch: 15 Global Step: 157140 Fp16 Grad Scale: 8192 Required: 109 hours Training: 2021-06-14 12:31:27,977-Speed 133.72 samples/sec Loss 3.4721 Epoch: 15 Global Step: 157145 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:31:47,535-Speed 130.90 samples/sec Loss 3.4139 Epoch: 15 Global Step: 157150 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:32:07,144-Speed 130.56 samples/sec Loss 3.2729 Epoch: 15 Global Step: 157155 Fp16 Grad Scale: 4096 Required: 109 hours Training: 2021-06-14 12:32:26,716-Speed 130.81 samples/sec Loss 3.2428 Epoch: 15 Global Step: 157160 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:32:46,240-Speed 131.12 samples/sec Loss 3.4137 Epoch: 15 Global Step: 157165 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:33:06,013-Speed 129.48 samples/sec Loss 3.2827 Epoch: 15 Global Step: 157170 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:33:25,410-Speed 131.99 samples/sec Loss 3.6435 Epoch: 15 Global Step: 157175 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:33:45,186-Speed 129.46 samples/sec Loss 3.6917 Epoch: 15 Global Step: 157180 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:34:04,961-Speed 129.47 samples/sec Loss 3.2579 Epoch: 15 Global Step: 157185 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:34:24,502-Speed 131.01 samples/sec Loss 3.3781 Epoch: 15 Global Step: 157190 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:34:44,045-Speed 131.00 samples/sec Loss 3.3901 Epoch: 15 Global Step: 157195 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:35:03,648-Speed 130.59 samples/sec Loss 3.1896 Epoch: 15 Global Step: 157200 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:35:23,181-Speed 131.07 samples/sec Loss 3.2494 Epoch: 15 Global Step: 157205 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:35:42,886-Speed 129.92 samples/sec Loss 3.3906 Epoch: 15 Global Step: 157210 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:36:02,430-Speed 131.00 samples/sec Loss 3.4557 Epoch: 15 Global Step: 157215 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:36:22,035-Speed 130.58 samples/sec Loss 3.7241 Epoch: 15 Global Step: 157220 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:36:41,890-Speed 128.94 samples/sec Loss 3.5316 Epoch: 15 Global Step: 157225 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:37:01,508-Speed 130.50 samples/sec Loss 3.3563 Epoch: 15 Global Step: 157230 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:37:21,040-Speed 131.07 samples/sec Loss 3.4137 Epoch: 15 Global Step: 157235 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:37:40,682-Speed 130.34 samples/sec Loss 3.4776 Epoch: 15 Global Step: 157240 Fp16 Grad Scale: 4096 Required: 108 hours Training: 2021-06-14 12:38:00,250-Speed 130.83 samples/sec Loss 3.4145 Epoch: 15 Global Step: 157245 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:38:19,845-Speed 130.65 samples/sec Loss 3.3896 Epoch: 15 Global Step: 157250 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:38:39,369-Speed 131.13 samples/sec Loss 3.1663 Epoch: 15 Global Step: 157255 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:38:58,959-Speed 130.69 samples/sec Loss 3.3036 Epoch: 15 Global Step: 157260 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:39:18,541-Speed 130.74 samples/sec Loss 3.4953 Epoch: 15 Global Step: 157265 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:39:38,431-Speed 128.71 samples/sec Loss 3.3974 Epoch: 15 Global Step: 157270 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:39:57,848-Speed 131.85 samples/sec Loss 3.4090 Epoch: 15 Global Step: 157275 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:40:17,569-Speed 129.82 samples/sec Loss 3.3948 Epoch: 15 Global Step: 157280 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:40:37,230-Speed 130.22 samples/sec Loss 3.3724 Epoch: 15 Global Step: 157285 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:40:56,781-Speed 130.94 samples/sec Loss 3.3385 Epoch: 15 Global Step: 157290 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:41:16,354-Speed 130.80 samples/sec Loss 3.2867 Epoch: 15 Global Step: 157295 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:41:35,911-Speed 130.91 samples/sec Loss 3.4516 Epoch: 15 Global Step: 157300 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:41:55,708-Speed 129.32 samples/sec Loss 3.3495 Epoch: 15 Global Step: 157305 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:42:15,327-Speed 130.50 samples/sec Loss 3.4231 Epoch: 15 Global Step: 157310 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:42:35,202-Speed 128.82 samples/sec Loss 3.3061 Epoch: 15 Global Step: 157315 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:42:54,977-Speed 129.46 samples/sec Loss 3.5215 Epoch: 15 Global Step: 157320 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:43:14,683-Speed 129.91 samples/sec Loss 3.4820 Epoch: 15 Global Step: 157325 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:43:34,265-Speed 130.74 samples/sec Loss 3.5114 Epoch: 15 Global Step: 157330 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:43:53,850-Speed 130.72 samples/sec Loss 3.3878 Epoch: 15 Global Step: 157335 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:44:13,384-Speed 131.06 samples/sec Loss 3.3876 Epoch: 15 Global Step: 157340 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:44:32,924-Speed 131.02 samples/sec Loss 3.3263 Epoch: 15 Global Step: 157345 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:44:52,519-Speed 130.66 samples/sec Loss 3.4308 Epoch: 15 Global Step: 157350 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:45:12,040-Speed 131.15 samples/sec Loss 3.6387 Epoch: 15 Global Step: 157355 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:45:31,857-Speed 129.19 samples/sec Loss 3.2986 Epoch: 15 Global Step: 157360 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:45:51,504-Speed 130.31 samples/sec Loss 3.4419 Epoch: 15 Global Step: 157365 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:46:11,158-Speed 130.26 samples/sec Loss 3.5208 Epoch: 15 Global Step: 157370 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:46:30,691-Speed 131.07 samples/sec Loss 3.6400 Epoch: 15 Global Step: 157375 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:46:50,292-Speed 130.61 samples/sec Loss 3.3293 Epoch: 15 Global Step: 157380 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:47:09,831-Speed 131.03 samples/sec Loss 3.3378 Epoch: 15 Global Step: 157385 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:47:29,369-Speed 131.04 samples/sec Loss 3.2715 Epoch: 15 Global Step: 157390 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:47:48,981-Speed 130.55 samples/sec Loss 3.4103 Epoch: 15 Global Step: 157395 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:48:08,480-Speed 131.29 samples/sec Loss 3.4118 Epoch: 15 Global Step: 157400 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:48:28,299-Speed 129.18 samples/sec Loss 3.2728 Epoch: 15 Global Step: 157405 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:48:47,906-Speed 130.58 samples/sec Loss 3.4439 Epoch: 15 Global Step: 157410 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:49:07,519-Speed 130.53 samples/sec Loss 3.3658 Epoch: 15 Global Step: 157415 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:49:26,987-Speed 131.51 samples/sec Loss 3.5208 Epoch: 15 Global Step: 157420 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:49:46,599-Speed 130.54 samples/sec Loss 3.4932 Epoch: 15 Global Step: 157425 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:50:06,141-Speed 131.01 samples/sec Loss 3.4239 Epoch: 15 Global Step: 157430 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:50:25,655-Speed 131.20 samples/sec Loss 3.4205 Epoch: 15 Global Step: 157435 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:50:45,267-Speed 130.54 samples/sec Loss 3.2674 Epoch: 15 Global Step: 157440 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:51:04,802-Speed 131.06 samples/sec Loss 3.5056 Epoch: 15 Global Step: 157445 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:51:24,606-Speed 129.27 samples/sec Loss 3.3604 Epoch: 15 Global Step: 157450 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:51:44,310-Speed 129.93 samples/sec Loss 3.5261 Epoch: 15 Global Step: 157455 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:52:03,877-Speed 130.84 samples/sec Loss 3.3318 Epoch: 15 Global Step: 157460 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:52:23,503-Speed 130.45 samples/sec Loss 3.3600 Epoch: 15 Global Step: 157465 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:52:43,102-Speed 130.63 samples/sec Loss 3.3297 Epoch: 15 Global Step: 157470 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:53:02,629-Speed 131.11 samples/sec Loss 3.5176 Epoch: 15 Global Step: 157475 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:53:22,231-Speed 130.61 samples/sec Loss 3.3260 Epoch: 15 Global Step: 157480 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:53:41,791-Speed 130.89 samples/sec Loss 3.2813 Epoch: 15 Global Step: 157485 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:54:01,352-Speed 130.88 samples/sec Loss 3.5003 Epoch: 15 Global Step: 157490 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:54:21,192-Speed 129.05 samples/sec Loss 3.0985 Epoch: 15 Global Step: 157495 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:54:40,888-Speed 129.98 samples/sec Loss 3.3913 Epoch: 15 Global Step: 157500 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:55:00,415-Speed 131.11 samples/sec Loss 3.4865 Epoch: 15 Global Step: 157505 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:55:19,946-Speed 131.08 samples/sec Loss 3.3466 Epoch: 15 Global Step: 157510 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:55:39,525-Speed 130.76 samples/sec Loss 3.3498 Epoch: 15 Global Step: 157515 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:55:59,137-Speed 130.54 samples/sec Loss 3.4908 Epoch: 15 Global Step: 157520 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:56:19,223-Speed 127.46 samples/sec Loss 3.2329 Epoch: 15 Global Step: 157525 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:56:38,946-Speed 129.80 samples/sec Loss 3.0767 Epoch: 15 Global Step: 157530 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:56:58,413-Speed 131.51 samples/sec Loss 3.4064 Epoch: 15 Global Step: 157535 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:57:18,393-Speed 128.14 samples/sec Loss 3.3175 Epoch: 15 Global Step: 157540 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:57:38,200-Speed 129.25 samples/sec Loss 3.3852 Epoch: 15 Global Step: 157545 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:57:57,753-Speed 130.93 samples/sec Loss 3.3415 Epoch: 15 Global Step: 157550 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:58:17,366-Speed 130.54 samples/sec Loss 3.4904 Epoch: 15 Global Step: 157555 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:58:36,896-Speed 131.09 samples/sec Loss 3.3302 Epoch: 15 Global Step: 157560 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:58:56,526-Speed 130.42 samples/sec Loss 3.3149 Epoch: 15 Global Step: 157565 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:59:16,094-Speed 130.83 samples/sec Loss 3.2495 Epoch: 15 Global Step: 157570 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:59:35,609-Speed 131.19 samples/sec Loss 3.6502 Epoch: 15 Global Step: 157575 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 12:59:55,190-Speed 130.75 samples/sec Loss 3.2571 Epoch: 15 Global Step: 157580 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:00:14,974-Speed 129.40 samples/sec Loss 3.4166 Epoch: 15 Global Step: 157585 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:00:34,648-Speed 130.13 samples/sec Loss 3.3268 Epoch: 15 Global Step: 157590 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:00:54,113-Speed 131.52 samples/sec Loss 3.4788 Epoch: 15 Global Step: 157595 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:01:13,706-Speed 130.66 samples/sec Loss 3.2269 Epoch: 15 Global Step: 157600 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:01:33,241-Speed 131.06 samples/sec Loss 3.3201 Epoch: 15 Global Step: 157605 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:01:52,852-Speed 130.54 samples/sec Loss 3.6204 Epoch: 15 Global Step: 157610 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:02:12,904-Speed 127.68 samples/sec Loss 3.3944 Epoch: 15 Global Step: 157615 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:02:32,391-Speed 131.38 samples/sec Loss 3.3127 Epoch: 15 Global Step: 157620 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:02:51,967-Speed 130.78 samples/sec Loss 3.3255 Epoch: 15 Global Step: 157625 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:03:11,896-Speed 128.46 samples/sec Loss 3.4381 Epoch: 15 Global Step: 157630 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:03:31,444-Speed 130.97 samples/sec Loss 3.3145 Epoch: 15 Global Step: 157635 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:03:50,910-Speed 131.52 samples/sec Loss 3.4841 Epoch: 15 Global Step: 157640 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:04:10,521-Speed 130.54 samples/sec Loss 3.3654 Epoch: 15 Global Step: 157645 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:04:30,130-Speed 130.56 samples/sec Loss 3.2412 Epoch: 15 Global Step: 157650 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:04:49,670-Speed 131.02 samples/sec Loss 3.3443 Epoch: 15 Global Step: 157655 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:05:09,210-Speed 131.02 samples/sec Loss 3.4492 Epoch: 15 Global Step: 157660 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:05:28,714-Speed 131.27 samples/sec Loss 3.6489 Epoch: 15 Global Step: 157665 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:05:48,284-Speed 130.82 samples/sec Loss 3.2730 Epoch: 15 Global Step: 157670 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:06:08,193-Speed 128.59 samples/sec Loss 3.5612 Epoch: 15 Global Step: 157675 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:06:27,631-Speed 131.71 samples/sec Loss 3.5271 Epoch: 15 Global Step: 157680 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:06:47,244-Speed 130.53 samples/sec Loss 3.3464 Epoch: 15 Global Step: 157685 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:07:06,851-Speed 130.57 samples/sec Loss 3.5430 Epoch: 15 Global Step: 157690 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:07:26,480-Speed 130.43 samples/sec Loss 3.2890 Epoch: 15 Global Step: 157695 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:07:46,073-Speed 130.66 samples/sec Loss 3.4594 Epoch: 15 Global Step: 157700 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:08:05,539-Speed 131.52 samples/sec Loss 3.4573 Epoch: 15 Global Step: 157705 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:08:25,090-Speed 130.95 samples/sec Loss 3.2054 Epoch: 15 Global Step: 157710 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:08:44,688-Speed 130.64 samples/sec Loss 3.8224 Epoch: 15 Global Step: 157715 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:09:04,617-Speed 128.46 samples/sec Loss 3.4379 Epoch: 15 Global Step: 157720 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:09:24,050-Speed 131.74 samples/sec Loss 3.3504 Epoch: 15 Global Step: 157725 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:09:43,833-Speed 129.41 samples/sec Loss 3.2068 Epoch: 15 Global Step: 157730 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:10:03,455-Speed 130.48 samples/sec Loss 3.7279 Epoch: 15 Global Step: 157735 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:10:22,997-Speed 131.01 samples/sec Loss 3.4297 Epoch: 15 Global Step: 157740 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:10:42,653-Speed 130.25 samples/sec Loss 3.3125 Epoch: 15 Global Step: 157745 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:11:02,190-Speed 131.04 samples/sec Loss 3.4369 Epoch: 15 Global Step: 157750 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:11:21,811-Speed 130.48 samples/sec Loss 3.3376 Epoch: 15 Global Step: 157755 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:11:41,329-Speed 131.17 samples/sec Loss 3.5163 Epoch: 15 Global Step: 157760 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:12:01,136-Speed 129.25 samples/sec Loss 3.5141 Epoch: 15 Global Step: 157765 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:12:20,604-Speed 131.51 samples/sec Loss 3.2824 Epoch: 15 Global Step: 157770 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:12:40,316-Speed 129.88 samples/sec Loss 3.3861 Epoch: 15 Global Step: 157775 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:12:59,796-Speed 131.43 samples/sec Loss 3.5265 Epoch: 15 Global Step: 157780 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:13:19,316-Speed 131.15 samples/sec Loss 3.4512 Epoch: 15 Global Step: 157785 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:13:38,882-Speed 130.84 samples/sec Loss 3.3227 Epoch: 15 Global Step: 157790 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:13:58,406-Speed 131.13 samples/sec Loss 3.3679 Epoch: 15 Global Step: 157795 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:14:18,022-Speed 130.51 samples/sec Loss 3.4185 Epoch: 15 Global Step: 157800 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:14:37,546-Speed 131.13 samples/sec Loss 3.5029 Epoch: 15 Global Step: 157805 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:14:57,333-Speed 129.39 samples/sec Loss 3.2545 Epoch: 15 Global Step: 157810 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:15:16,903-Speed 130.82 samples/sec Loss 3.1810 Epoch: 15 Global Step: 157815 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:15:36,526-Speed 130.46 samples/sec Loss 3.3314 Epoch: 15 Global Step: 157820 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:15:56,057-Speed 131.08 samples/sec Loss 3.2259 Epoch: 15 Global Step: 157825 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:16:15,660-Speed 130.60 samples/sec Loss 3.3831 Epoch: 15 Global Step: 157830 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:16:35,195-Speed 131.05 samples/sec Loss 3.6085 Epoch: 15 Global Step: 157835 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:16:54,855-Speed 130.22 samples/sec Loss 3.3404 Epoch: 15 Global Step: 157840 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:17:14,479-Speed 130.46 samples/sec Loss 3.4112 Epoch: 15 Global Step: 157845 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:17:33,928-Speed 131.64 samples/sec Loss 3.5992 Epoch: 15 Global Step: 157850 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:17:53,742-Speed 129.21 samples/sec Loss 3.4950 Epoch: 15 Global Step: 157855 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:18:13,455-Speed 129.87 samples/sec Loss 3.3721 Epoch: 15 Global Step: 157860 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:18:33,061-Speed 130.58 samples/sec Loss 3.3674 Epoch: 15 Global Step: 157865 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:18:52,542-Speed 131.42 samples/sec Loss 3.4931 Epoch: 15 Global Step: 157870 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:19:12,102-Speed 130.88 samples/sec Loss 3.4784 Epoch: 15 Global Step: 157875 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:19:31,679-Speed 130.78 samples/sec Loss 3.4052 Epoch: 15 Global Step: 157880 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:19:51,266-Speed 130.70 samples/sec Loss 3.2252 Epoch: 15 Global Step: 157885 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:20:10,873-Speed 130.58 samples/sec Loss 3.3953 Epoch: 15 Global Step: 157890 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:20:30,334-Speed 131.55 samples/sec Loss 3.4565 Epoch: 15 Global Step: 157895 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:20:50,097-Speed 129.54 samples/sec Loss 3.2289 Epoch: 15 Global Step: 157900 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:21:09,814-Speed 129.85 samples/sec Loss 3.3203 Epoch: 15 Global Step: 157905 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:21:29,347-Speed 131.08 samples/sec Loss 3.5076 Epoch: 15 Global Step: 157910 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:21:48,968-Speed 130.48 samples/sec Loss 3.3895 Epoch: 15 Global Step: 157915 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:22:08,413-Speed 131.66 samples/sec Loss 3.6696 Epoch: 15 Global Step: 157920 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:22:27,938-Speed 131.12 samples/sec Loss 3.5334 Epoch: 15 Global Step: 157925 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:22:47,522-Speed 130.73 samples/sec Loss 3.3029 Epoch: 15 Global Step: 157930 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:23:07,040-Speed 131.17 samples/sec Loss 3.2951 Epoch: 15 Global Step: 157935 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:23:26,647-Speed 130.57 samples/sec Loss 3.6236 Epoch: 15 Global Step: 157940 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:23:46,466-Speed 129.18 samples/sec Loss 3.5129 Epoch: 15 Global Step: 157945 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:24:05,980-Speed 131.19 samples/sec Loss 3.2709 Epoch: 15 Global Step: 157950 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:24:25,531-Speed 130.95 samples/sec Loss 3.5209 Epoch: 15 Global Step: 157955 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:24:45,133-Speed 130.61 samples/sec Loss 3.6274 Epoch: 15 Global Step: 157960 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:25:04,653-Speed 131.15 samples/sec Loss 3.1499 Epoch: 15 Global Step: 157965 Fp16 Grad Scale: 8192 Required: 108 hours Training: 2021-06-14 13:25:24,263-Speed 130.55 samples/sec Loss 3.3121 Epoch: 15 Global Step: 157970 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:25:43,865-Speed 130.61 samples/sec Loss 3.4927 Epoch: 15 Global Step: 157975 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:26:03,396-Speed 131.08 samples/sec Loss 3.4544 Epoch: 15 Global Step: 157980 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:26:22,958-Speed 130.87 samples/sec Loss 3.3493 Epoch: 15 Global Step: 157985 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:26:42,892-Speed 128.43 samples/sec Loss 3.2392 Epoch: 15 Global Step: 157990 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:27:02,331-Speed 131.70 samples/sec Loss 3.5072 Epoch: 15 Global Step: 157995 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:27:22,129-Speed 129.32 samples/sec Loss 3.5154 Epoch: 15 Global Step: 158000 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:27:41,707-Speed 130.76 samples/sec Loss 3.3037 Epoch: 15 Global Step: 158005 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:28:01,247-Speed 131.02 samples/sec Loss 3.7908 Epoch: 15 Global Step: 158010 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:28:20,862-Speed 130.53 samples/sec Loss 3.2710 Epoch: 15 Global Step: 158015 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:28:40,366-Speed 131.26 samples/sec Loss 3.4264 Epoch: 15 Global Step: 158020 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:28:59,920-Speed 130.92 samples/sec Loss 3.3941 Epoch: 15 Global Step: 158025 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:29:19,482-Speed 130.87 samples/sec Loss 3.3656 Epoch: 15 Global Step: 158030 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:29:39,400-Speed 128.54 samples/sec Loss 3.5384 Epoch: 15 Global Step: 158035 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:29:58,422-Speed 134.58 samples/sec Loss 3.4283 Epoch: 15 Global Step: 158040 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:30:18,051-Speed 130.43 samples/sec Loss 3.4146 Epoch: 15 Global Step: 158045 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:30:37,598-Speed 130.98 samples/sec Loss 3.6440 Epoch: 15 Global Step: 158050 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:30:57,147-Speed 130.96 samples/sec Loss 3.5860 Epoch: 15 Global Step: 158055 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:31:16,799-Speed 130.27 samples/sec Loss 3.3983 Epoch: 15 Global Step: 158060 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:31:36,329-Speed 131.09 samples/sec Loss 3.4919 Epoch: 15 Global Step: 158065 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:31:55,860-Speed 131.08 samples/sec Loss 3.3610 Epoch: 15 Global Step: 158070 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:32:15,431-Speed 130.81 samples/sec Loss 3.4062 Epoch: 15 Global Step: 158075 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:32:35,351-Speed 128.52 samples/sec Loss 3.2889 Epoch: 15 Global Step: 158080 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:32:54,777-Speed 131.79 samples/sec Loss 3.3967 Epoch: 15 Global Step: 158085 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:33:14,327-Speed 130.96 samples/sec Loss 3.5448 Epoch: 15 Global Step: 158090 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:33:33,885-Speed 130.90 samples/sec Loss 3.2095 Epoch: 15 Global Step: 158095 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:33:53,403-Speed 131.17 samples/sec Loss 3.2609 Epoch: 15 Global Step: 158100 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:34:13,031-Speed 130.43 samples/sec Loss 3.2792 Epoch: 15 Global Step: 158105 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:34:32,456-Speed 131.80 samples/sec Loss 3.1667 Epoch: 15 Global Step: 158110 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:34:51,992-Speed 131.05 samples/sec Loss 3.4519 Epoch: 15 Global Step: 158115 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:35:11,563-Speed 130.82 samples/sec Loss 3.4394 Epoch: 15 Global Step: 158120 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:35:31,538-Speed 128.16 samples/sec Loss 3.3469 Epoch: 15 Global Step: 158125 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:35:50,963-Speed 131.80 samples/sec Loss 3.2591 Epoch: 15 Global Step: 158130 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:36:10,604-Speed 130.34 samples/sec Loss 3.3155 Epoch: 15 Global Step: 158135 Fp16 Grad Scale: 4096 Required: 107 hours Training: 2021-06-14 13:36:30,185-Speed 130.74 samples/sec Loss 3.6618 Epoch: 15 Global Step: 158140 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:36:49,809-Speed 130.46 samples/sec Loss 3.2176 Epoch: 15 Global Step: 158145 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:37:09,401-Speed 130.67 samples/sec Loss 3.2994 Epoch: 15 Global Step: 158150 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:37:28,901-Speed 131.29 samples/sec Loss 3.2173 Epoch: 15 Global Step: 158155 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:37:48,491-Speed 130.69 samples/sec Loss 3.3437 Epoch: 15 Global Step: 158160 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:38:08,037-Speed 130.98 samples/sec Loss 3.5300 Epoch: 15 Global Step: 158165 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:38:28,016-Speed 128.14 samples/sec Loss 3.2601 Epoch: 15 Global Step: 158170 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:38:47,567-Speed 130.95 samples/sec Loss 3.2826 Epoch: 15 Global Step: 158175 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:39:07,097-Speed 131.09 samples/sec Loss 3.5533 Epoch: 15 Global Step: 158180 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:39:26,664-Speed 130.84 samples/sec Loss 3.4204 Epoch: 15 Global Step: 158185 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:39:46,290-Speed 130.44 samples/sec Loss 3.4407 Epoch: 15 Global Step: 158190 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:40:05,883-Speed 130.67 samples/sec Loss 3.2131 Epoch: 15 Global Step: 158195 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:40:25,402-Speed 131.16 samples/sec Loss 3.4070 Epoch: 15 Global Step: 158200 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:40:44,977-Speed 130.79 samples/sec Loss 3.5916 Epoch: 15 Global Step: 158205 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:41:04,557-Speed 130.75 samples/sec Loss 3.1939 Epoch: 15 Global Step: 158210 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:41:24,471-Speed 128.56 samples/sec Loss 3.2837 Epoch: 15 Global Step: 158215 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:41:43,916-Speed 131.66 samples/sec Loss 3.2886 Epoch: 15 Global Step: 158220 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:42:03,547-Speed 130.41 samples/sec Loss 3.2456 Epoch: 15 Global Step: 158225 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:42:23,159-Speed 130.54 samples/sec Loss 3.6413 Epoch: 15 Global Step: 158230 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:42:42,604-Speed 131.66 samples/sec Loss 3.4804 Epoch: 15 Global Step: 158235 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:43:02,204-Speed 130.62 samples/sec Loss 3.4777 Epoch: 15 Global Step: 158240 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:43:21,742-Speed 131.03 samples/sec Loss 3.4479 Epoch: 15 Global Step: 158245 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:43:41,364-Speed 130.47 samples/sec Loss 3.2956 Epoch: 15 Global Step: 158250 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:44:00,940-Speed 130.78 samples/sec Loss 3.1032 Epoch: 15 Global Step: 158255 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:44:20,839-Speed 128.66 samples/sec Loss 3.3264 Epoch: 15 Global Step: 158260 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:44:40,263-Speed 131.80 samples/sec Loss 3.3420 Epoch: 15 Global Step: 158265 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:45:00,084-Speed 129.16 samples/sec Loss 3.3733 Epoch: 15 Global Step: 158270 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:45:19,613-Speed 131.10 samples/sec Loss 3.6009 Epoch: 15 Global Step: 158275 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:45:39,103-Speed 131.36 samples/sec Loss 3.4565 Epoch: 15 Global Step: 158280 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:45:58,668-Speed 130.85 samples/sec Loss 3.4280 Epoch: 15 Global Step: 158285 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:46:18,212-Speed 130.99 samples/sec Loss 3.6954 Epoch: 15 Global Step: 158290 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:46:37,815-Speed 130.60 samples/sec Loss 3.3600 Epoch: 15 Global Step: 158295 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:46:57,316-Speed 131.29 samples/sec Loss 3.3200 Epoch: 15 Global Step: 158300 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:47:17,125-Speed 129.24 samples/sec Loss 3.3895 Epoch: 15 Global Step: 158305 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:47:36,619-Speed 131.33 samples/sec Loss 3.4306 Epoch: 15 Global Step: 158310 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:47:56,289-Speed 130.16 samples/sec Loss 3.3999 Epoch: 15 Global Step: 158315 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:48:15,728-Speed 131.70 samples/sec Loss 3.4724 Epoch: 15 Global Step: 158320 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:48:35,351-Speed 130.47 samples/sec Loss 3.3094 Epoch: 15 Global Step: 158325 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:48:54,874-Speed 131.14 samples/sec Loss 3.2526 Epoch: 15 Global Step: 158330 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:49:14,443-Speed 130.83 samples/sec Loss 3.3030 Epoch: 15 Global Step: 158335 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:49:34,056-Speed 130.53 samples/sec Loss 3.3433 Epoch: 15 Global Step: 158340 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:49:53,493-Speed 131.71 samples/sec Loss 3.4640 Epoch: 15 Global Step: 158345 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:50:13,299-Speed 129.26 samples/sec Loss 3.3019 Epoch: 15 Global Step: 158350 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:50:32,967-Speed 130.17 samples/sec Loss 3.5189 Epoch: 15 Global Step: 158355 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:50:52,578-Speed 130.55 samples/sec Loss 3.2775 Epoch: 15 Global Step: 158360 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:51:12,285-Speed 129.91 samples/sec Loss 3.2567 Epoch: 15 Global Step: 158365 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:51:31,854-Speed 130.82 samples/sec Loss 3.6258 Epoch: 15 Global Step: 158370 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:51:51,418-Speed 130.86 samples/sec Loss 3.3433 Epoch: 15 Global Step: 158375 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:52:11,024-Speed 130.58 samples/sec Loss 3.2621 Epoch: 15 Global Step: 158380 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:52:30,642-Speed 130.50 samples/sec Loss 3.4859 Epoch: 15 Global Step: 158385 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:52:50,399-Speed 129.58 samples/sec Loss 3.3644 Epoch: 15 Global Step: 158390 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:53:10,147-Speed 129.64 samples/sec Loss 3.3066 Epoch: 15 Global Step: 158395 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:53:29,851-Speed 129.93 samples/sec Loss 3.4782 Epoch: 15 Global Step: 158400 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:53:49,279-Speed 131.77 samples/sec Loss 3.5598 Epoch: 15 Global Step: 158405 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:54:08,901-Speed 130.47 samples/sec Loss 3.3855 Epoch: 15 Global Step: 158410 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:54:28,434-Speed 131.06 samples/sec Loss 3.4125 Epoch: 15 Global Step: 158415 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:54:47,982-Speed 130.97 samples/sec Loss 3.3492 Epoch: 15 Global Step: 158420 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:55:08,124-Speed 127.11 samples/sec Loss 3.6547 Epoch: 15 Global Step: 158425 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:55:27,678-Speed 130.92 samples/sec Loss 3.4611 Epoch: 15 Global Step: 158430 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:55:47,284-Speed 130.59 samples/sec Loss 3.2403 Epoch: 15 Global Step: 158435 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:56:07,044-Speed 129.56 samples/sec Loss 3.2327 Epoch: 15 Global Step: 158440 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:56:26,702-Speed 130.23 samples/sec Loss 3.2428 Epoch: 15 Global Step: 158445 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:56:46,276-Speed 130.79 samples/sec Loss 3.3843 Epoch: 15 Global Step: 158450 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:57:06,068-Speed 129.35 samples/sec Loss 3.2454 Epoch: 15 Global Step: 158455 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:57:25,568-Speed 131.29 samples/sec Loss 3.3391 Epoch: 15 Global Step: 158460 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:57:45,166-Speed 130.63 samples/sec Loss 3.5025 Epoch: 15 Global Step: 158465 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:58:04,714-Speed 130.97 samples/sec Loss 3.4638 Epoch: 15 Global Step: 158470 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:58:24,328-Speed 130.53 samples/sec Loss 3.2882 Epoch: 15 Global Step: 158475 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:58:43,893-Speed 130.85 samples/sec Loss 3.6097 Epoch: 15 Global Step: 158480 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:59:03,798-Speed 128.61 samples/sec Loss 3.6543 Epoch: 15 Global Step: 158485 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:59:23,231-Speed 131.74 samples/sec Loss 3.3909 Epoch: 15 Global Step: 158490 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 13:59:42,888-Speed 130.24 samples/sec Loss 3.4121 Epoch: 15 Global Step: 158495 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:00:02,452-Speed 130.86 samples/sec Loss 3.3347 Epoch: 15 Global Step: 158500 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:00:22,012-Speed 130.88 samples/sec Loss 3.3649 Epoch: 15 Global Step: 158505 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:00:41,618-Speed 130.58 samples/sec Loss 3.3915 Epoch: 15 Global Step: 158510 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:01:01,074-Speed 131.58 samples/sec Loss 3.3113 Epoch: 15 Global Step: 158515 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:01:20,627-Speed 130.94 samples/sec Loss 3.4606 Epoch: 15 Global Step: 158520 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:01:40,129-Speed 131.28 samples/sec Loss 3.6535 Epoch: 15 Global Step: 158525 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:02:00,032-Speed 128.63 samples/sec Loss 3.3604 Epoch: 15 Global Step: 158530 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:02:19,473-Speed 131.68 samples/sec Loss 3.3522 Epoch: 15 Global Step: 158535 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:02:39,085-Speed 130.54 samples/sec Loss 3.3931 Epoch: 15 Global Step: 158540 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:02:58,638-Speed 130.93 samples/sec Loss 3.5903 Epoch: 15 Global Step: 158545 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:03:18,086-Speed 131.64 samples/sec Loss 3.1583 Epoch: 15 Global Step: 158550 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:03:37,717-Speed 130.41 samples/sec Loss 3.3162 Epoch: 15 Global Step: 158555 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:03:57,137-Speed 131.83 samples/sec Loss 3.6729 Epoch: 15 Global Step: 158560 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:04:16,666-Speed 131.09 samples/sec Loss 3.3025 Epoch: 15 Global Step: 158565 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:04:36,220-Speed 130.94 samples/sec Loss 3.1575 Epoch: 15 Global Step: 158570 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:04:56,113-Speed 128.70 samples/sec Loss 3.4184 Epoch: 15 Global Step: 158575 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:05:15,530-Speed 131.85 samples/sec Loss 3.3966 Epoch: 15 Global Step: 158580 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:05:35,139-Speed 130.56 samples/sec Loss 3.4306 Epoch: 15 Global Step: 158585 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:05:54,694-Speed 130.92 samples/sec Loss 3.3541 Epoch: 15 Global Step: 158590 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:06:14,313-Speed 130.49 samples/sec Loss 3.2922 Epoch: 15 Global Step: 158595 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:06:33,915-Speed 130.61 samples/sec Loss 3.3916 Epoch: 15 Global Step: 158600 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:06:53,451-Speed 131.05 samples/sec Loss 3.6365 Epoch: 15 Global Step: 158605 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:07:12,972-Speed 131.15 samples/sec Loss 3.3506 Epoch: 15 Global Step: 158610 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:07:32,531-Speed 130.89 samples/sec Loss 3.4540 Epoch: 15 Global Step: 158615 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:07:52,517-Speed 128.09 samples/sec Loss 3.4980 Epoch: 15 Global Step: 158620 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:08:12,228-Speed 129.88 samples/sec Loss 3.3590 Epoch: 15 Global Step: 158625 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:08:31,798-Speed 130.82 samples/sec Loss 3.3329 Epoch: 15 Global Step: 158630 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:08:51,378-Speed 130.76 samples/sec Loss 3.7003 Epoch: 15 Global Step: 158635 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:09:11,000-Speed 130.47 samples/sec Loss 3.5596 Epoch: 15 Global Step: 158640 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:09:30,585-Speed 130.72 samples/sec Loss 3.6237 Epoch: 15 Global Step: 158645 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:09:50,058-Speed 131.47 samples/sec Loss 3.4351 Epoch: 15 Global Step: 158650 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:10:09,655-Speed 130.64 samples/sec Loss 3.4034 Epoch: 15 Global Step: 158655 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:10:29,190-Speed 131.05 samples/sec Loss 3.5711 Epoch: 15 Global Step: 158660 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:10:49,094-Speed 128.63 samples/sec Loss 3.4829 Epoch: 15 Global Step: 158665 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:11:08,512-Speed 131.84 samples/sec Loss 3.3202 Epoch: 15 Global Step: 158670 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:11:28,151-Speed 130.36 samples/sec Loss 3.2770 Epoch: 15 Global Step: 158675 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:11:47,680-Speed 131.09 samples/sec Loss 3.6129 Epoch: 15 Global Step: 158680 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:12:07,291-Speed 130.55 samples/sec Loss 3.6665 Epoch: 15 Global Step: 158685 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:12:26,826-Speed 131.05 samples/sec Loss 3.4057 Epoch: 15 Global Step: 158690 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:12:46,296-Speed 131.49 samples/sec Loss 3.2617 Epoch: 15 Global Step: 158695 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:13:05,860-Speed 130.86 samples/sec Loss 3.2669 Epoch: 15 Global Step: 158700 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:13:25,395-Speed 131.06 samples/sec Loss 3.3894 Epoch: 15 Global Step: 158705 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:13:45,186-Speed 129.36 samples/sec Loss 3.3162 Epoch: 15 Global Step: 158710 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:14:04,907-Speed 129.82 samples/sec Loss 3.3820 Epoch: 15 Global Step: 158715 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:14:24,450-Speed 131.00 samples/sec Loss 3.3748 Epoch: 15 Global Step: 158720 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:14:44,053-Speed 130.60 samples/sec Loss 3.5150 Epoch: 15 Global Step: 158725 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:15:03,622-Speed 130.82 samples/sec Loss 3.5095 Epoch: 15 Global Step: 158730 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:15:23,288-Speed 130.18 samples/sec Loss 3.5226 Epoch: 15 Global Step: 158735 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:15:42,902-Speed 130.52 samples/sec Loss 3.4107 Epoch: 15 Global Step: 158740 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:16:02,335-Speed 131.75 samples/sec Loss 3.2333 Epoch: 15 Global Step: 158745 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:16:21,949-Speed 130.52 samples/sec Loss 3.3983 Epoch: 15 Global Step: 158750 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:16:41,726-Speed 129.45 samples/sec Loss 3.3731 Epoch: 15 Global Step: 158755 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:17:01,392-Speed 130.18 samples/sec Loss 3.4800 Epoch: 15 Global Step: 158760 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:17:20,991-Speed 130.62 samples/sec Loss 3.3658 Epoch: 15 Global Step: 158765 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:17:40,607-Speed 130.51 samples/sec Loss 3.4340 Epoch: 15 Global Step: 158770 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:18:00,165-Speed 130.91 samples/sec Loss 3.4764 Epoch: 15 Global Step: 158775 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:18:19,697-Speed 131.08 samples/sec Loss 3.3750 Epoch: 15 Global Step: 158780 Fp16 Grad Scale: 8192 Required: 107 hours Training: 2021-06-14 14:18:39,498-Speed 129.29 samples/sec Loss 3.3668 Epoch: 15 Global Step: 158785 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:18:58,961-Speed 131.54 samples/sec Loss 3.4073 Epoch: 15 Global Step: 158790 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:19:18,507-Speed 130.98 samples/sec Loss 3.5297 Epoch: 15 Global Step: 158795 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:19:38,399-Speed 128.70 samples/sec Loss 3.5321 Epoch: 15 Global Step: 158800 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:19:58,114-Speed 129.86 samples/sec Loss 3.3301 Epoch: 15 Global Step: 158805 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:20:17,624-Speed 131.22 samples/sec Loss 3.3794 Epoch: 15 Global Step: 158810 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:20:37,165-Speed 131.02 samples/sec Loss 3.6850 Epoch: 15 Global Step: 158815 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:20:56,862-Speed 129.97 samples/sec Loss 3.6688 Epoch: 15 Global Step: 158820 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:21:16,402-Speed 131.02 samples/sec Loss 3.3644 Epoch: 15 Global Step: 158825 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:21:36,008-Speed 130.58 samples/sec Loss 3.2978 Epoch: 15 Global Step: 158830 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:21:55,617-Speed 130.56 samples/sec Loss 3.4580 Epoch: 15 Global Step: 158835 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:22:15,145-Speed 131.10 samples/sec Loss 3.5116 Epoch: 15 Global Step: 158840 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:22:34,877-Speed 129.74 samples/sec Loss 3.4512 Epoch: 15 Global Step: 158845 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:22:54,600-Speed 129.81 samples/sec Loss 3.3951 Epoch: 15 Global Step: 158850 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:23:14,191-Speed 130.68 samples/sec Loss 3.0814 Epoch: 15 Global Step: 158855 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:23:33,804-Speed 130.53 samples/sec Loss 3.3358 Epoch: 15 Global Step: 158860 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:23:53,335-Speed 131.08 samples/sec Loss 3.4244 Epoch: 15 Global Step: 158865 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:24:12,917-Speed 130.74 samples/sec Loss 3.4487 Epoch: 15 Global Step: 158870 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:24:32,521-Speed 130.60 samples/sec Loss 3.5152 Epoch: 15 Global Step: 158875 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:24:52,134-Speed 130.53 samples/sec Loss 3.3908 Epoch: 15 Global Step: 158880 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:25:11,719-Speed 130.72 samples/sec Loss 3.4170 Epoch: 15 Global Step: 158885 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:25:31,870-Speed 127.04 samples/sec Loss 3.4606 Epoch: 15 Global Step: 158890 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:25:51,511-Speed 130.35 samples/sec Loss 3.3782 Epoch: 15 Global Step: 158895 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:26:11,121-Speed 130.56 samples/sec Loss 3.3009 Epoch: 15 Global Step: 158900 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:26:30,728-Speed 130.57 samples/sec Loss 3.4579 Epoch: 15 Global Step: 158905 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:26:50,275-Speed 130.98 samples/sec Loss 3.5275 Epoch: 15 Global Step: 158910 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:27:09,873-Speed 130.63 samples/sec Loss 3.3671 Epoch: 15 Global Step: 158915 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:27:29,448-Speed 130.79 samples/sec Loss 3.4757 Epoch: 15 Global Step: 158920 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:27:49,038-Speed 130.69 samples/sec Loss 3.3208 Epoch: 15 Global Step: 158925 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:28:08,646-Speed 130.57 samples/sec Loss 3.4302 Epoch: 15 Global Step: 158930 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:28:28,541-Speed 128.68 samples/sec Loss 3.2326 Epoch: 15 Global Step: 158935 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:28:48,111-Speed 130.82 samples/sec Loss 3.3842 Epoch: 15 Global Step: 158940 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:29:07,649-Speed 131.04 samples/sec Loss 3.5779 Epoch: 15 Global Step: 158945 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:29:27,252-Speed 130.60 samples/sec Loss 3.5758 Epoch: 15 Global Step: 158950 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:29:46,796-Speed 131.00 samples/sec Loss 3.3348 Epoch: 15 Global Step: 158955 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:30:06,406-Speed 130.55 samples/sec Loss 3.5680 Epoch: 15 Global Step: 158960 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:30:25,939-Speed 131.07 samples/sec Loss 3.5139 Epoch: 15 Global Step: 158965 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:30:45,481-Speed 131.01 samples/sec Loss 3.6488 Epoch: 15 Global Step: 158970 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:31:05,085-Speed 130.59 samples/sec Loss 3.2264 Epoch: 15 Global Step: 158975 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:31:24,933-Speed 128.99 samples/sec Loss 3.3319 Epoch: 15 Global Step: 158980 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:31:44,368-Speed 131.72 samples/sec Loss 3.3491 Epoch: 15 Global Step: 158985 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:32:03,689-Speed 132.51 samples/sec Loss 3.3482 Epoch: 15 Global Step: 158990 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:32:23,253-Speed 130.86 samples/sec Loss 3.3162 Epoch: 15 Global Step: 158995 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:32:42,692-Speed 131.70 samples/sec Loss 3.5800 Epoch: 15 Global Step: 159000 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:33:02,329-Speed 130.38 samples/sec Loss 3.5243 Epoch: 15 Global Step: 159005 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:33:21,957-Speed 130.43 samples/sec Loss 3.5240 Epoch: 15 Global Step: 159010 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:33:41,574-Speed 130.51 samples/sec Loss 3.3666 Epoch: 15 Global Step: 159015 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:34:01,120-Speed 130.98 samples/sec Loss 3.4381 Epoch: 15 Global Step: 159020 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:34:20,973-Speed 128.95 samples/sec Loss 3.2947 Epoch: 15 Global Step: 159025 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:34:40,409-Speed 131.72 samples/sec Loss 3.3499 Epoch: 15 Global Step: 159030 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:35:00,190-Speed 129.43 samples/sec Loss 3.3249 Epoch: 15 Global Step: 159035 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:35:19,737-Speed 130.97 samples/sec Loss 3.2849 Epoch: 15 Global Step: 159040 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:35:39,279-Speed 131.01 samples/sec Loss 3.4264 Epoch: 15 Global Step: 159045 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:35:58,836-Speed 130.91 samples/sec Loss 3.5431 Epoch: 15 Global Step: 159050 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:36:18,375-Speed 131.03 samples/sec Loss 3.4579 Epoch: 15 Global Step: 159055 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:36:37,973-Speed 130.63 samples/sec Loss 3.5057 Epoch: 15 Global Step: 159060 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:36:57,518-Speed 130.99 samples/sec Loss 3.4252 Epoch: 15 Global Step: 159065 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:37:17,520-Speed 127.99 samples/sec Loss 3.4417 Epoch: 15 Global Step: 159070 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:37:37,304-Speed 129.41 samples/sec Loss 3.5038 Epoch: 15 Global Step: 159075 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:37:56,761-Speed 131.57 samples/sec Loss 3.3781 Epoch: 15 Global Step: 159080 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:38:16,317-Speed 130.91 samples/sec Loss 3.3611 Epoch: 15 Global Step: 159085 Fp16 Grad Scale: 4096 Required: 106 hours Training: 2021-06-14 14:38:35,890-Speed 130.80 samples/sec Loss 3.3593 Epoch: 15 Global Step: 159090 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:38:55,513-Speed 130.47 samples/sec Loss 3.4794 Epoch: 15 Global Step: 159095 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:39:15,113-Speed 130.62 samples/sec Loss 3.4694 Epoch: 15 Global Step: 159100 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:39:34,557-Speed 131.66 samples/sec Loss 3.3895 Epoch: 15 Global Step: 159105 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:39:54,088-Speed 131.08 samples/sec Loss 3.1531 Epoch: 15 Global Step: 159110 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:40:13,883-Speed 129.33 samples/sec Loss 3.3842 Epoch: 15 Global Step: 159115 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:40:33,688-Speed 129.26 samples/sec Loss 3.4349 Epoch: 15 Global Step: 159120 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:40:53,213-Speed 131.12 samples/sec Loss 3.2376 Epoch: 15 Global Step: 159125 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:41:12,817-Speed 130.59 samples/sec Loss 3.4589 Epoch: 15 Global Step: 159130 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:41:32,351-Speed 131.06 samples/sec Loss 3.5750 Epoch: 15 Global Step: 159135 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:41:51,928-Speed 130.78 samples/sec Loss 3.4873 Epoch: 15 Global Step: 159140 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:42:11,545-Speed 130.50 samples/sec Loss 3.5929 Epoch: 15 Global Step: 159145 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:42:31,114-Speed 130.83 samples/sec Loss 3.5195 Epoch: 15 Global Step: 159150 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:42:50,709-Speed 130.65 samples/sec Loss 3.2705 Epoch: 15 Global Step: 159155 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:43:10,539-Speed 129.10 samples/sec Loss 3.3416 Epoch: 15 Global Step: 159160 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:43:30,170-Speed 130.41 samples/sec Loss 3.5253 Epoch: 15 Global Step: 159165 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:43:49,704-Speed 131.06 samples/sec Loss 3.1993 Epoch: 15 Global Step: 159170 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:44:09,302-Speed 130.64 samples/sec Loss 3.4207 Epoch: 15 Global Step: 159175 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:44:28,896-Speed 130.66 samples/sec Loss 3.4009 Epoch: 15 Global Step: 159180 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:44:48,404-Speed 131.23 samples/sec Loss 3.5781 Epoch: 15 Global Step: 159185 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:45:08,084-Speed 130.09 samples/sec Loss 3.4754 Epoch: 15 Global Step: 159190 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:45:27,626-Speed 131.00 samples/sec Loss 3.4763 Epoch: 15 Global Step: 159195 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:45:47,232-Speed 130.58 samples/sec Loss 3.3023 Epoch: 15 Global Step: 159200 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:46:07,124-Speed 128.70 samples/sec Loss 3.3030 Epoch: 15 Global Step: 159205 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:46:26,624-Speed 131.29 samples/sec Loss 3.4671 Epoch: 15 Global Step: 159210 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:46:46,128-Speed 131.26 samples/sec Loss 3.6938 Epoch: 15 Global Step: 159215 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:47:05,727-Speed 130.62 samples/sec Loss 3.3428 Epoch: 15 Global Step: 159220 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:47:25,319-Speed 130.68 samples/sec Loss 3.4460 Epoch: 15 Global Step: 159225 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:47:44,935-Speed 130.51 samples/sec Loss 3.3261 Epoch: 15 Global Step: 159230 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:48:04,486-Speed 130.94 samples/sec Loss 3.5397 Epoch: 15 Global Step: 159235 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:48:24,006-Speed 131.16 samples/sec Loss 3.4456 Epoch: 15 Global Step: 159240 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:48:43,563-Speed 130.91 samples/sec Loss 3.4930 Epoch: 15 Global Step: 159245 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:49:03,310-Speed 129.65 samples/sec Loss 3.3560 Epoch: 15 Global Step: 159250 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:49:22,968-Speed 130.24 samples/sec Loss 3.1548 Epoch: 15 Global Step: 159255 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:49:42,415-Speed 131.64 samples/sec Loss 3.6040 Epoch: 15 Global Step: 159260 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:50:02,030-Speed 130.52 samples/sec Loss 3.2470 Epoch: 15 Global Step: 159265 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:50:21,554-Speed 131.13 samples/sec Loss 3.3482 Epoch: 15 Global Step: 159270 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:50:41,174-Speed 130.49 samples/sec Loss 3.3647 Epoch: 15 Global Step: 159275 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:51:00,705-Speed 131.08 samples/sec Loss 3.5237 Epoch: 15 Global Step: 159280 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:51:20,226-Speed 131.15 samples/sec Loss 3.5222 Epoch: 15 Global Step: 159285 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:51:39,829-Speed 130.60 samples/sec Loss 3.2779 Epoch: 15 Global Step: 159290 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:51:59,769-Speed 128.39 samples/sec Loss 3.3397 Epoch: 15 Global Step: 159295 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:52:19,291-Speed 131.14 samples/sec Loss 3.3973 Epoch: 15 Global Step: 159300 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:52:38,939-Speed 130.30 samples/sec Loss 3.3664 Epoch: 15 Global Step: 159305 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:52:58,529-Speed 130.69 samples/sec Loss 3.3178 Epoch: 15 Global Step: 159310 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:53:18,064-Speed 131.05 samples/sec Loss 3.1902 Epoch: 15 Global Step: 159315 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:53:37,667-Speed 130.60 samples/sec Loss 3.2221 Epoch: 15 Global Step: 159320 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:53:57,224-Speed 130.91 samples/sec Loss 3.1953 Epoch: 15 Global Step: 159325 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:54:16,747-Speed 131.13 samples/sec Loss 3.2246 Epoch: 15 Global Step: 159330 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:54:36,305-Speed 130.90 samples/sec Loss 3.5104 Epoch: 15 Global Step: 159335 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:54:56,282-Speed 128.16 samples/sec Loss 3.5228 Epoch: 15 Global Step: 159340 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:55:15,733-Speed 131.62 samples/sec Loss 3.4608 Epoch: 15 Global Step: 159345 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:55:35,531-Speed 129.31 samples/sec Loss 3.4333 Epoch: 15 Global Step: 159350 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:55:55,100-Speed 130.83 samples/sec Loss 3.4644 Epoch: 15 Global Step: 159355 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:56:14,636-Speed 131.05 samples/sec Loss 3.4944 Epoch: 15 Global Step: 159360 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:56:34,212-Speed 130.78 samples/sec Loss 3.4097 Epoch: 15 Global Step: 159365 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:56:53,722-Speed 131.22 samples/sec Loss 3.3894 Epoch: 15 Global Step: 159370 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:57:13,330-Speed 130.57 samples/sec Loss 3.4376 Epoch: 15 Global Step: 159375 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:57:32,878-Speed 130.96 samples/sec Loss 3.5200 Epoch: 15 Global Step: 159380 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:57:52,688-Speed 129.24 samples/sec Loss 3.2965 Epoch: 15 Global Step: 159385 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:58:12,110-Speed 131.81 samples/sec Loss 3.2982 Epoch: 15 Global Step: 159390 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:58:32,041-Speed 128.45 samples/sec Loss 3.4560 Epoch: 15 Global Step: 159395 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:58:51,592-Speed 130.94 samples/sec Loss 3.4008 Epoch: 15 Global Step: 159400 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:59:11,128-Speed 131.05 samples/sec Loss 3.4455 Epoch: 15 Global Step: 159405 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:59:30,707-Speed 130.76 samples/sec Loss 3.4459 Epoch: 15 Global Step: 159410 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 14:59:50,266-Speed 130.90 samples/sec Loss 3.5777 Epoch: 15 Global Step: 159415 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:00:09,865-Speed 130.62 samples/sec Loss 3.3882 Epoch: 15 Global Step: 159420 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:01:00,666-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-14 15:01:11,548-Speed 41.50 samples/sec Loss 3.0393 Epoch: 16 Global Step: 159425 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:01:31,652-Speed 127.35 samples/sec Loss 2.6810 Epoch: 16 Global Step: 159430 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:01:51,550-Speed 128.66 samples/sec Loss 2.4562 Epoch: 16 Global Step: 159435 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:02:11,982-Speed 125.30 samples/sec Loss 2.5164 Epoch: 16 Global Step: 159440 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:02:31,926-Speed 128.36 samples/sec Loss 2.5460 Epoch: 16 Global Step: 159445 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:02:51,750-Speed 129.15 samples/sec Loss 2.5513 Epoch: 16 Global Step: 159450 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:03:11,615-Speed 128.88 samples/sec Loss 2.6045 Epoch: 16 Global Step: 159455 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:03:31,395-Speed 129.43 samples/sec Loss 2.5604 Epoch: 16 Global Step: 159460 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:03:51,150-Speed 129.59 samples/sec Loss 2.2903 Epoch: 16 Global Step: 159465 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:04:11,052-Speed 128.64 samples/sec Loss 2.5979 Epoch: 16 Global Step: 159470 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:04:30,743-Speed 130.02 samples/sec Loss 2.4602 Epoch: 16 Global Step: 159475 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:04:50,540-Speed 129.32 samples/sec Loss 2.4760 Epoch: 16 Global Step: 159480 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:05:10,929-Speed 125.56 samples/sec Loss 2.3903 Epoch: 16 Global Step: 159485 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:05:30,870-Speed 128.38 samples/sec Loss 2.4117 Epoch: 16 Global Step: 159490 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:05:50,510-Speed 130.35 samples/sec Loss 2.3206 Epoch: 16 Global Step: 159495 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:06:10,355-Speed 129.00 samples/sec Loss 2.3000 Epoch: 16 Global Step: 159500 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:06:29,999-Speed 130.33 samples/sec Loss 2.1488 Epoch: 16 Global Step: 159505 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:06:49,775-Speed 129.46 samples/sec Loss 2.3057 Epoch: 16 Global Step: 159510 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:07:09,643-Speed 128.86 samples/sec Loss 2.1251 Epoch: 16 Global Step: 159515 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:07:29,369-Speed 129.78 samples/sec Loss 2.3277 Epoch: 16 Global Step: 159520 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:07:49,266-Speed 128.67 samples/sec Loss 2.2749 Epoch: 16 Global Step: 159525 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:08:09,541-Speed 126.27 samples/sec Loss 2.1507 Epoch: 16 Global Step: 159530 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:08:29,287-Speed 129.65 samples/sec Loss 2.4469 Epoch: 16 Global Step: 159535 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:08:49,094-Speed 129.25 samples/sec Loss 2.1292 Epoch: 16 Global Step: 159540 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:09:08,921-Speed 129.13 samples/sec Loss 2.2246 Epoch: 16 Global Step: 159545 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:09:28,657-Speed 129.72 samples/sec Loss 2.1797 Epoch: 16 Global Step: 159550 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:09:48,574-Speed 128.54 samples/sec Loss 2.2849 Epoch: 16 Global Step: 159555 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:10:08,312-Speed 129.71 samples/sec Loss 2.1315 Epoch: 16 Global Step: 159560 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:10:28,223-Speed 128.58 samples/sec Loss 2.1380 Epoch: 16 Global Step: 159565 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:10:47,858-Speed 130.39 samples/sec Loss 2.2046 Epoch: 16 Global Step: 159570 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:11:08,299-Speed 125.24 samples/sec Loss 2.0808 Epoch: 16 Global Step: 159575 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:11:28,104-Speed 129.27 samples/sec Loss 2.0682 Epoch: 16 Global Step: 159580 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:11:47,972-Speed 128.85 samples/sec Loss 2.1930 Epoch: 16 Global Step: 159585 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:12:07,661-Speed 130.03 samples/sec Loss 2.2298 Epoch: 16 Global Step: 159590 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:12:27,639-Speed 128.15 samples/sec Loss 2.2923 Epoch: 16 Global Step: 159595 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:12:47,370-Speed 129.76 samples/sec Loss 2.3281 Epoch: 16 Global Step: 159600 Fp16 Grad Scale: 8192 Required: 106 hours Training: 2021-06-14 15:13:07,275-Speed 128.62 samples/sec Loss 2.0387 Epoch: 16 Global Step: 159605 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:13:26,912-Speed 130.37 samples/sec Loss 2.2164 Epoch: 16 Global Step: 159610 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:13:46,845-Speed 128.44 samples/sec Loss 2.1961 Epoch: 16 Global Step: 159615 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:14:07,268-Speed 125.36 samples/sec Loss 2.1945 Epoch: 16 Global Step: 159620 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:14:27,166-Speed 128.66 samples/sec Loss 2.1584 Epoch: 16 Global Step: 159625 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:14:46,770-Speed 130.59 samples/sec Loss 2.1930 Epoch: 16 Global Step: 159630 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:15:06,665-Speed 128.69 samples/sec Loss 2.2116 Epoch: 16 Global Step: 159635 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:15:26,408-Speed 129.67 samples/sec Loss 1.9599 Epoch: 16 Global Step: 159640 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:15:46,262-Speed 128.95 samples/sec Loss 2.2038 Epoch: 16 Global Step: 159645 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:16:05,937-Speed 130.12 samples/sec Loss 2.2934 Epoch: 16 Global Step: 159650 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:16:25,651-Speed 129.86 samples/sec Loss 2.1703 Epoch: 16 Global Step: 159655 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:16:45,595-Speed 128.37 samples/sec Loss 2.0307 Epoch: 16 Global Step: 159660 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:17:05,961-Speed 125.71 samples/sec Loss 2.2011 Epoch: 16 Global Step: 159665 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:17:25,567-Speed 130.57 samples/sec Loss 2.1616 Epoch: 16 Global Step: 159670 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:17:45,492-Speed 128.49 samples/sec Loss 2.0384 Epoch: 16 Global Step: 159675 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:18:05,382-Speed 128.71 samples/sec Loss 2.4425 Epoch: 16 Global Step: 159680 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:18:25,148-Speed 129.52 samples/sec Loss 2.0595 Epoch: 16 Global Step: 159685 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:18:45,011-Speed 128.89 samples/sec Loss 2.3691 Epoch: 16 Global Step: 159690 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:19:04,704-Speed 130.00 samples/sec Loss 2.1810 Epoch: 16 Global Step: 159695 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:19:24,635-Speed 128.45 samples/sec Loss 2.1059 Epoch: 16 Global Step: 159700 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:19:44,350-Speed 129.85 samples/sec Loss 2.0243 Epoch: 16 Global Step: 159705 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:20:04,584-Speed 126.52 samples/sec Loss 2.0429 Epoch: 16 Global Step: 159710 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:20:24,379-Speed 129.33 samples/sec Loss 1.8433 Epoch: 16 Global Step: 159715 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:20:44,269-Speed 128.71 samples/sec Loss 2.1203 Epoch: 16 Global Step: 159720 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:21:03,987-Speed 129.84 samples/sec Loss 2.0306 Epoch: 16 Global Step: 159725 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:21:23,780-Speed 129.34 samples/sec Loss 2.0008 Epoch: 16 Global Step: 159730 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:21:43,460-Speed 130.08 samples/sec Loss 2.1123 Epoch: 16 Global Step: 159735 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:22:03,356-Speed 128.68 samples/sec Loss 1.9901 Epoch: 16 Global Step: 159740 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:22:23,084-Speed 129.77 samples/sec Loss 2.1319 Epoch: 16 Global Step: 159745 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:22:42,817-Speed 129.74 samples/sec Loss 2.2149 Epoch: 16 Global Step: 159750 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:23:03,063-Speed 126.45 samples/sec Loss 2.0007 Epoch: 16 Global Step: 159755 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:23:23,035-Speed 128.18 samples/sec Loss 2.1129 Epoch: 16 Global Step: 159760 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:23:42,858-Speed 129.15 samples/sec Loss 1.9563 Epoch: 16 Global Step: 159765 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:24:02,897-Speed 127.76 samples/sec Loss 2.1161 Epoch: 16 Global Step: 159770 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:24:22,535-Speed 130.37 samples/sec Loss 1.9615 Epoch: 16 Global Step: 159775 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:24:42,376-Speed 129.03 samples/sec Loss 2.1950 Epoch: 16 Global Step: 159780 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:25:02,171-Speed 129.33 samples/sec Loss 2.0711 Epoch: 16 Global Step: 159785 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:25:21,820-Speed 130.30 samples/sec Loss 2.0489 Epoch: 16 Global Step: 159790 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:25:41,726-Speed 128.61 samples/sec Loss 1.8720 Epoch: 16 Global Step: 159795 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:26:02,008-Speed 126.23 samples/sec Loss 2.0406 Epoch: 16 Global Step: 159800 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:26:21,777-Speed 129.51 samples/sec Loss 1.9656 Epoch: 16 Global Step: 159805 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:26:41,688-Speed 128.58 samples/sec Loss 2.0246 Epoch: 16 Global Step: 159810 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:27:01,511-Speed 129.15 samples/sec Loss 2.0329 Epoch: 16 Global Step: 159815 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:27:21,103-Speed 130.67 samples/sec Loss 2.0502 Epoch: 16 Global Step: 159820 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:27:41,173-Speed 127.56 samples/sec Loss 1.9868 Epoch: 16 Global Step: 159825 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:28:00,897-Speed 129.80 samples/sec Loss 1.9173 Epoch: 16 Global Step: 159830 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:28:20,812-Speed 128.55 samples/sec Loss 2.3302 Epoch: 16 Global Step: 159835 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:28:40,409-Speed 130.63 samples/sec Loss 2.0122 Epoch: 16 Global Step: 159840 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:29:00,782-Speed 125.66 samples/sec Loss 2.0246 Epoch: 16 Global Step: 159845 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:29:20,534-Speed 129.61 samples/sec Loss 1.9003 Epoch: 16 Global Step: 159850 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:29:40,390-Speed 128.94 samples/sec Loss 2.0077 Epoch: 16 Global Step: 159855 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:30:00,142-Speed 129.61 samples/sec Loss 2.0878 Epoch: 16 Global Step: 159860 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:30:20,040-Speed 128.67 samples/sec Loss 2.1117 Epoch: 16 Global Step: 159865 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:30:39,755-Speed 129.86 samples/sec Loss 1.9665 Epoch: 16 Global Step: 159870 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:30:59,524-Speed 129.50 samples/sec Loss 1.9085 Epoch: 16 Global Step: 159875 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:31:19,366-Speed 129.03 samples/sec Loss 2.1106 Epoch: 16 Global Step: 159880 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:31:39,177-Speed 129.23 samples/sec Loss 2.0468 Epoch: 16 Global Step: 159885 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:31:59,520-Speed 125.84 samples/sec Loss 2.1059 Epoch: 16 Global Step: 159890 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:32:19,497-Speed 128.15 samples/sec Loss 2.0220 Epoch: 16 Global Step: 159895 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:32:39,113-Speed 130.51 samples/sec Loss 1.9445 Epoch: 16 Global Step: 159900 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:32:58,873-Speed 129.56 samples/sec Loss 2.0335 Epoch: 16 Global Step: 159905 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:33:18,738-Speed 128.88 samples/sec Loss 2.0210 Epoch: 16 Global Step: 159910 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:33:38,464-Speed 129.78 samples/sec Loss 2.0886 Epoch: 16 Global Step: 159915 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:33:58,256-Speed 129.36 samples/sec Loss 2.0368 Epoch: 16 Global Step: 159920 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:34:17,964-Speed 129.90 samples/sec Loss 1.8701 Epoch: 16 Global Step: 159925 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:34:37,690-Speed 129.78 samples/sec Loss 2.2220 Epoch: 16 Global Step: 159930 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:34:58,008-Speed 126.00 samples/sec Loss 1.9452 Epoch: 16 Global Step: 159935 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:35:17,894-Speed 128.74 samples/sec Loss 2.0540 Epoch: 16 Global Step: 159940 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:35:37,602-Speed 129.90 samples/sec Loss 1.8572 Epoch: 16 Global Step: 159945 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:35:57,497-Speed 128.69 samples/sec Loss 2.0131 Epoch: 16 Global Step: 159950 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:36:17,211-Speed 129.86 samples/sec Loss 1.9235 Epoch: 16 Global Step: 159955 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:36:36,948-Speed 129.72 samples/sec Loss 2.1137 Epoch: 16 Global Step: 159960 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:36:56,700-Speed 129.62 samples/sec Loss 1.9540 Epoch: 16 Global Step: 159965 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:37:16,423-Speed 129.80 samples/sec Loss 2.0990 Epoch: 16 Global Step: 159970 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:37:36,323-Speed 128.65 samples/sec Loss 2.0067 Epoch: 16 Global Step: 159975 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:37:56,750-Speed 125.33 samples/sec Loss 1.9043 Epoch: 16 Global Step: 159980 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:38:16,462-Speed 129.87 samples/sec Loss 1.8385 Epoch: 16 Global Step: 159985 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:38:36,218-Speed 129.59 samples/sec Loss 1.8866 Epoch: 16 Global Step: 159990 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:38:56,078-Speed 128.91 samples/sec Loss 1.9706 Epoch: 16 Global Step: 159995 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:39:15,859-Speed 129.42 samples/sec Loss 1.9541 Epoch: 16 Global Step: 160000 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 15:54:04,359-[lfw][160000]XNorm: 22.419967 Training: 2021-06-14 15:54:04,359-[lfw][160000]Accuracy-Flip: 0.99817+-0.00203 Training: 2021-06-14 15:54:04,359-[lfw][160000]Accuracy-Highest: 0.99833 Training: 2021-06-14 16:11:20,977-[cfp_fp][160000]XNorm: 22.515405 Training: 2021-06-14 16:11:20,977-[cfp_fp][160000]Accuracy-Flip: 0.99057+-0.00265 Training: 2021-06-14 16:11:20,977-[cfp_fp][160000]Accuracy-Highest: 0.99071 Training: 2021-06-14 16:26:09,448-[agedb_30][160000]XNorm: 23.046653 Training: 2021-06-14 16:26:09,448-[agedb_30][160000]Accuracy-Flip: 0.98383+-0.00592 Training: 2021-06-14 16:26:09,448-[agedb_30][160000]Accuracy-Highest: 0.98417 Training: 2021-06-14 16:26:29,117-Speed 0.90 samples/sec Loss 1.9589 Epoch: 16 Global Step: 160005 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:26:48,991-Speed 128.82 samples/sec Loss 1.9704 Epoch: 16 Global Step: 160010 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:27:08,508-Speed 131.18 samples/sec Loss 1.8380 Epoch: 16 Global Step: 160015 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:27:28,500-Speed 128.05 samples/sec Loss 2.1076 Epoch: 16 Global Step: 160020 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:27:48,105-Speed 130.59 samples/sec Loss 1.9755 Epoch: 16 Global Step: 160025 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:28:07,733-Speed 130.43 samples/sec Loss 1.9505 Epoch: 16 Global Step: 160030 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:28:27,345-Speed 130.53 samples/sec Loss 1.9753 Epoch: 16 Global Step: 160035 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:28:46,874-Speed 131.10 samples/sec Loss 1.9319 Epoch: 16 Global Step: 160040 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:29:06,647-Speed 129.48 samples/sec Loss 2.0509 Epoch: 16 Global Step: 160045 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:29:26,173-Speed 131.12 samples/sec Loss 1.9480 Epoch: 16 Global Step: 160050 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:29:46,099-Speed 128.48 samples/sec Loss 2.0414 Epoch: 16 Global Step: 160055 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:30:05,585-Speed 131.38 samples/sec Loss 1.9115 Epoch: 16 Global Step: 160060 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:30:25,629-Speed 127.73 samples/sec Loss 1.9577 Epoch: 16 Global Step: 160065 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:30:45,398-Speed 129.50 samples/sec Loss 1.8725 Epoch: 16 Global Step: 160070 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:31:05,015-Speed 130.50 samples/sec Loss 2.0851 Epoch: 16 Global Step: 160075 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:31:24,678-Speed 130.20 samples/sec Loss 1.9154 Epoch: 16 Global Step: 160080 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:31:44,290-Speed 130.54 samples/sec Loss 1.9263 Epoch: 16 Global Step: 160085 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:32:03,988-Speed 129.97 samples/sec Loss 2.0698 Epoch: 16 Global Step: 160090 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:32:23,531-Speed 131.00 samples/sec Loss 1.8663 Epoch: 16 Global Step: 160095 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:32:43,395-Speed 128.88 samples/sec Loss 2.0973 Epoch: 16 Global Step: 160100 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:33:02,933-Speed 131.05 samples/sec Loss 1.9427 Epoch: 16 Global Step: 160105 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:33:22,913-Speed 128.13 samples/sec Loss 2.0520 Epoch: 16 Global Step: 160110 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:33:42,942-Speed 127.82 samples/sec Loss 1.9425 Epoch: 16 Global Step: 160115 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:34:02,452-Speed 131.22 samples/sec Loss 2.0858 Epoch: 16 Global Step: 160120 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:34:22,303-Speed 128.97 samples/sec Loss 1.8750 Epoch: 16 Global Step: 160125 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:34:41,806-Speed 131.27 samples/sec Loss 1.8523 Epoch: 16 Global Step: 160130 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:35:01,572-Speed 129.52 samples/sec Loss 1.9546 Epoch: 16 Global Step: 160135 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:35:21,302-Speed 129.77 samples/sec Loss 1.9479 Epoch: 16 Global Step: 160140 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:35:40,920-Speed 130.49 samples/sec Loss 2.0196 Epoch: 16 Global Step: 160145 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:36:00,525-Speed 130.59 samples/sec Loss 2.0802 Epoch: 16 Global Step: 160150 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:36:20,320-Speed 129.33 samples/sec Loss 1.9789 Epoch: 16 Global Step: 160155 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:36:40,415-Speed 127.40 samples/sec Loss 2.0221 Epoch: 16 Global Step: 160160 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:36:59,928-Speed 131.21 samples/sec Loss 1.8948 Epoch: 16 Global Step: 160165 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:37:19,761-Speed 129.08 samples/sec Loss 1.9551 Epoch: 16 Global Step: 160170 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:37:39,290-Speed 131.09 samples/sec Loss 1.7783 Epoch: 16 Global Step: 160175 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:37:59,056-Speed 129.52 samples/sec Loss 1.8612 Epoch: 16 Global Step: 160180 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:38:18,663-Speed 130.58 samples/sec Loss 1.9422 Epoch: 16 Global Step: 160185 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:38:38,562-Speed 128.66 samples/sec Loss 1.9170 Epoch: 16 Global Step: 160190 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:38:58,164-Speed 130.61 samples/sec Loss 2.1675 Epoch: 16 Global Step: 160195 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:39:18,174-Speed 127.94 samples/sec Loss 2.0063 Epoch: 16 Global Step: 160200 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:39:37,951-Speed 129.45 samples/sec Loss 1.8934 Epoch: 16 Global Step: 160205 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:39:57,633-Speed 130.07 samples/sec Loss 2.0040 Epoch: 16 Global Step: 160210 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:40:17,396-Speed 129.54 samples/sec Loss 1.8509 Epoch: 16 Global Step: 160215 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:40:36,936-Speed 131.02 samples/sec Loss 1.8297 Epoch: 16 Global Step: 160220 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:40:56,905-Speed 128.21 samples/sec Loss 1.9957 Epoch: 16 Global Step: 160225 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:41:16,434-Speed 131.10 samples/sec Loss 2.1144 Epoch: 16 Global Step: 160230 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:41:36,301-Speed 128.86 samples/sec Loss 1.9981 Epoch: 16 Global Step: 160235 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:41:55,824-Speed 131.14 samples/sec Loss 1.9079 Epoch: 16 Global Step: 160240 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:42:15,931-Speed 127.32 samples/sec Loss 1.9523 Epoch: 16 Global Step: 160245 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:42:35,653-Speed 129.82 samples/sec Loss 1.7993 Epoch: 16 Global Step: 160250 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:42:55,267-Speed 130.52 samples/sec Loss 1.9115 Epoch: 16 Global Step: 160255 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:43:14,978-Speed 129.89 samples/sec Loss 2.0047 Epoch: 16 Global Step: 160260 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:43:34,534-Speed 130.92 samples/sec Loss 1.9802 Epoch: 16 Global Step: 160265 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:43:54,469-Speed 128.42 samples/sec Loss 2.0448 Epoch: 16 Global Step: 160270 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:44:14,032-Speed 130.87 samples/sec Loss 1.8671 Epoch: 16 Global Step: 160275 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:44:33,826-Speed 129.34 samples/sec Loss 1.8378 Epoch: 16 Global Step: 160280 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:44:53,377-Speed 130.95 samples/sec Loss 1.9635 Epoch: 16 Global Step: 160285 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:45:13,472-Speed 127.40 samples/sec Loss 1.8853 Epoch: 16 Global Step: 160290 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:45:33,451-Speed 128.14 samples/sec Loss 1.9376 Epoch: 16 Global Step: 160295 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:45:53,100-Speed 130.30 samples/sec Loss 1.8719 Epoch: 16 Global Step: 160300 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:46:12,793-Speed 130.00 samples/sec Loss 2.0305 Epoch: 16 Global Step: 160305 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:46:32,406-Speed 130.53 samples/sec Loss 1.8999 Epoch: 16 Global Step: 160310 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:46:52,150-Speed 129.67 samples/sec Loss 1.8438 Epoch: 16 Global Step: 160315 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:47:11,686-Speed 131.05 samples/sec Loss 1.9730 Epoch: 16 Global Step: 160320 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:47:31,395-Speed 129.90 samples/sec Loss 2.0408 Epoch: 16 Global Step: 160325 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:47:50,958-Speed 130.87 samples/sec Loss 1.8507 Epoch: 16 Global Step: 160330 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:48:10,999-Speed 127.74 samples/sec Loss 1.9727 Epoch: 16 Global Step: 160335 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:48:31,009-Speed 127.95 samples/sec Loss 2.0755 Epoch: 16 Global Step: 160340 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:48:50,535-Speed 131.11 samples/sec Loss 1.8073 Epoch: 16 Global Step: 160345 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:49:10,431-Speed 128.68 samples/sec Loss 2.0571 Epoch: 16 Global Step: 160350 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:49:30,019-Speed 130.70 samples/sec Loss 1.9492 Epoch: 16 Global Step: 160355 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:49:49,835-Speed 129.19 samples/sec Loss 1.9084 Epoch: 16 Global Step: 160360 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:50:09,358-Speed 131.14 samples/sec Loss 1.9830 Epoch: 16 Global Step: 160365 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:50:29,114-Speed 129.59 samples/sec Loss 1.9128 Epoch: 16 Global Step: 160370 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:50:48,754-Speed 130.35 samples/sec Loss 1.7665 Epoch: 16 Global Step: 160375 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:51:08,904-Speed 127.05 samples/sec Loss 1.8979 Epoch: 16 Global Step: 160380 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:51:28,695-Speed 129.36 samples/sec Loss 1.9090 Epoch: 16 Global Step: 160385 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:51:48,251-Speed 130.91 samples/sec Loss 1.8397 Epoch: 16 Global Step: 160390 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:52:08,109-Speed 128.93 samples/sec Loss 1.8320 Epoch: 16 Global Step: 160395 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:52:27,714-Speed 130.58 samples/sec Loss 1.8092 Epoch: 16 Global Step: 160400 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:52:47,514-Speed 129.30 samples/sec Loss 1.6843 Epoch: 16 Global Step: 160405 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:53:07,034-Speed 131.15 samples/sec Loss 1.7801 Epoch: 16 Global Step: 160410 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:53:26,942-Speed 128.60 samples/sec Loss 1.8354 Epoch: 16 Global Step: 160415 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:53:46,563-Speed 130.48 samples/sec Loss 1.8348 Epoch: 16 Global Step: 160420 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:54:06,915-Speed 125.79 samples/sec Loss 1.9077 Epoch: 16 Global Step: 160425 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:54:26,550-Speed 130.39 samples/sec Loss 1.8687 Epoch: 16 Global Step: 160430 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:54:46,180-Speed 130.42 samples/sec Loss 1.9939 Epoch: 16 Global Step: 160435 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:55:05,875-Speed 129.99 samples/sec Loss 1.8735 Epoch: 16 Global Step: 160440 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:55:25,484-Speed 130.56 samples/sec Loss 1.9666 Epoch: 16 Global Step: 160445 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:55:45,097-Speed 130.53 samples/sec Loss 1.8065 Epoch: 16 Global Step: 160450 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:56:04,689-Speed 130.67 samples/sec Loss 1.9158 Epoch: 16 Global Step: 160455 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:56:24,558-Speed 128.85 samples/sec Loss 1.9179 Epoch: 16 Global Step: 160460 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:56:44,172-Speed 130.52 samples/sec Loss 1.8016 Epoch: 16 Global Step: 160465 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:57:04,262-Speed 127.44 samples/sec Loss 1.7329 Epoch: 16 Global Step: 160470 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:57:24,379-Speed 127.27 samples/sec Loss 1.8129 Epoch: 16 Global Step: 160475 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:57:43,861-Speed 131.41 samples/sec Loss 1.8535 Epoch: 16 Global Step: 160480 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:58:03,654-Speed 129.34 samples/sec Loss 1.9543 Epoch: 16 Global Step: 160485 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:58:23,267-Speed 130.53 samples/sec Loss 2.0087 Epoch: 16 Global Step: 160490 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:58:42,967-Speed 129.96 samples/sec Loss 1.8836 Epoch: 16 Global Step: 160495 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:59:02,518-Speed 130.95 samples/sec Loss 1.8668 Epoch: 16 Global Step: 160500 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:59:22,224-Speed 129.92 samples/sec Loss 1.9424 Epoch: 16 Global Step: 160505 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 16:59:41,809-Speed 130.72 samples/sec Loss 1.9875 Epoch: 16 Global Step: 160510 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:00:01,849-Speed 127.75 samples/sec Loss 1.9263 Epoch: 16 Global Step: 160515 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:00:21,870-Speed 127.87 samples/sec Loss 1.8666 Epoch: 16 Global Step: 160520 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:00:41,333-Speed 131.54 samples/sec Loss 1.6659 Epoch: 16 Global Step: 160525 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:01:01,190-Speed 128.93 samples/sec Loss 1.8548 Epoch: 16 Global Step: 160530 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:01:20,740-Speed 130.95 samples/sec Loss 1.8721 Epoch: 16 Global Step: 160535 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:01:40,508-Speed 129.51 samples/sec Loss 1.8078 Epoch: 16 Global Step: 160540 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:02:00,090-Speed 130.74 samples/sec Loss 1.8940 Epoch: 16 Global Step: 160545 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:02:19,725-Speed 130.39 samples/sec Loss 1.7196 Epoch: 16 Global Step: 160550 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:02:39,345-Speed 130.49 samples/sec Loss 1.8167 Epoch: 16 Global Step: 160555 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:02:59,300-Speed 128.29 samples/sec Loss 1.8313 Epoch: 16 Global Step: 160560 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:03:20,861-Speed 118.74 samples/sec Loss 2.0360 Epoch: 16 Global Step: 160565 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:03:40,454-Speed 130.67 samples/sec Loss 1.9554 Epoch: 16 Global Step: 160570 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:04:00,251-Speed 129.32 samples/sec Loss 1.9071 Epoch: 16 Global Step: 160575 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:04:19,785-Speed 131.06 samples/sec Loss 1.8411 Epoch: 16 Global Step: 160580 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:04:39,684-Speed 128.66 samples/sec Loss 2.1020 Epoch: 16 Global Step: 160585 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:04:59,118-Speed 131.73 samples/sec Loss 1.9475 Epoch: 16 Global Step: 160590 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:05:18,949-Speed 129.10 samples/sec Loss 2.0004 Epoch: 16 Global Step: 160595 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:05:38,533-Speed 130.72 samples/sec Loss 1.9199 Epoch: 16 Global Step: 160600 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:05:58,628-Speed 127.40 samples/sec Loss 1.8421 Epoch: 16 Global Step: 160605 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:06:18,255-Speed 130.44 samples/sec Loss 1.9625 Epoch: 16 Global Step: 160610 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:06:37,872-Speed 130.51 samples/sec Loss 1.9525 Epoch: 16 Global Step: 160615 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:06:57,650-Speed 129.45 samples/sec Loss 1.8897 Epoch: 16 Global Step: 160620 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:07:17,210-Speed 130.88 samples/sec Loss 1.7031 Epoch: 16 Global Step: 160625 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:07:37,060-Speed 128.98 samples/sec Loss 1.9750 Epoch: 16 Global Step: 160630 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:07:56,605-Speed 130.98 samples/sec Loss 1.6509 Epoch: 16 Global Step: 160635 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:08:16,476-Speed 128.84 samples/sec Loss 1.8496 Epoch: 16 Global Step: 160640 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:08:36,013-Speed 131.04 samples/sec Loss 1.8989 Epoch: 16 Global Step: 160645 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:08:56,110-Speed 127.39 samples/sec Loss 1.7729 Epoch: 16 Global Step: 160650 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:09:15,836-Speed 129.78 samples/sec Loss 1.8864 Epoch: 16 Global Step: 160655 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:09:35,446-Speed 130.55 samples/sec Loss 1.8478 Epoch: 16 Global Step: 160660 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:09:55,239-Speed 129.34 samples/sec Loss 1.9549 Epoch: 16 Global Step: 160665 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:10:14,785-Speed 130.98 samples/sec Loss 2.0175 Epoch: 16 Global Step: 160670 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:10:34,587-Speed 129.29 samples/sec Loss 1.9450 Epoch: 16 Global Step: 160675 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:10:54,183-Speed 130.64 samples/sec Loss 1.8484 Epoch: 16 Global Step: 160680 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:11:13,981-Speed 129.31 samples/sec Loss 1.8789 Epoch: 16 Global Step: 160685 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:11:33,538-Speed 130.91 samples/sec Loss 1.9066 Epoch: 16 Global Step: 160690 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:11:53,524-Speed 128.10 samples/sec Loss 1.8107 Epoch: 16 Global Step: 160695 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:12:13,452-Speed 128.47 samples/sec Loss 1.8801 Epoch: 16 Global Step: 160700 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:12:33,028-Speed 130.78 samples/sec Loss 1.7844 Epoch: 16 Global Step: 160705 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:12:52,748-Speed 129.82 samples/sec Loss 1.7712 Epoch: 16 Global Step: 160710 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:13:12,359-Speed 130.55 samples/sec Loss 1.6915 Epoch: 16 Global Step: 160715 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:13:32,000-Speed 130.35 samples/sec Loss 1.8603 Epoch: 16 Global Step: 160720 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:13:51,554-Speed 130.93 samples/sec Loss 1.8451 Epoch: 16 Global Step: 160725 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:14:11,406-Speed 128.96 samples/sec Loss 1.8076 Epoch: 16 Global Step: 160730 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:14:30,971-Speed 130.85 samples/sec Loss 1.8803 Epoch: 16 Global Step: 160735 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:14:50,950-Speed 128.14 samples/sec Loss 1.7862 Epoch: 16 Global Step: 160740 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:15:10,885-Speed 128.42 samples/sec Loss 1.6678 Epoch: 16 Global Step: 160745 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:15:30,434-Speed 130.96 samples/sec Loss 1.7983 Epoch: 16 Global Step: 160750 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:15:50,093-Speed 130.23 samples/sec Loss 1.9089 Epoch: 16 Global Step: 160755 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:16:09,667-Speed 130.79 samples/sec Loss 1.9611 Epoch: 16 Global Step: 160760 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:16:29,394-Speed 129.78 samples/sec Loss 1.8505 Epoch: 16 Global Step: 160765 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:16:49,014-Speed 130.49 samples/sec Loss 1.7607 Epoch: 16 Global Step: 160770 Fp16 Grad Scale: 8192 Required: 105 hours Training: 2021-06-14 17:17:08,738-Speed 129.80 samples/sec Loss 1.9172 Epoch: 16 Global Step: 160775 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:17:28,339-Speed 130.61 samples/sec Loss 1.7458 Epoch: 16 Global Step: 160780 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:17:48,215-Speed 128.80 samples/sec Loss 1.8762 Epoch: 16 Global Step: 160785 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:18:08,178-Speed 128.25 samples/sec Loss 1.8136 Epoch: 16 Global Step: 160790 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:18:27,705-Speed 131.10 samples/sec Loss 1.7263 Epoch: 16 Global Step: 160795 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:18:47,490-Speed 129.40 samples/sec Loss 1.7285 Epoch: 16 Global Step: 160800 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:19:06,928-Speed 131.71 samples/sec Loss 1.8328 Epoch: 16 Global Step: 160805 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:19:26,871-Speed 128.37 samples/sec Loss 1.8066 Epoch: 16 Global Step: 160810 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:19:46,409-Speed 131.03 samples/sec Loss 1.9333 Epoch: 16 Global Step: 160815 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:20:06,116-Speed 129.91 samples/sec Loss 1.8688 Epoch: 16 Global Step: 160820 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:20:25,733-Speed 130.50 samples/sec Loss 1.8576 Epoch: 16 Global Step: 160825 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:20:45,759-Speed 127.84 samples/sec Loss 1.8540 Epoch: 16 Global Step: 160830 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:21:05,480-Speed 129.82 samples/sec Loss 1.8684 Epoch: 16 Global Step: 160835 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:21:25,023-Speed 131.00 samples/sec Loss 1.9350 Epoch: 16 Global Step: 160840 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:21:44,756-Speed 129.74 samples/sec Loss 1.8158 Epoch: 16 Global Step: 160845 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:22:04,327-Speed 130.81 samples/sec Loss 1.9203 Epoch: 16 Global Step: 160850 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:22:24,294-Speed 128.22 samples/sec Loss 1.8278 Epoch: 16 Global Step: 160855 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:22:43,812-Speed 131.17 samples/sec Loss 1.9020 Epoch: 16 Global Step: 160860 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:23:04,023-Speed 126.67 samples/sec Loss 1.7965 Epoch: 16 Global Step: 160865 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:23:23,597-Speed 130.79 samples/sec Loss 1.7839 Epoch: 16 Global Step: 160870 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:23:43,711-Speed 127.29 samples/sec Loss 1.7744 Epoch: 16 Global Step: 160875 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:24:03,441-Speed 129.75 samples/sec Loss 1.7978 Epoch: 16 Global Step: 160880 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:24:23,048-Speed 130.57 samples/sec Loss 2.0400 Epoch: 16 Global Step: 160885 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:24:42,673-Speed 130.46 samples/sec Loss 1.7313 Epoch: 16 Global Step: 160890 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:25:02,219-Speed 130.98 samples/sec Loss 1.8075 Epoch: 16 Global Step: 160895 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:25:22,092-Speed 128.83 samples/sec Loss 1.7710 Epoch: 16 Global Step: 160900 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:25:41,636-Speed 130.99 samples/sec Loss 1.8843 Epoch: 16 Global Step: 160905 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:26:01,577-Speed 128.39 samples/sec Loss 1.9208 Epoch: 16 Global Step: 160910 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:26:21,112-Speed 131.06 samples/sec Loss 1.8019 Epoch: 16 Global Step: 160915 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:26:41,191-Speed 127.50 samples/sec Loss 1.7016 Epoch: 16 Global Step: 160920 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:27:00,910-Speed 129.83 samples/sec Loss 1.7586 Epoch: 16 Global Step: 160925 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:27:20,522-Speed 130.54 samples/sec Loss 1.8738 Epoch: 16 Global Step: 160930 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:27:40,229-Speed 129.91 samples/sec Loss 2.1100 Epoch: 16 Global Step: 160935 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:27:59,841-Speed 130.54 samples/sec Loss 1.7266 Epoch: 16 Global Step: 160940 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:28:19,571-Speed 129.76 samples/sec Loss 1.8886 Epoch: 16 Global Step: 160945 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:28:39,107-Speed 131.05 samples/sec Loss 1.8005 Epoch: 16 Global Step: 160950 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:28:58,886-Speed 129.44 samples/sec Loss 1.7661 Epoch: 16 Global Step: 160955 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:29:18,460-Speed 130.79 samples/sec Loss 1.6559 Epoch: 16 Global Step: 160960 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:29:38,592-Speed 127.17 samples/sec Loss 1.6897 Epoch: 16 Global Step: 160965 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:29:58,338-Speed 129.66 samples/sec Loss 1.8223 Epoch: 16 Global Step: 160970 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:30:17,915-Speed 130.77 samples/sec Loss 1.7248 Epoch: 16 Global Step: 160975 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:30:37,641-Speed 129.79 samples/sec Loss 1.6803 Epoch: 16 Global Step: 160980 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:30:57,252-Speed 130.55 samples/sec Loss 1.8697 Epoch: 16 Global Step: 160985 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:31:17,092-Speed 129.04 samples/sec Loss 1.5878 Epoch: 16 Global Step: 160990 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:31:36,542-Speed 131.63 samples/sec Loss 1.9346 Epoch: 16 Global Step: 160995 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:31:56,396-Speed 128.95 samples/sec Loss 1.6812 Epoch: 16 Global Step: 161000 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:32:16,038-Speed 130.34 samples/sec Loss 1.7073 Epoch: 16 Global Step: 161005 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:32:36,199-Speed 126.98 samples/sec Loss 1.8108 Epoch: 16 Global Step: 161010 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:32:56,068-Speed 128.85 samples/sec Loss 1.8138 Epoch: 16 Global Step: 161015 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:33:15,608-Speed 131.02 samples/sec Loss 2.0124 Epoch: 16 Global Step: 161020 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:33:35,312-Speed 129.93 samples/sec Loss 1.8141 Epoch: 16 Global Step: 161025 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:33:54,936-Speed 130.46 samples/sec Loss 1.6950 Epoch: 16 Global Step: 161030 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:34:14,595-Speed 130.23 samples/sec Loss 1.9273 Epoch: 16 Global Step: 161035 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:34:34,064-Speed 131.50 samples/sec Loss 1.7650 Epoch: 16 Global Step: 161040 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:34:53,937-Speed 128.83 samples/sec Loss 1.7767 Epoch: 16 Global Step: 161045 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:35:13,516-Speed 130.76 samples/sec Loss 1.8764 Epoch: 16 Global Step: 161050 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:35:33,665-Speed 127.05 samples/sec Loss 2.0687 Epoch: 16 Global Step: 161055 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:35:53,592-Speed 128.48 samples/sec Loss 1.8329 Epoch: 16 Global Step: 161060 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:36:13,145-Speed 130.93 samples/sec Loss 1.7392 Epoch: 16 Global Step: 161065 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:36:32,918-Speed 129.48 samples/sec Loss 1.8402 Epoch: 16 Global Step: 161070 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:36:52,510-Speed 130.67 samples/sec Loss 1.8194 Epoch: 16 Global Step: 161075 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:37:12,170-Speed 130.22 samples/sec Loss 1.5728 Epoch: 16 Global Step: 161080 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:37:31,766-Speed 130.65 samples/sec Loss 1.9874 Epoch: 16 Global Step: 161085 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:37:51,456-Speed 130.02 samples/sec Loss 1.7936 Epoch: 16 Global Step: 161090 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:38:11,003-Speed 130.97 samples/sec Loss 1.9058 Epoch: 16 Global Step: 161095 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:38:31,090-Speed 127.45 samples/sec Loss 1.8019 Epoch: 16 Global Step: 161100 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:38:50,928-Speed 129.05 samples/sec Loss 1.7262 Epoch: 16 Global Step: 161105 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:39:10,469-Speed 131.01 samples/sec Loss 1.7891 Epoch: 16 Global Step: 161110 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:39:30,212-Speed 129.68 samples/sec Loss 1.8197 Epoch: 16 Global Step: 161115 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:39:49,788-Speed 130.78 samples/sec Loss 1.7902 Epoch: 16 Global Step: 161120 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:40:09,510-Speed 129.81 samples/sec Loss 1.8800 Epoch: 16 Global Step: 161125 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:40:29,127-Speed 130.51 samples/sec Loss 1.6800 Epoch: 16 Global Step: 161130 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:40:48,848-Speed 129.82 samples/sec Loss 1.5810 Epoch: 16 Global Step: 161135 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:41:08,463-Speed 130.52 samples/sec Loss 1.8958 Epoch: 16 Global Step: 161140 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:41:28,280-Speed 129.19 samples/sec Loss 1.7629 Epoch: 16 Global Step: 161145 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:41:48,377-Speed 127.39 samples/sec Loss 1.7809 Epoch: 16 Global Step: 161150 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:42:07,962-Speed 130.72 samples/sec Loss 1.7678 Epoch: 16 Global Step: 161155 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:42:28,088-Speed 127.21 samples/sec Loss 1.7582 Epoch: 16 Global Step: 161160 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:42:47,609-Speed 131.15 samples/sec Loss 1.8353 Epoch: 16 Global Step: 161165 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:43:07,260-Speed 130.28 samples/sec Loss 1.7513 Epoch: 16 Global Step: 161170 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:43:26,829-Speed 130.83 samples/sec Loss 1.9096 Epoch: 16 Global Step: 161175 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:43:46,546-Speed 129.84 samples/sec Loss 1.8287 Epoch: 16 Global Step: 161180 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:44:06,156-Speed 130.56 samples/sec Loss 1.7364 Epoch: 16 Global Step: 161185 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:44:26,163-Speed 127.96 samples/sec Loss 1.6794 Epoch: 16 Global Step: 161190 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:44:46,109-Speed 128.36 samples/sec Loss 1.6978 Epoch: 16 Global Step: 161195 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:45:05,595-Speed 131.38 samples/sec Loss 1.7166 Epoch: 16 Global Step: 161200 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:45:25,576-Speed 128.13 samples/sec Loss 1.8736 Epoch: 16 Global Step: 161205 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:45:45,020-Speed 131.67 samples/sec Loss 1.9425 Epoch: 16 Global Step: 161210 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:46:04,758-Speed 129.70 samples/sec Loss 2.0020 Epoch: 16 Global Step: 161215 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:46:24,360-Speed 130.61 samples/sec Loss 1.8500 Epoch: 16 Global Step: 161220 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:46:43,966-Speed 130.58 samples/sec Loss 1.7139 Epoch: 16 Global Step: 161225 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:47:03,702-Speed 129.72 samples/sec Loss 1.7370 Epoch: 16 Global Step: 161230 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:47:23,623-Speed 128.51 samples/sec Loss 1.9404 Epoch: 16 Global Step: 161235 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:47:43,447-Speed 129.14 samples/sec Loss 1.7387 Epoch: 16 Global Step: 161240 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:48:02,976-Speed 131.09 samples/sec Loss 1.8910 Epoch: 16 Global Step: 161245 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:48:22,959-Speed 128.12 samples/sec Loss 1.6173 Epoch: 16 Global Step: 161250 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:48:42,387-Speed 131.77 samples/sec Loss 1.8300 Epoch: 16 Global Step: 161255 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:49:02,302-Speed 128.55 samples/sec Loss 1.8551 Epoch: 16 Global Step: 161260 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:49:21,854-Speed 130.94 samples/sec Loss 1.7063 Epoch: 16 Global Step: 161265 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:49:41,718-Speed 128.89 samples/sec Loss 1.8819 Epoch: 16 Global Step: 161270 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:50:01,332-Speed 130.53 samples/sec Loss 1.8016 Epoch: 16 Global Step: 161275 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:50:21,353-Speed 127.87 samples/sec Loss 1.8695 Epoch: 16 Global Step: 161280 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:50:41,101-Speed 129.64 samples/sec Loss 1.7876 Epoch: 16 Global Step: 161285 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:51:00,682-Speed 130.75 samples/sec Loss 1.6609 Epoch: 16 Global Step: 161290 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:51:20,459-Speed 129.45 samples/sec Loss 1.7701 Epoch: 16 Global Step: 161295 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:51:40,005-Speed 130.98 samples/sec Loss 1.6822 Epoch: 16 Global Step: 161300 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:51:59,903-Speed 128.66 samples/sec Loss 1.7795 Epoch: 16 Global Step: 161305 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:52:19,403-Speed 131.29 samples/sec Loss 1.8993 Epoch: 16 Global Step: 161310 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:52:39,162-Speed 129.57 samples/sec Loss 1.9121 Epoch: 16 Global Step: 161315 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:52:58,744-Speed 130.74 samples/sec Loss 1.8592 Epoch: 16 Global Step: 161320 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:53:18,859-Speed 127.28 samples/sec Loss 1.7364 Epoch: 16 Global Step: 161325 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:53:38,474-Speed 130.52 samples/sec Loss 1.6931 Epoch: 16 Global Step: 161330 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:53:58,183-Speed 129.90 samples/sec Loss 1.7240 Epoch: 16 Global Step: 161335 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:54:17,895-Speed 129.88 samples/sec Loss 1.7356 Epoch: 16 Global Step: 161340 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:54:37,461-Speed 130.84 samples/sec Loss 1.8527 Epoch: 16 Global Step: 161345 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:54:57,371-Speed 128.59 samples/sec Loss 1.7947 Epoch: 16 Global Step: 161350 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:55:16,920-Speed 130.96 samples/sec Loss 1.7726 Epoch: 16 Global Step: 161355 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:55:36,878-Speed 128.28 samples/sec Loss 1.6712 Epoch: 16 Global Step: 161360 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:55:56,431-Speed 130.94 samples/sec Loss 1.7038 Epoch: 16 Global Step: 161365 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:56:16,521-Speed 127.43 samples/sec Loss 1.6454 Epoch: 16 Global Step: 161370 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:56:36,256-Speed 129.73 samples/sec Loss 1.8142 Epoch: 16 Global Step: 161375 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:56:55,896-Speed 130.35 samples/sec Loss 1.7024 Epoch: 16 Global Step: 161380 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:57:15,613-Speed 129.85 samples/sec Loss 1.7420 Epoch: 16 Global Step: 161385 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:57:35,162-Speed 130.96 samples/sec Loss 1.7203 Epoch: 16 Global Step: 161390 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:57:54,918-Speed 129.59 samples/sec Loss 1.6944 Epoch: 16 Global Step: 161395 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:58:14,530-Speed 130.54 samples/sec Loss 1.7633 Epoch: 16 Global Step: 161400 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:58:34,335-Speed 129.26 samples/sec Loss 1.7655 Epoch: 16 Global Step: 161405 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:58:53,893-Speed 130.90 samples/sec Loss 1.7361 Epoch: 16 Global Step: 161410 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:59:13,996-Speed 127.35 samples/sec Loss 1.8945 Epoch: 16 Global Step: 161415 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:59:33,989-Speed 128.05 samples/sec Loss 1.8207 Epoch: 16 Global Step: 161420 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 17:59:53,564-Speed 130.79 samples/sec Loss 1.7185 Epoch: 16 Global Step: 161425 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:00:13,298-Speed 129.74 samples/sec Loss 1.6803 Epoch: 16 Global Step: 161430 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:00:32,946-Speed 130.30 samples/sec Loss 1.7303 Epoch: 16 Global Step: 161435 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:00:52,642-Speed 129.98 samples/sec Loss 1.7365 Epoch: 16 Global Step: 161440 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:01:12,269-Speed 130.44 samples/sec Loss 1.6626 Epoch: 16 Global Step: 161445 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:01:32,073-Speed 129.28 samples/sec Loss 1.8625 Epoch: 16 Global Step: 161450 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:01:51,641-Speed 130.83 samples/sec Loss 1.6412 Epoch: 16 Global Step: 161455 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:02:11,906-Speed 126.33 samples/sec Loss 1.8615 Epoch: 16 Global Step: 161460 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:02:31,980-Speed 127.54 samples/sec Loss 1.6614 Epoch: 16 Global Step: 161465 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:02:51,517-Speed 131.04 samples/sec Loss 1.6311 Epoch: 16 Global Step: 161470 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:03:11,265-Speed 129.64 samples/sec Loss 1.7754 Epoch: 16 Global Step: 161475 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:03:30,830-Speed 130.85 samples/sec Loss 1.7641 Epoch: 16 Global Step: 161480 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:03:50,441-Speed 130.55 samples/sec Loss 1.6358 Epoch: 16 Global Step: 161485 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:04:10,082-Speed 130.35 samples/sec Loss 1.5730 Epoch: 16 Global Step: 161490 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:04:29,802-Speed 129.82 samples/sec Loss 1.7067 Epoch: 16 Global Step: 161495 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:04:49,355-Speed 130.94 samples/sec Loss 1.5926 Epoch: 16 Global Step: 161500 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:05:09,390-Speed 127.78 samples/sec Loss 1.7308 Epoch: 16 Global Step: 161505 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:05:29,348-Speed 128.27 samples/sec Loss 1.7385 Epoch: 16 Global Step: 161510 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:05:48,894-Speed 130.98 samples/sec Loss 1.5022 Epoch: 16 Global Step: 161515 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:06:08,826-Speed 128.44 samples/sec Loss 1.7032 Epoch: 16 Global Step: 161520 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:06:28,349-Speed 131.13 samples/sec Loss 1.7691 Epoch: 16 Global Step: 161525 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:06:48,100-Speed 129.62 samples/sec Loss 1.6825 Epoch: 16 Global Step: 161530 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:07:07,667-Speed 130.84 samples/sec Loss 1.8187 Epoch: 16 Global Step: 161535 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:07:27,411-Speed 129.67 samples/sec Loss 1.6402 Epoch: 16 Global Step: 161540 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:07:47,035-Speed 130.46 samples/sec Loss 1.6650 Epoch: 16 Global Step: 161545 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:08:07,075-Speed 127.75 samples/sec Loss 1.6297 Epoch: 16 Global Step: 161550 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:08:26,791-Speed 129.85 samples/sec Loss 1.8262 Epoch: 16 Global Step: 161555 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:08:46,342-Speed 130.95 samples/sec Loss 1.6150 Epoch: 16 Global Step: 161560 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:09:06,193-Speed 128.97 samples/sec Loss 1.5977 Epoch: 16 Global Step: 161565 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:09:25,632-Speed 131.70 samples/sec Loss 1.7902 Epoch: 16 Global Step: 161570 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:09:45,612-Speed 128.14 samples/sec Loss 1.7069 Epoch: 16 Global Step: 161575 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:10:05,140-Speed 131.10 samples/sec Loss 1.8642 Epoch: 16 Global Step: 161580 Fp16 Grad Scale: 8192 Required: 104 hours Training: 2021-06-14 18:10:24,910-Speed 129.50 samples/sec Loss 1.4974 Epoch: 16 Global Step: 161585 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:10:44,502-Speed 130.67 samples/sec Loss 1.7863 Epoch: 16 Global Step: 161590 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:11:04,471-Speed 128.21 samples/sec Loss 1.9061 Epoch: 16 Global Step: 161595 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:11:24,166-Speed 129.98 samples/sec Loss 1.7097 Epoch: 16 Global Step: 161600 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:11:43,706-Speed 131.02 samples/sec Loss 1.8271 Epoch: 16 Global Step: 161605 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:12:03,566-Speed 128.91 samples/sec Loss 1.7322 Epoch: 16 Global Step: 161610 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:12:23,074-Speed 131.24 samples/sec Loss 1.8338 Epoch: 16 Global Step: 161615 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:12:43,136-Speed 127.61 samples/sec Loss 1.7599 Epoch: 16 Global Step: 161620 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:13:02,956-Speed 129.17 samples/sec Loss 1.8759 Epoch: 16 Global Step: 161625 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:13:22,787-Speed 129.10 samples/sec Loss 1.7248 Epoch: 16 Global Step: 161630 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:13:42,249-Speed 131.55 samples/sec Loss 1.6811 Epoch: 16 Global Step: 161635 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:14:02,339-Speed 127.43 samples/sec Loss 1.7404 Epoch: 16 Global Step: 161640 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:14:22,095-Speed 129.58 samples/sec Loss 1.5537 Epoch: 16 Global Step: 161645 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:14:41,651-Speed 130.92 samples/sec Loss 1.7626 Epoch: 16 Global Step: 161650 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:15:01,324-Speed 130.13 samples/sec Loss 1.6924 Epoch: 16 Global Step: 161655 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:15:20,993-Speed 130.16 samples/sec Loss 1.7775 Epoch: 16 Global Step: 161660 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:15:40,895-Speed 128.64 samples/sec Loss 1.8505 Epoch: 16 Global Step: 161665 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:16:00,435-Speed 131.02 samples/sec Loss 1.7081 Epoch: 16 Global Step: 161670 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:16:20,323-Speed 128.73 samples/sec Loss 1.7882 Epoch: 16 Global Step: 161675 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:16:39,848-Speed 131.12 samples/sec Loss 1.7365 Epoch: 16 Global Step: 161680 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:16:59,992-Speed 127.09 samples/sec Loss 1.6881 Epoch: 16 Global Step: 161685 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:17:19,709-Speed 129.84 samples/sec Loss 1.7599 Epoch: 16 Global Step: 161690 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:17:39,511-Speed 129.29 samples/sec Loss 1.5964 Epoch: 16 Global Step: 161695 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:17:59,219-Speed 129.90 samples/sec Loss 1.6988 Epoch: 16 Global Step: 161700 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:18:18,831-Speed 130.54 samples/sec Loss 1.7939 Epoch: 16 Global Step: 161705 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:18:38,703-Speed 128.83 samples/sec Loss 1.7751 Epoch: 16 Global Step: 161710 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:18:58,234-Speed 131.08 samples/sec Loss 1.6695 Epoch: 16 Global Step: 161715 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:19:18,217-Speed 128.11 samples/sec Loss 1.6042 Epoch: 16 Global Step: 161720 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:19:37,767-Speed 130.95 samples/sec Loss 1.6136 Epoch: 16 Global Step: 161725 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:19:57,877-Speed 127.31 samples/sec Loss 1.6842 Epoch: 16 Global Step: 161730 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:20:17,764-Speed 128.74 samples/sec Loss 1.6718 Epoch: 16 Global Step: 161735 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:20:37,346-Speed 130.73 samples/sec Loss 1.7652 Epoch: 16 Global Step: 161740 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:20:57,020-Speed 130.13 samples/sec Loss 1.6422 Epoch: 16 Global Step: 161745 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:21:16,624-Speed 130.59 samples/sec Loss 1.7484 Epoch: 16 Global Step: 161750 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:21:36,356-Speed 129.74 samples/sec Loss 1.8270 Epoch: 16 Global Step: 161755 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:21:55,896-Speed 131.02 samples/sec Loss 1.8020 Epoch: 16 Global Step: 161760 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:22:15,704-Speed 129.25 samples/sec Loss 1.6263 Epoch: 16 Global Step: 161765 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:22:35,251-Speed 130.97 samples/sec Loss 1.7402 Epoch: 16 Global Step: 161770 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:22:55,348-Speed 127.39 samples/sec Loss 1.6466 Epoch: 16 Global Step: 161775 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:23:15,327-Speed 128.15 samples/sec Loss 1.7701 Epoch: 16 Global Step: 161780 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:23:34,872-Speed 130.98 samples/sec Loss 1.7402 Epoch: 16 Global Step: 161785 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:23:54,684-Speed 129.22 samples/sec Loss 1.7464 Epoch: 16 Global Step: 161790 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:24:14,256-Speed 130.81 samples/sec Loss 1.6347 Epoch: 16 Global Step: 161795 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:24:33,888-Speed 130.41 samples/sec Loss 1.7311 Epoch: 16 Global Step: 161800 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:24:53,491-Speed 130.60 samples/sec Loss 1.7598 Epoch: 16 Global Step: 161805 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:25:13,217-Speed 129.78 samples/sec Loss 1.6566 Epoch: 16 Global Step: 161810 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:25:32,818-Speed 130.61 samples/sec Loss 1.8032 Epoch: 16 Global Step: 161815 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:25:52,773-Speed 128.30 samples/sec Loss 1.7900 Epoch: 16 Global Step: 161820 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:26:12,735-Speed 128.25 samples/sec Loss 1.7642 Epoch: 16 Global Step: 161825 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:26:32,281-Speed 130.98 samples/sec Loss 1.7822 Epoch: 16 Global Step: 161830 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:26:52,276-Speed 128.03 samples/sec Loss 1.7310 Epoch: 16 Global Step: 161835 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:27:11,821-Speed 130.99 samples/sec Loss 1.6861 Epoch: 16 Global Step: 161840 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:27:31,656-Speed 129.07 samples/sec Loss 1.6683 Epoch: 16 Global Step: 161845 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:27:51,187-Speed 131.09 samples/sec Loss 1.6196 Epoch: 16 Global Step: 161850 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:28:10,934-Speed 129.64 samples/sec Loss 1.8425 Epoch: 16 Global Step: 161855 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:28:30,546-Speed 130.54 samples/sec Loss 1.7468 Epoch: 16 Global Step: 161860 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:28:50,525-Speed 128.14 samples/sec Loss 1.5945 Epoch: 16 Global Step: 161865 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:29:10,256-Speed 129.75 samples/sec Loss 1.7000 Epoch: 16 Global Step: 161870 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:29:29,759-Speed 131.27 samples/sec Loss 1.7158 Epoch: 16 Global Step: 161875 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:29:49,692-Speed 128.43 samples/sec Loss 1.7785 Epoch: 16 Global Step: 161880 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:30:09,142-Speed 131.63 samples/sec Loss 1.5470 Epoch: 16 Global Step: 161885 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:30:29,044-Speed 128.64 samples/sec Loss 1.6755 Epoch: 16 Global Step: 161890 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:30:48,586-Speed 131.01 samples/sec Loss 1.5949 Epoch: 16 Global Step: 161895 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:31:08,236-Speed 130.29 samples/sec Loss 1.7051 Epoch: 16 Global Step: 161900 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:31:27,844-Speed 130.57 samples/sec Loss 1.6608 Epoch: 16 Global Step: 161905 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:31:47,831-Speed 128.09 samples/sec Loss 1.7320 Epoch: 16 Global Step: 161910 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:32:07,567-Speed 129.72 samples/sec Loss 1.6182 Epoch: 16 Global Step: 161915 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:32:27,188-Speed 130.48 samples/sec Loss 1.6448 Epoch: 16 Global Step: 161920 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:32:46,923-Speed 129.72 samples/sec Loss 1.5461 Epoch: 16 Global Step: 161925 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:33:06,412-Speed 131.37 samples/sec Loss 1.7850 Epoch: 16 Global Step: 161930 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:33:26,261-Speed 128.98 samples/sec Loss 1.5780 Epoch: 16 Global Step: 161935 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:33:45,773-Speed 131.21 samples/sec Loss 1.6826 Epoch: 16 Global Step: 161940 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:34:05,664-Speed 128.71 samples/sec Loss 1.6910 Epoch: 16 Global Step: 161945 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:34:25,222-Speed 130.91 samples/sec Loss 1.6361 Epoch: 16 Global Step: 161950 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:34:45,294-Speed 127.55 samples/sec Loss 1.6583 Epoch: 16 Global Step: 161955 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:35:05,118-Speed 129.14 samples/sec Loss 1.6915 Epoch: 16 Global Step: 161960 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:35:24,831-Speed 129.87 samples/sec Loss 1.6469 Epoch: 16 Global Step: 161965 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:35:44,473-Speed 130.35 samples/sec Loss 1.7783 Epoch: 16 Global Step: 161970 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:36:04,035-Speed 130.87 samples/sec Loss 1.7206 Epoch: 16 Global Step: 161975 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:36:23,770-Speed 129.73 samples/sec Loss 1.6445 Epoch: 16 Global Step: 161980 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:36:43,382-Speed 130.54 samples/sec Loss 1.7949 Epoch: 16 Global Step: 161985 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:37:03,162-Speed 129.43 samples/sec Loss 1.7212 Epoch: 16 Global Step: 161990 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:37:22,640-Speed 131.44 samples/sec Loss 1.6993 Epoch: 16 Global Step: 161995 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:37:42,810-Speed 126.93 samples/sec Loss 1.5548 Epoch: 16 Global Step: 162000 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:38:02,748-Speed 128.41 samples/sec Loss 1.7196 Epoch: 16 Global Step: 162005 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:38:22,389-Speed 130.34 samples/sec Loss 1.6648 Epoch: 16 Global Step: 162010 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:38:42,110-Speed 129.82 samples/sec Loss 1.5348 Epoch: 16 Global Step: 162015 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:39:01,651-Speed 131.01 samples/sec Loss 1.6698 Epoch: 16 Global Step: 162020 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:39:21,262-Speed 130.55 samples/sec Loss 1.4744 Epoch: 16 Global Step: 162025 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:39:40,881-Speed 130.49 samples/sec Loss 1.6673 Epoch: 16 Global Step: 162030 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:40:00,671-Speed 129.36 samples/sec Loss 1.7074 Epoch: 16 Global Step: 162035 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:40:20,229-Speed 130.91 samples/sec Loss 1.8535 Epoch: 16 Global Step: 162040 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:40:40,265-Speed 127.77 samples/sec Loss 1.7562 Epoch: 16 Global Step: 162045 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:41:00,150-Speed 128.75 samples/sec Loss 1.5625 Epoch: 16 Global Step: 162050 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:41:19,671-Speed 131.15 samples/sec Loss 1.6980 Epoch: 16 Global Step: 162055 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:41:39,579-Speed 128.60 samples/sec Loss 1.7269 Epoch: 16 Global Step: 162060 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:41:59,124-Speed 130.98 samples/sec Loss 1.7629 Epoch: 16 Global Step: 162065 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:42:18,772-Speed 130.31 samples/sec Loss 1.5540 Epoch: 16 Global Step: 162070 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:42:38,473-Speed 129.95 samples/sec Loss 1.6019 Epoch: 16 Global Step: 162075 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:42:58,104-Speed 130.41 samples/sec Loss 1.6384 Epoch: 16 Global Step: 162080 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:43:17,719-Speed 130.52 samples/sec Loss 1.7306 Epoch: 16 Global Step: 162085 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:43:37,691-Speed 128.19 samples/sec Loss 1.6858 Epoch: 16 Global Step: 162090 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:43:57,455-Speed 129.54 samples/sec Loss 1.7395 Epoch: 16 Global Step: 162095 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:44:16,965-Speed 131.21 samples/sec Loss 1.7058 Epoch: 16 Global Step: 162100 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:44:36,840-Speed 128.81 samples/sec Loss 1.7634 Epoch: 16 Global Step: 162105 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:44:56,258-Speed 131.85 samples/sec Loss 1.7308 Epoch: 16 Global Step: 162110 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:45:16,276-Speed 127.89 samples/sec Loss 1.7289 Epoch: 16 Global Step: 162115 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:45:35,881-Speed 130.59 samples/sec Loss 1.5742 Epoch: 16 Global Step: 162120 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:45:55,644-Speed 129.55 samples/sec Loss 1.8405 Epoch: 16 Global Step: 162125 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:46:15,216-Speed 130.80 samples/sec Loss 1.7296 Epoch: 16 Global Step: 162130 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:46:35,205-Speed 128.07 samples/sec Loss 1.5664 Epoch: 16 Global Step: 162135 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:46:54,953-Speed 129.64 samples/sec Loss 1.7990 Epoch: 16 Global Step: 162140 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:47:14,509-Speed 130.92 samples/sec Loss 1.8727 Epoch: 16 Global Step: 162145 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:47:34,411-Speed 128.64 samples/sec Loss 1.6142 Epoch: 16 Global Step: 162150 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:47:53,944-Speed 131.07 samples/sec Loss 1.6213 Epoch: 16 Global Step: 162155 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:48:13,884-Speed 128.39 samples/sec Loss 1.6884 Epoch: 16 Global Step: 162160 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:48:33,405-Speed 131.15 samples/sec Loss 1.6832 Epoch: 16 Global Step: 162165 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:48:53,224-Speed 129.17 samples/sec Loss 1.6894 Epoch: 16 Global Step: 162170 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:49:12,736-Speed 131.21 samples/sec Loss 1.5999 Epoch: 16 Global Step: 162175 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:49:32,834-Speed 127.38 samples/sec Loss 1.7109 Epoch: 16 Global Step: 162180 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:49:52,562-Speed 129.77 samples/sec Loss 1.6947 Epoch: 16 Global Step: 162185 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:50:12,252-Speed 130.02 samples/sec Loss 1.6035 Epoch: 16 Global Step: 162190 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:50:31,991-Speed 129.70 samples/sec Loss 1.7154 Epoch: 16 Global Step: 162195 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:50:51,570-Speed 130.76 samples/sec Loss 1.7149 Epoch: 16 Global Step: 162200 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:51:11,524-Speed 128.30 samples/sec Loss 1.7130 Epoch: 16 Global Step: 162205 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:51:31,092-Speed 130.83 samples/sec Loss 1.6126 Epoch: 16 Global Step: 162210 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:51:50,953-Speed 128.90 samples/sec Loss 1.7301 Epoch: 16 Global Step: 162215 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:52:10,398-Speed 131.66 samples/sec Loss 1.6273 Epoch: 16 Global Step: 162220 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:52:30,488-Speed 127.43 samples/sec Loss 1.7697 Epoch: 16 Global Step: 162225 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:52:50,253-Speed 129.53 samples/sec Loss 1.7035 Epoch: 16 Global Step: 162230 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:53:09,882-Speed 130.43 samples/sec Loss 1.6582 Epoch: 16 Global Step: 162235 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:53:29,591-Speed 129.90 samples/sec Loss 1.5575 Epoch: 16 Global Step: 162240 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:53:49,133-Speed 131.01 samples/sec Loss 1.5778 Epoch: 16 Global Step: 162245 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:54:09,084-Speed 128.32 samples/sec Loss 1.6626 Epoch: 16 Global Step: 162250 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:54:28,636-Speed 130.94 samples/sec Loss 1.8143 Epoch: 16 Global Step: 162255 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:54:48,426-Speed 129.37 samples/sec Loss 1.8412 Epoch: 16 Global Step: 162260 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:55:07,986-Speed 130.89 samples/sec Loss 1.6351 Epoch: 16 Global Step: 162265 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:55:28,148-Speed 126.98 samples/sec Loss 1.7560 Epoch: 16 Global Step: 162270 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:55:48,283-Speed 127.14 samples/sec Loss 1.8329 Epoch: 16 Global Step: 162275 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:56:07,780-Speed 131.31 samples/sec Loss 1.7127 Epoch: 16 Global Step: 162280 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:56:27,550-Speed 129.50 samples/sec Loss 1.6817 Epoch: 16 Global Step: 162285 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:56:47,136-Speed 130.71 samples/sec Loss 1.6677 Epoch: 16 Global Step: 162290 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:57:06,958-Speed 129.15 samples/sec Loss 1.6569 Epoch: 16 Global Step: 162295 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:57:26,558-Speed 130.62 samples/sec Loss 1.5553 Epoch: 16 Global Step: 162300 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:57:46,176-Speed 130.50 samples/sec Loss 1.7482 Epoch: 16 Global Step: 162305 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:58:05,719-Speed 131.00 samples/sec Loss 1.7118 Epoch: 16 Global Step: 162310 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:58:25,760-Speed 127.75 samples/sec Loss 1.6690 Epoch: 16 Global Step: 162315 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:58:45,778-Speed 127.89 samples/sec Loss 1.7509 Epoch: 16 Global Step: 162320 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:59:05,309-Speed 131.08 samples/sec Loss 1.7873 Epoch: 16 Global Step: 162325 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:59:25,111-Speed 129.29 samples/sec Loss 1.4841 Epoch: 16 Global Step: 162330 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 18:59:44,718-Speed 130.57 samples/sec Loss 1.8244 Epoch: 16 Global Step: 162335 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:00:04,427-Speed 129.90 samples/sec Loss 1.6816 Epoch: 16 Global Step: 162340 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:00:24,034-Speed 130.57 samples/sec Loss 1.5633 Epoch: 16 Global Step: 162345 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:00:43,754-Speed 129.82 samples/sec Loss 1.5859 Epoch: 16 Global Step: 162350 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:01:03,313-Speed 130.89 samples/sec Loss 1.6514 Epoch: 16 Global Step: 162355 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:01:23,423-Speed 127.31 samples/sec Loss 1.5685 Epoch: 16 Global Step: 162360 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:01:43,335-Speed 128.57 samples/sec Loss 1.6618 Epoch: 16 Global Step: 162365 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:02:02,904-Speed 130.83 samples/sec Loss 1.5845 Epoch: 16 Global Step: 162370 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:02:22,733-Speed 129.11 samples/sec Loss 1.7818 Epoch: 16 Global Step: 162375 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:02:42,299-Speed 130.85 samples/sec Loss 1.5666 Epoch: 16 Global Step: 162380 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:03:02,082-Speed 129.41 samples/sec Loss 1.5412 Epoch: 16 Global Step: 162385 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:03:21,667-Speed 130.72 samples/sec Loss 1.7533 Epoch: 16 Global Step: 162390 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:03:41,371-Speed 129.93 samples/sec Loss 1.6660 Epoch: 16 Global Step: 162395 Fp16 Grad Scale: 8192 Required: 103 hours Training: 2021-06-14 19:04:01,001-Speed 130.42 samples/sec Loss 1.6489 Epoch: 16 Global Step: 162400 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:04:20,938-Speed 128.41 samples/sec Loss 1.6130 Epoch: 16 Global Step: 162405 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:04:40,877-Speed 128.40 samples/sec Loss 1.5455 Epoch: 16 Global Step: 162410 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:05:00,474-Speed 130.64 samples/sec Loss 1.7348 Epoch: 16 Global Step: 162415 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:05:20,376-Speed 128.64 samples/sec Loss 1.6675 Epoch: 16 Global Step: 162420 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:05:39,916-Speed 131.02 samples/sec Loss 1.6745 Epoch: 16 Global Step: 162425 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:05:59,700-Speed 129.41 samples/sec Loss 1.6918 Epoch: 16 Global Step: 162430 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:06:19,297-Speed 130.64 samples/sec Loss 1.6726 Epoch: 16 Global Step: 162435 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:06:39,096-Speed 129.31 samples/sec Loss 1.7566 Epoch: 16 Global Step: 162440 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:06:58,710-Speed 130.52 samples/sec Loss 1.7086 Epoch: 16 Global Step: 162445 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:07:18,812-Speed 127.36 samples/sec Loss 1.6695 Epoch: 16 Global Step: 162450 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:07:38,533-Speed 129.81 samples/sec Loss 1.7902 Epoch: 16 Global Step: 162455 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:07:58,065-Speed 131.08 samples/sec Loss 1.5765 Epoch: 16 Global Step: 162460 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:08:17,989-Speed 128.50 samples/sec Loss 1.6548 Epoch: 16 Global Step: 162465 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:08:37,533-Speed 130.99 samples/sec Loss 1.7259 Epoch: 16 Global Step: 162470 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:08:57,324-Speed 129.36 samples/sec Loss 1.7752 Epoch: 16 Global Step: 162475 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:09:16,859-Speed 131.06 samples/sec Loss 1.7615 Epoch: 16 Global Step: 162480 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:09:36,635-Speed 129.46 samples/sec Loss 1.7396 Epoch: 16 Global Step: 162485 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:09:56,203-Speed 130.83 samples/sec Loss 1.8272 Epoch: 16 Global Step: 162490 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:10:16,239-Speed 127.78 samples/sec Loss 1.5192 Epoch: 16 Global Step: 162495 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:10:35,951-Speed 129.88 samples/sec Loss 1.7692 Epoch: 16 Global Step: 162500 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:10:55,567-Speed 130.51 samples/sec Loss 1.6463 Epoch: 16 Global Step: 162505 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:11:15,280-Speed 129.87 samples/sec Loss 1.7216 Epoch: 16 Global Step: 162510 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:11:34,709-Speed 131.77 samples/sec Loss 1.5845 Epoch: 16 Global Step: 162515 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:11:54,623-Speed 128.56 samples/sec Loss 1.6235 Epoch: 16 Global Step: 162520 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:12:14,212-Speed 130.69 samples/sec Loss 1.7563 Epoch: 16 Global Step: 162525 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:12:34,167-Speed 128.30 samples/sec Loss 1.8681 Epoch: 16 Global Step: 162530 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:12:53,726-Speed 130.90 samples/sec Loss 1.7060 Epoch: 16 Global Step: 162535 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:13:13,696-Speed 128.20 samples/sec Loss 1.5286 Epoch: 16 Global Step: 162540 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:13:33,427-Speed 129.75 samples/sec Loss 1.6405 Epoch: 16 Global Step: 162545 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:13:53,063-Speed 130.38 samples/sec Loss 1.6954 Epoch: 16 Global Step: 162550 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:14:12,765-Speed 129.94 samples/sec Loss 1.6990 Epoch: 16 Global Step: 162555 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:14:32,362-Speed 130.63 samples/sec Loss 1.5960 Epoch: 16 Global Step: 162560 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:14:52,079-Speed 129.85 samples/sec Loss 1.6574 Epoch: 16 Global Step: 162565 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:15:11,550-Speed 131.48 samples/sec Loss 1.5633 Epoch: 16 Global Step: 162570 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:15:31,448-Speed 128.67 samples/sec Loss 1.5724 Epoch: 16 Global Step: 162575 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:15:50,989-Speed 131.01 samples/sec Loss 1.7258 Epoch: 16 Global Step: 162580 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:16:11,065-Speed 127.52 samples/sec Loss 1.6203 Epoch: 16 Global Step: 162585 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:16:30,941-Speed 128.81 samples/sec Loss 1.5737 Epoch: 16 Global Step: 162590 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:16:50,510-Speed 130.82 samples/sec Loss 1.6128 Epoch: 16 Global Step: 162595 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:17:10,161-Speed 130.28 samples/sec Loss 1.5913 Epoch: 16 Global Step: 162600 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:17:29,749-Speed 130.70 samples/sec Loss 1.5717 Epoch: 16 Global Step: 162605 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:17:49,627-Speed 128.79 samples/sec Loss 1.7514 Epoch: 16 Global Step: 162610 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:18:09,392-Speed 129.53 samples/sec Loss 1.5042 Epoch: 16 Global Step: 162615 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:18:29,131-Speed 129.70 samples/sec Loss 1.4785 Epoch: 16 Global Step: 162620 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:18:48,640-Speed 131.23 samples/sec Loss 1.7050 Epoch: 16 Global Step: 162625 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:19:08,717-Speed 127.52 samples/sec Loss 1.6488 Epoch: 16 Global Step: 162630 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:19:28,665-Speed 128.34 samples/sec Loss 1.6598 Epoch: 16 Global Step: 162635 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:19:48,194-Speed 131.09 samples/sec Loss 1.6170 Epoch: 16 Global Step: 162640 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:20:08,104-Speed 128.59 samples/sec Loss 1.6087 Epoch: 16 Global Step: 162645 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:20:27,657-Speed 130.93 samples/sec Loss 1.5473 Epoch: 16 Global Step: 162650 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:20:47,398-Speed 129.69 samples/sec Loss 1.6271 Epoch: 16 Global Step: 162655 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:21:07,405-Speed 127.96 samples/sec Loss 1.6529 Epoch: 16 Global Step: 162660 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:21:27,068-Speed 130.20 samples/sec Loss 1.4549 Epoch: 16 Global Step: 162665 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:21:46,660-Speed 130.68 samples/sec Loss 1.6216 Epoch: 16 Global Step: 162670 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:22:06,708-Speed 127.70 samples/sec Loss 1.7630 Epoch: 16 Global Step: 162675 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:22:26,504-Speed 129.33 samples/sec Loss 1.5770 Epoch: 16 Global Step: 162680 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:22:46,041-Speed 131.04 samples/sec Loss 1.6713 Epoch: 16 Global Step: 162685 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:23:05,837-Speed 129.33 samples/sec Loss 1.6201 Epoch: 16 Global Step: 162690 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:23:25,323-Speed 131.39 samples/sec Loss 1.6326 Epoch: 16 Global Step: 162695 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:23:45,138-Speed 129.20 samples/sec Loss 1.5311 Epoch: 16 Global Step: 162700 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:24:04,741-Speed 130.60 samples/sec Loss 1.7975 Epoch: 16 Global Step: 162705 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:24:24,534-Speed 129.35 samples/sec Loss 1.5778 Epoch: 16 Global Step: 162710 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:24:44,088-Speed 130.93 samples/sec Loss 1.6883 Epoch: 16 Global Step: 162715 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:25:04,052-Speed 128.23 samples/sec Loss 1.7169 Epoch: 16 Global Step: 162720 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:25:23,795-Speed 129.68 samples/sec Loss 1.6582 Epoch: 16 Global Step: 162725 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:25:43,414-Speed 130.49 samples/sec Loss 1.6979 Epoch: 16 Global Step: 162730 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:26:03,134-Speed 129.83 samples/sec Loss 1.7753 Epoch: 16 Global Step: 162735 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:26:22,746-Speed 130.54 samples/sec Loss 1.7857 Epoch: 16 Global Step: 162740 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:26:42,476-Speed 129.75 samples/sec Loss 1.4853 Epoch: 16 Global Step: 162745 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:27:02,011-Speed 131.06 samples/sec Loss 1.6549 Epoch: 16 Global Step: 162750 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:27:21,769-Speed 129.57 samples/sec Loss 1.5872 Epoch: 16 Global Step: 162755 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:27:41,344-Speed 130.79 samples/sec Loss 1.4072 Epoch: 16 Global Step: 162760 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:28:01,325-Speed 128.13 samples/sec Loss 1.5587 Epoch: 16 Global Step: 162765 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:28:21,133-Speed 129.25 samples/sec Loss 1.6012 Epoch: 16 Global Step: 162770 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:28:40,712-Speed 130.75 samples/sec Loss 1.5970 Epoch: 16 Global Step: 162775 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:29:00,389-Speed 130.11 samples/sec Loss 1.6893 Epoch: 16 Global Step: 162780 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:29:20,004-Speed 130.52 samples/sec Loss 1.6155 Epoch: 16 Global Step: 162785 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:29:39,795-Speed 129.35 samples/sec Loss 1.6855 Epoch: 16 Global Step: 162790 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:29:59,262-Speed 131.51 samples/sec Loss 1.5488 Epoch: 16 Global Step: 162795 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:30:19,112-Speed 128.98 samples/sec Loss 1.6221 Epoch: 16 Global Step: 162800 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:30:38,577-Speed 131.53 samples/sec Loss 1.7151 Epoch: 16 Global Step: 162805 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:30:58,706-Speed 127.19 samples/sec Loss 1.6566 Epoch: 16 Global Step: 162810 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:31:18,682-Speed 128.16 samples/sec Loss 1.5874 Epoch: 16 Global Step: 162815 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:31:38,232-Speed 130.95 samples/sec Loss 1.7880 Epoch: 16 Global Step: 162820 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:31:57,891-Speed 130.22 samples/sec Loss 1.7350 Epoch: 16 Global Step: 162825 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:32:17,468-Speed 130.77 samples/sec Loss 1.7493 Epoch: 16 Global Step: 162830 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:32:37,248-Speed 129.43 samples/sec Loss 1.6595 Epoch: 16 Global Step: 162835 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:32:56,855-Speed 130.57 samples/sec Loss 1.7282 Epoch: 16 Global Step: 162840 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:33:16,609-Speed 129.60 samples/sec Loss 1.6925 Epoch: 16 Global Step: 162845 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:33:36,157-Speed 130.97 samples/sec Loss 1.5790 Epoch: 16 Global Step: 162850 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:33:56,215-Speed 127.64 samples/sec Loss 1.6877 Epoch: 16 Global Step: 162855 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:34:16,183-Speed 128.21 samples/sec Loss 1.5837 Epoch: 16 Global Step: 162860 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:34:35,738-Speed 130.92 samples/sec Loss 1.6473 Epoch: 16 Global Step: 162865 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:34:55,482-Speed 129.67 samples/sec Loss 1.5839 Epoch: 16 Global Step: 162870 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:35:15,051-Speed 130.83 samples/sec Loss 1.6554 Epoch: 16 Global Step: 162875 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:35:34,753-Speed 129.94 samples/sec Loss 1.6939 Epoch: 16 Global Step: 162880 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:35:54,362-Speed 130.57 samples/sec Loss 1.6725 Epoch: 16 Global Step: 162885 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:36:14,085-Speed 129.80 samples/sec Loss 1.4739 Epoch: 16 Global Step: 162890 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:36:33,706-Speed 130.48 samples/sec Loss 1.6851 Epoch: 16 Global Step: 162895 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:36:53,617-Speed 128.57 samples/sec Loss 1.6193 Epoch: 16 Global Step: 162900 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:37:13,506-Speed 128.72 samples/sec Loss 1.6040 Epoch: 16 Global Step: 162905 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:37:33,179-Speed 130.14 samples/sec Loss 1.7683 Epoch: 16 Global Step: 162910 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:37:53,088-Speed 128.60 samples/sec Loss 1.5478 Epoch: 16 Global Step: 162915 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:38:12,688-Speed 130.62 samples/sec Loss 1.6078 Epoch: 16 Global Step: 162920 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:38:32,470-Speed 129.42 samples/sec Loss 1.5812 Epoch: 16 Global Step: 162925 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:38:52,012-Speed 131.00 samples/sec Loss 1.6268 Epoch: 16 Global Step: 162930 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:39:11,764-Speed 129.62 samples/sec Loss 1.5136 Epoch: 16 Global Step: 162935 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:39:31,340-Speed 130.78 samples/sec Loss 1.7462 Epoch: 16 Global Step: 162940 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:39:51,520-Speed 126.86 samples/sec Loss 1.6595 Epoch: 16 Global Step: 162945 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:40:11,265-Speed 129.66 samples/sec Loss 1.6630 Epoch: 16 Global Step: 162950 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:40:30,815-Speed 130.95 samples/sec Loss 1.7310 Epoch: 16 Global Step: 162955 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:40:50,565-Speed 129.63 samples/sec Loss 1.7370 Epoch: 16 Global Step: 162960 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:41:10,176-Speed 130.55 samples/sec Loss 1.5112 Epoch: 16 Global Step: 162965 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:41:29,943-Speed 129.52 samples/sec Loss 1.6712 Epoch: 16 Global Step: 162970 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:41:49,487-Speed 131.00 samples/sec Loss 1.4601 Epoch: 16 Global Step: 162975 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:42:09,381-Speed 128.69 samples/sec Loss 1.7452 Epoch: 16 Global Step: 162980 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:42:28,974-Speed 130.66 samples/sec Loss 1.6225 Epoch: 16 Global Step: 162985 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:42:48,949-Speed 128.17 samples/sec Loss 1.6917 Epoch: 16 Global Step: 162990 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:43:08,657-Speed 129.90 samples/sec Loss 1.6064 Epoch: 16 Global Step: 162995 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:43:28,278-Speed 130.47 samples/sec Loss 1.4451 Epoch: 16 Global Step: 163000 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:43:48,056-Speed 129.44 samples/sec Loss 1.7928 Epoch: 16 Global Step: 163005 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:44:07,705-Speed 130.30 samples/sec Loss 1.6471 Epoch: 16 Global Step: 163010 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:44:27,501-Speed 129.33 samples/sec Loss 1.6622 Epoch: 16 Global Step: 163015 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:44:47,039-Speed 131.03 samples/sec Loss 1.4589 Epoch: 16 Global Step: 163020 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:45:06,888-Speed 128.98 samples/sec Loss 1.5440 Epoch: 16 Global Step: 163025 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:45:26,606-Speed 129.84 samples/sec Loss 1.6592 Epoch: 16 Global Step: 163030 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:45:46,578-Speed 128.19 samples/sec Loss 1.7251 Epoch: 16 Global Step: 163035 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:46:06,533-Speed 128.29 samples/sec Loss 1.4593 Epoch: 16 Global Step: 163040 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:46:25,618-Speed 134.15 samples/sec Loss 1.5304 Epoch: 16 Global Step: 163045 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:46:45,223-Speed 130.58 samples/sec Loss 1.6838 Epoch: 16 Global Step: 163050 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:47:04,796-Speed 130.80 samples/sec Loss 1.7491 Epoch: 16 Global Step: 163055 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:47:24,338-Speed 131.01 samples/sec Loss 1.6723 Epoch: 16 Global Step: 163060 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:47:43,938-Speed 130.62 samples/sec Loss 1.5965 Epoch: 16 Global Step: 163065 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:48:03,562-Speed 130.46 samples/sec Loss 1.6277 Epoch: 16 Global Step: 163070 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:48:23,266-Speed 129.93 samples/sec Loss 1.5502 Epoch: 16 Global Step: 163075 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:48:42,803-Speed 131.04 samples/sec Loss 1.6027 Epoch: 16 Global Step: 163080 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:49:02,539-Speed 129.72 samples/sec Loss 1.6041 Epoch: 16 Global Step: 163085 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:49:22,013-Speed 131.47 samples/sec Loss 1.8059 Epoch: 16 Global Step: 163090 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:49:41,571-Speed 130.90 samples/sec Loss 1.7321 Epoch: 16 Global Step: 163095 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:50:01,337-Speed 129.52 samples/sec Loss 1.6239 Epoch: 16 Global Step: 163100 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:50:21,048-Speed 129.88 samples/sec Loss 1.6894 Epoch: 16 Global Step: 163105 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:50:40,468-Speed 131.83 samples/sec Loss 1.6980 Epoch: 16 Global Step: 163110 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:51:00,089-Speed 130.48 samples/sec Loss 1.6499 Epoch: 16 Global Step: 163115 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:51:19,611-Speed 131.14 samples/sec Loss 1.5781 Epoch: 16 Global Step: 163120 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:51:39,148-Speed 131.04 samples/sec Loss 1.6100 Epoch: 16 Global Step: 163125 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:51:58,689-Speed 131.01 samples/sec Loss 1.5451 Epoch: 16 Global Step: 163130 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:52:18,224-Speed 131.05 samples/sec Loss 1.7277 Epoch: 16 Global Step: 163135 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:52:37,792-Speed 130.83 samples/sec Loss 1.5216 Epoch: 16 Global Step: 163140 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:52:57,702-Speed 128.59 samples/sec Loss 1.6551 Epoch: 16 Global Step: 163145 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:53:17,123-Speed 131.82 samples/sec Loss 1.5033 Epoch: 16 Global Step: 163150 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:53:36,841-Speed 129.84 samples/sec Loss 1.5967 Epoch: 16 Global Step: 163155 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:53:56,434-Speed 130.67 samples/sec Loss 1.5027 Epoch: 16 Global Step: 163160 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:54:16,107-Speed 130.13 samples/sec Loss 1.6960 Epoch: 16 Global Step: 163165 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:54:36,126-Speed 127.88 samples/sec Loss 1.6321 Epoch: 16 Global Step: 163170 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:54:55,579-Speed 131.61 samples/sec Loss 1.6270 Epoch: 16 Global Step: 163175 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:55:15,206-Speed 130.44 samples/sec Loss 1.7647 Epoch: 16 Global Step: 163180 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:55:34,733-Speed 131.11 samples/sec Loss 1.7574 Epoch: 16 Global Step: 163185 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:55:54,642-Speed 128.59 samples/sec Loss 1.6655 Epoch: 16 Global Step: 163190 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:56:14,080-Speed 131.71 samples/sec Loss 1.7221 Epoch: 16 Global Step: 163195 Fp16 Grad Scale: 8192 Required: 102 hours Training: 2021-06-14 19:56:33,283-Speed 133.32 samples/sec Loss 1.5317 Epoch: 16 Global Step: 163200 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:56:56,996-Speed 107.96 samples/sec Loss 1.5085 Epoch: 16 Global Step: 163205 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:57:16,555-Speed 130.89 samples/sec Loss 1.8170 Epoch: 16 Global Step: 163210 Fp16 Grad Scale: 4096 Required: 102 hours Training: 2021-06-14 19:57:36,133-Speed 130.77 samples/sec Loss 1.6534 Epoch: 16 Global Step: 163215 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 19:57:55,736-Speed 130.60 samples/sec Loss 1.7195 Epoch: 16 Global Step: 163220 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 19:58:15,458-Speed 129.81 samples/sec Loss 1.6753 Epoch: 16 Global Step: 163225 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 19:58:34,983-Speed 131.12 samples/sec Loss 1.5802 Epoch: 16 Global Step: 163230 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 19:58:54,861-Speed 128.79 samples/sec Loss 1.6172 Epoch: 16 Global Step: 163235 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 19:59:14,290-Speed 131.77 samples/sec Loss 1.5850 Epoch: 16 Global Step: 163240 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 19:59:34,076-Speed 129.40 samples/sec Loss 1.6437 Epoch: 16 Global Step: 163245 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 19:59:53,609-Speed 131.07 samples/sec Loss 1.6326 Epoch: 16 Global Step: 163250 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:00:13,125-Speed 131.18 samples/sec Loss 1.6475 Epoch: 16 Global Step: 163255 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:00:32,688-Speed 130.87 samples/sec Loss 1.5779 Epoch: 16 Global Step: 163260 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:00:52,282-Speed 130.66 samples/sec Loss 1.5326 Epoch: 16 Global Step: 163265 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:01:11,879-Speed 130.64 samples/sec Loss 1.5553 Epoch: 16 Global Step: 163270 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:01:31,425-Speed 130.98 samples/sec Loss 1.6018 Epoch: 16 Global Step: 163275 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:01:51,279-Speed 128.95 samples/sec Loss 1.7571 Epoch: 16 Global Step: 163280 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:02:10,717-Speed 131.70 samples/sec Loss 1.6831 Epoch: 16 Global Step: 163285 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:02:30,379-Speed 130.21 samples/sec Loss 1.6027 Epoch: 16 Global Step: 163290 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:02:49,948-Speed 130.83 samples/sec Loss 1.5950 Epoch: 16 Global Step: 163295 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:03:09,486-Speed 131.03 samples/sec Loss 1.5345 Epoch: 16 Global Step: 163300 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:03:29,193-Speed 129.91 samples/sec Loss 1.6277 Epoch: 16 Global Step: 163305 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:03:48,783-Speed 130.68 samples/sec Loss 1.4901 Epoch: 16 Global Step: 163310 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:04:08,387-Speed 130.60 samples/sec Loss 1.6424 Epoch: 16 Global Step: 163315 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:04:27,929-Speed 131.00 samples/sec Loss 1.5658 Epoch: 16 Global Step: 163320 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:04:47,802-Speed 128.83 samples/sec Loss 1.7076 Epoch: 16 Global Step: 163325 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:05:07,278-Speed 131.45 samples/sec Loss 1.5311 Epoch: 16 Global Step: 163330 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:05:27,353-Speed 127.52 samples/sec Loss 1.5937 Epoch: 16 Global Step: 163335 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:05:46,913-Speed 130.89 samples/sec Loss 1.4830 Epoch: 16 Global Step: 163340 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:06:06,413-Speed 131.29 samples/sec Loss 1.6275 Epoch: 16 Global Step: 163345 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:06:25,977-Speed 130.86 samples/sec Loss 1.5910 Epoch: 16 Global Step: 163350 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:06:45,583-Speed 130.58 samples/sec Loss 1.5893 Epoch: 16 Global Step: 163355 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:07:05,170-Speed 130.71 samples/sec Loss 1.7006 Epoch: 16 Global Step: 163360 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:07:24,946-Speed 129.46 samples/sec Loss 1.6504 Epoch: 16 Global Step: 163365 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:07:44,790-Speed 129.01 samples/sec Loss 1.6180 Epoch: 16 Global Step: 163370 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:08:04,259-Speed 131.50 samples/sec Loss 1.6045 Epoch: 16 Global Step: 163375 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:08:23,925-Speed 130.18 samples/sec Loss 1.6568 Epoch: 16 Global Step: 163380 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:08:43,451-Speed 131.12 samples/sec Loss 1.6900 Epoch: 16 Global Step: 163385 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:09:02,940-Speed 131.36 samples/sec Loss 1.7572 Epoch: 16 Global Step: 163390 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:09:22,524-Speed 130.73 samples/sec Loss 1.5922 Epoch: 16 Global Step: 163395 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:09:42,047-Speed 131.13 samples/sec Loss 1.6091 Epoch: 16 Global Step: 163400 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:10:01,655-Speed 130.57 samples/sec Loss 1.5959 Epoch: 16 Global Step: 163405 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:10:21,373-Speed 129.84 samples/sec Loss 1.6092 Epoch: 16 Global Step: 163410 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:10:41,247-Speed 128.82 samples/sec Loss 1.6586 Epoch: 16 Global Step: 163415 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:11:00,690-Speed 131.67 samples/sec Loss 1.6198 Epoch: 16 Global Step: 163420 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:11:20,399-Speed 129.90 samples/sec Loss 1.4553 Epoch: 16 Global Step: 163425 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:11:39,926-Speed 131.11 samples/sec Loss 1.7166 Epoch: 16 Global Step: 163430 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:11:59,449-Speed 131.14 samples/sec Loss 1.5630 Epoch: 16 Global Step: 163435 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:12:19,044-Speed 130.65 samples/sec Loss 1.6478 Epoch: 16 Global Step: 163440 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:12:38,714-Speed 130.16 samples/sec Loss 1.6559 Epoch: 16 Global Step: 163445 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:12:58,312-Speed 130.63 samples/sec Loss 1.6309 Epoch: 16 Global Step: 163450 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:13:17,780-Speed 131.51 samples/sec Loss 1.5997 Epoch: 16 Global Step: 163455 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:13:37,610-Speed 129.10 samples/sec Loss 1.8540 Epoch: 16 Global Step: 163460 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:13:57,369-Speed 129.57 samples/sec Loss 1.7400 Epoch: 16 Global Step: 163465 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:14:16,911-Speed 131.01 samples/sec Loss 1.7146 Epoch: 16 Global Step: 163470 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:14:36,452-Speed 131.01 samples/sec Loss 1.6839 Epoch: 16 Global Step: 163475 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:14:56,020-Speed 130.84 samples/sec Loss 1.6166 Epoch: 16 Global Step: 163480 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:15:15,635-Speed 130.52 samples/sec Loss 1.5294 Epoch: 16 Global Step: 163485 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:15:35,254-Speed 130.49 samples/sec Loss 1.7113 Epoch: 16 Global Step: 163490 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:15:54,785-Speed 131.08 samples/sec Loss 1.7046 Epoch: 16 Global Step: 163495 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:16:14,309-Speed 131.13 samples/sec Loss 1.5146 Epoch: 16 Global Step: 163500 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:16:34,126-Speed 129.19 samples/sec Loss 1.5615 Epoch: 16 Global Step: 163505 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:16:53,858-Speed 129.75 samples/sec Loss 1.6011 Epoch: 16 Global Step: 163510 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:17:13,501-Speed 130.33 samples/sec Loss 1.5795 Epoch: 16 Global Step: 163515 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:17:33,030-Speed 131.10 samples/sec Loss 1.6458 Epoch: 16 Global Step: 163520 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:17:53,165-Speed 127.15 samples/sec Loss 1.6944 Epoch: 16 Global Step: 163525 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:18:12,707-Speed 131.01 samples/sec Loss 1.5732 Epoch: 16 Global Step: 163530 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:18:32,289-Speed 130.74 samples/sec Loss 1.6006 Epoch: 16 Global Step: 163535 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:18:51,817-Speed 131.10 samples/sec Loss 1.5972 Epoch: 16 Global Step: 163540 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:19:11,316-Speed 131.30 samples/sec Loss 1.6286 Epoch: 16 Global Step: 163545 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:19:31,287-Speed 128.19 samples/sec Loss 1.8266 Epoch: 16 Global Step: 163550 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:19:51,014-Speed 129.78 samples/sec Loss 1.6503 Epoch: 16 Global Step: 163555 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:20:10,647-Speed 130.40 samples/sec Loss 1.6932 Epoch: 16 Global Step: 163560 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:20:30,171-Speed 131.13 samples/sec Loss 1.6844 Epoch: 16 Global Step: 163565 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:20:49,774-Speed 130.60 samples/sec Loss 1.5064 Epoch: 16 Global Step: 163570 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:21:09,323-Speed 130.96 samples/sec Loss 1.5357 Epoch: 16 Global Step: 163575 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:21:28,934-Speed 130.55 samples/sec Loss 1.7029 Epoch: 16 Global Step: 163580 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:21:48,465-Speed 131.08 samples/sec Loss 1.6993 Epoch: 16 Global Step: 163585 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:22:08,258-Speed 129.34 samples/sec Loss 1.5022 Epoch: 16 Global Step: 163590 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:22:28,177-Speed 128.53 samples/sec Loss 1.4523 Epoch: 16 Global Step: 163595 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:22:47,669-Speed 131.34 samples/sec Loss 1.5099 Epoch: 16 Global Step: 163600 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:23:07,158-Speed 131.36 samples/sec Loss 1.7311 Epoch: 16 Global Step: 163605 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:23:26,763-Speed 130.59 samples/sec Loss 1.7253 Epoch: 16 Global Step: 163610 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:23:46,356-Speed 130.67 samples/sec Loss 1.5704 Epoch: 16 Global Step: 163615 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:24:05,960-Speed 130.59 samples/sec Loss 1.6078 Epoch: 16 Global Step: 163620 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:24:25,451-Speed 131.35 samples/sec Loss 1.5396 Epoch: 16 Global Step: 163625 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:24:44,914-Speed 131.54 samples/sec Loss 1.5867 Epoch: 16 Global Step: 163630 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:25:04,483-Speed 130.83 samples/sec Loss 1.6337 Epoch: 16 Global Step: 163635 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:25:24,396-Speed 128.57 samples/sec Loss 1.6967 Epoch: 16 Global Step: 163640 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:25:43,900-Speed 131.26 samples/sec Loss 1.5191 Epoch: 16 Global Step: 163645 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:26:03,362-Speed 131.55 samples/sec Loss 1.6092 Epoch: 16 Global Step: 163650 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:26:22,957-Speed 130.65 samples/sec Loss 1.5167 Epoch: 16 Global Step: 163655 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:26:42,575-Speed 130.50 samples/sec Loss 1.7039 Epoch: 16 Global Step: 163660 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:27:02,197-Speed 130.48 samples/sec Loss 1.5283 Epoch: 16 Global Step: 163665 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:27:21,691-Speed 131.33 samples/sec Loss 1.6956 Epoch: 16 Global Step: 163670 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:27:41,284-Speed 130.66 samples/sec Loss 1.6381 Epoch: 16 Global Step: 163675 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:28:00,812-Speed 131.10 samples/sec Loss 1.4315 Epoch: 16 Global Step: 163680 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:28:20,719-Speed 128.60 samples/sec Loss 1.6776 Epoch: 16 Global Step: 163685 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:28:40,151-Speed 131.75 samples/sec Loss 1.5096 Epoch: 16 Global Step: 163690 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:28:59,762-Speed 130.55 samples/sec Loss 1.5045 Epoch: 16 Global Step: 163695 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:29:19,603-Speed 129.03 samples/sec Loss 1.5870 Epoch: 16 Global Step: 163700 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:29:39,162-Speed 130.89 samples/sec Loss 1.4804 Epoch: 16 Global Step: 163705 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:29:58,706-Speed 130.99 samples/sec Loss 1.5500 Epoch: 16 Global Step: 163710 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:30:18,245-Speed 131.03 samples/sec Loss 1.5384 Epoch: 16 Global Step: 163715 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:30:37,844-Speed 130.63 samples/sec Loss 1.7219 Epoch: 16 Global Step: 163720 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:30:57,497-Speed 130.27 samples/sec Loss 1.5867 Epoch: 16 Global Step: 163725 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:31:18,162-Speed 123.89 samples/sec Loss 1.7623 Epoch: 16 Global Step: 163730 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:31:37,766-Speed 130.59 samples/sec Loss 1.5068 Epoch: 16 Global Step: 163735 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:31:57,300-Speed 131.06 samples/sec Loss 1.7127 Epoch: 16 Global Step: 163740 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:32:16,923-Speed 130.47 samples/sec Loss 1.6004 Epoch: 16 Global Step: 163745 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:32:36,492-Speed 130.82 samples/sec Loss 1.4734 Epoch: 16 Global Step: 163750 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:32:55,927-Speed 131.74 samples/sec Loss 1.6565 Epoch: 16 Global Step: 163755 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:33:15,536-Speed 130.56 samples/sec Loss 1.4848 Epoch: 16 Global Step: 163760 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:33:35,076-Speed 131.02 samples/sec Loss 1.4943 Epoch: 16 Global Step: 163765 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:33:54,623-Speed 130.97 samples/sec Loss 1.5925 Epoch: 16 Global Step: 163770 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:34:14,404-Speed 129.43 samples/sec Loss 1.7037 Epoch: 16 Global Step: 163775 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:34:34,004-Speed 130.62 samples/sec Loss 1.5879 Epoch: 16 Global Step: 163780 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:34:53,549-Speed 130.98 samples/sec Loss 1.5373 Epoch: 16 Global Step: 163785 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:35:13,148-Speed 130.63 samples/sec Loss 1.7760 Epoch: 16 Global Step: 163790 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:35:32,716-Speed 130.83 samples/sec Loss 1.6473 Epoch: 16 Global Step: 163795 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:35:52,120-Speed 131.94 samples/sec Loss 1.6888 Epoch: 16 Global Step: 163800 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:36:11,657-Speed 131.05 samples/sec Loss 1.5463 Epoch: 16 Global Step: 163805 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:36:31,190-Speed 131.06 samples/sec Loss 1.5781 Epoch: 16 Global Step: 163810 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:36:50,745-Speed 130.92 samples/sec Loss 1.4789 Epoch: 16 Global Step: 163815 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:37:10,545-Speed 129.31 samples/sec Loss 1.6250 Epoch: 16 Global Step: 163820 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:37:29,993-Speed 131.64 samples/sec Loss 1.7050 Epoch: 16 Global Step: 163825 Fp16 Grad Scale: 4096 Required: 101 hours Training: 2021-06-14 20:37:49,622-Speed 130.43 samples/sec Loss 1.5457 Epoch: 16 Global Step: 163830 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:38:09,186-Speed 130.86 samples/sec Loss 1.5580 Epoch: 16 Global Step: 163835 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:38:28,804-Speed 130.50 samples/sec Loss 1.6260 Epoch: 16 Global Step: 163840 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:38:48,264-Speed 131.56 samples/sec Loss 1.4987 Epoch: 16 Global Step: 163845 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:39:07,802-Speed 131.03 samples/sec Loss 1.5709 Epoch: 16 Global Step: 163850 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:39:27,376-Speed 130.79 samples/sec Loss 1.5977 Epoch: 16 Global Step: 163855 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:39:46,939-Speed 130.87 samples/sec Loss 1.5464 Epoch: 16 Global Step: 163860 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:40:06,732-Speed 129.35 samples/sec Loss 1.5564 Epoch: 16 Global Step: 163865 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:40:26,235-Speed 131.27 samples/sec Loss 1.6306 Epoch: 16 Global Step: 163870 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:40:45,800-Speed 130.85 samples/sec Loss 1.6623 Epoch: 16 Global Step: 163875 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:41:05,361-Speed 130.88 samples/sec Loss 1.5706 Epoch: 16 Global Step: 163880 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:41:24,974-Speed 130.53 samples/sec Loss 1.5563 Epoch: 16 Global Step: 163885 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:41:44,525-Speed 130.95 samples/sec Loss 1.7013 Epoch: 16 Global Step: 163890 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:42:04,128-Speed 130.60 samples/sec Loss 1.4617 Epoch: 16 Global Step: 163895 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:42:23,730-Speed 130.61 samples/sec Loss 1.4789 Epoch: 16 Global Step: 163900 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:42:43,266-Speed 131.04 samples/sec Loss 1.7221 Epoch: 16 Global Step: 163905 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:43:03,052-Speed 129.40 samples/sec Loss 1.6555 Epoch: 16 Global Step: 163910 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:43:22,673-Speed 130.48 samples/sec Loss 1.5835 Epoch: 16 Global Step: 163915 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:43:42,280-Speed 130.57 samples/sec Loss 1.4511 Epoch: 16 Global Step: 163920 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:44:01,904-Speed 130.46 samples/sec Loss 1.5711 Epoch: 16 Global Step: 163925 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:44:21,350-Speed 131.65 samples/sec Loss 1.6617 Epoch: 16 Global Step: 163930 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:44:40,882-Speed 131.08 samples/sec Loss 1.6065 Epoch: 16 Global Step: 163935 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:45:00,510-Speed 130.44 samples/sec Loss 1.5578 Epoch: 16 Global Step: 163940 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:45:20,177-Speed 130.17 samples/sec Loss 1.5108 Epoch: 16 Global Step: 163945 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:45:39,778-Speed 130.62 samples/sec Loss 1.5433 Epoch: 16 Global Step: 163950 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:45:59,439-Speed 130.21 samples/sec Loss 1.6045 Epoch: 16 Global Step: 163955 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:46:19,091-Speed 130.27 samples/sec Loss 1.5901 Epoch: 16 Global Step: 163960 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:46:38,671-Speed 130.75 samples/sec Loss 1.5398 Epoch: 16 Global Step: 163965 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:46:58,448-Speed 129.45 samples/sec Loss 1.5335 Epoch: 16 Global Step: 163970 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:47:17,917-Speed 131.49 samples/sec Loss 1.5251 Epoch: 16 Global Step: 163975 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:47:37,452-Speed 131.06 samples/sec Loss 1.5422 Epoch: 16 Global Step: 163980 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:47:57,560-Speed 127.32 samples/sec Loss 1.6460 Epoch: 16 Global Step: 163985 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:48:17,136-Speed 130.78 samples/sec Loss 1.6295 Epoch: 16 Global Step: 163990 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:48:36,757-Speed 130.48 samples/sec Loss 1.6633 Epoch: 16 Global Step: 163995 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:48:56,435-Speed 130.10 samples/sec Loss 1.5316 Epoch: 16 Global Step: 164000 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:49:16,086-Speed 130.28 samples/sec Loss 1.7025 Epoch: 16 Global Step: 164005 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:49:35,595-Speed 131.23 samples/sec Loss 1.6047 Epoch: 16 Global Step: 164010 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:49:55,188-Speed 130.66 samples/sec Loss 1.8083 Epoch: 16 Global Step: 164015 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:50:14,646-Speed 131.57 samples/sec Loss 1.4758 Epoch: 16 Global Step: 164020 Fp16 Grad Scale: 8192 Required: 101 hours Training: 2021-06-14 20:50:34,232-Speed 130.71 samples/sec Loss 1.5918 Epoch: 16 Global Step: 164025 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:50:53,846-Speed 130.53 samples/sec Loss 1.6558 Epoch: 16 Global Step: 164030 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:51:13,382-Speed 131.05 samples/sec Loss 1.5096 Epoch: 16 Global Step: 164035 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:51:33,031-Speed 130.29 samples/sec Loss 1.5450 Epoch: 16 Global Step: 164040 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:51:52,810-Speed 129.44 samples/sec Loss 1.5659 Epoch: 16 Global Step: 164045 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:52:12,687-Speed 128.80 samples/sec Loss 1.6309 Epoch: 16 Global Step: 164050 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:52:32,216-Speed 131.09 samples/sec Loss 1.6787 Epoch: 16 Global Step: 164055 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:52:51,830-Speed 130.53 samples/sec Loss 1.4676 Epoch: 16 Global Step: 164060 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:53:11,420-Speed 130.68 samples/sec Loss 1.5443 Epoch: 16 Global Step: 164065 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:53:30,950-Speed 131.09 samples/sec Loss 1.6235 Epoch: 16 Global Step: 164070 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:53:50,565-Speed 130.52 samples/sec Loss 1.5462 Epoch: 16 Global Step: 164075 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:54:10,058-Speed 131.34 samples/sec Loss 1.4763 Epoch: 16 Global Step: 164080 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:54:29,772-Speed 129.86 samples/sec Loss 1.3660 Epoch: 16 Global Step: 164085 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:54:49,532-Speed 129.57 samples/sec Loss 1.5642 Epoch: 16 Global Step: 164090 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:55:09,109-Speed 130.77 samples/sec Loss 1.6639 Epoch: 16 Global Step: 164095 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:55:28,576-Speed 131.51 samples/sec Loss 1.5843 Epoch: 16 Global Step: 164100 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:55:48,195-Speed 130.49 samples/sec Loss 1.4087 Epoch: 16 Global Step: 164105 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:56:07,727-Speed 131.08 samples/sec Loss 1.7707 Epoch: 16 Global Step: 164110 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:56:27,447-Speed 129.82 samples/sec Loss 1.7131 Epoch: 16 Global Step: 164115 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:56:46,979-Speed 131.08 samples/sec Loss 1.7819 Epoch: 16 Global Step: 164120 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:57:06,453-Speed 131.47 samples/sec Loss 1.5349 Epoch: 16 Global Step: 164125 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:57:26,028-Speed 130.79 samples/sec Loss 1.6439 Epoch: 16 Global Step: 164130 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:57:45,946-Speed 128.53 samples/sec Loss 1.6349 Epoch: 16 Global Step: 164135 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:58:05,381-Speed 131.73 samples/sec Loss 1.6097 Epoch: 16 Global Step: 164140 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:58:24,975-Speed 130.66 samples/sec Loss 1.5725 Epoch: 16 Global Step: 164145 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:58:44,569-Speed 130.65 samples/sec Loss 1.7062 Epoch: 16 Global Step: 164150 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:59:03,993-Speed 131.81 samples/sec Loss 1.5674 Epoch: 16 Global Step: 164155 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:59:23,588-Speed 130.65 samples/sec Loss 1.6052 Epoch: 16 Global Step: 164160 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 20:59:43,056-Speed 131.51 samples/sec Loss 1.5272 Epoch: 16 Global Step: 164165 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:00:02,598-Speed 131.01 samples/sec Loss 1.6909 Epoch: 16 Global Step: 164170 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:00:22,352-Speed 129.60 samples/sec Loss 1.5892 Epoch: 16 Global Step: 164175 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:00:42,164-Speed 129.22 samples/sec Loss 1.5148 Epoch: 16 Global Step: 164180 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:01:01,600-Speed 131.72 samples/sec Loss 1.5021 Epoch: 16 Global Step: 164185 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:01:21,119-Speed 131.16 samples/sec Loss 1.4884 Epoch: 16 Global Step: 164190 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:01:40,664-Speed 130.99 samples/sec Loss 1.6695 Epoch: 16 Global Step: 164195 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:02:00,092-Speed 131.78 samples/sec Loss 1.6378 Epoch: 16 Global Step: 164200 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:02:19,699-Speed 130.57 samples/sec Loss 1.6006 Epoch: 16 Global Step: 164205 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:02:39,240-Speed 131.01 samples/sec Loss 1.5874 Epoch: 16 Global Step: 164210 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:02:58,837-Speed 130.64 samples/sec Loss 1.4438 Epoch: 16 Global Step: 164215 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:03:18,295-Speed 131.58 samples/sec Loss 1.5849 Epoch: 16 Global Step: 164220 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:03:38,087-Speed 129.35 samples/sec Loss 1.4579 Epoch: 16 Global Step: 164225 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:03:57,534-Speed 131.65 samples/sec Loss 1.6833 Epoch: 16 Global Step: 164230 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:04:17,261-Speed 129.78 samples/sec Loss 1.5075 Epoch: 16 Global Step: 164235 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:04:36,757-Speed 131.32 samples/sec Loss 1.6712 Epoch: 16 Global Step: 164240 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:04:56,268-Speed 131.21 samples/sec Loss 1.5533 Epoch: 16 Global Step: 164245 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:05:15,863-Speed 130.66 samples/sec Loss 1.5085 Epoch: 16 Global Step: 164250 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:05:35,388-Speed 131.12 samples/sec Loss 1.5318 Epoch: 16 Global Step: 164255 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:05:54,975-Speed 130.71 samples/sec Loss 1.5689 Epoch: 16 Global Step: 164260 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:06:14,390-Speed 131.87 samples/sec Loss 1.6719 Epoch: 16 Global Step: 164265 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:06:34,228-Speed 129.05 samples/sec Loss 1.6102 Epoch: 16 Global Step: 164270 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:06:53,651-Speed 131.81 samples/sec Loss 1.5277 Epoch: 16 Global Step: 164275 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:07:13,371-Speed 129.82 samples/sec Loss 1.4630 Epoch: 16 Global Step: 164280 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:07:32,790-Speed 131.84 samples/sec Loss 1.5447 Epoch: 16 Global Step: 164285 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:07:52,280-Speed 131.36 samples/sec Loss 1.5543 Epoch: 16 Global Step: 164290 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:08:11,848-Speed 130.83 samples/sec Loss 1.5295 Epoch: 16 Global Step: 164295 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:08:31,404-Speed 130.91 samples/sec Loss 1.5357 Epoch: 16 Global Step: 164300 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:08:51,008-Speed 130.60 samples/sec Loss 1.4519 Epoch: 16 Global Step: 164305 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:09:10,452-Speed 131.66 samples/sec Loss 1.4962 Epoch: 16 Global Step: 164310 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:09:30,255-Speed 129.28 samples/sec Loss 1.6519 Epoch: 16 Global Step: 164315 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:09:49,681-Speed 131.79 samples/sec Loss 1.4778 Epoch: 16 Global Step: 164320 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:10:09,368-Speed 130.04 samples/sec Loss 1.4787 Epoch: 16 Global Step: 164325 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:10:28,798-Speed 131.76 samples/sec Loss 1.5550 Epoch: 16 Global Step: 164330 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:10:48,405-Speed 130.58 samples/sec Loss 1.5339 Epoch: 16 Global Step: 164335 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:11:07,977-Speed 130.80 samples/sec Loss 1.6776 Epoch: 16 Global Step: 164340 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:11:27,459-Speed 131.41 samples/sec Loss 1.5311 Epoch: 16 Global Step: 164345 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:11:47,060-Speed 130.61 samples/sec Loss 1.5216 Epoch: 16 Global Step: 164350 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:12:06,491-Speed 131.75 samples/sec Loss 1.6261 Epoch: 16 Global Step: 164355 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:12:26,427-Speed 128.42 samples/sec Loss 1.5501 Epoch: 16 Global Step: 164360 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:12:45,862-Speed 131.73 samples/sec Loss 1.6160 Epoch: 16 Global Step: 164365 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:13:05,702-Speed 129.04 samples/sec Loss 1.5404 Epoch: 16 Global Step: 164370 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:13:25,253-Speed 130.95 samples/sec Loss 1.5805 Epoch: 16 Global Step: 164375 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:13:44,858-Speed 130.59 samples/sec Loss 1.4887 Epoch: 16 Global Step: 164380 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:14:04,385-Speed 131.11 samples/sec Loss 1.6037 Epoch: 16 Global Step: 164385 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:14:23,925-Speed 131.02 samples/sec Loss 1.3819 Epoch: 16 Global Step: 164390 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:14:43,524-Speed 130.62 samples/sec Loss 1.5066 Epoch: 16 Global Step: 164395 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:15:03,116-Speed 130.67 samples/sec Loss 1.7050 Epoch: 16 Global Step: 164400 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:15:22,927-Speed 129.24 samples/sec Loss 1.6389 Epoch: 16 Global Step: 164405 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:15:43,885-Speed 122.15 samples/sec Loss 1.6441 Epoch: 16 Global Step: 164410 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:16:03,525-Speed 130.35 samples/sec Loss 1.5811 Epoch: 16 Global Step: 164415 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:16:23,067-Speed 131.01 samples/sec Loss 1.5387 Epoch: 16 Global Step: 164420 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:16:42,851-Speed 129.40 samples/sec Loss 1.5112 Epoch: 16 Global Step: 164425 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:17:02,374-Speed 131.13 samples/sec Loss 1.3607 Epoch: 16 Global Step: 164430 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:17:21,906-Speed 131.08 samples/sec Loss 1.6778 Epoch: 16 Global Step: 164435 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:17:41,466-Speed 130.89 samples/sec Loss 1.5928 Epoch: 16 Global Step: 164440 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:18:01,086-Speed 130.49 samples/sec Loss 1.6995 Epoch: 16 Global Step: 164445 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:18:20,889-Speed 129.28 samples/sec Loss 1.4957 Epoch: 16 Global Step: 164450 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:18:40,708-Speed 129.18 samples/sec Loss 1.5681 Epoch: 16 Global Step: 164455 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:19:00,246-Speed 131.04 samples/sec Loss 1.6458 Epoch: 16 Global Step: 164460 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:19:19,854-Speed 130.56 samples/sec Loss 1.6844 Epoch: 16 Global Step: 164465 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:19:39,455-Speed 130.62 samples/sec Loss 1.5182 Epoch: 16 Global Step: 164470 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:19:58,988-Speed 131.07 samples/sec Loss 1.6362 Epoch: 16 Global Step: 164475 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:20:18,594-Speed 130.58 samples/sec Loss 1.6494 Epoch: 16 Global Step: 164480 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:20:38,139-Speed 130.98 samples/sec Loss 1.4639 Epoch: 16 Global Step: 164485 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:20:57,748-Speed 130.56 samples/sec Loss 1.5109 Epoch: 16 Global Step: 164490 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:21:17,517-Speed 129.51 samples/sec Loss 1.6409 Epoch: 16 Global Step: 164495 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:21:37,221-Speed 129.93 samples/sec Loss 1.7088 Epoch: 16 Global Step: 164500 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:21:56,701-Speed 131.43 samples/sec Loss 1.4205 Epoch: 16 Global Step: 164505 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:22:16,297-Speed 130.65 samples/sec Loss 1.6058 Epoch: 16 Global Step: 164510 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:22:35,867-Speed 130.82 samples/sec Loss 1.5321 Epoch: 16 Global Step: 164515 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:22:55,411-Speed 130.99 samples/sec Loss 1.4905 Epoch: 16 Global Step: 164520 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:23:15,016-Speed 130.58 samples/sec Loss 1.3786 Epoch: 16 Global Step: 164525 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:23:34,569-Speed 130.94 samples/sec Loss 1.3568 Epoch: 16 Global Step: 164530 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:23:54,192-Speed 130.47 samples/sec Loss 1.4802 Epoch: 16 Global Step: 164535 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:24:13,993-Speed 129.29 samples/sec Loss 1.7156 Epoch: 16 Global Step: 164540 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:24:33,633-Speed 130.36 samples/sec Loss 1.5571 Epoch: 16 Global Step: 164545 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:24:53,163-Speed 131.08 samples/sec Loss 1.6797 Epoch: 16 Global Step: 164550 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:25:12,773-Speed 130.55 samples/sec Loss 1.7297 Epoch: 16 Global Step: 164555 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:25:32,308-Speed 131.06 samples/sec Loss 1.3863 Epoch: 16 Global Step: 164560 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:25:51,720-Speed 131.89 samples/sec Loss 1.5683 Epoch: 16 Global Step: 164565 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:26:11,325-Speed 130.58 samples/sec Loss 1.5037 Epoch: 16 Global Step: 164570 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:26:30,880-Speed 130.92 samples/sec Loss 1.6439 Epoch: 16 Global Step: 164575 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:26:50,412-Speed 131.08 samples/sec Loss 1.6204 Epoch: 16 Global Step: 164580 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:27:10,170-Speed 129.57 samples/sec Loss 1.6447 Epoch: 16 Global Step: 164585 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:27:30,082-Speed 128.57 samples/sec Loss 1.4672 Epoch: 16 Global Step: 164590 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:27:49,641-Speed 130.89 samples/sec Loss 1.5687 Epoch: 16 Global Step: 164595 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:28:09,244-Speed 130.60 samples/sec Loss 1.6544 Epoch: 16 Global Step: 164600 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:28:28,739-Speed 131.32 samples/sec Loss 1.5699 Epoch: 16 Global Step: 164605 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:28:48,273-Speed 131.06 samples/sec Loss 1.5027 Epoch: 16 Global Step: 164610 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:29:07,836-Speed 130.86 samples/sec Loss 1.6860 Epoch: 16 Global Step: 164615 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:29:27,435-Speed 130.63 samples/sec Loss 1.5596 Epoch: 16 Global Step: 164620 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:29:47,036-Speed 130.61 samples/sec Loss 1.6506 Epoch: 16 Global Step: 164625 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:30:06,712-Speed 130.12 samples/sec Loss 1.4996 Epoch: 16 Global Step: 164630 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:30:26,316-Speed 130.59 samples/sec Loss 1.4707 Epoch: 16 Global Step: 164635 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:30:45,924-Speed 130.57 samples/sec Loss 1.5975 Epoch: 16 Global Step: 164640 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:31:05,514-Speed 130.69 samples/sec Loss 1.6244 Epoch: 16 Global Step: 164645 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:31:25,050-Speed 131.04 samples/sec Loss 1.4847 Epoch: 16 Global Step: 164650 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:31:44,493-Speed 131.67 samples/sec Loss 1.7186 Epoch: 16 Global Step: 164655 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:32:04,119-Speed 130.45 samples/sec Loss 1.5294 Epoch: 16 Global Step: 164660 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:32:23,769-Speed 130.29 samples/sec Loss 1.7213 Epoch: 16 Global Step: 164665 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:32:44,035-Speed 126.32 samples/sec Loss 1.4477 Epoch: 16 Global Step: 164670 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:33:03,783-Speed 129.64 samples/sec Loss 1.5189 Epoch: 16 Global Step: 164675 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:33:23,404-Speed 130.48 samples/sec Loss 1.6362 Epoch: 16 Global Step: 164680 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:33:43,017-Speed 130.53 samples/sec Loss 1.6107 Epoch: 16 Global Step: 164685 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:34:02,146-Speed 133.83 samples/sec Loss 1.5388 Epoch: 16 Global Step: 164690 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:34:21,765-Speed 130.50 samples/sec Loss 1.5373 Epoch: 16 Global Step: 164695 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:34:41,646-Speed 128.77 samples/sec Loss 1.5017 Epoch: 16 Global Step: 164700 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:35:01,225-Speed 130.76 samples/sec Loss 1.4750 Epoch: 16 Global Step: 164705 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:35:20,848-Speed 130.47 samples/sec Loss 1.5614 Epoch: 16 Global Step: 164710 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:35:40,482-Speed 130.39 samples/sec Loss 1.4305 Epoch: 16 Global Step: 164715 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:35:59,916-Speed 131.73 samples/sec Loss 1.7100 Epoch: 16 Global Step: 164720 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:36:19,523-Speed 130.58 samples/sec Loss 1.5761 Epoch: 16 Global Step: 164725 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:36:39,320-Speed 129.32 samples/sec Loss 1.6330 Epoch: 16 Global Step: 164730 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:36:59,045-Speed 129.79 samples/sec Loss 1.4198 Epoch: 16 Global Step: 164735 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:37:18,470-Speed 131.80 samples/sec Loss 1.6006 Epoch: 16 Global Step: 164740 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:37:38,081-Speed 130.54 samples/sec Loss 1.5362 Epoch: 16 Global Step: 164745 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:37:57,690-Speed 130.56 samples/sec Loss 1.6052 Epoch: 16 Global Step: 164750 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:38:17,213-Speed 131.14 samples/sec Loss 1.5866 Epoch: 16 Global Step: 164755 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:38:36,789-Speed 130.78 samples/sec Loss 1.4599 Epoch: 16 Global Step: 164760 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:38:56,238-Speed 131.63 samples/sec Loss 1.5379 Epoch: 16 Global Step: 164765 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:39:15,876-Speed 130.37 samples/sec Loss 1.4983 Epoch: 16 Global Step: 164770 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:39:35,738-Speed 128.89 samples/sec Loss 1.6307 Epoch: 16 Global Step: 164775 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:40:01,275-Speed 100.25 samples/sec Loss 1.7033 Epoch: 16 Global Step: 164780 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:40:39,191-Speed 67.52 samples/sec Loss 1.5600 Epoch: 16 Global Step: 164785 Fp16 Grad Scale: 4096 Required: 100 hours Training: 2021-06-14 21:41:01,510-Speed 114.73 samples/sec Loss 1.4387 Epoch: 16 Global Step: 164790 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:41:21,172-Speed 130.21 samples/sec Loss 1.4936 Epoch: 16 Global Step: 164795 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:41:40,736-Speed 130.86 samples/sec Loss 1.3852 Epoch: 16 Global Step: 164800 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:42:00,285-Speed 130.96 samples/sec Loss 1.6343 Epoch: 16 Global Step: 164805 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:42:19,844-Speed 130.90 samples/sec Loss 1.5711 Epoch: 16 Global Step: 164810 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:42:39,420-Speed 130.78 samples/sec Loss 1.5217 Epoch: 16 Global Step: 164815 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:42:59,220-Speed 129.30 samples/sec Loss 1.6595 Epoch: 16 Global Step: 164820 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:43:18,768-Speed 130.96 samples/sec Loss 1.7129 Epoch: 16 Global Step: 164825 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:43:38,445-Speed 130.11 samples/sec Loss 1.6032 Epoch: 16 Global Step: 164830 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:43:57,974-Speed 131.09 samples/sec Loss 1.5084 Epoch: 16 Global Step: 164835 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:44:17,582-Speed 130.57 samples/sec Loss 1.4981 Epoch: 16 Global Step: 164840 Fp16 Grad Scale: 8192 Required: 100 hours Training: 2021-06-14 21:44:37,306-Speed 129.80 samples/sec Loss 1.6937 Epoch: 16 Global Step: 164845 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:44:56,881-Speed 130.79 samples/sec Loss 1.5022 Epoch: 16 Global Step: 164850 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:45:16,480-Speed 130.62 samples/sec Loss 1.4879 Epoch: 16 Global Step: 164855 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:45:35,923-Speed 131.68 samples/sec Loss 1.5948 Epoch: 16 Global Step: 164860 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:45:55,721-Speed 129.31 samples/sec Loss 1.5703 Epoch: 16 Global Step: 164865 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:46:15,560-Speed 129.05 samples/sec Loss 1.6867 Epoch: 16 Global Step: 164870 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:46:35,170-Speed 130.55 samples/sec Loss 1.5136 Epoch: 16 Global Step: 164875 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:46:54,706-Speed 131.05 samples/sec Loss 1.4856 Epoch: 16 Global Step: 164880 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:47:14,270-Speed 130.86 samples/sec Loss 1.4660 Epoch: 16 Global Step: 164885 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:47:33,813-Speed 131.00 samples/sec Loss 1.4855 Epoch: 16 Global Step: 164890 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:47:53,415-Speed 130.61 samples/sec Loss 1.5685 Epoch: 16 Global Step: 164895 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:48:13,002-Speed 130.71 samples/sec Loss 1.4114 Epoch: 16 Global Step: 164900 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:48:32,445-Speed 131.68 samples/sec Loss 1.6165 Epoch: 16 Global Step: 164905 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:48:52,237-Speed 129.35 samples/sec Loss 1.6180 Epoch: 16 Global Step: 164910 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:49:11,880-Speed 130.33 samples/sec Loss 1.5723 Epoch: 16 Global Step: 164915 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:49:31,482-Speed 130.61 samples/sec Loss 1.5654 Epoch: 16 Global Step: 164920 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:49:50,943-Speed 131.55 samples/sec Loss 1.4663 Epoch: 16 Global Step: 164925 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:50:10,523-Speed 130.75 samples/sec Loss 1.5938 Epoch: 16 Global Step: 164930 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:50:30,068-Speed 130.99 samples/sec Loss 1.5367 Epoch: 16 Global Step: 164935 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:50:49,644-Speed 130.78 samples/sec Loss 1.5333 Epoch: 16 Global Step: 164940 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:51:09,240-Speed 130.64 samples/sec Loss 1.5920 Epoch: 16 Global Step: 164945 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:51:28,689-Speed 131.63 samples/sec Loss 1.3680 Epoch: 16 Global Step: 164950 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:51:48,495-Speed 129.26 samples/sec Loss 1.6034 Epoch: 16 Global Step: 164955 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:52:08,234-Speed 129.70 samples/sec Loss 1.4799 Epoch: 16 Global Step: 164960 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:52:27,836-Speed 130.60 samples/sec Loss 1.5497 Epoch: 16 Global Step: 164965 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:52:47,385-Speed 130.96 samples/sec Loss 1.5589 Epoch: 16 Global Step: 164970 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:53:06,946-Speed 130.88 samples/sec Loss 1.4843 Epoch: 16 Global Step: 164975 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:53:26,544-Speed 130.64 samples/sec Loss 1.5706 Epoch: 16 Global Step: 164980 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:53:46,176-Speed 130.41 samples/sec Loss 1.4548 Epoch: 16 Global Step: 164985 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:54:05,797-Speed 130.48 samples/sec Loss 1.4451 Epoch: 16 Global Step: 164990 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:54:25,347-Speed 130.95 samples/sec Loss 1.5712 Epoch: 16 Global Step: 164995 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:54:45,107-Speed 129.56 samples/sec Loss 1.5637 Epoch: 16 Global Step: 165000 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:55:04,806-Speed 129.96 samples/sec Loss 1.6044 Epoch: 16 Global Step: 165005 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:55:24,347-Speed 131.01 samples/sec Loss 1.6030 Epoch: 16 Global Step: 165010 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:55:43,887-Speed 131.02 samples/sec Loss 1.6092 Epoch: 16 Global Step: 165015 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:56:03,464-Speed 130.77 samples/sec Loss 1.4629 Epoch: 16 Global Step: 165020 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:56:23,702-Speed 126.50 samples/sec Loss 1.5636 Epoch: 16 Global Step: 165025 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:56:43,336-Speed 130.40 samples/sec Loss 1.4452 Epoch: 16 Global Step: 165030 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:57:02,921-Speed 130.71 samples/sec Loss 1.5124 Epoch: 16 Global Step: 165035 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:57:22,394-Speed 131.48 samples/sec Loss 1.5710 Epoch: 16 Global Step: 165040 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:57:42,160-Speed 129.52 samples/sec Loss 1.6607 Epoch: 16 Global Step: 165045 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:58:01,881-Speed 129.82 samples/sec Loss 1.5789 Epoch: 16 Global Step: 165050 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:58:21,449-Speed 130.83 samples/sec Loss 1.5376 Epoch: 16 Global Step: 165055 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:58:41,067-Speed 130.50 samples/sec Loss 1.5528 Epoch: 16 Global Step: 165060 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:59:00,587-Speed 131.15 samples/sec Loss 1.5767 Epoch: 16 Global Step: 165065 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:59:20,136-Speed 130.96 samples/sec Loss 1.4928 Epoch: 16 Global Step: 165070 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:59:39,714-Speed 130.77 samples/sec Loss 1.3931 Epoch: 16 Global Step: 165075 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 21:59:59,264-Speed 130.96 samples/sec Loss 1.5702 Epoch: 16 Global Step: 165080 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:00:18,872-Speed 130.57 samples/sec Loss 1.5137 Epoch: 16 Global Step: 165085 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:00:38,196-Speed 132.48 samples/sec Loss 1.7190 Epoch: 16 Global Step: 165090 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:00:57,886-Speed 130.02 samples/sec Loss 1.5796 Epoch: 16 Global Step: 165095 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:01:17,334-Speed 131.64 samples/sec Loss 1.4819 Epoch: 16 Global Step: 165100 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:01:36,935-Speed 130.61 samples/sec Loss 1.4700 Epoch: 16 Global Step: 165105 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:01:56,479-Speed 131.00 samples/sec Loss 1.3715 Epoch: 16 Global Step: 165110 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:02:15,935-Speed 131.58 samples/sec Loss 1.6670 Epoch: 16 Global Step: 165115 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:02:35,532-Speed 130.64 samples/sec Loss 1.6058 Epoch: 16 Global Step: 165120 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:02:55,027-Speed 131.33 samples/sec Loss 1.5144 Epoch: 16 Global Step: 165125 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:03:14,631-Speed 130.60 samples/sec Loss 1.7026 Epoch: 16 Global Step: 165130 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:03:34,399-Speed 129.51 samples/sec Loss 1.5730 Epoch: 16 Global Step: 165135 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:03:53,969-Speed 130.82 samples/sec Loss 1.5322 Epoch: 16 Global Step: 165140 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:04:13,403-Speed 131.73 samples/sec Loss 1.5817 Epoch: 16 Global Step: 165145 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:04:33,011-Speed 130.56 samples/sec Loss 1.7330 Epoch: 16 Global Step: 165150 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:04:52,543-Speed 131.07 samples/sec Loss 1.5889 Epoch: 16 Global Step: 165155 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:05:12,118-Speed 130.79 samples/sec Loss 1.4701 Epoch: 16 Global Step: 165160 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:05:31,716-Speed 130.64 samples/sec Loss 1.4240 Epoch: 16 Global Step: 165165 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:05:51,147-Speed 131.75 samples/sec Loss 1.4827 Epoch: 16 Global Step: 165170 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:06:10,759-Speed 130.55 samples/sec Loss 1.4794 Epoch: 16 Global Step: 165175 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:06:30,731-Speed 128.18 samples/sec Loss 1.5720 Epoch: 16 Global Step: 165180 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:06:50,182-Speed 131.62 samples/sec Loss 1.4744 Epoch: 16 Global Step: 165185 Fp16 Grad Scale: 4096 Required: 99 hours Training: 2021-06-14 22:07:09,767-Speed 130.72 samples/sec Loss 1.4436 Epoch: 16 Global Step: 165190 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:07:29,369-Speed 130.61 samples/sec Loss 1.6226 Epoch: 16 Global Step: 165195 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:07:48,816-Speed 131.65 samples/sec Loss 1.5923 Epoch: 16 Global Step: 165200 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:08:08,349-Speed 131.07 samples/sec Loss 1.6177 Epoch: 16 Global Step: 165205 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:08:27,922-Speed 130.80 samples/sec Loss 1.5855 Epoch: 16 Global Step: 165210 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:08:47,382-Speed 131.56 samples/sec Loss 1.6320 Epoch: 16 Global Step: 165215 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:09:06,985-Speed 130.60 samples/sec Loss 1.5583 Epoch: 16 Global Step: 165220 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:09:26,737-Speed 129.61 samples/sec Loss 1.6330 Epoch: 16 Global Step: 165225 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:09:46,224-Speed 131.38 samples/sec Loss 1.6069 Epoch: 16 Global Step: 165230 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:10:05,861-Speed 130.38 samples/sec Loss 1.3901 Epoch: 16 Global Step: 165235 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:10:25,440-Speed 130.76 samples/sec Loss 1.5432 Epoch: 16 Global Step: 165240 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:10:44,870-Speed 131.76 samples/sec Loss 1.5037 Epoch: 16 Global Step: 165245 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:11:04,470-Speed 130.62 samples/sec Loss 1.3738 Epoch: 16 Global Step: 165250 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:11:24,042-Speed 130.81 samples/sec Loss 1.4615 Epoch: 16 Global Step: 165255 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:11:43,589-Speed 130.97 samples/sec Loss 1.4358 Epoch: 16 Global Step: 165260 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:12:03,148-Speed 130.90 samples/sec Loss 1.4634 Epoch: 16 Global Step: 165265 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:12:22,949-Speed 129.29 samples/sec Loss 1.4368 Epoch: 16 Global Step: 165270 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:12:42,391-Speed 131.68 samples/sec Loss 1.5440 Epoch: 16 Global Step: 165275 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:13:02,085-Speed 130.00 samples/sec Loss 1.4241 Epoch: 16 Global Step: 165280 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:13:21,779-Speed 129.99 samples/sec Loss 1.5703 Epoch: 16 Global Step: 165285 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:13:41,239-Speed 131.56 samples/sec Loss 1.4914 Epoch: 16 Global Step: 165290 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:14:00,832-Speed 130.67 samples/sec Loss 1.5456 Epoch: 16 Global Step: 165295 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:14:20,377-Speed 130.99 samples/sec Loss 1.4696 Epoch: 16 Global Step: 165300 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:14:39,997-Speed 130.48 samples/sec Loss 1.6867 Epoch: 16 Global Step: 165305 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:14:59,441-Speed 131.67 samples/sec Loss 1.5235 Epoch: 16 Global Step: 165310 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:15:19,315-Speed 128.82 samples/sec Loss 1.6043 Epoch: 16 Global Step: 165315 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:15:38,759-Speed 131.66 samples/sec Loss 1.5486 Epoch: 16 Global Step: 165320 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:15:58,481-Speed 129.81 samples/sec Loss 1.4199 Epoch: 16 Global Step: 165325 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:16:17,916-Speed 131.73 samples/sec Loss 1.4035 Epoch: 16 Global Step: 165330 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:16:37,522-Speed 130.58 samples/sec Loss 1.5118 Epoch: 16 Global Step: 165335 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:16:57,118-Speed 130.65 samples/sec Loss 1.4909 Epoch: 16 Global Step: 165340 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:17:16,643-Speed 131.12 samples/sec Loss 1.5689 Epoch: 16 Global Step: 165345 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:17:36,242-Speed 130.63 samples/sec Loss 1.5103 Epoch: 16 Global Step: 165350 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:17:55,678-Speed 131.72 samples/sec Loss 1.6293 Epoch: 16 Global Step: 165355 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:18:15,482-Speed 129.27 samples/sec Loss 1.5206 Epoch: 16 Global Step: 165360 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:18:35,117-Speed 130.39 samples/sec Loss 1.5463 Epoch: 16 Global Step: 165365 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:18:54,772-Speed 130.25 samples/sec Loss 1.4712 Epoch: 16 Global Step: 165370 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:19:14,395-Speed 130.47 samples/sec Loss 1.5762 Epoch: 16 Global Step: 165375 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:19:33,983-Speed 130.70 samples/sec Loss 1.4503 Epoch: 16 Global Step: 165380 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:19:53,584-Speed 130.61 samples/sec Loss 1.5856 Epoch: 16 Global Step: 165385 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:20:13,146-Speed 130.87 samples/sec Loss 1.5869 Epoch: 16 Global Step: 165390 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:20:32,748-Speed 130.61 samples/sec Loss 1.4130 Epoch: 16 Global Step: 165395 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:20:52,268-Speed 131.16 samples/sec Loss 1.4763 Epoch: 16 Global Step: 165400 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:21:12,067-Speed 129.31 samples/sec Loss 1.4798 Epoch: 16 Global Step: 165405 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:21:31,707-Speed 130.35 samples/sec Loss 1.6214 Epoch: 16 Global Step: 165410 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:21:51,349-Speed 130.34 samples/sec Loss 1.3486 Epoch: 16 Global Step: 165415 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:22:10,878-Speed 131.09 samples/sec Loss 1.5969 Epoch: 16 Global Step: 165420 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:22:30,485-Speed 130.57 samples/sec Loss 1.4763 Epoch: 16 Global Step: 165425 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:22:50,218-Speed 129.74 samples/sec Loss 1.5461 Epoch: 16 Global Step: 165430 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:23:09,776-Speed 130.90 samples/sec Loss 1.4976 Epoch: 16 Global Step: 165435 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:23:29,379-Speed 130.60 samples/sec Loss 1.5598 Epoch: 16 Global Step: 165440 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:23:48,824-Speed 131.66 samples/sec Loss 1.4590 Epoch: 16 Global Step: 165445 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:24:08,616-Speed 129.35 samples/sec Loss 1.4123 Epoch: 16 Global Step: 165450 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:24:28,183-Speed 130.84 samples/sec Loss 1.4926 Epoch: 16 Global Step: 165455 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:24:47,858-Speed 130.13 samples/sec Loss 1.4471 Epoch: 16 Global Step: 165460 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:25:07,419-Speed 130.87 samples/sec Loss 1.6653 Epoch: 16 Global Step: 165465 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:25:27,021-Speed 130.61 samples/sec Loss 1.3732 Epoch: 16 Global Step: 165470 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:25:46,564-Speed 131.00 samples/sec Loss 1.5396 Epoch: 16 Global Step: 165475 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:26:06,230-Speed 130.18 samples/sec Loss 1.5651 Epoch: 16 Global Step: 165480 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:26:25,756-Speed 131.11 samples/sec Loss 1.4821 Epoch: 16 Global Step: 165485 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:26:45,198-Speed 131.68 samples/sec Loss 1.5358 Epoch: 16 Global Step: 165490 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:27:05,000-Speed 129.29 samples/sec Loss 1.6359 Epoch: 16 Global Step: 165495 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:27:24,696-Speed 129.98 samples/sec Loss 1.4133 Epoch: 16 Global Step: 165500 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:27:44,278-Speed 130.74 samples/sec Loss 1.6435 Epoch: 16 Global Step: 165505 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:28:03,755-Speed 131.45 samples/sec Loss 1.4683 Epoch: 16 Global Step: 165510 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:28:23,361-Speed 130.58 samples/sec Loss 1.6140 Epoch: 16 Global Step: 165515 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:28:42,913-Speed 130.94 samples/sec Loss 1.4572 Epoch: 16 Global Step: 165520 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:29:02,511-Speed 130.63 samples/sec Loss 1.4430 Epoch: 16 Global Step: 165525 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:29:21,955-Speed 131.67 samples/sec Loss 1.4512 Epoch: 16 Global Step: 165530 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:29:41,424-Speed 131.50 samples/sec Loss 1.5034 Epoch: 16 Global Step: 165535 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:30:01,217-Speed 129.35 samples/sec Loss 1.5456 Epoch: 16 Global Step: 165540 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:30:20,956-Speed 129.70 samples/sec Loss 1.5209 Epoch: 16 Global Step: 165545 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:30:40,513-Speed 130.90 samples/sec Loss 1.5354 Epoch: 16 Global Step: 165550 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:31:00,940-Speed 125.33 samples/sec Loss 1.3958 Epoch: 16 Global Step: 165555 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:31:20,547-Speed 130.58 samples/sec Loss 1.6483 Epoch: 16 Global Step: 165560 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:31:40,093-Speed 130.98 samples/sec Loss 1.5704 Epoch: 16 Global Step: 165565 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:31:59,861-Speed 129.51 samples/sec Loss 1.3459 Epoch: 16 Global Step: 165570 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:32:19,289-Speed 131.78 samples/sec Loss 1.4249 Epoch: 16 Global Step: 165575 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:32:38,896-Speed 130.58 samples/sec Loss 1.6283 Epoch: 16 Global Step: 165580 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:32:58,798-Speed 128.63 samples/sec Loss 1.6424 Epoch: 16 Global Step: 165585 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:33:18,243-Speed 131.67 samples/sec Loss 1.5012 Epoch: 16 Global Step: 165590 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:33:37,719-Speed 131.45 samples/sec Loss 1.5035 Epoch: 16 Global Step: 165595 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:33:57,373-Speed 130.26 samples/sec Loss 1.5836 Epoch: 16 Global Step: 165600 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:34:16,921-Speed 130.97 samples/sec Loss 1.4870 Epoch: 16 Global Step: 165605 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:34:36,528-Speed 130.57 samples/sec Loss 1.6085 Epoch: 16 Global Step: 165610 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:34:55,972-Speed 131.67 samples/sec Loss 1.4703 Epoch: 16 Global Step: 165615 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:35:15,579-Speed 130.57 samples/sec Loss 1.5478 Epoch: 16 Global Step: 165620 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:35:35,208-Speed 130.43 samples/sec Loss 1.3069 Epoch: 16 Global Step: 165625 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:35:55,097-Speed 128.72 samples/sec Loss 1.5781 Epoch: 16 Global Step: 165630 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:36:14,521-Speed 131.81 samples/sec Loss 1.5375 Epoch: 16 Global Step: 165635 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:36:34,120-Speed 130.62 samples/sec Loss 1.6113 Epoch: 16 Global Step: 165640 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:36:53,739-Speed 130.49 samples/sec Loss 1.4555 Epoch: 16 Global Step: 165645 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:37:13,272-Speed 131.07 samples/sec Loss 1.6469 Epoch: 16 Global Step: 165650 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:37:32,870-Speed 130.64 samples/sec Loss 1.5941 Epoch: 16 Global Step: 165655 Fp16 Grad Scale: 8192 Required: 99 hours Training: 2021-06-14 22:37:52,309-Speed 131.70 samples/sec Loss 1.4239 Epoch: 16 Global Step: 165660 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:38:11,909-Speed 130.62 samples/sec Loss 1.4935 Epoch: 16 Global Step: 165665 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:38:31,526-Speed 130.50 samples/sec Loss 1.4034 Epoch: 16 Global Step: 165670 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:38:51,462-Speed 128.42 samples/sec Loss 1.5092 Epoch: 16 Global Step: 165675 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:39:10,917-Speed 131.59 samples/sec Loss 1.7886 Epoch: 16 Global Step: 165680 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:39:30,473-Speed 130.91 samples/sec Loss 1.4683 Epoch: 16 Global Step: 165685 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:39:50,092-Speed 130.49 samples/sec Loss 1.3552 Epoch: 16 Global Step: 165690 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:40:09,712-Speed 130.49 samples/sec Loss 1.6855 Epoch: 16 Global Step: 165695 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:40:29,142-Speed 131.76 samples/sec Loss 1.4157 Epoch: 16 Global Step: 165700 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:40:48,634-Speed 131.34 samples/sec Loss 1.4299 Epoch: 16 Global Step: 165705 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:41:08,172-Speed 131.03 samples/sec Loss 1.5720 Epoch: 16 Global Step: 165710 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:41:27,709-Speed 131.05 samples/sec Loss 1.4376 Epoch: 16 Global Step: 165715 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:41:47,517-Speed 129.24 samples/sec Loss 1.6602 Epoch: 16 Global Step: 165720 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:42:06,959-Speed 131.68 samples/sec Loss 1.5568 Epoch: 16 Global Step: 165725 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:42:26,524-Speed 130.85 samples/sec Loss 1.5178 Epoch: 16 Global Step: 165730 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:42:46,236-Speed 129.88 samples/sec Loss 1.5762 Epoch: 16 Global Step: 165735 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:43:05,979-Speed 129.68 samples/sec Loss 1.4673 Epoch: 16 Global Step: 165740 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:43:25,456-Speed 131.44 samples/sec Loss 1.5726 Epoch: 16 Global Step: 165745 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:43:44,929-Speed 131.47 samples/sec Loss 1.4747 Epoch: 16 Global Step: 165750 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:44:04,521-Speed 130.67 samples/sec Loss 1.6111 Epoch: 16 Global Step: 165755 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:44:24,051-Speed 131.09 samples/sec Loss 1.5630 Epoch: 16 Global Step: 165760 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:44:43,872-Speed 129.16 samples/sec Loss 1.4299 Epoch: 16 Global Step: 165765 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:45:03,394-Speed 131.14 samples/sec Loss 1.4333 Epoch: 16 Global Step: 165770 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:45:23,014-Speed 130.49 samples/sec Loss 1.4124 Epoch: 16 Global Step: 165775 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:45:42,573-Speed 130.89 samples/sec Loss 1.6536 Epoch: 16 Global Step: 165780 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:46:02,189-Speed 130.51 samples/sec Loss 1.6395 Epoch: 16 Global Step: 165785 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:46:21,627-Speed 131.71 samples/sec Loss 1.5915 Epoch: 16 Global Step: 165790 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:46:41,231-Speed 130.59 samples/sec Loss 1.4243 Epoch: 16 Global Step: 165795 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:47:00,832-Speed 130.61 samples/sec Loss 1.3821 Epoch: 16 Global Step: 165800 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:47:20,358-Speed 131.12 samples/sec Loss 1.5597 Epoch: 16 Global Step: 165805 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:47:40,455-Speed 127.39 samples/sec Loss 1.4885 Epoch: 16 Global Step: 165810 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:48:00,064-Speed 130.56 samples/sec Loss 1.4699 Epoch: 16 Global Step: 165815 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:48:19,660-Speed 130.65 samples/sec Loss 1.4933 Epoch: 16 Global Step: 165820 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:48:39,292-Speed 130.41 samples/sec Loss 1.3824 Epoch: 16 Global Step: 165825 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:48:58,721-Speed 131.77 samples/sec Loss 1.5157 Epoch: 16 Global Step: 165830 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:49:18,192-Speed 131.49 samples/sec Loss 1.4121 Epoch: 16 Global Step: 165835 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:49:37,762-Speed 130.82 samples/sec Loss 1.2751 Epoch: 16 Global Step: 165840 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:49:57,363-Speed 130.61 samples/sec Loss 1.6524 Epoch: 16 Global Step: 165845 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:50:17,097-Speed 129.73 samples/sec Loss 1.3733 Epoch: 16 Global Step: 165850 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:50:36,775-Speed 130.11 samples/sec Loss 1.3528 Epoch: 16 Global Step: 165855 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:50:56,354-Speed 130.76 samples/sec Loss 1.4853 Epoch: 16 Global Step: 165860 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:51:15,957-Speed 130.59 samples/sec Loss 1.4881 Epoch: 16 Global Step: 165865 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:51:35,577-Speed 130.49 samples/sec Loss 1.5527 Epoch: 16 Global Step: 165870 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:51:55,068-Speed 131.35 samples/sec Loss 1.4461 Epoch: 16 Global Step: 165875 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:52:14,763-Speed 129.99 samples/sec Loss 1.5510 Epoch: 16 Global Step: 165880 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:52:34,343-Speed 130.76 samples/sec Loss 1.5321 Epoch: 16 Global Step: 165885 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:52:54,186-Speed 129.02 samples/sec Loss 1.5544 Epoch: 16 Global Step: 165890 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:53:13,787-Speed 130.61 samples/sec Loss 1.4653 Epoch: 16 Global Step: 165895 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:53:33,589-Speed 129.29 samples/sec Loss 1.4499 Epoch: 16 Global Step: 165900 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:53:53,009-Speed 131.83 samples/sec Loss 1.4312 Epoch: 16 Global Step: 165905 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:54:12,755-Speed 129.65 samples/sec Loss 1.4546 Epoch: 16 Global Step: 165910 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:54:32,692-Speed 128.41 samples/sec Loss 1.5172 Epoch: 16 Global Step: 165915 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:54:52,205-Speed 131.20 samples/sec Loss 1.4710 Epoch: 16 Global Step: 165920 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:55:14,668-Speed 113.97 samples/sec Loss 1.3488 Epoch: 16 Global Step: 165925 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:55:34,314-Speed 130.31 samples/sec Loss 1.4578 Epoch: 16 Global Step: 165930 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:55:53,932-Speed 130.50 samples/sec Loss 1.4770 Epoch: 16 Global Step: 165935 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:56:13,391-Speed 131.56 samples/sec Loss 1.5633 Epoch: 16 Global Step: 165940 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:56:33,292-Speed 128.65 samples/sec Loss 1.4050 Epoch: 16 Global Step: 165945 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:56:53,478-Speed 126.83 samples/sec Loss 1.4471 Epoch: 16 Global Step: 165950 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:57:12,938-Speed 131.56 samples/sec Loss 1.4067 Epoch: 16 Global Step: 165955 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:57:32,520-Speed 130.74 samples/sec Loss 1.5331 Epoch: 16 Global Step: 165960 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:57:52,140-Speed 130.49 samples/sec Loss 1.5804 Epoch: 16 Global Step: 165965 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:58:11,687-Speed 130.97 samples/sec Loss 1.4146 Epoch: 16 Global Step: 165970 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:58:31,302-Speed 130.52 samples/sec Loss 1.4395 Epoch: 16 Global Step: 165975 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:58:50,748-Speed 131.66 samples/sec Loss 1.4472 Epoch: 16 Global Step: 165980 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:59:10,319-Speed 130.81 samples/sec Loss 1.3333 Epoch: 16 Global Step: 165985 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:59:30,089-Speed 129.50 samples/sec Loss 1.6628 Epoch: 16 Global Step: 165990 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 22:59:49,778-Speed 130.03 samples/sec Loss 1.4943 Epoch: 16 Global Step: 165995 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:00:09,882-Speed 127.34 samples/sec Loss 1.5631 Epoch: 16 Global Step: 166000 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:00:29,487-Speed 130.59 samples/sec Loss 1.5069 Epoch: 16 Global Step: 166005 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:00:49,021-Speed 131.06 samples/sec Loss 1.6392 Epoch: 16 Global Step: 166010 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:01:08,603-Speed 130.74 samples/sec Loss 1.5388 Epoch: 16 Global Step: 166015 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:01:28,174-Speed 130.81 samples/sec Loss 1.6139 Epoch: 16 Global Step: 166020 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:01:47,719-Speed 130.99 samples/sec Loss 1.5632 Epoch: 16 Global Step: 166025 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:02:07,297-Speed 130.76 samples/sec Loss 1.4963 Epoch: 16 Global Step: 166030 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:02:27,054-Speed 129.58 samples/sec Loss 1.5723 Epoch: 16 Global Step: 166035 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:02:46,759-Speed 129.93 samples/sec Loss 1.6301 Epoch: 16 Global Step: 166040 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:03:06,312-Speed 130.93 samples/sec Loss 1.5196 Epoch: 16 Global Step: 166045 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:03:25,925-Speed 130.54 samples/sec Loss 1.5467 Epoch: 16 Global Step: 166050 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:03:45,455-Speed 131.09 samples/sec Loss 1.3799 Epoch: 16 Global Step: 166055 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:04:05,066-Speed 130.54 samples/sec Loss 1.6890 Epoch: 16 Global Step: 166060 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:04:24,718-Speed 130.27 samples/sec Loss 1.5550 Epoch: 16 Global Step: 166065 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:04:44,256-Speed 131.03 samples/sec Loss 1.4024 Epoch: 16 Global Step: 166070 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:05:03,822-Speed 130.85 samples/sec Loss 1.5113 Epoch: 16 Global Step: 166075 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:05:23,751-Speed 128.47 samples/sec Loss 1.4379 Epoch: 16 Global Step: 166080 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:05:43,197-Speed 131.65 samples/sec Loss 1.5299 Epoch: 16 Global Step: 166085 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:06:02,816-Speed 130.49 samples/sec Loss 1.4475 Epoch: 16 Global Step: 166090 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:06:22,373-Speed 130.91 samples/sec Loss 1.5070 Epoch: 16 Global Step: 166095 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:06:41,808-Speed 131.73 samples/sec Loss 1.6724 Epoch: 16 Global Step: 166100 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:07:01,463-Speed 130.26 samples/sec Loss 1.4775 Epoch: 16 Global Step: 166105 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:07:21,021-Speed 130.90 samples/sec Loss 1.6568 Epoch: 16 Global Step: 166110 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:07:40,644-Speed 130.46 samples/sec Loss 1.4639 Epoch: 16 Global Step: 166115 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:08:00,189-Speed 130.99 samples/sec Loss 1.4801 Epoch: 16 Global Step: 166120 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:08:20,004-Speed 129.20 samples/sec Loss 1.5474 Epoch: 16 Global Step: 166125 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:08:39,434-Speed 131.76 samples/sec Loss 1.5278 Epoch: 16 Global Step: 166130 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:08:59,183-Speed 129.63 samples/sec Loss 1.5274 Epoch: 16 Global Step: 166135 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:09:18,716-Speed 131.07 samples/sec Loss 1.5973 Epoch: 16 Global Step: 166140 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:09:38,321-Speed 130.58 samples/sec Loss 1.6212 Epoch: 16 Global Step: 166145 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:09:57,849-Speed 131.10 samples/sec Loss 1.4624 Epoch: 16 Global Step: 166150 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:10:17,413-Speed 130.86 samples/sec Loss 1.5660 Epoch: 16 Global Step: 166155 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:10:36,967-Speed 130.93 samples/sec Loss 1.4785 Epoch: 16 Global Step: 166160 Fp16 Grad Scale: 4096 Required: 98 hours Training: 2021-06-14 23:10:56,491-Speed 131.13 samples/sec Loss 1.4133 Epoch: 16 Global Step: 166165 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:11:16,295-Speed 129.27 samples/sec Loss 1.5324 Epoch: 16 Global Step: 166170 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:11:35,862-Speed 130.84 samples/sec Loss 1.5238 Epoch: 16 Global Step: 166175 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:11:55,513-Speed 130.28 samples/sec Loss 1.5642 Epoch: 16 Global Step: 166180 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:12:15,034-Speed 131.15 samples/sec Loss 1.5366 Epoch: 16 Global Step: 166185 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:12:34,629-Speed 130.65 samples/sec Loss 1.4507 Epoch: 16 Global Step: 166190 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:12:54,156-Speed 131.11 samples/sec Loss 1.4470 Epoch: 16 Global Step: 166195 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:13:13,718-Speed 130.87 samples/sec Loss 1.5151 Epoch: 16 Global Step: 166200 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:13:33,287-Speed 130.83 samples/sec Loss 1.5191 Epoch: 16 Global Step: 166205 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:13:52,844-Speed 130.90 samples/sec Loss 1.4660 Epoch: 16 Global Step: 166210 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:14:12,647-Speed 129.28 samples/sec Loss 1.4342 Epoch: 16 Global Step: 166215 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:14:32,344-Speed 129.98 samples/sec Loss 1.4520 Epoch: 16 Global Step: 166220 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:14:52,101-Speed 129.58 samples/sec Loss 1.4421 Epoch: 16 Global Step: 166225 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:15:11,579-Speed 131.44 samples/sec Loss 1.3752 Epoch: 16 Global Step: 166230 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:15:31,183-Speed 130.59 samples/sec Loss 1.4911 Epoch: 16 Global Step: 166235 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:15:50,742-Speed 130.90 samples/sec Loss 1.4680 Epoch: 16 Global Step: 166240 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:16:10,353-Speed 130.55 samples/sec Loss 1.5931 Epoch: 16 Global Step: 166245 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:16:29,856-Speed 131.26 samples/sec Loss 1.6052 Epoch: 16 Global Step: 166250 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:16:49,353-Speed 131.31 samples/sec Loss 1.4655 Epoch: 16 Global Step: 166255 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:17:09,159-Speed 129.26 samples/sec Loss 1.4257 Epoch: 16 Global Step: 166260 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:17:28,887-Speed 129.77 samples/sec Loss 1.4021 Epoch: 16 Global Step: 166265 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:17:48,701-Speed 129.20 samples/sec Loss 1.3260 Epoch: 16 Global Step: 166270 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:18:08,236-Speed 131.06 samples/sec Loss 1.3991 Epoch: 16 Global Step: 166275 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:18:28,028-Speed 129.35 samples/sec Loss 1.4045 Epoch: 16 Global Step: 166280 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:18:47,870-Speed 129.02 samples/sec Loss 1.4184 Epoch: 16 Global Step: 166285 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:19:07,508-Speed 130.37 samples/sec Loss 1.5209 Epoch: 16 Global Step: 166290 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:19:26,946-Speed 131.71 samples/sec Loss 1.3939 Epoch: 16 Global Step: 166295 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:19:46,501-Speed 130.92 samples/sec Loss 1.3719 Epoch: 16 Global Step: 166300 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:20:06,296-Speed 129.33 samples/sec Loss 1.4625 Epoch: 16 Global Step: 166305 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:20:25,966-Speed 130.15 samples/sec Loss 1.3917 Epoch: 16 Global Step: 166310 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:20:45,386-Speed 131.83 samples/sec Loss 1.4969 Epoch: 16 Global Step: 166315 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:21:04,936-Speed 130.96 samples/sec Loss 1.4636 Epoch: 16 Global Step: 166320 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:21:24,496-Speed 130.88 samples/sec Loss 1.5559 Epoch: 16 Global Step: 166325 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:21:44,010-Speed 131.20 samples/sec Loss 1.3642 Epoch: 16 Global Step: 166330 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:22:03,650-Speed 130.35 samples/sec Loss 1.6080 Epoch: 16 Global Step: 166335 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:22:23,059-Speed 131.91 samples/sec Loss 1.4695 Epoch: 16 Global Step: 166340 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:22:42,666-Speed 130.57 samples/sec Loss 1.4734 Epoch: 16 Global Step: 166345 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:23:02,419-Speed 129.61 samples/sec Loss 1.4610 Epoch: 16 Global Step: 166350 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:23:22,124-Speed 129.92 samples/sec Loss 1.6096 Epoch: 16 Global Step: 166355 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:23:41,660-Speed 131.05 samples/sec Loss 1.4270 Epoch: 16 Global Step: 166360 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:24:01,264-Speed 130.60 samples/sec Loss 1.3888 Epoch: 16 Global Step: 166365 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:24:20,834-Speed 130.82 samples/sec Loss 1.4420 Epoch: 16 Global Step: 166370 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:24:40,378-Speed 130.99 samples/sec Loss 1.3936 Epoch: 16 Global Step: 166375 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:25:00,236-Speed 128.92 samples/sec Loss 1.3970 Epoch: 16 Global Step: 166380 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:25:19,787-Speed 130.95 samples/sec Loss 1.5742 Epoch: 16 Global Step: 166385 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:25:39,392-Speed 130.59 samples/sec Loss 1.4279 Epoch: 16 Global Step: 166390 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:25:59,132-Speed 129.69 samples/sec Loss 1.4734 Epoch: 16 Global Step: 166395 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:26:18,842-Speed 129.89 samples/sec Loss 1.4152 Epoch: 16 Global Step: 166400 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:26:38,277-Speed 131.72 samples/sec Loss 1.5831 Epoch: 16 Global Step: 166405 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:26:57,878-Speed 130.62 samples/sec Loss 1.6290 Epoch: 16 Global Step: 166410 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:27:17,397-Speed 131.16 samples/sec Loss 1.5174 Epoch: 16 Global Step: 166415 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:27:36,932-Speed 131.05 samples/sec Loss 1.5916 Epoch: 16 Global Step: 166420 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:27:56,498-Speed 130.85 samples/sec Loss 1.4362 Epoch: 16 Global Step: 166425 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:28:16,069-Speed 130.81 samples/sec Loss 1.4411 Epoch: 16 Global Step: 166430 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:28:35,679-Speed 130.56 samples/sec Loss 1.5247 Epoch: 16 Global Step: 166435 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:28:55,538-Speed 128.92 samples/sec Loss 1.4443 Epoch: 16 Global Step: 166440 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:29:15,024-Speed 131.38 samples/sec Loss 1.5578 Epoch: 16 Global Step: 166445 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:29:34,635-Speed 130.55 samples/sec Loss 1.5175 Epoch: 16 Global Step: 166450 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:29:54,325-Speed 130.06 samples/sec Loss 1.4436 Epoch: 16 Global Step: 166455 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:30:13,893-Speed 130.83 samples/sec Loss 1.4152 Epoch: 16 Global Step: 166460 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:30:33,489-Speed 130.65 samples/sec Loss 1.5567 Epoch: 16 Global Step: 166465 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:30:52,907-Speed 131.85 samples/sec Loss 1.5105 Epoch: 16 Global Step: 166470 Fp16 Grad Scale: 8192 Required: 98 hours Training: 2021-06-14 23:31:12,499-Speed 130.67 samples/sec Loss 1.5732 Epoch: 16 Global Step: 166475 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:31:32,085-Speed 130.72 samples/sec Loss 1.5554 Epoch: 16 Global Step: 166480 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:31:51,981-Speed 128.67 samples/sec Loss 1.6183 Epoch: 16 Global Step: 166485 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:32:11,397-Speed 131.86 samples/sec Loss 1.4037 Epoch: 16 Global Step: 166490 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:32:31,054-Speed 130.24 samples/sec Loss 1.4063 Epoch: 16 Global Step: 166495 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:32:50,634-Speed 130.75 samples/sec Loss 1.6043 Epoch: 16 Global Step: 166500 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:33:10,173-Speed 131.03 samples/sec Loss 1.3051 Epoch: 16 Global Step: 166505 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:33:29,774-Speed 130.61 samples/sec Loss 1.4758 Epoch: 16 Global Step: 166510 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:33:49,302-Speed 131.10 samples/sec Loss 1.6443 Epoch: 16 Global Step: 166515 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:34:08,908-Speed 130.58 samples/sec Loss 1.5187 Epoch: 16 Global Step: 166520 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:34:28,446-Speed 131.03 samples/sec Loss 1.4296 Epoch: 16 Global Step: 166525 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:34:48,325-Speed 128.79 samples/sec Loss 1.3854 Epoch: 16 Global Step: 166530 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:35:07,746-Speed 131.82 samples/sec Loss 1.4773 Epoch: 16 Global Step: 166535 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:35:27,461-Speed 129.85 samples/sec Loss 1.5611 Epoch: 16 Global Step: 166540 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:35:47,008-Speed 130.98 samples/sec Loss 1.4642 Epoch: 16 Global Step: 166545 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:36:06,486-Speed 131.44 samples/sec Loss 1.5737 Epoch: 16 Global Step: 166550 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:36:26,053-Speed 130.84 samples/sec Loss 1.5596 Epoch: 16 Global Step: 166555 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:36:45,599-Speed 130.98 samples/sec Loss 1.4164 Epoch: 16 Global Step: 166560 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:37:05,189-Speed 130.69 samples/sec Loss 1.6110 Epoch: 16 Global Step: 166565 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:37:24,723-Speed 131.06 samples/sec Loss 1.3744 Epoch: 16 Global Step: 166570 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:37:44,613-Speed 128.71 samples/sec Loss 1.4804 Epoch: 16 Global Step: 166575 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:38:04,016-Speed 131.94 samples/sec Loss 1.4099 Epoch: 16 Global Step: 166580 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:38:23,726-Speed 129.89 samples/sec Loss 1.4458 Epoch: 16 Global Step: 166585 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:38:43,263-Speed 131.04 samples/sec Loss 1.5121 Epoch: 16 Global Step: 166590 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:39:02,723-Speed 131.56 samples/sec Loss 1.4762 Epoch: 16 Global Step: 166595 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:39:22,900-Speed 126.88 samples/sec Loss 1.5253 Epoch: 16 Global Step: 166600 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:39:42,610-Speed 129.90 samples/sec Loss 1.3962 Epoch: 16 Global Step: 166605 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:40:02,208-Speed 130.63 samples/sec Loss 1.6533 Epoch: 16 Global Step: 166610 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:40:21,804-Speed 130.65 samples/sec Loss 1.4591 Epoch: 16 Global Step: 166615 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:40:41,608-Speed 129.27 samples/sec Loss 1.4153 Epoch: 16 Global Step: 166620 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:41:01,047-Speed 131.70 samples/sec Loss 1.3906 Epoch: 16 Global Step: 166625 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:41:20,845-Speed 129.31 samples/sec Loss 1.4499 Epoch: 16 Global Step: 166630 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:41:40,382-Speed 131.04 samples/sec Loss 1.5273 Epoch: 16 Global Step: 166635 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:41:59,925-Speed 131.00 samples/sec Loss 1.4273 Epoch: 16 Global Step: 166640 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:42:19,423-Speed 131.30 samples/sec Loss 1.4475 Epoch: 16 Global Step: 166645 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:42:39,042-Speed 130.50 samples/sec Loss 1.4779 Epoch: 16 Global Step: 166650 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:42:58,639-Speed 130.64 samples/sec Loss 1.4441 Epoch: 16 Global Step: 166655 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:43:18,130-Speed 131.35 samples/sec Loss 1.4875 Epoch: 16 Global Step: 166660 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:43:37,933-Speed 129.28 samples/sec Loss 1.4936 Epoch: 16 Global Step: 166665 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:43:57,362-Speed 131.77 samples/sec Loss 1.4327 Epoch: 16 Global Step: 166670 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:44:17,133-Speed 129.49 samples/sec Loss 1.4135 Epoch: 16 Global Step: 166675 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:44:36,575-Speed 131.68 samples/sec Loss 1.6147 Epoch: 16 Global Step: 166680 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:44:56,177-Speed 130.60 samples/sec Loss 1.5164 Epoch: 16 Global Step: 166685 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:45:15,640-Speed 131.54 samples/sec Loss 1.4957 Epoch: 16 Global Step: 166690 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:45:35,225-Speed 130.72 samples/sec Loss 1.5932 Epoch: 16 Global Step: 166695 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:45:54,829-Speed 130.60 samples/sec Loss 1.4722 Epoch: 16 Global Step: 166700 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:46:14,261-Speed 131.75 samples/sec Loss 1.4047 Epoch: 16 Global Step: 166705 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:46:34,049-Speed 129.38 samples/sec Loss 1.4475 Epoch: 16 Global Step: 166710 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:46:53,762-Speed 129.87 samples/sec Loss 1.4699 Epoch: 16 Global Step: 166715 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:47:13,284-Speed 131.14 samples/sec Loss 1.5897 Epoch: 16 Global Step: 166720 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:47:32,773-Speed 131.37 samples/sec Loss 1.5034 Epoch: 16 Global Step: 166725 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:47:52,339-Speed 130.84 samples/sec Loss 1.4664 Epoch: 16 Global Step: 166730 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:48:11,935-Speed 130.65 samples/sec Loss 1.3880 Epoch: 16 Global Step: 166735 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:48:31,482-Speed 130.97 samples/sec Loss 1.4262 Epoch: 16 Global Step: 166740 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:48:51,037-Speed 130.93 samples/sec Loss 1.4724 Epoch: 16 Global Step: 166745 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:49:10,578-Speed 131.01 samples/sec Loss 1.4530 Epoch: 16 Global Step: 166750 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:49:30,379-Speed 129.30 samples/sec Loss 1.6182 Epoch: 16 Global Step: 166755 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:49:50,039-Speed 130.22 samples/sec Loss 1.3185 Epoch: 16 Global Step: 166760 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:50:09,649-Speed 130.56 samples/sec Loss 1.3996 Epoch: 16 Global Step: 166765 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:50:29,311-Speed 130.21 samples/sec Loss 1.5108 Epoch: 16 Global Step: 166770 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:50:48,903-Speed 130.67 samples/sec Loss 1.6226 Epoch: 16 Global Step: 166775 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:51:08,506-Speed 130.60 samples/sec Loss 1.3982 Epoch: 16 Global Step: 166780 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:51:27,989-Speed 131.41 samples/sec Loss 1.4572 Epoch: 16 Global Step: 166785 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:51:47,557-Speed 130.84 samples/sec Loss 1.4296 Epoch: 16 Global Step: 166790 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:52:06,995-Speed 131.71 samples/sec Loss 1.4655 Epoch: 16 Global Step: 166795 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:52:26,780-Speed 129.40 samples/sec Loss 1.5290 Epoch: 16 Global Step: 166800 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:52:46,362-Speed 130.74 samples/sec Loss 1.3862 Epoch: 16 Global Step: 166805 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:53:05,991-Speed 130.42 samples/sec Loss 1.5067 Epoch: 16 Global Step: 166810 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:53:25,530-Speed 131.03 samples/sec Loss 1.3603 Epoch: 16 Global Step: 166815 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:53:45,064-Speed 131.06 samples/sec Loss 1.5406 Epoch: 16 Global Step: 166820 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:54:04,671-Speed 130.58 samples/sec Loss 1.3581 Epoch: 16 Global Step: 166825 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:54:24,214-Speed 131.00 samples/sec Loss 1.5463 Epoch: 16 Global Step: 166830 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:54:43,846-Speed 130.41 samples/sec Loss 1.4125 Epoch: 16 Global Step: 166835 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:55:03,289-Speed 131.67 samples/sec Loss 1.4300 Epoch: 16 Global Step: 166840 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:55:23,096-Speed 129.26 samples/sec Loss 1.5410 Epoch: 16 Global Step: 166845 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:55:42,750-Speed 130.26 samples/sec Loss 1.4650 Epoch: 16 Global Step: 166850 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:56:02,419-Speed 130.16 samples/sec Loss 1.4092 Epoch: 16 Global Step: 166855 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:56:21,963-Speed 131.00 samples/sec Loss 1.4160 Epoch: 16 Global Step: 166860 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:56:41,765-Speed 129.28 samples/sec Loss 1.4335 Epoch: 16 Global Step: 166865 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:57:01,375-Speed 130.56 samples/sec Loss 1.5994 Epoch: 16 Global Step: 166870 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:57:20,854-Speed 131.43 samples/sec Loss 1.4743 Epoch: 16 Global Step: 166875 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:57:40,407-Speed 130.94 samples/sec Loss 1.3190 Epoch: 16 Global Step: 166880 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:57:59,946-Speed 131.03 samples/sec Loss 1.4588 Epoch: 16 Global Step: 166885 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:58:19,747-Speed 129.30 samples/sec Loss 1.5016 Epoch: 16 Global Step: 166890 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:58:39,530-Speed 129.41 samples/sec Loss 1.5370 Epoch: 16 Global Step: 166895 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:58:58,961-Speed 131.76 samples/sec Loss 1.4137 Epoch: 16 Global Step: 166900 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:59:18,498-Speed 131.04 samples/sec Loss 1.4057 Epoch: 16 Global Step: 166905 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:59:38,119-Speed 130.48 samples/sec Loss 1.4943 Epoch: 16 Global Step: 166910 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-14 23:59:57,722-Speed 130.60 samples/sec Loss 1.4329 Epoch: 16 Global Step: 166915 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:00:17,268-Speed 130.99 samples/sec Loss 1.4104 Epoch: 16 Global Step: 166920 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:00:36,794-Speed 131.11 samples/sec Loss 1.2895 Epoch: 16 Global Step: 166925 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:00:56,377-Speed 130.73 samples/sec Loss 1.4898 Epoch: 16 Global Step: 166930 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:01:16,140-Speed 129.55 samples/sec Loss 1.3631 Epoch: 16 Global Step: 166935 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:01:35,848-Speed 129.90 samples/sec Loss 1.4244 Epoch: 16 Global Step: 166940 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:01:55,411-Speed 130.87 samples/sec Loss 1.3992 Epoch: 16 Global Step: 166945 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:02:15,021-Speed 130.55 samples/sec Loss 1.4837 Epoch: 16 Global Step: 166950 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:02:34,498-Speed 131.44 samples/sec Loss 1.3047 Epoch: 16 Global Step: 166955 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:02:54,069-Speed 130.81 samples/sec Loss 1.4618 Epoch: 16 Global Step: 166960 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:03:13,599-Speed 131.09 samples/sec Loss 1.3844 Epoch: 16 Global Step: 166965 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:03:33,178-Speed 130.76 samples/sec Loss 1.4778 Epoch: 16 Global Step: 166970 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:03:52,799-Speed 130.48 samples/sec Loss 1.5524 Epoch: 16 Global Step: 166975 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:04:12,898-Speed 127.37 samples/sec Loss 1.3821 Epoch: 16 Global Step: 166980 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:04:32,440-Speed 131.01 samples/sec Loss 1.4528 Epoch: 16 Global Step: 166985 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:04:51,913-Speed 131.48 samples/sec Loss 1.5159 Epoch: 16 Global Step: 166990 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:05:11,518-Speed 130.58 samples/sec Loss 1.6461 Epoch: 16 Global Step: 166995 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:05:31,157-Speed 130.36 samples/sec Loss 1.3648 Epoch: 16 Global Step: 167000 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:05:50,807-Speed 130.29 samples/sec Loss 1.4123 Epoch: 16 Global Step: 167005 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:06:10,342-Speed 131.05 samples/sec Loss 1.3578 Epoch: 16 Global Step: 167010 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:06:29,945-Speed 130.60 samples/sec Loss 1.3864 Epoch: 16 Global Step: 167015 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:06:49,478-Speed 131.07 samples/sec Loss 1.5133 Epoch: 16 Global Step: 167020 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:07:09,454-Speed 128.16 samples/sec Loss 1.3617 Epoch: 16 Global Step: 167025 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:07:28,886-Speed 131.75 samples/sec Loss 1.4579 Epoch: 16 Global Step: 167030 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:07:48,579-Speed 130.00 samples/sec Loss 1.3924 Epoch: 16 Global Step: 167035 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:08:08,090-Speed 131.22 samples/sec Loss 1.4185 Epoch: 16 Global Step: 167040 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:08:27,712-Speed 130.47 samples/sec Loss 1.4436 Epoch: 16 Global Step: 167045 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:08:47,300-Speed 130.70 samples/sec Loss 1.4701 Epoch: 16 Global Step: 167050 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:09:06,845-Speed 130.98 samples/sec Loss 1.3519 Epoch: 16 Global Step: 167055 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:09:26,525-Speed 130.09 samples/sec Loss 1.3685 Epoch: 16 Global Step: 167060 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:09:45,977-Speed 131.62 samples/sec Loss 1.4887 Epoch: 16 Global Step: 167065 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:10:05,784-Speed 129.25 samples/sec Loss 1.3350 Epoch: 16 Global Step: 167070 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:10:25,216-Speed 131.75 samples/sec Loss 1.4584 Epoch: 16 Global Step: 167075 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:10:44,920-Speed 129.93 samples/sec Loss 1.4872 Epoch: 16 Global Step: 167080 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:11:04,465-Speed 130.99 samples/sec Loss 1.3732 Epoch: 16 Global Step: 167085 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:11:24,006-Speed 131.01 samples/sec Loss 1.4995 Epoch: 16 Global Step: 167090 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:11:43,601-Speed 130.65 samples/sec Loss 1.5352 Epoch: 16 Global Step: 167095 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:12:03,130-Speed 131.10 samples/sec Loss 1.4068 Epoch: 16 Global Step: 167100 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:12:22,696-Speed 130.84 samples/sec Loss 1.3266 Epoch: 16 Global Step: 167105 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:12:42,303-Speed 130.57 samples/sec Loss 1.5380 Epoch: 16 Global Step: 167110 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:13:02,097-Speed 129.34 samples/sec Loss 1.4223 Epoch: 16 Global Step: 167115 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:13:21,669-Speed 130.81 samples/sec Loss 1.5168 Epoch: 16 Global Step: 167120 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:13:41,681-Speed 127.93 samples/sec Loss 1.4480 Epoch: 16 Global Step: 167125 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:14:01,284-Speed 130.60 samples/sec Loss 1.4455 Epoch: 16 Global Step: 167130 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:14:20,902-Speed 130.50 samples/sec Loss 1.4202 Epoch: 16 Global Step: 167135 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:14:40,422-Speed 131.15 samples/sec Loss 1.5889 Epoch: 16 Global Step: 167140 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:14:59,981-Speed 130.90 samples/sec Loss 1.3951 Epoch: 16 Global Step: 167145 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:15:19,548-Speed 130.84 samples/sec Loss 1.6076 Epoch: 16 Global Step: 167150 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:15:39,157-Speed 130.56 samples/sec Loss 1.5033 Epoch: 16 Global Step: 167155 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:15:58,957-Speed 129.30 samples/sec Loss 1.4573 Epoch: 16 Global Step: 167160 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:16:18,490-Speed 131.07 samples/sec Loss 1.4575 Epoch: 16 Global Step: 167165 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:16:38,022-Speed 131.08 samples/sec Loss 1.4379 Epoch: 16 Global Step: 167170 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:16:57,626-Speed 130.59 samples/sec Loss 1.5453 Epoch: 16 Global Step: 167175 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:17:17,194-Speed 130.83 samples/sec Loss 1.5104 Epoch: 16 Global Step: 167180 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:17:36,620-Speed 131.79 samples/sec Loss 1.3896 Epoch: 16 Global Step: 167185 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:17:56,219-Speed 130.62 samples/sec Loss 1.2993 Epoch: 16 Global Step: 167190 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:18:15,722-Speed 131.27 samples/sec Loss 1.4783 Epoch: 16 Global Step: 167195 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:18:35,332-Speed 130.56 samples/sec Loss 1.4057 Epoch: 16 Global Step: 167200 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:18:55,126-Speed 129.34 samples/sec Loss 1.5180 Epoch: 16 Global Step: 167205 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:19:14,734-Speed 130.56 samples/sec Loss 1.4123 Epoch: 16 Global Step: 167210 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:19:34,298-Speed 130.86 samples/sec Loss 1.5908 Epoch: 16 Global Step: 167215 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:19:53,914-Speed 130.51 samples/sec Loss 1.3956 Epoch: 16 Global Step: 167220 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:20:13,484-Speed 130.82 samples/sec Loss 1.4457 Epoch: 16 Global Step: 167225 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:20:33,025-Speed 131.01 samples/sec Loss 1.4746 Epoch: 16 Global Step: 167230 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:20:53,136-Speed 127.30 samples/sec Loss 1.4953 Epoch: 16 Global Step: 167235 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:21:12,669-Speed 131.06 samples/sec Loss 1.4264 Epoch: 16 Global Step: 167240 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:21:32,200-Speed 131.08 samples/sec Loss 1.5368 Epoch: 16 Global Step: 167245 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:21:51,952-Speed 129.62 samples/sec Loss 1.3472 Epoch: 16 Global Step: 167250 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:22:11,667-Speed 129.85 samples/sec Loss 1.4088 Epoch: 16 Global Step: 167255 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:22:31,289-Speed 130.48 samples/sec Loss 1.5507 Epoch: 16 Global Step: 167260 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:22:50,897-Speed 130.57 samples/sec Loss 1.3978 Epoch: 16 Global Step: 167265 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:23:10,425-Speed 131.10 samples/sec Loss 1.3784 Epoch: 16 Global Step: 167270 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:23:30,016-Speed 130.68 samples/sec Loss 1.4645 Epoch: 16 Global Step: 167275 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:23:49,738-Speed 129.81 samples/sec Loss 1.4785 Epoch: 16 Global Step: 167280 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:24:09,335-Speed 130.64 samples/sec Loss 1.5413 Epoch: 16 Global Step: 167285 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:24:28,972-Speed 130.38 samples/sec Loss 1.4171 Epoch: 16 Global Step: 167290 Fp16 Grad Scale: 8192 Required: 97 hours Training: 2021-06-15 00:24:48,781-Speed 129.24 samples/sec Loss 1.4882 Epoch: 16 Global Step: 167295 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:25:08,344-Speed 130.86 samples/sec Loss 1.5757 Epoch: 16 Global Step: 167300 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:25:27,957-Speed 130.53 samples/sec Loss 1.3580 Epoch: 16 Global Step: 167305 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:25:47,550-Speed 130.67 samples/sec Loss 1.2821 Epoch: 16 Global Step: 167310 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:26:07,194-Speed 130.32 samples/sec Loss 1.3836 Epoch: 16 Global Step: 167315 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:26:26,816-Speed 130.47 samples/sec Loss 1.4396 Epoch: 16 Global Step: 167320 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:26:46,251-Speed 131.73 samples/sec Loss 1.5289 Epoch: 16 Global Step: 167325 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:27:05,857-Speed 130.58 samples/sec Loss 1.6723 Epoch: 16 Global Step: 167330 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:27:25,402-Speed 130.99 samples/sec Loss 1.5181 Epoch: 16 Global Step: 167335 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:27:45,425-Speed 127.86 samples/sec Loss 1.3871 Epoch: 16 Global Step: 167340 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:28:04,850-Speed 131.80 samples/sec Loss 1.4380 Epoch: 16 Global Step: 167345 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:28:24,564-Speed 129.86 samples/sec Loss 1.5874 Epoch: 16 Global Step: 167350 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:28:44,160-Speed 130.65 samples/sec Loss 1.5191 Epoch: 16 Global Step: 167355 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:29:03,737-Speed 130.77 samples/sec Loss 1.5496 Epoch: 16 Global Step: 167360 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:29:23,345-Speed 130.57 samples/sec Loss 1.4869 Epoch: 16 Global Step: 167365 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:29:42,809-Speed 131.53 samples/sec Loss 1.3124 Epoch: 16 Global Step: 167370 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:30:02,395-Speed 130.71 samples/sec Loss 1.5441 Epoch: 16 Global Step: 167375 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:30:21,839-Speed 131.66 samples/sec Loss 1.3919 Epoch: 16 Global Step: 167380 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:30:41,750-Speed 128.58 samples/sec Loss 1.4702 Epoch: 16 Global Step: 167385 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:31:01,161-Speed 131.89 samples/sec Loss 1.3658 Epoch: 16 Global Step: 167390 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:31:20,854-Speed 130.00 samples/sec Loss 1.5544 Epoch: 16 Global Step: 167395 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:31:40,426-Speed 130.81 samples/sec Loss 1.6334 Epoch: 16 Global Step: 167400 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:31:59,986-Speed 130.88 samples/sec Loss 1.5147 Epoch: 16 Global Step: 167405 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:32:19,589-Speed 130.60 samples/sec Loss 1.5273 Epoch: 16 Global Step: 167410 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:32:39,129-Speed 131.02 samples/sec Loss 1.5699 Epoch: 16 Global Step: 167415 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:32:58,727-Speed 130.63 samples/sec Loss 1.4378 Epoch: 16 Global Step: 167420 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:33:18,266-Speed 131.03 samples/sec Loss 1.3761 Epoch: 16 Global Step: 167425 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:33:38,059-Speed 129.35 samples/sec Loss 1.3080 Epoch: 16 Global Step: 167430 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:33:57,475-Speed 131.86 samples/sec Loss 1.5096 Epoch: 16 Global Step: 167435 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:34:17,185-Speed 129.89 samples/sec Loss 1.4503 Epoch: 16 Global Step: 167440 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:34:36,733-Speed 130.97 samples/sec Loss 1.3606 Epoch: 16 Global Step: 167445 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:34:56,182-Speed 131.63 samples/sec Loss 1.5053 Epoch: 16 Global Step: 167450 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:35:15,933-Speed 129.62 samples/sec Loss 1.4593 Epoch: 16 Global Step: 167455 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:35:35,554-Speed 130.48 samples/sec Loss 1.3980 Epoch: 16 Global Step: 167460 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:35:55,147-Speed 130.66 samples/sec Loss 1.5146 Epoch: 16 Global Step: 167465 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:36:14,957-Speed 129.24 samples/sec Loss 1.3776 Epoch: 16 Global Step: 167470 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:36:34,792-Speed 129.07 samples/sec Loss 1.4561 Epoch: 16 Global Step: 167475 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:36:54,231-Speed 131.70 samples/sec Loss 1.4773 Epoch: 16 Global Step: 167480 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:37:13,936-Speed 129.92 samples/sec Loss 1.3891 Epoch: 16 Global Step: 167485 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:37:33,481-Speed 130.99 samples/sec Loss 1.4230 Epoch: 16 Global Step: 167490 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:37:53,021-Speed 131.02 samples/sec Loss 1.3062 Epoch: 16 Global Step: 167495 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:38:12,530-Speed 131.23 samples/sec Loss 1.5584 Epoch: 16 Global Step: 167500 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:38:32,189-Speed 130.23 samples/sec Loss 1.3859 Epoch: 16 Global Step: 167505 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:38:51,757-Speed 130.83 samples/sec Loss 1.4314 Epoch: 16 Global Step: 167510 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:39:11,286-Speed 131.10 samples/sec Loss 1.5643 Epoch: 16 Global Step: 167515 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:39:31,071-Speed 129.40 samples/sec Loss 1.5088 Epoch: 16 Global Step: 167520 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:39:50,649-Speed 130.76 samples/sec Loss 1.4621 Epoch: 16 Global Step: 167525 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:40:10,357-Speed 129.91 samples/sec Loss 1.4466 Epoch: 16 Global Step: 167530 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:40:29,825-Speed 131.50 samples/sec Loss 1.3691 Epoch: 16 Global Step: 167535 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:40:49,425-Speed 130.62 samples/sec Loss 1.4901 Epoch: 16 Global Step: 167540 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:41:08,953-Speed 131.10 samples/sec Loss 1.5412 Epoch: 16 Global Step: 167545 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:41:28,562-Speed 130.56 samples/sec Loss 1.4029 Epoch: 16 Global Step: 167550 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:41:48,153-Speed 130.68 samples/sec Loss 1.4349 Epoch: 16 Global Step: 167555 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:42:07,580-Speed 131.78 samples/sec Loss 1.3919 Epoch: 16 Global Step: 167560 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:42:27,361-Speed 129.43 samples/sec Loss 1.4926 Epoch: 16 Global Step: 167565 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:42:46,934-Speed 130.80 samples/sec Loss 1.4394 Epoch: 16 Global Step: 167570 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:43:06,957-Speed 127.86 samples/sec Loss 1.6778 Epoch: 16 Global Step: 167575 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:43:26,375-Speed 131.84 samples/sec Loss 1.3996 Epoch: 16 Global Step: 167580 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:43:45,991-Speed 130.52 samples/sec Loss 1.4159 Epoch: 16 Global Step: 167585 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:44:05,493-Speed 131.27 samples/sec Loss 1.3975 Epoch: 16 Global Step: 167590 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:44:25,109-Speed 130.52 samples/sec Loss 1.4443 Epoch: 16 Global Step: 167595 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:44:44,670-Speed 130.88 samples/sec Loss 1.3235 Epoch: 16 Global Step: 167600 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:45:04,111-Speed 131.69 samples/sec Loss 1.5219 Epoch: 16 Global Step: 167605 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:45:23,907-Speed 129.33 samples/sec Loss 1.4398 Epoch: 16 Global Step: 167610 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:45:43,551-Speed 130.33 samples/sec Loss 1.4774 Epoch: 16 Global Step: 167615 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:46:03,052-Speed 131.28 samples/sec Loss 1.3464 Epoch: 16 Global Step: 167620 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:46:22,671-Speed 130.50 samples/sec Loss 1.3985 Epoch: 16 Global Step: 167625 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:46:42,214-Speed 131.00 samples/sec Loss 1.4697 Epoch: 16 Global Step: 167630 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:47:01,741-Speed 131.11 samples/sec Loss 1.4337 Epoch: 16 Global Step: 167635 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:47:21,313-Speed 130.81 samples/sec Loss 1.3912 Epoch: 16 Global Step: 167640 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:47:40,765-Speed 131.62 samples/sec Loss 1.4366 Epoch: 16 Global Step: 167645 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:48:00,345-Speed 130.75 samples/sec Loss 1.4348 Epoch: 16 Global Step: 167650 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:48:20,105-Speed 129.56 samples/sec Loss 1.7024 Epoch: 16 Global Step: 167655 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:48:39,823-Speed 129.84 samples/sec Loss 1.4796 Epoch: 16 Global Step: 167660 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:48:59,264-Speed 131.69 samples/sec Loss 1.4041 Epoch: 16 Global Step: 167665 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:49:18,905-Speed 130.35 samples/sec Loss 1.3222 Epoch: 16 Global Step: 167670 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:49:38,435-Speed 131.08 samples/sec Loss 1.4495 Epoch: 16 Global Step: 167675 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:49:58,220-Speed 129.40 samples/sec Loss 1.5003 Epoch: 16 Global Step: 167680 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:50:17,761-Speed 131.01 samples/sec Loss 1.3340 Epoch: 16 Global Step: 167685 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:50:37,224-Speed 131.54 samples/sec Loss 1.3653 Epoch: 16 Global Step: 167690 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:50:56,797-Speed 130.80 samples/sec Loss 1.5111 Epoch: 16 Global Step: 167695 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:51:16,672-Speed 128.81 samples/sec Loss 1.3345 Epoch: 16 Global Step: 167700 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:51:36,316-Speed 130.32 samples/sec Loss 1.5168 Epoch: 16 Global Step: 167705 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:51:55,917-Speed 130.61 samples/sec Loss 1.5243 Epoch: 16 Global Step: 167710 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:52:15,523-Speed 130.58 samples/sec Loss 1.5420 Epoch: 16 Global Step: 167715 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:52:34,944-Speed 131.82 samples/sec Loss 1.5065 Epoch: 16 Global Step: 167720 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:52:54,554-Speed 130.56 samples/sec Loss 1.4288 Epoch: 16 Global Step: 167725 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:53:14,234-Speed 130.09 samples/sec Loss 1.4720 Epoch: 16 Global Step: 167730 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:53:33,711-Speed 131.45 samples/sec Loss 1.5590 Epoch: 16 Global Step: 167735 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:53:53,279-Speed 130.84 samples/sec Loss 1.4259 Epoch: 16 Global Step: 167740 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:54:13,088-Speed 129.24 samples/sec Loss 1.3392 Epoch: 16 Global Step: 167745 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:54:32,523-Speed 131.72 samples/sec Loss 1.4885 Epoch: 16 Global Step: 167750 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:54:52,047-Speed 131.13 samples/sec Loss 1.4151 Epoch: 16 Global Step: 167755 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:55:11,598-Speed 130.94 samples/sec Loss 1.4377 Epoch: 16 Global Step: 167760 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:55:31,030-Speed 131.75 samples/sec Loss 1.4553 Epoch: 16 Global Step: 167765 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:55:50,631-Speed 130.61 samples/sec Loss 1.5083 Epoch: 16 Global Step: 167770 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:56:10,044-Speed 131.88 samples/sec Loss 1.3452 Epoch: 16 Global Step: 167775 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:56:29,531-Speed 131.38 samples/sec Loss 1.5260 Epoch: 16 Global Step: 167780 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:56:49,109-Speed 130.77 samples/sec Loss 1.4550 Epoch: 16 Global Step: 167785 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:57:08,960-Speed 128.97 samples/sec Loss 1.4323 Epoch: 16 Global Step: 167790 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:57:28,380-Speed 131.83 samples/sec Loss 1.4021 Epoch: 16 Global Step: 167795 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:57:47,980-Speed 130.62 samples/sec Loss 1.5156 Epoch: 16 Global Step: 167800 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:58:07,405-Speed 131.80 samples/sec Loss 1.4330 Epoch: 16 Global Step: 167805 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:58:27,034-Speed 130.42 samples/sec Loss 1.4695 Epoch: 16 Global Step: 167810 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:58:46,631-Speed 130.64 samples/sec Loss 1.5248 Epoch: 16 Global Step: 167815 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:59:06,872-Speed 126.49 samples/sec Loss 1.3647 Epoch: 16 Global Step: 167820 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:59:26,479-Speed 130.57 samples/sec Loss 1.4536 Epoch: 16 Global Step: 167825 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 00:59:45,971-Speed 131.35 samples/sec Loss 1.4823 Epoch: 16 Global Step: 167830 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:00:05,803-Speed 129.09 samples/sec Loss 1.5643 Epoch: 16 Global Step: 167835 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:00:25,258-Speed 131.59 samples/sec Loss 1.4578 Epoch: 16 Global Step: 167840 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:00:44,852-Speed 130.66 samples/sec Loss 1.4720 Epoch: 16 Global Step: 167845 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:01:04,407-Speed 130.92 samples/sec Loss 1.6000 Epoch: 16 Global Step: 167850 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:01:24,015-Speed 130.56 samples/sec Loss 1.5120 Epoch: 16 Global Step: 167855 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:01:43,546-Speed 131.08 samples/sec Loss 1.3993 Epoch: 16 Global Step: 167860 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:02:03,162-Speed 130.51 samples/sec Loss 1.6081 Epoch: 16 Global Step: 167865 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:02:22,905-Speed 129.67 samples/sec Loss 1.4356 Epoch: 16 Global Step: 167870 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:02:42,359-Speed 131.61 samples/sec Loss 1.3229 Epoch: 16 Global Step: 167875 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:03:02,167-Speed 129.24 samples/sec Loss 1.5815 Epoch: 16 Global Step: 167880 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:03:21,652-Speed 131.39 samples/sec Loss 1.3620 Epoch: 16 Global Step: 167885 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:03:41,267-Speed 130.52 samples/sec Loss 1.3826 Epoch: 16 Global Step: 167890 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:04:00,768-Speed 131.28 samples/sec Loss 1.5330 Epoch: 16 Global Step: 167895 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:04:20,564-Speed 129.33 samples/sec Loss 1.4939 Epoch: 16 Global Step: 167900 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:04:40,049-Speed 131.39 samples/sec Loss 1.4127 Epoch: 16 Global Step: 167905 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:04:59,662-Speed 130.53 samples/sec Loss 1.4117 Epoch: 16 Global Step: 167910 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:05:19,221-Speed 130.89 samples/sec Loss 1.3712 Epoch: 16 Global Step: 167915 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:05:38,690-Speed 131.50 samples/sec Loss 1.3901 Epoch: 16 Global Step: 167920 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:05:58,475-Speed 129.40 samples/sec Loss 1.4121 Epoch: 16 Global Step: 167925 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:06:18,106-Speed 130.41 samples/sec Loss 1.4385 Epoch: 16 Global Step: 167930 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:06:37,533-Speed 131.78 samples/sec Loss 1.3701 Epoch: 16 Global Step: 167935 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:06:57,154-Speed 130.48 samples/sec Loss 1.5659 Epoch: 16 Global Step: 167940 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:07:16,755-Speed 130.61 samples/sec Loss 1.4550 Epoch: 16 Global Step: 167945 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:07:36,276-Speed 131.15 samples/sec Loss 1.5208 Epoch: 16 Global Step: 167950 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:07:55,906-Speed 130.42 samples/sec Loss 1.4507 Epoch: 16 Global Step: 167955 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:08:15,349-Speed 131.68 samples/sec Loss 1.4426 Epoch: 16 Global Step: 167960 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:08:34,960-Speed 130.55 samples/sec Loss 1.3674 Epoch: 16 Global Step: 167965 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:08:54,931-Speed 128.20 samples/sec Loss 1.4180 Epoch: 16 Global Step: 167970 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:09:14,420-Speed 131.36 samples/sec Loss 1.4646 Epoch: 16 Global Step: 167975 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:09:34,011-Speed 130.68 samples/sec Loss 1.4957 Epoch: 16 Global Step: 167980 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:09:53,632-Speed 130.48 samples/sec Loss 1.4408 Epoch: 16 Global Step: 167985 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:10:13,146-Speed 131.19 samples/sec Loss 1.3769 Epoch: 16 Global Step: 167990 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:10:32,726-Speed 130.76 samples/sec Loss 1.5566 Epoch: 16 Global Step: 167995 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:10:52,216-Speed 131.35 samples/sec Loss 1.3427 Epoch: 16 Global Step: 168000 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:11:11,745-Speed 131.10 samples/sec Loss 1.4713 Epoch: 16 Global Step: 168005 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:11:31,323-Speed 130.76 samples/sec Loss 1.4477 Epoch: 16 Global Step: 168010 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:11:51,291-Speed 128.21 samples/sec Loss 1.4808 Epoch: 16 Global Step: 168015 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:12:10,722-Speed 131.76 samples/sec Loss 1.5232 Epoch: 16 Global Step: 168020 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:12:30,265-Speed 131.00 samples/sec Loss 1.4249 Epoch: 16 Global Step: 168025 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:12:49,875-Speed 130.55 samples/sec Loss 1.3887 Epoch: 16 Global Step: 168030 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:13:09,411-Speed 131.05 samples/sec Loss 1.4165 Epoch: 16 Global Step: 168035 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:13:29,023-Speed 130.54 samples/sec Loss 1.3258 Epoch: 16 Global Step: 168040 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:13:48,447-Speed 131.80 samples/sec Loss 1.4598 Epoch: 16 Global Step: 168045 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:14:07,989-Speed 131.01 samples/sec Loss 1.5072 Epoch: 16 Global Step: 168050 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:14:27,580-Speed 130.68 samples/sec Loss 1.5174 Epoch: 16 Global Step: 168055 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:14:47,465-Speed 128.75 samples/sec Loss 1.4121 Epoch: 16 Global Step: 168060 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:15:06,894-Speed 131.77 samples/sec Loss 1.4185 Epoch: 16 Global Step: 168065 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:15:26,488-Speed 130.66 samples/sec Loss 1.3088 Epoch: 16 Global Step: 168070 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:15:46,125-Speed 130.38 samples/sec Loss 1.4611 Epoch: 16 Global Step: 168075 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:16:05,791-Speed 130.18 samples/sec Loss 1.3734 Epoch: 16 Global Step: 168080 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:16:25,409-Speed 130.50 samples/sec Loss 1.5117 Epoch: 16 Global Step: 168085 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:16:44,929-Speed 131.15 samples/sec Loss 1.3729 Epoch: 16 Global Step: 168090 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:17:04,530-Speed 130.61 samples/sec Loss 1.4748 Epoch: 16 Global Step: 168095 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:17:24,118-Speed 130.70 samples/sec Loss 1.3754 Epoch: 16 Global Step: 168100 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:17:44,039-Speed 128.51 samples/sec Loss 1.4611 Epoch: 16 Global Step: 168105 Fp16 Grad Scale: 8192 Required: 96 hours Training: 2021-06-15 01:18:03,474-Speed 131.73 samples/sec Loss 1.4077 Epoch: 16 Global Step: 168110 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:18:23,372-Speed 128.66 samples/sec Loss 1.4527 Epoch: 16 Global Step: 168115 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:18:42,975-Speed 130.60 samples/sec Loss 1.3158 Epoch: 16 Global Step: 168120 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:19:02,646-Speed 130.15 samples/sec Loss 1.3342 Epoch: 16 Global Step: 168125 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:19:22,238-Speed 130.68 samples/sec Loss 1.5232 Epoch: 16 Global Step: 168130 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:19:41,755-Speed 131.17 samples/sec Loss 1.3810 Epoch: 16 Global Step: 168135 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:20:01,385-Speed 130.42 samples/sec Loss 1.4447 Epoch: 16 Global Step: 168140 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:20:21,001-Speed 130.51 samples/sec Loss 1.4994 Epoch: 16 Global Step: 168145 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:20:40,891-Speed 128.72 samples/sec Loss 1.3277 Epoch: 16 Global Step: 168150 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:21:00,428-Speed 131.04 samples/sec Loss 1.4032 Epoch: 16 Global Step: 168155 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:21:20,144-Speed 129.85 samples/sec Loss 1.4208 Epoch: 16 Global Step: 168160 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:21:39,707-Speed 130.86 samples/sec Loss 1.5192 Epoch: 16 Global Step: 168165 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:21:59,329-Speed 130.48 samples/sec Loss 1.3874 Epoch: 16 Global Step: 168170 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:22:18,929-Speed 130.62 samples/sec Loss 1.4526 Epoch: 16 Global Step: 168175 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:22:37,963-Speed 134.51 samples/sec Loss 1.3928 Epoch: 16 Global Step: 168180 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:22:57,678-Speed 129.85 samples/sec Loss 1.4489 Epoch: 16 Global Step: 168185 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:23:17,204-Speed 131.12 samples/sec Loss 1.3720 Epoch: 16 Global Step: 168190 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:23:36,814-Speed 130.55 samples/sec Loss 1.4881 Epoch: 16 Global Step: 168195 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:23:56,337-Speed 131.14 samples/sec Loss 1.3502 Epoch: 16 Global Step: 168200 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:24:15,914-Speed 130.77 samples/sec Loss 1.4455 Epoch: 16 Global Step: 168205 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:24:35,469-Speed 130.92 samples/sec Loss 1.3981 Epoch: 16 Global Step: 168210 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:24:54,904-Speed 131.73 samples/sec Loss 1.4248 Epoch: 16 Global Step: 168215 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:25:14,703-Speed 129.31 samples/sec Loss 1.4860 Epoch: 16 Global Step: 168220 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:25:34,397-Speed 130.00 samples/sec Loss 1.4844 Epoch: 16 Global Step: 168225 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:25:54,141-Speed 129.67 samples/sec Loss 1.3265 Epoch: 16 Global Step: 168230 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:26:13,653-Speed 131.21 samples/sec Loss 1.4999 Epoch: 16 Global Step: 168235 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:26:33,283-Speed 130.42 samples/sec Loss 1.5151 Epoch: 16 Global Step: 168240 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:26:52,825-Speed 131.01 samples/sec Loss 1.4289 Epoch: 16 Global Step: 168245 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:27:12,433-Speed 130.56 samples/sec Loss 1.5052 Epoch: 16 Global Step: 168250 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:27:32,051-Speed 130.50 samples/sec Loss 1.4856 Epoch: 16 Global Step: 168255 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:27:51,502-Speed 131.62 samples/sec Loss 1.5898 Epoch: 16 Global Step: 168260 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:28:11,294-Speed 129.36 samples/sec Loss 1.5525 Epoch: 16 Global Step: 168265 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:28:31,024-Speed 129.76 samples/sec Loss 1.4017 Epoch: 16 Global Step: 168270 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:28:50,651-Speed 130.44 samples/sec Loss 1.3794 Epoch: 16 Global Step: 168275 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:29:10,178-Speed 131.11 samples/sec Loss 1.4146 Epoch: 16 Global Step: 168280 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:29:29,737-Speed 130.90 samples/sec Loss 1.4429 Epoch: 16 Global Step: 168285 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:29:49,268-Speed 131.08 samples/sec Loss 1.4000 Epoch: 16 Global Step: 168290 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:30:08,882-Speed 130.53 samples/sec Loss 1.4149 Epoch: 16 Global Step: 168295 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:30:28,444-Speed 130.87 samples/sec Loss 1.4161 Epoch: 16 Global Step: 168300 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:30:47,984-Speed 131.03 samples/sec Loss 1.5302 Epoch: 16 Global Step: 168305 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:31:07,922-Speed 128.40 samples/sec Loss 1.4910 Epoch: 16 Global Step: 168310 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:31:27,581-Speed 130.23 samples/sec Loss 1.3618 Epoch: 16 Global Step: 168315 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:31:47,128-Speed 130.97 samples/sec Loss 1.5276 Epoch: 16 Global Step: 168320 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:32:06,738-Speed 130.55 samples/sec Loss 1.4965 Epoch: 16 Global Step: 168325 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:32:26,277-Speed 131.02 samples/sec Loss 1.4211 Epoch: 16 Global Step: 168330 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:32:45,818-Speed 131.01 samples/sec Loss 1.3179 Epoch: 16 Global Step: 168335 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:33:05,388-Speed 130.82 samples/sec Loss 1.3250 Epoch: 16 Global Step: 168340 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:33:24,478-Speed 134.12 samples/sec Loss 1.3014 Epoch: 16 Global Step: 168345 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:33:44,092-Speed 130.52 samples/sec Loss 1.2770 Epoch: 16 Global Step: 168350 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:34:04,037-Speed 128.36 samples/sec Loss 1.4428 Epoch: 16 Global Step: 168355 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:34:23,574-Speed 131.04 samples/sec Loss 1.3960 Epoch: 16 Global Step: 168360 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:34:43,155-Speed 130.75 samples/sec Loss 1.4298 Epoch: 16 Global Step: 168365 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:35:02,751-Speed 130.64 samples/sec Loss 1.4768 Epoch: 16 Global Step: 168370 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:35:22,286-Speed 131.06 samples/sec Loss 1.3138 Epoch: 16 Global Step: 168375 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:35:41,866-Speed 130.75 samples/sec Loss 1.3139 Epoch: 16 Global Step: 168380 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:36:01,352-Speed 131.39 samples/sec Loss 1.2808 Epoch: 16 Global Step: 168385 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:36:21,011-Speed 130.23 samples/sec Loss 1.6000 Epoch: 16 Global Step: 168390 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:36:40,629-Speed 130.50 samples/sec Loss 1.2670 Epoch: 16 Global Step: 168395 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:37:00,618-Speed 128.07 samples/sec Loss 1.3767 Epoch: 16 Global Step: 168400 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:37:20,061-Speed 131.67 samples/sec Loss 1.5637 Epoch: 16 Global Step: 168405 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:37:39,750-Speed 130.03 samples/sec Loss 1.3140 Epoch: 16 Global Step: 168410 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:37:59,333-Speed 130.73 samples/sec Loss 1.3492 Epoch: 16 Global Step: 168415 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:38:18,895-Speed 130.88 samples/sec Loss 1.4872 Epoch: 16 Global Step: 168420 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:38:38,495-Speed 130.62 samples/sec Loss 1.4512 Epoch: 16 Global Step: 168425 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:38:58,096-Speed 130.61 samples/sec Loss 1.4684 Epoch: 16 Global Step: 168430 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:39:17,644-Speed 130.97 samples/sec Loss 1.3992 Epoch: 16 Global Step: 168435 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:39:37,177-Speed 131.07 samples/sec Loss 1.4109 Epoch: 16 Global Step: 168440 Fp16 Grad Scale: 4096 Required: 95 hours Training: 2021-06-15 01:39:57,077-Speed 128.65 samples/sec Loss 1.4572 Epoch: 16 Global Step: 168445 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:40:16,502-Speed 131.80 samples/sec Loss 1.4184 Epoch: 16 Global Step: 168450 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:40:36,282-Speed 129.43 samples/sec Loss 1.3547 Epoch: 16 Global Step: 168455 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:40:55,817-Speed 131.05 samples/sec Loss 1.3155 Epoch: 16 Global Step: 168460 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:41:15,401-Speed 130.72 samples/sec Loss 1.3693 Epoch: 16 Global Step: 168465 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:41:34,960-Speed 130.90 samples/sec Loss 1.3110 Epoch: 16 Global Step: 168470 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:41:54,557-Speed 130.64 samples/sec Loss 1.3884 Epoch: 16 Global Step: 168475 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:42:14,145-Speed 130.70 samples/sec Loss 1.4815 Epoch: 16 Global Step: 168480 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:42:33,699-Speed 130.93 samples/sec Loss 1.4640 Epoch: 16 Global Step: 168485 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:42:53,576-Speed 128.80 samples/sec Loss 1.5749 Epoch: 16 Global Step: 168490 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:43:13,015-Speed 131.70 samples/sec Loss 1.4241 Epoch: 16 Global Step: 168495 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:43:32,811-Speed 129.33 samples/sec Loss 1.3883 Epoch: 16 Global Step: 168500 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:43:52,354-Speed 131.00 samples/sec Loss 1.4386 Epoch: 16 Global Step: 168505 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:44:11,890-Speed 131.05 samples/sec Loss 1.5232 Epoch: 16 Global Step: 168510 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:44:31,422-Speed 131.08 samples/sec Loss 1.3610 Epoch: 16 Global Step: 168515 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:44:51,035-Speed 130.53 samples/sec Loss 1.4879 Epoch: 16 Global Step: 168520 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:45:10,790-Speed 129.60 samples/sec Loss 1.4037 Epoch: 16 Global Step: 168525 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:45:30,550-Speed 129.56 samples/sec Loss 1.5573 Epoch: 16 Global Step: 168530 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:45:50,337-Speed 129.39 samples/sec Loss 1.3832 Epoch: 16 Global Step: 168535 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:46:10,089-Speed 129.61 samples/sec Loss 1.4033 Epoch: 16 Global Step: 168540 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:46:29,670-Speed 130.75 samples/sec Loss 1.5077 Epoch: 16 Global Step: 168545 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:46:49,212-Speed 131.01 samples/sec Loss 1.4395 Epoch: 16 Global Step: 168550 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:47:08,829-Speed 130.51 samples/sec Loss 1.5154 Epoch: 16 Global Step: 168555 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:47:28,275-Speed 131.65 samples/sec Loss 1.3716 Epoch: 16 Global Step: 168560 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:47:47,888-Speed 130.54 samples/sec Loss 1.4734 Epoch: 16 Global Step: 168565 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:48:07,431-Speed 131.00 samples/sec Loss 1.4344 Epoch: 16 Global Step: 168570 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:48:26,942-Speed 131.21 samples/sec Loss 1.3630 Epoch: 16 Global Step: 168575 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:48:46,701-Speed 129.57 samples/sec Loss 1.5235 Epoch: 16 Global Step: 168580 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:49:06,369-Speed 130.17 samples/sec Loss 1.4236 Epoch: 16 Global Step: 168585 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:49:25,932-Speed 130.86 samples/sec Loss 1.4585 Epoch: 16 Global Step: 168590 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:49:45,556-Speed 130.46 samples/sec Loss 1.3773 Epoch: 16 Global Step: 168595 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:50:05,024-Speed 131.51 samples/sec Loss 1.4631 Epoch: 16 Global Step: 168600 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:50:24,547-Speed 131.14 samples/sec Loss 1.5127 Epoch: 16 Global Step: 168605 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:50:44,158-Speed 130.55 samples/sec Loss 1.4055 Epoch: 16 Global Step: 168610 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:51:03,705-Speed 130.97 samples/sec Loss 1.4953 Epoch: 16 Global Step: 168615 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:51:23,253-Speed 130.97 samples/sec Loss 1.3786 Epoch: 16 Global Step: 168620 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:51:43,006-Speed 129.61 samples/sec Loss 1.5177 Epoch: 16 Global Step: 168625 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:52:02,673-Speed 130.18 samples/sec Loss 1.4679 Epoch: 16 Global Step: 168630 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:52:22,191-Speed 131.16 samples/sec Loss 1.2993 Epoch: 16 Global Step: 168635 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:52:41,795-Speed 130.59 samples/sec Loss 1.3781 Epoch: 16 Global Step: 168640 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:53:01,251-Speed 131.58 samples/sec Loss 1.2810 Epoch: 16 Global Step: 168645 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:53:20,865-Speed 130.53 samples/sec Loss 1.3847 Epoch: 16 Global Step: 168650 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:53:40,448-Speed 130.73 samples/sec Loss 1.3811 Epoch: 16 Global Step: 168655 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:53:59,966-Speed 131.17 samples/sec Loss 1.2354 Epoch: 16 Global Step: 168660 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:54:19,549-Speed 130.73 samples/sec Loss 1.4363 Epoch: 16 Global Step: 168665 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:54:39,274-Speed 129.79 samples/sec Loss 1.3654 Epoch: 16 Global Step: 168670 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:54:59,052-Speed 129.45 samples/sec Loss 1.2812 Epoch: 16 Global Step: 168675 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:55:18,468-Speed 131.86 samples/sec Loss 1.3910 Epoch: 16 Global Step: 168680 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:55:38,075-Speed 130.57 samples/sec Loss 1.4200 Epoch: 16 Global Step: 168685 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:55:57,725-Speed 130.29 samples/sec Loss 1.4740 Epoch: 16 Global Step: 168690 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:56:17,329-Speed 130.60 samples/sec Loss 1.4329 Epoch: 16 Global Step: 168695 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:56:36,892-Speed 130.86 samples/sec Loss 1.3881 Epoch: 16 Global Step: 168700 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:56:56,520-Speed 130.43 samples/sec Loss 1.6064 Epoch: 16 Global Step: 168705 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:57:16,112-Speed 130.68 samples/sec Loss 1.4966 Epoch: 16 Global Step: 168710 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:57:35,871-Speed 129.57 samples/sec Loss 1.4144 Epoch: 16 Global Step: 168715 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:57:55,519-Speed 130.30 samples/sec Loss 1.3269 Epoch: 16 Global Step: 168720 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:58:14,935-Speed 131.86 samples/sec Loss 1.4806 Epoch: 16 Global Step: 168725 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:58:34,532-Speed 130.64 samples/sec Loss 1.4225 Epoch: 16 Global Step: 168730 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:58:54,054-Speed 131.14 samples/sec Loss 1.4793 Epoch: 16 Global Step: 168735 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:59:13,572-Speed 131.17 samples/sec Loss 1.4198 Epoch: 16 Global Step: 168740 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:59:33,108-Speed 131.05 samples/sec Loss 1.2322 Epoch: 16 Global Step: 168745 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 01:59:52,534-Speed 131.80 samples/sec Loss 1.4025 Epoch: 16 Global Step: 168750 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:00:12,314-Speed 129.43 samples/sec Loss 1.3727 Epoch: 16 Global Step: 168755 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:00:32,225-Speed 128.58 samples/sec Loss 1.4563 Epoch: 16 Global Step: 168760 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:00:51,660-Speed 131.72 samples/sec Loss 1.4705 Epoch: 16 Global Step: 168765 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:01:11,133-Speed 131.48 samples/sec Loss 1.4951 Epoch: 16 Global Step: 168770 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:01:30,692-Speed 130.89 samples/sec Loss 1.3999 Epoch: 16 Global Step: 168775 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:01:50,723-Speed 127.81 samples/sec Loss 1.3193 Epoch: 16 Global Step: 168780 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:02:10,446-Speed 129.80 samples/sec Loss 1.3673 Epoch: 16 Global Step: 168785 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:02:29,873-Speed 131.78 samples/sec Loss 1.3843 Epoch: 16 Global Step: 168790 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:02:49,323-Speed 131.63 samples/sec Loss 1.3979 Epoch: 16 Global Step: 168795 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:03:08,889-Speed 130.85 samples/sec Loss 1.4093 Epoch: 16 Global Step: 168800 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:03:28,830-Speed 128.39 samples/sec Loss 1.3122 Epoch: 16 Global Step: 168805 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:03:48,396-Speed 130.85 samples/sec Loss 1.5043 Epoch: 16 Global Step: 168810 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:04:07,862-Speed 131.51 samples/sec Loss 1.4117 Epoch: 16 Global Step: 168815 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:04:27,443-Speed 130.75 samples/sec Loss 1.3881 Epoch: 16 Global Step: 168820 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:04:47,057-Speed 130.52 samples/sec Loss 1.5124 Epoch: 16 Global Step: 168825 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:05:06,652-Speed 130.66 samples/sec Loss 1.3708 Epoch: 16 Global Step: 168830 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:05:26,099-Speed 131.65 samples/sec Loss 1.4237 Epoch: 16 Global Step: 168835 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:05:45,695-Speed 130.65 samples/sec Loss 1.4665 Epoch: 16 Global Step: 168840 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:06:05,230-Speed 131.05 samples/sec Loss 1.3654 Epoch: 16 Global Step: 168845 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:06:25,128-Speed 128.66 samples/sec Loss 1.4610 Epoch: 16 Global Step: 168850 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:06:44,545-Speed 131.85 samples/sec Loss 1.5342 Epoch: 16 Global Step: 168855 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:07:04,157-Speed 130.54 samples/sec Loss 1.3746 Epoch: 16 Global Step: 168860 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:07:23,715-Speed 130.90 samples/sec Loss 1.4089 Epoch: 16 Global Step: 168865 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:07:43,262-Speed 130.97 samples/sec Loss 1.4382 Epoch: 16 Global Step: 168870 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:08:02,759-Speed 131.31 samples/sec Loss 1.4647 Epoch: 16 Global Step: 168875 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:08:22,208-Speed 131.63 samples/sec Loss 1.3467 Epoch: 16 Global Step: 168880 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:08:41,818-Speed 130.56 samples/sec Loss 1.3338 Epoch: 16 Global Step: 168885 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:09:01,340-Speed 131.14 samples/sec Loss 1.3306 Epoch: 16 Global Step: 168890 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:09:21,154-Speed 129.21 samples/sec Loss 1.4100 Epoch: 16 Global Step: 168895 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:09:40,632-Speed 131.44 samples/sec Loss 1.4777 Epoch: 16 Global Step: 168900 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:10:00,205-Speed 130.80 samples/sec Loss 1.3462 Epoch: 16 Global Step: 168905 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:10:19,849-Speed 130.33 samples/sec Loss 1.4271 Epoch: 16 Global Step: 168910 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:10:39,476-Speed 130.44 samples/sec Loss 1.4082 Epoch: 16 Global Step: 168915 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:10:58,891-Speed 131.87 samples/sec Loss 1.4340 Epoch: 16 Global Step: 168920 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:11:18,359-Speed 131.51 samples/sec Loss 1.5946 Epoch: 16 Global Step: 168925 Fp16 Grad Scale: 8192 Required: 95 hours Training: 2021-06-15 02:11:37,925-Speed 130.84 samples/sec Loss 1.3890 Epoch: 16 Global Step: 168930 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:11:57,452-Speed 131.11 samples/sec Loss 1.2507 Epoch: 16 Global Step: 168935 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:12:17,254-Speed 129.29 samples/sec Loss 1.3934 Epoch: 16 Global Step: 168940 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:12:36,849-Speed 130.65 samples/sec Loss 1.4689 Epoch: 16 Global Step: 168945 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:12:56,394-Speed 130.98 samples/sec Loss 1.3272 Epoch: 16 Global Step: 168950 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:13:15,923-Speed 131.10 samples/sec Loss 1.2824 Epoch: 16 Global Step: 168955 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:13:35,479-Speed 130.91 samples/sec Loss 1.4797 Epoch: 16 Global Step: 168960 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:13:54,900-Speed 131.83 samples/sec Loss 1.4083 Epoch: 16 Global Step: 168965 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:14:14,430-Speed 131.09 samples/sec Loss 1.5701 Epoch: 16 Global Step: 168970 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:14:34,017-Speed 130.70 samples/sec Loss 1.3911 Epoch: 16 Global Step: 168975 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:14:53,616-Speed 130.63 samples/sec Loss 1.4314 Epoch: 16 Global Step: 168980 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:15:13,407-Speed 129.36 samples/sec Loss 1.4783 Epoch: 16 Global Step: 168985 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:15:33,096-Speed 130.03 samples/sec Loss 1.3696 Epoch: 16 Global Step: 168990 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:15:52,567-Speed 131.49 samples/sec Loss 1.4715 Epoch: 16 Global Step: 168995 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:16:12,173-Speed 130.58 samples/sec Loss 1.3716 Epoch: 16 Global Step: 169000 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:16:31,743-Speed 130.82 samples/sec Loss 1.4074 Epoch: 16 Global Step: 169005 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:16:51,170-Speed 131.78 samples/sec Loss 1.5316 Epoch: 16 Global Step: 169010 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:17:10,748-Speed 130.77 samples/sec Loss 1.4504 Epoch: 16 Global Step: 169015 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:17:30,258-Speed 131.22 samples/sec Loss 1.4159 Epoch: 16 Global Step: 169020 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:17:49,871-Speed 130.53 samples/sec Loss 1.5179 Epoch: 16 Global Step: 169025 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:18:09,679-Speed 129.25 samples/sec Loss 1.3002 Epoch: 16 Global Step: 169030 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:18:29,485-Speed 129.26 samples/sec Loss 1.4736 Epoch: 16 Global Step: 169035 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:18:49,013-Speed 131.10 samples/sec Loss 1.3622 Epoch: 16 Global Step: 169040 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:19:08,602-Speed 130.70 samples/sec Loss 1.4210 Epoch: 16 Global Step: 169045 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:19:28,158-Speed 130.91 samples/sec Loss 1.4035 Epoch: 16 Global Step: 169050 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:19:47,678-Speed 131.16 samples/sec Loss 1.3422 Epoch: 16 Global Step: 169055 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:20:07,283-Speed 130.58 samples/sec Loss 1.4415 Epoch: 16 Global Step: 169060 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:20:26,794-Speed 131.21 samples/sec Loss 1.4710 Epoch: 16 Global Step: 169065 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:20:46,370-Speed 130.78 samples/sec Loss 1.4692 Epoch: 16 Global Step: 169070 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:21:06,142-Speed 129.48 samples/sec Loss 1.3846 Epoch: 16 Global Step: 169075 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:21:25,840-Speed 129.97 samples/sec Loss 1.3039 Epoch: 16 Global Step: 169080 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:21:45,351-Speed 131.21 samples/sec Loss 1.4267 Epoch: 16 Global Step: 169085 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:22:04,958-Speed 130.58 samples/sec Loss 1.3686 Epoch: 16 Global Step: 169090 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:22:24,394-Speed 131.72 samples/sec Loss 1.4712 Epoch: 16 Global Step: 169095 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:22:44,008-Speed 130.53 samples/sec Loss 1.4576 Epoch: 16 Global Step: 169100 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:23:03,648-Speed 130.35 samples/sec Loss 1.4435 Epoch: 16 Global Step: 169105 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:23:23,242-Speed 130.66 samples/sec Loss 1.2828 Epoch: 16 Global Step: 169110 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:23:42,754-Speed 131.21 samples/sec Loss 1.4253 Epoch: 16 Global Step: 169115 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:24:02,510-Speed 129.59 samples/sec Loss 1.3490 Epoch: 16 Global Step: 169120 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:24:22,178-Speed 130.17 samples/sec Loss 1.3525 Epoch: 16 Global Step: 169125 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:24:41,798-Speed 130.49 samples/sec Loss 1.4529 Epoch: 16 Global Step: 169130 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:25:01,441-Speed 130.33 samples/sec Loss 1.4389 Epoch: 16 Global Step: 169135 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:25:20,872-Speed 131.76 samples/sec Loss 1.3020 Epoch: 16 Global Step: 169140 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:25:40,496-Speed 130.46 samples/sec Loss 1.3587 Epoch: 16 Global Step: 169145 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:26:00,028-Speed 131.08 samples/sec Loss 1.4416 Epoch: 16 Global Step: 169150 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:26:19,644-Speed 130.51 samples/sec Loss 1.3891 Epoch: 16 Global Step: 169155 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:26:39,205-Speed 130.88 samples/sec Loss 1.4404 Epoch: 16 Global Step: 169160 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:26:59,095-Speed 128.72 samples/sec Loss 1.3658 Epoch: 16 Global Step: 169165 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:27:18,511-Speed 131.85 samples/sec Loss 1.4696 Epoch: 16 Global Step: 169170 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:27:38,214-Speed 129.94 samples/sec Loss 1.3392 Epoch: 16 Global Step: 169175 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:27:57,700-Speed 131.38 samples/sec Loss 1.4388 Epoch: 16 Global Step: 169180 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:28:17,766-Speed 127.59 samples/sec Loss 1.4788 Epoch: 16 Global Step: 169185 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:28:37,355-Speed 130.69 samples/sec Loss 1.3677 Epoch: 16 Global Step: 169190 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:28:56,848-Speed 131.34 samples/sec Loss 1.4541 Epoch: 16 Global Step: 169195 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:29:16,469-Speed 130.48 samples/sec Loss 1.3732 Epoch: 16 Global Step: 169200 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:29:35,900-Speed 131.75 samples/sec Loss 1.5056 Epoch: 16 Global Step: 169205 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:29:55,949-Speed 127.70 samples/sec Loss 1.3885 Epoch: 16 Global Step: 169210 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:30:15,375-Speed 131.79 samples/sec Loss 1.3867 Epoch: 16 Global Step: 169215 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:30:34,998-Speed 130.46 samples/sec Loss 1.4065 Epoch: 16 Global Step: 169220 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:30:54,445-Speed 131.65 samples/sec Loss 1.4720 Epoch: 16 Global Step: 169225 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:31:13,999-Speed 130.93 samples/sec Loss 1.3932 Epoch: 16 Global Step: 169230 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:31:33,557-Speed 130.90 samples/sec Loss 1.3616 Epoch: 16 Global Step: 169235 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:31:53,066-Speed 131.23 samples/sec Loss 1.3992 Epoch: 16 Global Step: 169240 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:32:12,662-Speed 130.64 samples/sec Loss 1.4726 Epoch: 16 Global Step: 169245 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:32:32,116-Speed 131.60 samples/sec Loss 1.4751 Epoch: 16 Global Step: 169250 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:32:51,912-Speed 129.33 samples/sec Loss 1.5177 Epoch: 16 Global Step: 169255 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:33:11,364-Speed 131.61 samples/sec Loss 1.4551 Epoch: 16 Global Step: 169260 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:33:31,077-Speed 129.87 samples/sec Loss 1.2796 Epoch: 16 Global Step: 169265 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:33:50,607-Speed 131.09 samples/sec Loss 1.3943 Epoch: 16 Global Step: 169270 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:34:10,223-Speed 130.51 samples/sec Loss 1.4996 Epoch: 16 Global Step: 169275 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:34:29,769-Speed 130.98 samples/sec Loss 1.3725 Epoch: 16 Global Step: 169280 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:34:49,404-Speed 130.38 samples/sec Loss 1.4544 Epoch: 16 Global Step: 169285 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:35:09,012-Speed 130.57 samples/sec Loss 1.3783 Epoch: 16 Global Step: 169290 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:35:28,548-Speed 131.05 samples/sec Loss 1.2975 Epoch: 16 Global Step: 169295 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:35:48,370-Speed 129.15 samples/sec Loss 1.4666 Epoch: 16 Global Step: 169300 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:36:08,084-Speed 129.86 samples/sec Loss 1.3492 Epoch: 16 Global Step: 169305 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:36:27,517-Speed 131.74 samples/sec Loss 1.4948 Epoch: 16 Global Step: 169310 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:36:47,057-Speed 131.02 samples/sec Loss 1.3474 Epoch: 16 Global Step: 169315 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:37:06,642-Speed 130.72 samples/sec Loss 1.3981 Epoch: 16 Global Step: 169320 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:37:26,162-Speed 131.15 samples/sec Loss 1.3123 Epoch: 16 Global Step: 169325 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:37:45,769-Speed 130.58 samples/sec Loss 1.4173 Epoch: 16 Global Step: 169330 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:38:05,327-Speed 130.90 samples/sec Loss 1.3693 Epoch: 16 Global Step: 169335 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:38:24,864-Speed 131.04 samples/sec Loss 1.4245 Epoch: 16 Global Step: 169340 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:38:44,599-Speed 129.72 samples/sec Loss 1.4590 Epoch: 16 Global Step: 169345 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:39:04,222-Speed 130.47 samples/sec Loss 1.4602 Epoch: 16 Global Step: 169350 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:39:23,793-Speed 130.81 samples/sec Loss 1.4429 Epoch: 16 Global Step: 169355 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:39:43,400-Speed 130.57 samples/sec Loss 1.3276 Epoch: 16 Global Step: 169360 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:40:02,933-Speed 131.07 samples/sec Loss 1.3215 Epoch: 16 Global Step: 169365 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:40:22,524-Speed 130.68 samples/sec Loss 1.4501 Epoch: 16 Global Step: 169370 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:40:42,134-Speed 130.55 samples/sec Loss 1.3819 Epoch: 16 Global Step: 169375 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:41:01,643-Speed 131.24 samples/sec Loss 1.2981 Epoch: 16 Global Step: 169380 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:41:21,161-Speed 131.17 samples/sec Loss 1.3620 Epoch: 16 Global Step: 169385 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:42:07,881-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-15 02:42:22,712-Speed 41.59 samples/sec Loss 1.3087 Epoch: 17 Global Step: 169390 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:42:42,752-Speed 127.75 samples/sec Loss 1.0918 Epoch: 17 Global Step: 169395 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:43:03,199-Speed 125.21 samples/sec Loss 1.1120 Epoch: 17 Global Step: 169400 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:43:23,170-Speed 128.20 samples/sec Loss 0.9954 Epoch: 17 Global Step: 169405 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:43:42,975-Speed 129.26 samples/sec Loss 1.0843 Epoch: 17 Global Step: 169410 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:44:02,725-Speed 129.63 samples/sec Loss 1.2288 Epoch: 17 Global Step: 169415 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:44:22,509-Speed 129.40 samples/sec Loss 1.1334 Epoch: 17 Global Step: 169420 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:44:42,236-Speed 129.78 samples/sec Loss 1.0248 Epoch: 17 Global Step: 169425 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:45:02,012-Speed 129.46 samples/sec Loss 1.1517 Epoch: 17 Global Step: 169430 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:45:21,894-Speed 128.77 samples/sec Loss 1.0523 Epoch: 17 Global Step: 169435 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:45:41,611-Speed 129.84 samples/sec Loss 1.2381 Epoch: 17 Global Step: 169440 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:46:02,002-Speed 125.55 samples/sec Loss 1.1565 Epoch: 17 Global Step: 169445 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:46:21,859-Speed 128.93 samples/sec Loss 0.9716 Epoch: 17 Global Step: 169450 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:46:41,569-Speed 129.88 samples/sec Loss 1.1119 Epoch: 17 Global Step: 169455 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:47:01,434-Speed 128.88 samples/sec Loss 1.1964 Epoch: 17 Global Step: 169460 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:47:21,234-Speed 129.30 samples/sec Loss 1.1150 Epoch: 17 Global Step: 169465 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:47:41,066-Speed 129.09 samples/sec Loss 1.0012 Epoch: 17 Global Step: 169470 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:48:00,917-Speed 128.97 samples/sec Loss 1.1179 Epoch: 17 Global Step: 169475 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:48:20,656-Speed 129.70 samples/sec Loss 1.2039 Epoch: 17 Global Step: 169480 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:48:40,321-Speed 130.19 samples/sec Loss 1.2179 Epoch: 17 Global Step: 169485 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:49:00,695-Speed 125.65 samples/sec Loss 1.0588 Epoch: 17 Global Step: 169490 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:49:20,599-Speed 128.62 samples/sec Loss 1.0482 Epoch: 17 Global Step: 169495 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:49:40,269-Speed 130.15 samples/sec Loss 1.2282 Epoch: 17 Global Step: 169500 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:50:00,105-Speed 129.06 samples/sec Loss 1.0679 Epoch: 17 Global Step: 169505 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:50:19,881-Speed 129.46 samples/sec Loss 1.1459 Epoch: 17 Global Step: 169510 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:50:39,667-Speed 129.39 samples/sec Loss 1.1171 Epoch: 17 Global Step: 169515 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:50:59,659-Speed 128.06 samples/sec Loss 1.0628 Epoch: 17 Global Step: 169520 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:51:19,400-Speed 129.69 samples/sec Loss 1.0911 Epoch: 17 Global Step: 169525 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:51:39,431-Speed 127.81 samples/sec Loss 1.0372 Epoch: 17 Global Step: 169530 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:51:59,806-Speed 125.65 samples/sec Loss 1.1619 Epoch: 17 Global Step: 169535 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:52:19,569-Speed 129.54 samples/sec Loss 1.0584 Epoch: 17 Global Step: 169540 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:52:39,245-Speed 130.11 samples/sec Loss 1.0834 Epoch: 17 Global Step: 169545 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:52:59,130-Speed 128.74 samples/sec Loss 0.9496 Epoch: 17 Global Step: 169550 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:53:18,852-Speed 129.82 samples/sec Loss 1.0831 Epoch: 17 Global Step: 169555 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:53:38,757-Speed 128.61 samples/sec Loss 1.0105 Epoch: 17 Global Step: 169560 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:53:58,476-Speed 129.83 samples/sec Loss 1.0634 Epoch: 17 Global Step: 169565 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:54:18,475-Speed 128.01 samples/sec Loss 1.0390 Epoch: 17 Global Step: 169570 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:54:38,245-Speed 129.50 samples/sec Loss 1.0505 Epoch: 17 Global Step: 169575 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:54:58,663-Speed 125.38 samples/sec Loss 1.0483 Epoch: 17 Global Step: 169580 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:55:18,449-Speed 129.39 samples/sec Loss 1.1655 Epoch: 17 Global Step: 169585 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:55:38,226-Speed 129.45 samples/sec Loss 1.0025 Epoch: 17 Global Step: 169590 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:55:58,049-Speed 129.15 samples/sec Loss 1.3196 Epoch: 17 Global Step: 169595 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:56:17,716-Speed 130.17 samples/sec Loss 0.9632 Epoch: 17 Global Step: 169600 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:56:37,609-Speed 128.70 samples/sec Loss 1.0089 Epoch: 17 Global Step: 169605 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:56:57,406-Speed 129.31 samples/sec Loss 1.1118 Epoch: 17 Global Step: 169610 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:57:17,300-Speed 128.69 samples/sec Loss 1.1260 Epoch: 17 Global Step: 169615 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:57:37,032-Speed 129.75 samples/sec Loss 1.1442 Epoch: 17 Global Step: 169620 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:57:57,461-Speed 125.32 samples/sec Loss 0.9863 Epoch: 17 Global Step: 169625 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:58:17,136-Speed 130.11 samples/sec Loss 1.0585 Epoch: 17 Global Step: 169630 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:58:37,147-Speed 127.94 samples/sec Loss 1.1514 Epoch: 17 Global Step: 169635 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:58:56,869-Speed 129.81 samples/sec Loss 1.0954 Epoch: 17 Global Step: 169640 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:59:16,768-Speed 128.65 samples/sec Loss 1.1003 Epoch: 17 Global Step: 169645 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:59:36,672-Speed 128.63 samples/sec Loss 0.9361 Epoch: 17 Global Step: 169650 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 02:59:56,344-Speed 130.14 samples/sec Loss 1.1720 Epoch: 17 Global Step: 169655 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:00:16,188-Speed 129.02 samples/sec Loss 1.0258 Epoch: 17 Global Step: 169660 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:00:35,845-Speed 130.24 samples/sec Loss 1.1082 Epoch: 17 Global Step: 169665 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:00:56,175-Speed 125.93 samples/sec Loss 1.0932 Epoch: 17 Global Step: 169670 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:01:16,171-Speed 128.03 samples/sec Loss 1.2117 Epoch: 17 Global Step: 169675 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:01:35,907-Speed 129.72 samples/sec Loss 1.1683 Epoch: 17 Global Step: 169680 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:01:55,668-Speed 129.56 samples/sec Loss 1.0817 Epoch: 17 Global Step: 169685 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:02:15,633-Speed 128.23 samples/sec Loss 1.1563 Epoch: 17 Global Step: 169690 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:02:35,248-Speed 130.52 samples/sec Loss 1.1099 Epoch: 17 Global Step: 169695 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:02:55,043-Speed 129.33 samples/sec Loss 0.9887 Epoch: 17 Global Step: 169700 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:03:14,887-Speed 129.02 samples/sec Loss 1.1457 Epoch: 17 Global Step: 169705 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:03:34,611-Speed 129.80 samples/sec Loss 0.9418 Epoch: 17 Global Step: 169710 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:03:54,938-Speed 125.94 samples/sec Loss 0.9422 Epoch: 17 Global Step: 169715 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:04:14,836-Speed 128.66 samples/sec Loss 1.1301 Epoch: 17 Global Step: 169720 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:04:34,455-Speed 130.49 samples/sec Loss 1.1257 Epoch: 17 Global Step: 169725 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:04:54,228-Speed 129.48 samples/sec Loss 1.0927 Epoch: 17 Global Step: 169730 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:05:14,036-Speed 129.25 samples/sec Loss 1.2545 Epoch: 17 Global Step: 169735 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:05:33,734-Speed 129.97 samples/sec Loss 1.0839 Epoch: 17 Global Step: 169740 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:05:53,620-Speed 128.74 samples/sec Loss 1.1374 Epoch: 17 Global Step: 169745 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:06:13,342-Speed 129.82 samples/sec Loss 1.1624 Epoch: 17 Global Step: 169750 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 03:06:33,258-Speed 128.55 samples/sec Loss 1.0164 Epoch: 17 Global Step: 169755 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:06:53,496-Speed 126.50 samples/sec Loss 1.1277 Epoch: 17 Global Step: 169760 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:07:13,373-Speed 128.79 samples/sec Loss 1.0930 Epoch: 17 Global Step: 169765 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:07:33,089-Speed 129.85 samples/sec Loss 0.9222 Epoch: 17 Global Step: 169770 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:07:53,022-Speed 128.44 samples/sec Loss 1.0413 Epoch: 17 Global Step: 169775 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:08:12,637-Speed 130.52 samples/sec Loss 1.1210 Epoch: 17 Global Step: 169780 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:08:32,387-Speed 129.63 samples/sec Loss 0.9401 Epoch: 17 Global Step: 169785 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:08:52,261-Speed 128.81 samples/sec Loss 1.0798 Epoch: 17 Global Step: 169790 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:09:12,060-Speed 129.31 samples/sec Loss 0.9547 Epoch: 17 Global Step: 169795 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:09:31,836-Speed 129.45 samples/sec Loss 1.1253 Epoch: 17 Global Step: 169800 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:09:52,055-Speed 126.62 samples/sec Loss 1.1547 Epoch: 17 Global Step: 169805 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:10:11,895-Speed 129.04 samples/sec Loss 0.9610 Epoch: 17 Global Step: 169810 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:10:31,687-Speed 129.35 samples/sec Loss 0.9836 Epoch: 17 Global Step: 169815 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:10:51,472-Speed 129.40 samples/sec Loss 1.0966 Epoch: 17 Global Step: 169820 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:11:11,223-Speed 129.62 samples/sec Loss 1.0450 Epoch: 17 Global Step: 169825 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:11:31,051-Speed 129.11 samples/sec Loss 1.0008 Epoch: 17 Global Step: 169830 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:11:50,842-Speed 129.36 samples/sec Loss 1.2228 Epoch: 17 Global Step: 169835 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:12:10,683-Speed 129.04 samples/sec Loss 1.0903 Epoch: 17 Global Step: 169840 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:12:30,447-Speed 129.53 samples/sec Loss 1.0684 Epoch: 17 Global Step: 169845 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:12:50,751-Speed 126.09 samples/sec Loss 1.0503 Epoch: 17 Global Step: 169850 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:13:10,600-Speed 128.98 samples/sec Loss 1.0904 Epoch: 17 Global Step: 169855 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:13:30,257-Speed 130.23 samples/sec Loss 0.9591 Epoch: 17 Global Step: 169860 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:13:50,345-Speed 127.45 samples/sec Loss 1.0561 Epoch: 17 Global Step: 169865 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:14:10,078-Speed 129.74 samples/sec Loss 1.1273 Epoch: 17 Global Step: 169870 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:14:29,893-Speed 129.20 samples/sec Loss 1.1010 Epoch: 17 Global Step: 169875 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:14:49,677-Speed 129.40 samples/sec Loss 1.1059 Epoch: 17 Global Step: 169880 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:15:09,507-Speed 129.11 samples/sec Loss 1.1588 Epoch: 17 Global Step: 169885 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:15:29,237-Speed 129.75 samples/sec Loss 1.0986 Epoch: 17 Global Step: 169890 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:15:49,643-Speed 125.46 samples/sec Loss 1.1722 Epoch: 17 Global Step: 169895 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:16:09,435-Speed 129.35 samples/sec Loss 1.1031 Epoch: 17 Global Step: 169900 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:16:29,238-Speed 129.28 samples/sec Loss 1.0253 Epoch: 17 Global Step: 169905 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:16:48,969-Speed 129.76 samples/sec Loss 1.0584 Epoch: 17 Global Step: 169910 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:17:08,750-Speed 129.42 samples/sec Loss 1.0711 Epoch: 17 Global Step: 169915 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:17:28,457-Speed 129.91 samples/sec Loss 1.0305 Epoch: 17 Global Step: 169920 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:17:48,289-Speed 129.10 samples/sec Loss 1.0682 Epoch: 17 Global Step: 169925 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:18:08,140-Speed 128.97 samples/sec Loss 1.0606 Epoch: 17 Global Step: 169930 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:18:28,078-Speed 128.40 samples/sec Loss 1.1859 Epoch: 17 Global Step: 169935 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:18:48,337-Speed 126.37 samples/sec Loss 1.0863 Epoch: 17 Global Step: 169940 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:19:08,318-Speed 128.13 samples/sec Loss 1.0473 Epoch: 17 Global Step: 169945 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:19:27,983-Speed 130.19 samples/sec Loss 1.2085 Epoch: 17 Global Step: 169950 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:19:47,786-Speed 129.28 samples/sec Loss 0.9753 Epoch: 17 Global Step: 169955 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:20:07,621-Speed 129.07 samples/sec Loss 1.2505 Epoch: 17 Global Step: 169960 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:20:27,253-Speed 130.41 samples/sec Loss 1.1172 Epoch: 17 Global Step: 169965 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:20:47,179-Speed 128.48 samples/sec Loss 1.0111 Epoch: 17 Global Step: 169970 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:21:07,047-Speed 128.85 samples/sec Loss 1.0551 Epoch: 17 Global Step: 169975 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:21:26,745-Speed 129.97 samples/sec Loss 1.1138 Epoch: 17 Global Step: 169980 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:21:47,028-Speed 126.21 samples/sec Loss 1.0107 Epoch: 17 Global Step: 169985 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:22:06,880-Speed 128.96 samples/sec Loss 1.1517 Epoch: 17 Global Step: 169990 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:22:26,511-Speed 130.41 samples/sec Loss 1.1186 Epoch: 17 Global Step: 169995 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:22:46,421-Speed 128.59 samples/sec Loss 1.0676 Epoch: 17 Global Step: 170000 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 03:37:30,627-[lfw][170000]XNorm: 22.014325 Training: 2021-06-15 03:37:30,628-[lfw][170000]Accuracy-Flip: 0.99817+-0.00203 Training: 2021-06-15 03:37:30,628-[lfw][170000]Accuracy-Highest: 0.99833 Training: 2021-06-15 03:54:41,696-[cfp_fp][170000]XNorm: 23.054696 Training: 2021-06-15 03:54:41,696-[cfp_fp][170000]Accuracy-Flip: 0.99229+-0.00273 Training: 2021-06-15 03:54:41,696-[cfp_fp][170000]Accuracy-Highest: 0.99229 Training: 2021-06-15 04:09:27,262-[agedb_30][170000]XNorm: 22.968422 Training: 2021-06-15 04:09:27,262-[agedb_30][170000]Accuracy-Flip: 0.98583+-0.00534 Training: 2021-06-15 04:09:27,262-[agedb_30][170000]Accuracy-Highest: 0.98583 Training: 2021-06-15 04:09:47,273-Speed 0.91 samples/sec Loss 1.0803 Epoch: 17 Global Step: 170005 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 04:10:06,859-Speed 130.71 samples/sec Loss 1.0751 Epoch: 17 Global Step: 170010 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 04:10:26,378-Speed 131.16 samples/sec Loss 1.2162 Epoch: 17 Global Step: 170015 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 04:10:46,180-Speed 129.29 samples/sec Loss 1.0934 Epoch: 17 Global Step: 170020 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 04:11:05,618-Speed 131.70 samples/sec Loss 1.0764 Epoch: 17 Global Step: 170025 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 04:11:25,210-Speed 130.68 samples/sec Loss 1.1120 Epoch: 17 Global Step: 170030 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 04:11:44,726-Speed 131.18 samples/sec Loss 1.1423 Epoch: 17 Global Step: 170035 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 04:12:04,213-Speed 131.38 samples/sec Loss 1.0318 Epoch: 17 Global Step: 170040 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 04:12:23,818-Speed 130.59 samples/sec Loss 1.0665 Epoch: 17 Global Step: 170045 Fp16 Grad Scale: 8192 Required: 94 hours Training: 2021-06-15 04:12:43,293-Speed 131.46 samples/sec Loss 1.0574 Epoch: 17 Global Step: 170050 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:13:02,884-Speed 130.68 samples/sec Loss 0.9680 Epoch: 17 Global Step: 170055 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:13:22,310-Speed 131.79 samples/sec Loss 1.0473 Epoch: 17 Global Step: 170060 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:13:42,108-Speed 129.31 samples/sec Loss 1.0479 Epoch: 17 Global Step: 170065 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:14:01,515-Speed 131.91 samples/sec Loss 1.1519 Epoch: 17 Global Step: 170070 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:14:21,217-Speed 129.94 samples/sec Loss 1.1368 Epoch: 17 Global Step: 170075 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:14:40,658-Speed 131.69 samples/sec Loss 1.1387 Epoch: 17 Global Step: 170080 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:15:00,128-Speed 131.49 samples/sec Loss 1.1692 Epoch: 17 Global Step: 170085 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:15:19,696-Speed 130.84 samples/sec Loss 0.9352 Epoch: 17 Global Step: 170090 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:15:39,244-Speed 130.97 samples/sec Loss 1.1160 Epoch: 17 Global Step: 170095 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:15:58,841-Speed 130.64 samples/sec Loss 0.9805 Epoch: 17 Global Step: 170100 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:16:18,278-Speed 131.71 samples/sec Loss 1.1461 Epoch: 17 Global Step: 170105 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:16:38,085-Speed 129.26 samples/sec Loss 1.2311 Epoch: 17 Global Step: 170110 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:16:57,655-Speed 130.82 samples/sec Loss 1.0580 Epoch: 17 Global Step: 170115 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:17:17,366-Speed 129.88 samples/sec Loss 1.1604 Epoch: 17 Global Step: 170120 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:17:36,801-Speed 131.73 samples/sec Loss 1.2270 Epoch: 17 Global Step: 170125 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:17:56,423-Speed 130.47 samples/sec Loss 1.1920 Epoch: 17 Global Step: 170130 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:18:15,930-Speed 131.24 samples/sec Loss 1.1137 Epoch: 17 Global Step: 170135 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:18:35,484-Speed 130.93 samples/sec Loss 0.9859 Epoch: 17 Global Step: 170140 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:18:54,965-Speed 131.42 samples/sec Loss 1.2107 Epoch: 17 Global Step: 170145 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:19:14,394-Speed 131.77 samples/sec Loss 1.1694 Epoch: 17 Global Step: 170150 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:19:34,211-Speed 129.19 samples/sec Loss 1.1439 Epoch: 17 Global Step: 170155 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:19:53,776-Speed 130.85 samples/sec Loss 0.9846 Epoch: 17 Global Step: 170160 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:20:13,405-Speed 130.43 samples/sec Loss 1.0629 Epoch: 17 Global Step: 170165 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:20:32,848-Speed 131.68 samples/sec Loss 1.0536 Epoch: 17 Global Step: 170170 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:20:52,431-Speed 130.74 samples/sec Loss 0.9253 Epoch: 17 Global Step: 170175 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:21:11,970-Speed 131.02 samples/sec Loss 1.0902 Epoch: 17 Global Step: 170180 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:21:31,469-Speed 131.30 samples/sec Loss 1.1658 Epoch: 17 Global Step: 170185 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:21:51,061-Speed 130.67 samples/sec Loss 1.2458 Epoch: 17 Global Step: 170190 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:22:10,508-Speed 131.65 samples/sec Loss 0.9481 Epoch: 17 Global Step: 170195 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:22:30,312-Speed 129.28 samples/sec Loss 1.1311 Epoch: 17 Global Step: 170200 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:22:50,151-Speed 129.05 samples/sec Loss 1.1455 Epoch: 17 Global Step: 170205 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:23:09,578-Speed 131.78 samples/sec Loss 1.1896 Epoch: 17 Global Step: 170210 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:23:29,104-Speed 131.12 samples/sec Loss 0.9773 Epoch: 17 Global Step: 170215 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:23:48,719-Speed 130.52 samples/sec Loss 1.0454 Epoch: 17 Global Step: 170220 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:24:08,328-Speed 130.56 samples/sec Loss 0.9895 Epoch: 17 Global Step: 170225 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:24:27,930-Speed 130.60 samples/sec Loss 1.1817 Epoch: 17 Global Step: 170230 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:24:47,355-Speed 131.79 samples/sec Loss 1.0110 Epoch: 17 Global Step: 170235 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:25:06,908-Speed 130.93 samples/sec Loss 1.1530 Epoch: 17 Global Step: 170240 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:25:26,731-Speed 129.15 samples/sec Loss 1.1675 Epoch: 17 Global Step: 170245 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:25:46,467-Speed 129.72 samples/sec Loss 1.1121 Epoch: 17 Global Step: 170250 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:26:05,891-Speed 131.80 samples/sec Loss 1.0481 Epoch: 17 Global Step: 170255 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:26:25,470-Speed 130.76 samples/sec Loss 1.0297 Epoch: 17 Global Step: 170260 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:26:45,040-Speed 130.82 samples/sec Loss 1.0673 Epoch: 17 Global Step: 170265 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:27:04,608-Speed 130.83 samples/sec Loss 1.0300 Epoch: 17 Global Step: 170270 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:27:24,188-Speed 130.75 samples/sec Loss 1.2100 Epoch: 17 Global Step: 170275 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:27:43,623-Speed 131.73 samples/sec Loss 1.1610 Epoch: 17 Global Step: 170280 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:28:03,338-Speed 129.86 samples/sec Loss 1.1879 Epoch: 17 Global Step: 170285 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:28:23,133-Speed 129.33 samples/sec Loss 1.0368 Epoch: 17 Global Step: 170290 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:28:42,929-Speed 129.33 samples/sec Loss 1.1611 Epoch: 17 Global Step: 170295 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:29:02,405-Speed 131.45 samples/sec Loss 1.0658 Epoch: 17 Global Step: 170300 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:29:22,001-Speed 130.65 samples/sec Loss 1.0443 Epoch: 17 Global Step: 170305 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:29:41,598-Speed 130.64 samples/sec Loss 1.0968 Epoch: 17 Global Step: 170310 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:30:01,083-Speed 131.39 samples/sec Loss 1.0381 Epoch: 17 Global Step: 170315 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:30:21,056-Speed 128.18 samples/sec Loss 1.0336 Epoch: 17 Global Step: 170320 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:30:40,586-Speed 131.09 samples/sec Loss 1.0662 Epoch: 17 Global Step: 170325 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:31:00,101-Speed 131.19 samples/sec Loss 1.1794 Epoch: 17 Global Step: 170330 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:31:19,869-Speed 129.52 samples/sec Loss 1.0724 Epoch: 17 Global Step: 170335 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:31:39,468-Speed 130.62 samples/sec Loss 1.2287 Epoch: 17 Global Step: 170340 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:31:59,000-Speed 131.08 samples/sec Loss 1.1231 Epoch: 17 Global Step: 170345 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:32:18,594-Speed 130.66 samples/sec Loss 1.1307 Epoch: 17 Global Step: 170350 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:32:38,158-Speed 130.86 samples/sec Loss 1.0797 Epoch: 17 Global Step: 170355 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:32:57,596-Speed 131.71 samples/sec Loss 1.1718 Epoch: 17 Global Step: 170360 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:33:17,212-Speed 130.51 samples/sec Loss 1.1235 Epoch: 17 Global Step: 170365 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:33:36,767-Speed 130.92 samples/sec Loss 1.1602 Epoch: 17 Global Step: 170370 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:33:56,365-Speed 130.63 samples/sec Loss 1.1899 Epoch: 17 Global Step: 170375 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:34:16,125-Speed 129.57 samples/sec Loss 1.0460 Epoch: 17 Global Step: 170380 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:34:35,781-Speed 130.24 samples/sec Loss 1.1503 Epoch: 17 Global Step: 170385 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:34:55,346-Speed 130.86 samples/sec Loss 1.1257 Epoch: 17 Global Step: 170390 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:35:14,986-Speed 130.35 samples/sec Loss 1.1281 Epoch: 17 Global Step: 170395 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:35:34,424-Speed 131.70 samples/sec Loss 1.1113 Epoch: 17 Global Step: 170400 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:35:54,005-Speed 130.75 samples/sec Loss 1.1087 Epoch: 17 Global Step: 170405 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:36:13,579-Speed 130.79 samples/sec Loss 1.1633 Epoch: 17 Global Step: 170410 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:36:33,154-Speed 130.79 samples/sec Loss 1.1414 Epoch: 17 Global Step: 170415 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:36:52,736-Speed 130.74 samples/sec Loss 1.0452 Epoch: 17 Global Step: 170420 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:37:12,401-Speed 130.19 samples/sec Loss 1.0964 Epoch: 17 Global Step: 170425 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:37:31,962-Speed 130.88 samples/sec Loss 1.1160 Epoch: 17 Global Step: 170430 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:37:51,504-Speed 131.00 samples/sec Loss 0.9956 Epoch: 17 Global Step: 170435 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:38:11,095-Speed 130.68 samples/sec Loss 1.1323 Epoch: 17 Global Step: 170440 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:38:30,536-Speed 131.68 samples/sec Loss 1.1656 Epoch: 17 Global Step: 170445 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:38:50,206-Speed 130.16 samples/sec Loss 1.2040 Epoch: 17 Global Step: 170450 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:39:09,731-Speed 131.12 samples/sec Loss 0.9771 Epoch: 17 Global Step: 170455 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:39:29,391-Speed 130.22 samples/sec Loss 1.0204 Epoch: 17 Global Step: 170460 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:39:48,952-Speed 130.89 samples/sec Loss 1.1727 Epoch: 17 Global Step: 170465 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:40:08,885-Speed 128.43 samples/sec Loss 1.1067 Epoch: 17 Global Step: 170470 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:40:28,318-Speed 131.74 samples/sec Loss 1.1173 Epoch: 17 Global Step: 170475 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:40:48,007-Speed 130.03 samples/sec Loss 1.1446 Epoch: 17 Global Step: 170480 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:41:07,885-Speed 128.79 samples/sec Loss 1.0102 Epoch: 17 Global Step: 170485 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:41:27,325-Speed 131.70 samples/sec Loss 1.0447 Epoch: 17 Global Step: 170490 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:41:46,934-Speed 130.56 samples/sec Loss 0.9792 Epoch: 17 Global Step: 170495 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:42:06,498-Speed 130.86 samples/sec Loss 1.0851 Epoch: 17 Global Step: 170500 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:42:26,107-Speed 130.55 samples/sec Loss 1.1447 Epoch: 17 Global Step: 170505 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:42:45,779-Speed 130.14 samples/sec Loss 1.0904 Epoch: 17 Global Step: 170510 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:43:05,598-Speed 129.18 samples/sec Loss 1.1086 Epoch: 17 Global Step: 170515 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:43:25,028-Speed 131.76 samples/sec Loss 1.1169 Epoch: 17 Global Step: 170520 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:43:44,988-Speed 128.26 samples/sec Loss 1.2016 Epoch: 17 Global Step: 170525 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:44:04,424-Speed 131.73 samples/sec Loss 1.2024 Epoch: 17 Global Step: 170530 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:44:23,889-Speed 131.53 samples/sec Loss 1.1150 Epoch: 17 Global Step: 170535 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:44:43,452-Speed 130.86 samples/sec Loss 1.1461 Epoch: 17 Global Step: 170540 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:45:03,052-Speed 130.62 samples/sec Loss 1.1091 Epoch: 17 Global Step: 170545 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:45:22,667-Speed 130.52 samples/sec Loss 0.9833 Epoch: 17 Global Step: 170550 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:45:42,101-Speed 131.73 samples/sec Loss 1.0246 Epoch: 17 Global Step: 170555 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:46:01,993-Speed 128.70 samples/sec Loss 1.1366 Epoch: 17 Global Step: 170560 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:46:21,430-Speed 131.72 samples/sec Loss 1.1327 Epoch: 17 Global Step: 170565 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:46:41,215-Speed 129.40 samples/sec Loss 0.9993 Epoch: 17 Global Step: 170570 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:47:00,671-Speed 131.59 samples/sec Loss 0.9973 Epoch: 17 Global Step: 170575 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:47:20,227-Speed 130.91 samples/sec Loss 1.0001 Epoch: 17 Global Step: 170580 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:47:39,831-Speed 130.59 samples/sec Loss 0.9937 Epoch: 17 Global Step: 170585 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:47:59,751-Speed 128.52 samples/sec Loss 1.1227 Epoch: 17 Global Step: 170590 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:48:19,357-Speed 130.58 samples/sec Loss 1.0317 Epoch: 17 Global Step: 170595 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:48:38,799-Speed 131.68 samples/sec Loss 1.1771 Epoch: 17 Global Step: 170600 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:48:58,831-Speed 127.80 samples/sec Loss 1.0724 Epoch: 17 Global Step: 170605 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:49:18,260-Speed 131.77 samples/sec Loss 1.2262 Epoch: 17 Global Step: 170610 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:49:38,044-Speed 129.40 samples/sec Loss 1.1417 Epoch: 17 Global Step: 170615 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:49:57,567-Speed 131.14 samples/sec Loss 1.0668 Epoch: 17 Global Step: 170620 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:50:17,108-Speed 131.01 samples/sec Loss 1.1008 Epoch: 17 Global Step: 170625 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:50:36,668-Speed 130.89 samples/sec Loss 1.0152 Epoch: 17 Global Step: 170630 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:50:56,173-Speed 131.26 samples/sec Loss 1.0660 Epoch: 17 Global Step: 170635 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:51:15,763-Speed 130.69 samples/sec Loss 1.0401 Epoch: 17 Global Step: 170640 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:51:35,300-Speed 131.04 samples/sec Loss 0.9216 Epoch: 17 Global Step: 170645 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:51:55,318-Speed 127.89 samples/sec Loss 1.0559 Epoch: 17 Global Step: 170650 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:52:14,749-Speed 131.76 samples/sec Loss 1.0530 Epoch: 17 Global Step: 170655 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:52:34,442-Speed 130.00 samples/sec Loss 1.1056 Epoch: 17 Global Step: 170660 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:52:53,979-Speed 131.04 samples/sec Loss 0.9847 Epoch: 17 Global Step: 170665 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:53:13,541-Speed 130.88 samples/sec Loss 1.1586 Epoch: 17 Global Step: 170670 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:53:33,132-Speed 130.68 samples/sec Loss 1.1277 Epoch: 17 Global Step: 170675 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:53:52,628-Speed 131.31 samples/sec Loss 1.0373 Epoch: 17 Global Step: 170680 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:54:12,239-Speed 130.55 samples/sec Loss 1.1181 Epoch: 17 Global Step: 170685 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:54:31,787-Speed 130.96 samples/sec Loss 1.1368 Epoch: 17 Global Step: 170690 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:54:51,583-Speed 129.33 samples/sec Loss 1.1990 Epoch: 17 Global Step: 170695 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:55:11,205-Speed 130.47 samples/sec Loss 1.1641 Epoch: 17 Global Step: 170700 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:55:30,780-Speed 130.78 samples/sec Loss 1.1381 Epoch: 17 Global Step: 170705 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:55:50,312-Speed 131.07 samples/sec Loss 1.1438 Epoch: 17 Global Step: 170710 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:56:09,939-Speed 130.44 samples/sec Loss 1.1003 Epoch: 17 Global Step: 170715 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:56:29,501-Speed 130.87 samples/sec Loss 1.0708 Epoch: 17 Global Step: 170720 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:56:48,984-Speed 131.40 samples/sec Loss 0.9461 Epoch: 17 Global Step: 170725 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:57:08,547-Speed 130.87 samples/sec Loss 1.0683 Epoch: 17 Global Step: 170730 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:57:28,108-Speed 130.88 samples/sec Loss 0.9681 Epoch: 17 Global Step: 170735 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:57:47,885-Speed 129.45 samples/sec Loss 1.0427 Epoch: 17 Global Step: 170740 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:58:07,615-Speed 129.75 samples/sec Loss 1.1612 Epoch: 17 Global Step: 170745 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:58:27,163-Speed 130.97 samples/sec Loss 1.0520 Epoch: 17 Global Step: 170750 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:58:46,775-Speed 130.54 samples/sec Loss 1.0408 Epoch: 17 Global Step: 170755 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:59:06,379-Speed 130.59 samples/sec Loss 1.0808 Epoch: 17 Global Step: 170760 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:59:25,839-Speed 131.56 samples/sec Loss 1.2161 Epoch: 17 Global Step: 170765 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 04:59:45,465-Speed 130.45 samples/sec Loss 1.0847 Epoch: 17 Global Step: 170770 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:00:05,015-Speed 130.95 samples/sec Loss 1.1291 Epoch: 17 Global Step: 170775 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:00:24,620-Speed 130.59 samples/sec Loss 1.0918 Epoch: 17 Global Step: 170780 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:00:44,454-Speed 129.08 samples/sec Loss 1.0996 Epoch: 17 Global Step: 170785 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:01:04,010-Speed 130.92 samples/sec Loss 1.1452 Epoch: 17 Global Step: 170790 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:01:23,531-Speed 131.15 samples/sec Loss 1.2479 Epoch: 17 Global Step: 170795 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:01:43,260-Speed 129.76 samples/sec Loss 1.0965 Epoch: 17 Global Step: 170800 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:02:02,869-Speed 130.56 samples/sec Loss 1.1234 Epoch: 17 Global Step: 170805 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:02:22,438-Speed 130.83 samples/sec Loss 0.9793 Epoch: 17 Global Step: 170810 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:02:42,018-Speed 130.76 samples/sec Loss 1.1371 Epoch: 17 Global Step: 170815 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:03:01,578-Speed 130.88 samples/sec Loss 1.0239 Epoch: 17 Global Step: 170820 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:03:21,186-Speed 130.57 samples/sec Loss 1.0330 Epoch: 17 Global Step: 170825 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:03:41,023-Speed 129.06 samples/sec Loss 1.0949 Epoch: 17 Global Step: 170830 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:04:00,666-Speed 130.33 samples/sec Loss 1.0255 Epoch: 17 Global Step: 170835 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:04:20,167-Speed 131.29 samples/sec Loss 1.1275 Epoch: 17 Global Step: 170840 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:04:39,784-Speed 130.51 samples/sec Loss 1.0679 Epoch: 17 Global Step: 170845 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:04:59,291-Speed 131.24 samples/sec Loss 0.9810 Epoch: 17 Global Step: 170850 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:05:18,885-Speed 130.66 samples/sec Loss 1.1263 Epoch: 17 Global Step: 170855 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:05:38,442-Speed 130.91 samples/sec Loss 1.0317 Epoch: 17 Global Step: 170860 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:05:57,921-Speed 131.43 samples/sec Loss 1.1352 Epoch: 17 Global Step: 170865 Fp16 Grad Scale: 8192 Required: 93 hours Training: 2021-06-15 05:06:17,544-Speed 130.46 samples/sec Loss 1.0294 Epoch: 17 Global Step: 170870 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:06:37,283-Speed 129.71 samples/sec Loss 1.2425 Epoch: 17 Global Step: 170875 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:06:56,797-Speed 131.19 samples/sec Loss 1.1421 Epoch: 17 Global Step: 170880 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:07:16,325-Speed 131.10 samples/sec Loss 1.1845 Epoch: 17 Global Step: 170885 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:07:35,966-Speed 130.34 samples/sec Loss 1.0377 Epoch: 17 Global Step: 170890 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:07:55,481-Speed 131.19 samples/sec Loss 1.2765 Epoch: 17 Global Step: 170895 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:08:15,013-Speed 131.07 samples/sec Loss 1.0869 Epoch: 17 Global Step: 170900 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:08:34,586-Speed 130.80 samples/sec Loss 1.0595 Epoch: 17 Global Step: 170905 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:08:54,202-Speed 130.51 samples/sec Loss 1.2028 Epoch: 17 Global Step: 170910 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:09:13,914-Speed 129.88 samples/sec Loss 1.0097 Epoch: 17 Global Step: 170915 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:09:33,688-Speed 129.47 samples/sec Loss 1.0270 Epoch: 17 Global Step: 170920 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:09:53,353-Speed 130.18 samples/sec Loss 1.1520 Epoch: 17 Global Step: 170925 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:10:12,972-Speed 130.49 samples/sec Loss 1.0701 Epoch: 17 Global Step: 170930 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:10:32,596-Speed 130.46 samples/sec Loss 1.1277 Epoch: 17 Global Step: 170935 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:10:52,099-Speed 131.27 samples/sec Loss 1.1838 Epoch: 17 Global Step: 170940 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:11:11,703-Speed 130.60 samples/sec Loss 1.0459 Epoch: 17 Global Step: 170945 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:11:31,142-Speed 131.70 samples/sec Loss 1.0348 Epoch: 17 Global Step: 170950 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:11:50,746-Speed 130.59 samples/sec Loss 1.0489 Epoch: 17 Global Step: 170955 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:12:10,320-Speed 130.79 samples/sec Loss 1.0350 Epoch: 17 Global Step: 170960 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:12:30,322-Speed 128.00 samples/sec Loss 1.0812 Epoch: 17 Global Step: 170965 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:12:49,752-Speed 131.76 samples/sec Loss 1.1526 Epoch: 17 Global Step: 170970 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:13:09,451-Speed 129.97 samples/sec Loss 1.1634 Epoch: 17 Global Step: 170975 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:13:28,979-Speed 131.10 samples/sec Loss 0.9860 Epoch: 17 Global Step: 170980 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:13:49,568-Speed 124.35 samples/sec Loss 1.0838 Epoch: 17 Global Step: 170985 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:14:09,175-Speed 130.57 samples/sec Loss 1.2214 Epoch: 17 Global Step: 170990 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:14:28,794-Speed 130.49 samples/sec Loss 1.0253 Epoch: 17 Global Step: 170995 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:14:48,437-Speed 130.34 samples/sec Loss 1.2276 Epoch: 17 Global Step: 171000 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:15:07,991-Speed 130.93 samples/sec Loss 1.0740 Epoch: 17 Global Step: 171005 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:15:27,831-Speed 129.03 samples/sec Loss 1.1200 Epoch: 17 Global Step: 171010 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:15:47,279-Speed 131.64 samples/sec Loss 1.1263 Epoch: 17 Global Step: 171015 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:16:07,067-Speed 129.38 samples/sec Loss 1.0019 Epoch: 17 Global Step: 171020 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:16:26,522-Speed 131.59 samples/sec Loss 1.0509 Epoch: 17 Global Step: 171025 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:16:46,066-Speed 131.00 samples/sec Loss 1.0731 Epoch: 17 Global Step: 171030 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:17:05,621-Speed 130.92 samples/sec Loss 1.1222 Epoch: 17 Global Step: 171035 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:17:25,979-Speed 125.76 samples/sec Loss 1.1462 Epoch: 17 Global Step: 171040 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:17:45,577-Speed 130.63 samples/sec Loss 1.0610 Epoch: 17 Global Step: 171045 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:18:05,023-Speed 131.65 samples/sec Loss 1.1210 Epoch: 17 Global Step: 171050 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:18:24,820-Speed 129.32 samples/sec Loss 1.1957 Epoch: 17 Global Step: 171055 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:18:44,345-Speed 131.12 samples/sec Loss 1.0223 Epoch: 17 Global Step: 171060 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:19:04,068-Speed 129.80 samples/sec Loss 1.0497 Epoch: 17 Global Step: 171065 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:19:23,740-Speed 130.14 samples/sec Loss 1.0592 Epoch: 17 Global Step: 171070 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:19:43,210-Speed 131.49 samples/sec Loss 1.0784 Epoch: 17 Global Step: 171075 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:20:02,816-Speed 130.58 samples/sec Loss 1.0086 Epoch: 17 Global Step: 171080 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:20:22,360-Speed 130.99 samples/sec Loss 1.1419 Epoch: 17 Global Step: 171085 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:20:41,977-Speed 130.51 samples/sec Loss 0.9370 Epoch: 17 Global Step: 171090 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:21:01,528-Speed 130.95 samples/sec Loss 1.0935 Epoch: 17 Global Step: 171095 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:21:21,310-Speed 129.42 samples/sec Loss 1.0517 Epoch: 17 Global Step: 171100 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:21:40,886-Speed 130.78 samples/sec Loss 1.0740 Epoch: 17 Global Step: 171105 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:22:00,515-Speed 130.43 samples/sec Loss 1.1295 Epoch: 17 Global Step: 171110 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:22:20,054-Speed 131.02 samples/sec Loss 1.0601 Epoch: 17 Global Step: 171115 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:22:39,665-Speed 130.55 samples/sec Loss 1.0051 Epoch: 17 Global Step: 171120 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:22:59,194-Speed 131.10 samples/sec Loss 1.0705 Epoch: 17 Global Step: 171125 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:23:18,740-Speed 130.98 samples/sec Loss 0.9632 Epoch: 17 Global Step: 171130 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:23:38,321-Speed 130.75 samples/sec Loss 1.1521 Epoch: 17 Global Step: 171135 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:23:57,830-Speed 131.23 samples/sec Loss 1.1388 Epoch: 17 Global Step: 171140 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:24:17,632-Speed 129.29 samples/sec Loss 1.1757 Epoch: 17 Global Step: 171145 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:24:37,199-Speed 130.84 samples/sec Loss 0.9618 Epoch: 17 Global Step: 171150 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:24:56,895-Speed 129.98 samples/sec Loss 1.0207 Epoch: 17 Global Step: 171155 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:25:16,409-Speed 131.20 samples/sec Loss 1.0557 Epoch: 17 Global Step: 171160 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:25:36,047-Speed 130.37 samples/sec Loss 1.0977 Epoch: 17 Global Step: 171165 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:25:55,648-Speed 130.61 samples/sec Loss 1.1166 Epoch: 17 Global Step: 171170 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:26:15,187-Speed 131.03 samples/sec Loss 1.2468 Epoch: 17 Global Step: 171175 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:26:34,753-Speed 130.85 samples/sec Loss 1.1536 Epoch: 17 Global Step: 171180 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:26:54,238-Speed 131.39 samples/sec Loss 1.1073 Epoch: 17 Global Step: 171185 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:27:14,043-Speed 129.26 samples/sec Loss 1.1483 Epoch: 17 Global Step: 171190 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:27:33,612-Speed 130.83 samples/sec Loss 1.0497 Epoch: 17 Global Step: 171195 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:27:53,284-Speed 130.14 samples/sec Loss 1.1437 Epoch: 17 Global Step: 171200 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:28:12,810-Speed 131.12 samples/sec Loss 1.1612 Epoch: 17 Global Step: 171205 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:28:32,420-Speed 130.55 samples/sec Loss 1.1822 Epoch: 17 Global Step: 171210 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:28:51,992-Speed 130.81 samples/sec Loss 1.0807 Epoch: 17 Global Step: 171215 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:29:11,596-Speed 130.59 samples/sec Loss 0.9839 Epoch: 17 Global Step: 171220 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:29:31,163-Speed 130.84 samples/sec Loss 1.1121 Epoch: 17 Global Step: 171225 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:29:50,772-Speed 130.57 samples/sec Loss 1.0963 Epoch: 17 Global Step: 171230 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:30:10,581-Speed 129.24 samples/sec Loss 1.1767 Epoch: 17 Global Step: 171235 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:30:30,116-Speed 131.05 samples/sec Loss 1.1207 Epoch: 17 Global Step: 171240 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:30:49,797-Speed 130.08 samples/sec Loss 1.0933 Epoch: 17 Global Step: 171245 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:31:09,400-Speed 130.60 samples/sec Loss 1.0072 Epoch: 17 Global Step: 171250 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:31:29,013-Speed 130.54 samples/sec Loss 1.0729 Epoch: 17 Global Step: 171255 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:31:48,530-Speed 131.17 samples/sec Loss 1.0060 Epoch: 17 Global Step: 171260 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:32:08,194-Speed 130.20 samples/sec Loss 1.0337 Epoch: 17 Global Step: 171265 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:32:27,870-Speed 130.12 samples/sec Loss 1.1527 Epoch: 17 Global Step: 171270 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:32:47,488-Speed 130.50 samples/sec Loss 1.1367 Epoch: 17 Global Step: 171275 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:33:07,274-Speed 129.39 samples/sec Loss 1.1212 Epoch: 17 Global Step: 171280 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:33:26,765-Speed 131.35 samples/sec Loss 1.0336 Epoch: 17 Global Step: 171285 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:33:46,407-Speed 130.34 samples/sec Loss 1.1271 Epoch: 17 Global Step: 171290 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:34:06,024-Speed 130.51 samples/sec Loss 1.0210 Epoch: 17 Global Step: 171295 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:34:25,636-Speed 130.54 samples/sec Loss 0.9895 Epoch: 17 Global Step: 171300 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:34:45,180-Speed 131.00 samples/sec Loss 1.0876 Epoch: 17 Global Step: 171305 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:35:04,864-Speed 130.06 samples/sec Loss 1.2174 Epoch: 17 Global Step: 171310 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:35:24,371-Speed 131.24 samples/sec Loss 1.0623 Epoch: 17 Global Step: 171315 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:35:43,920-Speed 130.96 samples/sec Loss 1.0220 Epoch: 17 Global Step: 171320 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:36:03,683-Speed 129.54 samples/sec Loss 1.1409 Epoch: 17 Global Step: 171325 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:36:23,447-Speed 129.53 samples/sec Loss 1.0445 Epoch: 17 Global Step: 171330 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:36:43,075-Speed 130.43 samples/sec Loss 1.1103 Epoch: 17 Global Step: 171335 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:37:02,691-Speed 130.52 samples/sec Loss 0.9844 Epoch: 17 Global Step: 171340 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:37:22,269-Speed 130.77 samples/sec Loss 1.0970 Epoch: 17 Global Step: 171345 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:37:41,821-Speed 130.94 samples/sec Loss 1.1592 Epoch: 17 Global Step: 171350 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:38:01,440-Speed 130.50 samples/sec Loss 0.9919 Epoch: 17 Global Step: 171355 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:38:21,744-Speed 126.09 samples/sec Loss 1.0096 Epoch: 17 Global Step: 171360 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:38:41,273-Speed 131.10 samples/sec Loss 1.1192 Epoch: 17 Global Step: 171365 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:39:01,047-Speed 129.47 samples/sec Loss 1.2264 Epoch: 17 Global Step: 171370 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:39:20,755-Speed 129.90 samples/sec Loss 1.1653 Epoch: 17 Global Step: 171375 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:39:40,299-Speed 130.99 samples/sec Loss 1.0724 Epoch: 17 Global Step: 171380 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:39:59,844-Speed 130.99 samples/sec Loss 1.1096 Epoch: 17 Global Step: 171385 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:40:19,398-Speed 130.93 samples/sec Loss 1.1540 Epoch: 17 Global Step: 171390 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:40:38,945-Speed 130.97 samples/sec Loss 1.0602 Epoch: 17 Global Step: 171395 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:40:58,537-Speed 130.68 samples/sec Loss 1.1381 Epoch: 17 Global Step: 171400 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:41:18,088-Speed 130.94 samples/sec Loss 1.0986 Epoch: 17 Global Step: 171405 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:41:37,697-Speed 130.56 samples/sec Loss 1.0768 Epoch: 17 Global Step: 171410 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:41:57,507-Speed 129.23 samples/sec Loss 1.2191 Epoch: 17 Global Step: 171415 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:42:17,160-Speed 130.27 samples/sec Loss 0.9983 Epoch: 17 Global Step: 171420 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:42:36,729-Speed 130.82 samples/sec Loss 1.1016 Epoch: 17 Global Step: 171425 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:42:56,336-Speed 130.57 samples/sec Loss 1.0792 Epoch: 17 Global Step: 171430 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:43:15,959-Speed 130.46 samples/sec Loss 1.1389 Epoch: 17 Global Step: 171435 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:43:35,456-Speed 131.31 samples/sec Loss 1.1122 Epoch: 17 Global Step: 171440 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:43:55,061-Speed 130.59 samples/sec Loss 1.1479 Epoch: 17 Global Step: 171445 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:44:14,596-Speed 131.05 samples/sec Loss 1.1020 Epoch: 17 Global Step: 171450 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:44:34,397-Speed 129.29 samples/sec Loss 1.1675 Epoch: 17 Global Step: 171455 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:44:54,251-Speed 128.95 samples/sec Loss 1.0697 Epoch: 17 Global Step: 171460 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:45:13,966-Speed 129.86 samples/sec Loss 1.0483 Epoch: 17 Global Step: 171465 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:45:33,408-Speed 131.68 samples/sec Loss 1.1516 Epoch: 17 Global Step: 171470 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:45:53,035-Speed 130.44 samples/sec Loss 1.2009 Epoch: 17 Global Step: 171475 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:46:12,592-Speed 130.90 samples/sec Loss 1.1969 Epoch: 17 Global Step: 171480 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:46:32,136-Speed 130.99 samples/sec Loss 1.2096 Epoch: 17 Global Step: 171485 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:46:51,707-Speed 130.82 samples/sec Loss 1.1566 Epoch: 17 Global Step: 171490 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:47:11,365-Speed 130.23 samples/sec Loss 1.0407 Epoch: 17 Global Step: 171495 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:47:30,984-Speed 130.50 samples/sec Loss 1.0717 Epoch: 17 Global Step: 171500 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:47:50,903-Speed 128.53 samples/sec Loss 0.9833 Epoch: 17 Global Step: 171505 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:48:10,347-Speed 131.67 samples/sec Loss 1.1176 Epoch: 17 Global Step: 171510 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:48:30,000-Speed 130.27 samples/sec Loss 1.1679 Epoch: 17 Global Step: 171515 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:48:49,577-Speed 130.78 samples/sec Loss 1.3602 Epoch: 17 Global Step: 171520 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:49:09,113-Speed 131.05 samples/sec Loss 1.1061 Epoch: 17 Global Step: 171525 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:49:28,721-Speed 130.57 samples/sec Loss 1.0169 Epoch: 17 Global Step: 171530 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:49:48,242-Speed 131.15 samples/sec Loss 1.0816 Epoch: 17 Global Step: 171535 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:50:07,908-Speed 130.18 samples/sec Loss 1.1761 Epoch: 17 Global Step: 171540 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:50:27,471-Speed 130.86 samples/sec Loss 1.0892 Epoch: 17 Global Step: 171545 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:50:47,305-Speed 129.08 samples/sec Loss 1.0459 Epoch: 17 Global Step: 171550 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:51:06,995-Speed 130.02 samples/sec Loss 1.1492 Epoch: 17 Global Step: 171555 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:51:26,680-Speed 130.06 samples/sec Loss 1.1952 Epoch: 17 Global Step: 171560 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:51:46,245-Speed 130.85 samples/sec Loss 1.1670 Epoch: 17 Global Step: 171565 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:52:06,015-Speed 129.49 samples/sec Loss 1.0711 Epoch: 17 Global Step: 171570 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:52:25,723-Speed 129.90 samples/sec Loss 1.1106 Epoch: 17 Global Step: 171575 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:52:45,188-Speed 131.53 samples/sec Loss 1.1099 Epoch: 17 Global Step: 171580 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:53:04,754-Speed 130.85 samples/sec Loss 1.0493 Epoch: 17 Global Step: 171585 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:53:24,356-Speed 130.61 samples/sec Loss 1.1401 Epoch: 17 Global Step: 171590 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:53:44,135-Speed 129.43 samples/sec Loss 1.1142 Epoch: 17 Global Step: 171595 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:54:03,765-Speed 130.42 samples/sec Loss 1.1442 Epoch: 17 Global Step: 171600 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:54:23,408-Speed 130.34 samples/sec Loss 1.1223 Epoch: 17 Global Step: 171605 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:54:42,913-Speed 131.25 samples/sec Loss 1.0262 Epoch: 17 Global Step: 171610 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:55:02,504-Speed 130.68 samples/sec Loss 1.0538 Epoch: 17 Global Step: 171615 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:55:22,110-Speed 130.58 samples/sec Loss 1.1114 Epoch: 17 Global Step: 171620 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:55:41,658-Speed 130.97 samples/sec Loss 1.0702 Epoch: 17 Global Step: 171625 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:56:01,214-Speed 130.92 samples/sec Loss 1.2313 Epoch: 17 Global Step: 171630 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:56:20,705-Speed 131.35 samples/sec Loss 1.1596 Epoch: 17 Global Step: 171635 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:56:40,522-Speed 129.19 samples/sec Loss 1.0187 Epoch: 17 Global Step: 171640 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:57:00,192-Speed 130.16 samples/sec Loss 1.0864 Epoch: 17 Global Step: 171645 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:57:19,758-Speed 130.85 samples/sec Loss 1.0466 Epoch: 17 Global Step: 171650 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:57:39,192-Speed 131.74 samples/sec Loss 1.1622 Epoch: 17 Global Step: 171655 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:57:58,815-Speed 130.46 samples/sec Loss 1.0557 Epoch: 17 Global Step: 171660 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:58:18,282-Speed 131.51 samples/sec Loss 1.0587 Epoch: 17 Global Step: 171665 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:58:37,812-Speed 131.09 samples/sec Loss 1.1269 Epoch: 17 Global Step: 171670 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:58:57,302-Speed 131.36 samples/sec Loss 1.0379 Epoch: 17 Global Step: 171675 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:59:16,777-Speed 131.46 samples/sec Loss 0.9671 Epoch: 17 Global Step: 171680 Fp16 Grad Scale: 8192 Required: 92 hours Training: 2021-06-15 05:59:36,570-Speed 129.35 samples/sec Loss 0.9845 Epoch: 17 Global Step: 171685 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 05:59:56,297-Speed 129.78 samples/sec Loss 1.1208 Epoch: 17 Global Step: 171690 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:00:15,734-Speed 131.71 samples/sec Loss 1.1216 Epoch: 17 Global Step: 171695 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:00:35,314-Speed 130.75 samples/sec Loss 1.1355 Epoch: 17 Global Step: 171700 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:00:54,873-Speed 130.89 samples/sec Loss 1.1704 Epoch: 17 Global Step: 171705 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:01:14,445-Speed 130.80 samples/sec Loss 1.1088 Epoch: 17 Global Step: 171710 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:01:34,041-Speed 130.65 samples/sec Loss 1.0580 Epoch: 17 Global Step: 171715 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:01:53,487-Speed 131.66 samples/sec Loss 0.9653 Epoch: 17 Global Step: 171720 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:02:13,073-Speed 130.71 samples/sec Loss 1.0779 Epoch: 17 Global Step: 171725 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:02:32,723-Speed 130.28 samples/sec Loss 1.1413 Epoch: 17 Global Step: 171730 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:02:52,433-Speed 129.89 samples/sec Loss 1.0217 Epoch: 17 Global Step: 171735 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:03:11,861-Speed 131.77 samples/sec Loss 1.0897 Epoch: 17 Global Step: 171740 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:03:31,477-Speed 130.52 samples/sec Loss 1.0951 Epoch: 17 Global Step: 171745 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:03:50,932-Speed 131.59 samples/sec Loss 1.1391 Epoch: 17 Global Step: 171750 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:04:10,453-Speed 131.14 samples/sec Loss 1.1311 Epoch: 17 Global Step: 171755 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:04:30,028-Speed 130.79 samples/sec Loss 1.0402 Epoch: 17 Global Step: 171760 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:04:49,475-Speed 131.65 samples/sec Loss 1.0582 Epoch: 17 Global Step: 171765 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:05:09,084-Speed 130.56 samples/sec Loss 0.9653 Epoch: 17 Global Step: 171770 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:05:28,877-Speed 129.35 samples/sec Loss 1.0769 Epoch: 17 Global Step: 171775 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:05:48,497-Speed 130.49 samples/sec Loss 1.0642 Epoch: 17 Global Step: 171780 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:06:07,987-Speed 131.36 samples/sec Loss 1.0494 Epoch: 17 Global Step: 171785 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:06:27,604-Speed 130.51 samples/sec Loss 1.0966 Epoch: 17 Global Step: 171790 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:06:47,136-Speed 131.07 samples/sec Loss 1.1034 Epoch: 17 Global Step: 171795 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:07:06,737-Speed 130.61 samples/sec Loss 1.0658 Epoch: 17 Global Step: 171800 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:07:26,173-Speed 131.72 samples/sec Loss 1.0568 Epoch: 17 Global Step: 171805 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:07:45,780-Speed 130.57 samples/sec Loss 1.1110 Epoch: 17 Global Step: 171810 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:08:05,349-Speed 130.83 samples/sec Loss 1.0891 Epoch: 17 Global Step: 171815 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:08:25,281-Speed 128.45 samples/sec Loss 1.1479 Epoch: 17 Global Step: 171820 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:08:44,711-Speed 131.76 samples/sec Loss 1.0908 Epoch: 17 Global Step: 171825 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:09:04,313-Speed 130.60 samples/sec Loss 1.1543 Epoch: 17 Global Step: 171830 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:09:23,858-Speed 130.99 samples/sec Loss 1.1468 Epoch: 17 Global Step: 171835 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:09:43,306-Speed 131.65 samples/sec Loss 1.2564 Epoch: 17 Global Step: 171840 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:10:02,899-Speed 130.66 samples/sec Loss 1.1668 Epoch: 17 Global Step: 171845 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:10:22,342-Speed 131.68 samples/sec Loss 1.1768 Epoch: 17 Global Step: 171850 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:10:41,953-Speed 130.54 samples/sec Loss 0.9917 Epoch: 17 Global Step: 171855 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:11:01,494-Speed 131.01 samples/sec Loss 1.0565 Epoch: 17 Global Step: 171860 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:11:21,386-Speed 128.71 samples/sec Loss 1.1882 Epoch: 17 Global Step: 171865 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:11:40,807-Speed 131.82 samples/sec Loss 1.1193 Epoch: 17 Global Step: 171870 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:12:00,420-Speed 130.54 samples/sec Loss 1.0728 Epoch: 17 Global Step: 171875 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:12:19,974-Speed 130.93 samples/sec Loss 0.9806 Epoch: 17 Global Step: 171880 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:12:39,521-Speed 130.97 samples/sec Loss 1.0606 Epoch: 17 Global Step: 171885 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:12:59,122-Speed 130.62 samples/sec Loss 1.1434 Epoch: 17 Global Step: 171890 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:13:18,661-Speed 131.03 samples/sec Loss 1.2582 Epoch: 17 Global Step: 171895 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:13:38,250-Speed 130.69 samples/sec Loss 1.1225 Epoch: 17 Global Step: 171900 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:13:57,723-Speed 131.47 samples/sec Loss 1.1122 Epoch: 17 Global Step: 171905 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:14:17,650-Speed 128.48 samples/sec Loss 1.1361 Epoch: 17 Global Step: 171910 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:14:37,120-Speed 131.49 samples/sec Loss 1.0489 Epoch: 17 Global Step: 171915 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:14:56,805-Speed 130.06 samples/sec Loss 1.0960 Epoch: 17 Global Step: 171920 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:15:16,365-Speed 130.89 samples/sec Loss 1.0205 Epoch: 17 Global Step: 171925 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:15:35,923-Speed 130.90 samples/sec Loss 1.1757 Epoch: 17 Global Step: 171930 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:15:55,522-Speed 130.63 samples/sec Loss 1.1578 Epoch: 17 Global Step: 171935 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:16:15,119-Speed 130.64 samples/sec Loss 1.0746 Epoch: 17 Global Step: 171940 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:16:34,236-Speed 133.93 samples/sec Loss 1.1246 Epoch: 17 Global Step: 171945 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:16:53,842-Speed 130.58 samples/sec Loss 1.1310 Epoch: 17 Global Step: 171950 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:17:13,754-Speed 128.58 samples/sec Loss 1.2588 Epoch: 17 Global Step: 171955 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:17:33,170-Speed 131.86 samples/sec Loss 1.0439 Epoch: 17 Global Step: 171960 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:17:52,948-Speed 129.44 samples/sec Loss 1.0790 Epoch: 17 Global Step: 171965 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:18:13,050-Speed 127.36 samples/sec Loss 1.0567 Epoch: 17 Global Step: 171970 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:18:32,594-Speed 131.00 samples/sec Loss 1.0470 Epoch: 17 Global Step: 171975 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:18:52,108-Speed 131.19 samples/sec Loss 1.2346 Epoch: 17 Global Step: 171980 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:19:11,786-Speed 130.10 samples/sec Loss 1.1063 Epoch: 17 Global Step: 171985 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:19:31,360-Speed 130.79 samples/sec Loss 1.0358 Epoch: 17 Global Step: 171990 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:19:50,916-Speed 130.91 samples/sec Loss 1.0811 Epoch: 17 Global Step: 171995 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:20:10,713-Speed 129.32 samples/sec Loss 1.1419 Epoch: 17 Global Step: 172000 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:20:30,280-Speed 130.84 samples/sec Loss 1.0358 Epoch: 17 Global Step: 172005 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:20:49,847-Speed 130.84 samples/sec Loss 1.1422 Epoch: 17 Global Step: 172010 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:21:09,391-Speed 130.99 samples/sec Loss 1.0099 Epoch: 17 Global Step: 172015 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:21:28,955-Speed 130.86 samples/sec Loss 1.1452 Epoch: 17 Global Step: 172020 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:21:48,537-Speed 130.74 samples/sec Loss 1.0164 Epoch: 17 Global Step: 172025 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:22:08,131-Speed 130.66 samples/sec Loss 0.9425 Epoch: 17 Global Step: 172030 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:22:27,782-Speed 130.28 samples/sec Loss 1.1145 Epoch: 17 Global Step: 172035 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:22:47,371-Speed 130.70 samples/sec Loss 1.0316 Epoch: 17 Global Step: 172040 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:23:07,353-Speed 128.12 samples/sec Loss 1.0952 Epoch: 17 Global Step: 172045 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:23:27,125-Speed 129.48 samples/sec Loss 1.0718 Epoch: 17 Global Step: 172050 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:23:46,583-Speed 131.57 samples/sec Loss 1.1186 Epoch: 17 Global Step: 172055 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:24:06,120-Speed 131.04 samples/sec Loss 1.2077 Epoch: 17 Global Step: 172060 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:24:25,793-Speed 130.14 samples/sec Loss 1.1900 Epoch: 17 Global Step: 172065 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:24:45,418-Speed 130.46 samples/sec Loss 1.1167 Epoch: 17 Global Step: 172070 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:25:05,030-Speed 130.54 samples/sec Loss 1.1391 Epoch: 17 Global Step: 172075 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:25:24,561-Speed 131.08 samples/sec Loss 1.1225 Epoch: 17 Global Step: 172080 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:25:44,034-Speed 131.47 samples/sec Loss 1.0333 Epoch: 17 Global Step: 172085 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:26:03,799-Speed 129.53 samples/sec Loss 1.1973 Epoch: 17 Global Step: 172090 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:26:23,589-Speed 129.36 samples/sec Loss 1.0046 Epoch: 17 Global Step: 172095 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:26:43,075-Speed 131.38 samples/sec Loss 1.0012 Epoch: 17 Global Step: 172100 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:27:02,554-Speed 131.43 samples/sec Loss 1.1218 Epoch: 17 Global Step: 172105 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:27:22,113-Speed 130.89 samples/sec Loss 1.1195 Epoch: 17 Global Step: 172110 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:27:41,732-Speed 130.49 samples/sec Loss 1.1322 Epoch: 17 Global Step: 172115 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:28:01,330-Speed 130.63 samples/sec Loss 1.0777 Epoch: 17 Global Step: 172120 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:28:20,876-Speed 130.98 samples/sec Loss 1.0887 Epoch: 17 Global Step: 172125 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:28:40,394-Speed 131.17 samples/sec Loss 1.1718 Epoch: 17 Global Step: 172130 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:29:00,191-Speed 129.32 samples/sec Loss 0.9935 Epoch: 17 Global Step: 172135 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:29:19,902-Speed 129.89 samples/sec Loss 1.1101 Epoch: 17 Global Step: 172140 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:29:39,470-Speed 130.83 samples/sec Loss 1.0855 Epoch: 17 Global Step: 172145 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:29:59,061-Speed 130.68 samples/sec Loss 1.1243 Epoch: 17 Global Step: 172150 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:30:18,660-Speed 130.63 samples/sec Loss 1.1609 Epoch: 17 Global Step: 172155 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:30:38,204-Speed 131.00 samples/sec Loss 1.0595 Epoch: 17 Global Step: 172160 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:30:57,779-Speed 130.79 samples/sec Loss 1.1995 Epoch: 17 Global Step: 172165 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:31:17,317-Speed 131.04 samples/sec Loss 1.0044 Epoch: 17 Global Step: 172170 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:31:36,939-Speed 130.47 samples/sec Loss 1.1037 Epoch: 17 Global Step: 172175 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:31:56,922-Speed 128.12 samples/sec Loss 1.1724 Epoch: 17 Global Step: 172180 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:32:16,349-Speed 131.78 samples/sec Loss 1.0830 Epoch: 17 Global Step: 172185 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:32:35,929-Speed 130.76 samples/sec Loss 0.9456 Epoch: 17 Global Step: 172190 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:32:55,497-Speed 130.83 samples/sec Loss 1.1409 Epoch: 17 Global Step: 172195 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:33:15,036-Speed 131.03 samples/sec Loss 0.9954 Epoch: 17 Global Step: 172200 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:33:34,654-Speed 130.50 samples/sec Loss 1.0514 Epoch: 17 Global Step: 172205 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:33:54,216-Speed 130.87 samples/sec Loss 1.0800 Epoch: 17 Global Step: 172210 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:34:13,743-Speed 131.11 samples/sec Loss 1.0362 Epoch: 17 Global Step: 172215 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:34:33,270-Speed 131.11 samples/sec Loss 1.0439 Epoch: 17 Global Step: 172220 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:34:53,205-Speed 128.42 samples/sec Loss 1.1067 Epoch: 17 Global Step: 172225 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:35:12,624-Speed 131.84 samples/sec Loss 1.0742 Epoch: 17 Global Step: 172230 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:35:32,280-Speed 130.25 samples/sec Loss 1.0438 Epoch: 17 Global Step: 172235 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:35:51,850-Speed 130.82 samples/sec Loss 1.0931 Epoch: 17 Global Step: 172240 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:36:11,388-Speed 131.03 samples/sec Loss 1.0012 Epoch: 17 Global Step: 172245 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:36:31,009-Speed 130.48 samples/sec Loss 1.1904 Epoch: 17 Global Step: 172250 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:36:50,550-Speed 131.01 samples/sec Loss 1.1065 Epoch: 17 Global Step: 172255 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:37:10,037-Speed 131.38 samples/sec Loss 1.1341 Epoch: 17 Global Step: 172260 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:37:29,644-Speed 130.57 samples/sec Loss 1.0901 Epoch: 17 Global Step: 172265 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:37:49,531-Speed 128.73 samples/sec Loss 0.9214 Epoch: 17 Global Step: 172270 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:38:08,988-Speed 131.58 samples/sec Loss 1.2285 Epoch: 17 Global Step: 172275 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:38:28,521-Speed 131.07 samples/sec Loss 1.1721 Epoch: 17 Global Step: 172280 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:38:48,120-Speed 130.62 samples/sec Loss 1.1203 Epoch: 17 Global Step: 172285 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:39:07,601-Speed 131.42 samples/sec Loss 1.2109 Epoch: 17 Global Step: 172290 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:39:27,202-Speed 130.61 samples/sec Loss 1.1070 Epoch: 17 Global Step: 172295 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:39:46,718-Speed 131.18 samples/sec Loss 1.1214 Epoch: 17 Global Step: 172300 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:40:06,304-Speed 130.71 samples/sec Loss 1.0608 Epoch: 17 Global Step: 172305 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:40:25,871-Speed 130.85 samples/sec Loss 1.0737 Epoch: 17 Global Step: 172310 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:40:45,715-Speed 129.01 samples/sec Loss 1.1481 Epoch: 17 Global Step: 172315 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:41:05,131-Speed 131.85 samples/sec Loss 1.0831 Epoch: 17 Global Step: 172320 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:41:24,844-Speed 129.87 samples/sec Loss 0.9836 Epoch: 17 Global Step: 172325 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:41:44,414-Speed 130.82 samples/sec Loss 1.0750 Epoch: 17 Global Step: 172330 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:42:03,852-Speed 131.71 samples/sec Loss 0.9988 Epoch: 17 Global Step: 172335 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:42:23,464-Speed 130.54 samples/sec Loss 1.0424 Epoch: 17 Global Step: 172340 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:42:42,934-Speed 131.49 samples/sec Loss 1.1665 Epoch: 17 Global Step: 172345 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:43:02,664-Speed 129.76 samples/sec Loss 1.0511 Epoch: 17 Global Step: 172350 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:43:22,167-Speed 131.27 samples/sec Loss 1.1555 Epoch: 17 Global Step: 172355 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:43:41,972-Speed 129.26 samples/sec Loss 1.0924 Epoch: 17 Global Step: 172360 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:44:01,393-Speed 131.83 samples/sec Loss 1.0146 Epoch: 17 Global Step: 172365 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:44:21,392-Speed 128.01 samples/sec Loss 0.9706 Epoch: 17 Global Step: 172370 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:44:40,921-Speed 131.09 samples/sec Loss 0.9013 Epoch: 17 Global Step: 172375 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:45:00,360-Speed 131.70 samples/sec Loss 1.0509 Epoch: 17 Global Step: 172380 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:45:19,981-Speed 130.48 samples/sec Loss 1.0271 Epoch: 17 Global Step: 172385 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:45:39,487-Speed 131.25 samples/sec Loss 1.1348 Epoch: 17 Global Step: 172390 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:45:59,218-Speed 129.75 samples/sec Loss 0.9928 Epoch: 17 Global Step: 172395 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:46:18,816-Speed 130.64 samples/sec Loss 1.1543 Epoch: 17 Global Step: 172400 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:46:38,725-Speed 128.59 samples/sec Loss 1.1461 Epoch: 17 Global Step: 172405 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:46:58,171-Speed 131.66 samples/sec Loss 1.1207 Epoch: 17 Global Step: 172410 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:47:17,875-Speed 129.93 samples/sec Loss 1.0896 Epoch: 17 Global Step: 172415 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:47:37,445-Speed 130.82 samples/sec Loss 1.1688 Epoch: 17 Global Step: 172420 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:47:56,987-Speed 131.00 samples/sec Loss 1.0781 Epoch: 17 Global Step: 172425 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:48:16,585-Speed 130.63 samples/sec Loss 1.1924 Epoch: 17 Global Step: 172430 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:48:36,170-Speed 130.72 samples/sec Loss 1.0028 Epoch: 17 Global Step: 172435 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:48:55,679-Speed 131.23 samples/sec Loss 1.1110 Epoch: 17 Global Step: 172440 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:49:15,199-Speed 131.16 samples/sec Loss 0.9969 Epoch: 17 Global Step: 172445 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:49:35,183-Speed 128.11 samples/sec Loss 1.0410 Epoch: 17 Global Step: 172450 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:49:54,623-Speed 131.70 samples/sec Loss 1.0415 Epoch: 17 Global Step: 172455 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:50:14,450-Speed 129.13 samples/sec Loss 1.0805 Epoch: 17 Global Step: 172460 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:50:34,060-Speed 130.55 samples/sec Loss 1.0121 Epoch: 17 Global Step: 172465 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:50:53,752-Speed 130.01 samples/sec Loss 1.1164 Epoch: 17 Global Step: 172470 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:51:13,333-Speed 130.74 samples/sec Loss 1.0838 Epoch: 17 Global Step: 172475 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:51:32,947-Speed 130.53 samples/sec Loss 1.0474 Epoch: 17 Global Step: 172480 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:51:52,542-Speed 130.65 samples/sec Loss 1.1711 Epoch: 17 Global Step: 172485 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:52:12,096-Speed 130.92 samples/sec Loss 1.1032 Epoch: 17 Global Step: 172490 Fp16 Grad Scale: 8192 Required: 91 hours Training: 2021-06-15 06:52:31,225-Speed 133.84 samples/sec Loss 1.2259 Epoch: 17 Global Step: 172495 Fp16 Grad Scale: 4096 Required: 91 hours Training: 2021-06-15 06:52:50,762-Speed 131.04 samples/sec Loss 1.1640 Epoch: 17 Global Step: 172500 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:53:10,650-Speed 128.72 samples/sec Loss 1.1420 Epoch: 17 Global Step: 172505 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:53:30,089-Speed 131.71 samples/sec Loss 1.1822 Epoch: 17 Global Step: 172510 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:53:49,777-Speed 130.03 samples/sec Loss 1.2355 Epoch: 17 Global Step: 172515 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:54:09,381-Speed 130.59 samples/sec Loss 1.0594 Epoch: 17 Global Step: 172520 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:54:28,822-Speed 131.69 samples/sec Loss 1.2147 Epoch: 17 Global Step: 172525 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:54:48,424-Speed 130.61 samples/sec Loss 1.1790 Epoch: 17 Global Step: 172530 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:55:07,943-Speed 131.16 samples/sec Loss 1.1086 Epoch: 17 Global Step: 172535 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:55:27,674-Speed 129.76 samples/sec Loss 1.0990 Epoch: 17 Global Step: 172540 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:55:47,302-Speed 130.43 samples/sec Loss 1.0130 Epoch: 17 Global Step: 172545 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:56:07,100-Speed 129.31 samples/sec Loss 1.1307 Epoch: 17 Global Step: 172550 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:56:26,669-Speed 130.83 samples/sec Loss 1.1598 Epoch: 17 Global Step: 172555 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:56:46,296-Speed 130.44 samples/sec Loss 1.0418 Epoch: 17 Global Step: 172560 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:57:05,914-Speed 130.50 samples/sec Loss 0.9917 Epoch: 17 Global Step: 172565 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:57:25,513-Speed 130.63 samples/sec Loss 1.2031 Epoch: 17 Global Step: 172570 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:57:45,054-Speed 131.01 samples/sec Loss 1.1508 Epoch: 17 Global Step: 172575 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:58:04,600-Speed 130.98 samples/sec Loss 1.1353 Epoch: 17 Global Step: 172580 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:58:24,164-Speed 130.86 samples/sec Loss 0.9526 Epoch: 17 Global Step: 172585 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:58:43,710-Speed 130.97 samples/sec Loss 0.9878 Epoch: 17 Global Step: 172590 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 06:59:03,500-Speed 129.37 samples/sec Loss 1.0617 Epoch: 17 Global Step: 172595 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 06:59:23,228-Speed 129.77 samples/sec Loss 1.0230 Epoch: 17 Global Step: 172600 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 06:59:42,770-Speed 131.00 samples/sec Loss 1.1067 Epoch: 17 Global Step: 172605 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:00:01,850-Speed 134.18 samples/sec Loss 0.9717 Epoch: 17 Global Step: 172610 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:00:21,412-Speed 130.87 samples/sec Loss 1.2048 Epoch: 17 Global Step: 172615 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:00:40,954-Speed 131.01 samples/sec Loss 1.0285 Epoch: 17 Global Step: 172620 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:01:00,563-Speed 130.56 samples/sec Loss 1.1318 Epoch: 17 Global Step: 172625 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:01:20,145-Speed 130.74 samples/sec Loss 1.1037 Epoch: 17 Global Step: 172630 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:01:39,752-Speed 130.57 samples/sec Loss 1.2003 Epoch: 17 Global Step: 172635 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:01:59,505-Speed 129.61 samples/sec Loss 0.9867 Epoch: 17 Global Step: 172640 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:02:19,190-Speed 130.05 samples/sec Loss 1.0805 Epoch: 17 Global Step: 172645 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:02:38,797-Speed 130.57 samples/sec Loss 1.0077 Epoch: 17 Global Step: 172650 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:02:58,404-Speed 130.58 samples/sec Loss 1.1288 Epoch: 17 Global Step: 172655 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:03:18,167-Speed 129.54 samples/sec Loss 1.1001 Epoch: 17 Global Step: 172660 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:03:37,699-Speed 131.07 samples/sec Loss 1.0778 Epoch: 17 Global Step: 172665 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:03:57,261-Speed 130.87 samples/sec Loss 1.0130 Epoch: 17 Global Step: 172670 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:04:16,880-Speed 130.50 samples/sec Loss 1.1205 Epoch: 17 Global Step: 172675 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:04:36,468-Speed 130.70 samples/sec Loss 1.2593 Epoch: 17 Global Step: 172680 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:04:56,247-Speed 129.44 samples/sec Loss 1.0424 Epoch: 17 Global Step: 172685 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:05:15,877-Speed 130.42 samples/sec Loss 1.2211 Epoch: 17 Global Step: 172690 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:05:35,435-Speed 130.90 samples/sec Loss 1.1249 Epoch: 17 Global Step: 172695 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:05:55,040-Speed 130.59 samples/sec Loss 1.0852 Epoch: 17 Global Step: 172700 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:06:14,580-Speed 131.02 samples/sec Loss 1.1543 Epoch: 17 Global Step: 172705 Fp16 Grad Scale: 4096 Required: 90 hours Training: 2021-06-15 07:06:34,126-Speed 130.98 samples/sec Loss 1.1774 Epoch: 17 Global Step: 172710 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:06:53,722-Speed 130.65 samples/sec Loss 1.1137 Epoch: 17 Global Step: 172715 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:07:13,254-Speed 131.07 samples/sec Loss 1.1189 Epoch: 17 Global Step: 172720 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:07:33,197-Speed 128.37 samples/sec Loss 1.0925 Epoch: 17 Global Step: 172725 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:07:53,030-Speed 129.09 samples/sec Loss 1.1319 Epoch: 17 Global Step: 172730 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:08:12,527-Speed 131.31 samples/sec Loss 1.1438 Epoch: 17 Global Step: 172735 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:08:32,171-Speed 130.32 samples/sec Loss 1.1079 Epoch: 17 Global Step: 172740 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:08:51,759-Speed 130.70 samples/sec Loss 1.1309 Epoch: 17 Global Step: 172745 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:09:11,305-Speed 130.98 samples/sec Loss 1.0968 Epoch: 17 Global Step: 172750 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:09:30,832-Speed 131.11 samples/sec Loss 1.1637 Epoch: 17 Global Step: 172755 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:09:50,371-Speed 131.03 samples/sec Loss 1.0127 Epoch: 17 Global Step: 172760 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:10:09,907-Speed 131.05 samples/sec Loss 1.1574 Epoch: 17 Global Step: 172765 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:10:29,504-Speed 130.64 samples/sec Loss 1.0777 Epoch: 17 Global Step: 172770 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:10:49,245-Speed 129.69 samples/sec Loss 1.1008 Epoch: 17 Global Step: 172775 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:11:08,870-Speed 130.45 samples/sec Loss 1.0940 Epoch: 17 Global Step: 172780 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:11:28,387-Speed 131.17 samples/sec Loss 0.9997 Epoch: 17 Global Step: 172785 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:11:47,994-Speed 130.58 samples/sec Loss 1.1650 Epoch: 17 Global Step: 172790 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:12:07,517-Speed 131.13 samples/sec Loss 1.0796 Epoch: 17 Global Step: 172795 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:12:27,132-Speed 130.52 samples/sec Loss 1.0007 Epoch: 17 Global Step: 172800 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:12:46,653-Speed 131.15 samples/sec Loss 1.0160 Epoch: 17 Global Step: 172805 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:13:06,192-Speed 131.02 samples/sec Loss 1.0777 Epoch: 17 Global Step: 172810 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:13:25,798-Speed 130.58 samples/sec Loss 1.1350 Epoch: 17 Global Step: 172815 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:13:45,673-Speed 128.81 samples/sec Loss 1.0307 Epoch: 17 Global Step: 172820 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:14:05,105-Speed 131.75 samples/sec Loss 1.2291 Epoch: 17 Global Step: 172825 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:14:24,742-Speed 130.38 samples/sec Loss 1.1947 Epoch: 17 Global Step: 172830 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:14:44,341-Speed 130.62 samples/sec Loss 1.1607 Epoch: 17 Global Step: 172835 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:15:03,857-Speed 131.18 samples/sec Loss 1.1486 Epoch: 17 Global Step: 172840 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:15:23,471-Speed 130.53 samples/sec Loss 1.1292 Epoch: 17 Global Step: 172845 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:15:43,014-Speed 131.00 samples/sec Loss 1.1099 Epoch: 17 Global Step: 172850 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:16:02,569-Speed 130.92 samples/sec Loss 1.0317 Epoch: 17 Global Step: 172855 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:16:22,135-Speed 130.85 samples/sec Loss 1.1811 Epoch: 17 Global Step: 172860 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:16:42,059-Speed 128.49 samples/sec Loss 1.0991 Epoch: 17 Global Step: 172865 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:17:01,492-Speed 131.74 samples/sec Loss 1.1022 Epoch: 17 Global Step: 172870 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:17:21,117-Speed 130.46 samples/sec Loss 1.0710 Epoch: 17 Global Step: 172875 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:17:40,681-Speed 130.86 samples/sec Loss 1.0274 Epoch: 17 Global Step: 172880 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:18:00,239-Speed 130.90 samples/sec Loss 1.0665 Epoch: 17 Global Step: 172885 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:18:19,885-Speed 130.31 samples/sec Loss 1.0550 Epoch: 17 Global Step: 172890 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:18:39,420-Speed 131.06 samples/sec Loss 1.1374 Epoch: 17 Global Step: 172895 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:18:59,012-Speed 130.67 samples/sec Loss 1.0876 Epoch: 17 Global Step: 172900 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:19:18,742-Speed 129.76 samples/sec Loss 0.9241 Epoch: 17 Global Step: 172905 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:19:38,662-Speed 128.53 samples/sec Loss 1.0444 Epoch: 17 Global Step: 172910 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:19:58,096-Speed 131.73 samples/sec Loss 1.0054 Epoch: 17 Global Step: 172915 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:20:17,824-Speed 129.77 samples/sec Loss 1.0068 Epoch: 17 Global Step: 172920 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:20:37,335-Speed 131.22 samples/sec Loss 0.9679 Epoch: 17 Global Step: 172925 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:20:56,921-Speed 130.71 samples/sec Loss 1.1428 Epoch: 17 Global Step: 172930 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:21:16,477-Speed 130.91 samples/sec Loss 1.1213 Epoch: 17 Global Step: 172935 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:21:36,029-Speed 130.94 samples/sec Loss 1.0625 Epoch: 17 Global Step: 172940 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:21:55,627-Speed 130.64 samples/sec Loss 1.0358 Epoch: 17 Global Step: 172945 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:22:15,175-Speed 130.97 samples/sec Loss 1.0973 Epoch: 17 Global Step: 172950 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:22:35,185-Speed 127.95 samples/sec Loss 1.0773 Epoch: 17 Global Step: 172955 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:22:54,620-Speed 131.72 samples/sec Loss 1.2196 Epoch: 17 Global Step: 172960 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:23:14,321-Speed 129.95 samples/sec Loss 0.9735 Epoch: 17 Global Step: 172965 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:23:33,865-Speed 130.99 samples/sec Loss 1.1183 Epoch: 17 Global Step: 172970 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:23:53,463-Speed 130.64 samples/sec Loss 1.1032 Epoch: 17 Global Step: 172975 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:24:13,031-Speed 130.83 samples/sec Loss 1.2033 Epoch: 17 Global Step: 172980 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:24:32,773-Speed 129.68 samples/sec Loss 1.1257 Epoch: 17 Global Step: 172985 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:24:52,360-Speed 130.71 samples/sec Loss 1.1453 Epoch: 17 Global Step: 172990 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:25:11,883-Speed 131.14 samples/sec Loss 1.2221 Epoch: 17 Global Step: 172995 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:25:31,828-Speed 128.36 samples/sec Loss 1.0634 Epoch: 17 Global Step: 173000 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:25:51,295-Speed 131.51 samples/sec Loss 1.0192 Epoch: 17 Global Step: 173005 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:26:10,993-Speed 129.97 samples/sec Loss 1.1343 Epoch: 17 Global Step: 173010 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:26:30,527-Speed 131.06 samples/sec Loss 1.0492 Epoch: 17 Global Step: 173015 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:26:50,125-Speed 130.64 samples/sec Loss 1.0187 Epoch: 17 Global Step: 173020 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:27:09,682-Speed 130.90 samples/sec Loss 1.0981 Epoch: 17 Global Step: 173025 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:27:29,184-Speed 131.28 samples/sec Loss 1.0637 Epoch: 17 Global Step: 173030 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:27:48,745-Speed 130.88 samples/sec Loss 1.0994 Epoch: 17 Global Step: 173035 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:28:08,827-Speed 127.49 samples/sec Loss 1.1144 Epoch: 17 Global Step: 173040 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:28:28,626-Speed 129.31 samples/sec Loss 1.1334 Epoch: 17 Global Step: 173045 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:28:48,329-Speed 129.94 samples/sec Loss 1.1627 Epoch: 17 Global Step: 173050 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:29:07,844-Speed 131.19 samples/sec Loss 1.0530 Epoch: 17 Global Step: 173055 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:29:27,428-Speed 130.72 samples/sec Loss 1.1489 Epoch: 17 Global Step: 173060 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:29:46,981-Speed 130.93 samples/sec Loss 0.9534 Epoch: 17 Global Step: 173065 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:30:06,532-Speed 130.95 samples/sec Loss 1.0776 Epoch: 17 Global Step: 173070 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:30:26,145-Speed 130.54 samples/sec Loss 1.0322 Epoch: 17 Global Step: 173075 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:30:45,667-Speed 131.14 samples/sec Loss 1.2256 Epoch: 17 Global Step: 173080 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:31:05,266-Speed 130.63 samples/sec Loss 1.1034 Epoch: 17 Global Step: 173085 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:31:25,027-Speed 129.56 samples/sec Loss 1.1203 Epoch: 17 Global Step: 173090 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:31:44,654-Speed 130.44 samples/sec Loss 1.1316 Epoch: 17 Global Step: 173095 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:32:04,169-Speed 131.19 samples/sec Loss 1.3001 Epoch: 17 Global Step: 173100 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:32:23,790-Speed 130.48 samples/sec Loss 1.0698 Epoch: 17 Global Step: 173105 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:32:43,362-Speed 130.81 samples/sec Loss 1.0452 Epoch: 17 Global Step: 173110 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:33:02,793-Speed 131.76 samples/sec Loss 1.1164 Epoch: 17 Global Step: 173115 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:33:22,390-Speed 130.64 samples/sec Loss 1.0740 Epoch: 17 Global Step: 173120 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:33:41,926-Speed 131.05 samples/sec Loss 1.1029 Epoch: 17 Global Step: 173125 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:34:01,530-Speed 130.60 samples/sec Loss 1.1188 Epoch: 17 Global Step: 173130 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:34:21,305-Speed 129.46 samples/sec Loss 1.1600 Epoch: 17 Global Step: 173135 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:34:40,937-Speed 130.41 samples/sec Loss 1.0855 Epoch: 17 Global Step: 173140 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:35:00,452-Speed 131.18 samples/sec Loss 1.0740 Epoch: 17 Global Step: 173145 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:35:20,062-Speed 130.55 samples/sec Loss 1.1191 Epoch: 17 Global Step: 173150 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:35:39,871-Speed 129.24 samples/sec Loss 1.1137 Epoch: 17 Global Step: 173155 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:35:59,413-Speed 131.00 samples/sec Loss 1.1187 Epoch: 17 Global Step: 173160 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:36:18,984-Speed 130.82 samples/sec Loss 1.0913 Epoch: 17 Global Step: 173165 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:36:38,503-Speed 131.16 samples/sec Loss 0.9639 Epoch: 17 Global Step: 173170 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:36:58,095-Speed 130.67 samples/sec Loss 1.1063 Epoch: 17 Global Step: 173175 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:37:17,774-Speed 130.10 samples/sec Loss 1.0436 Epoch: 17 Global Step: 173180 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:37:37,868-Speed 127.41 samples/sec Loss 1.0476 Epoch: 17 Global Step: 173185 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:37:57,355-Speed 131.37 samples/sec Loss 1.1104 Epoch: 17 Global Step: 173190 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:38:16,954-Speed 130.63 samples/sec Loss 1.0551 Epoch: 17 Global Step: 173195 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:38:36,399-Speed 131.66 samples/sec Loss 1.1793 Epoch: 17 Global Step: 173200 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:38:55,987-Speed 130.70 samples/sec Loss 1.1396 Epoch: 17 Global Step: 173205 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:39:15,520-Speed 131.07 samples/sec Loss 1.1386 Epoch: 17 Global Step: 173210 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:39:35,139-Speed 130.49 samples/sec Loss 1.1233 Epoch: 17 Global Step: 173215 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:39:54,736-Speed 130.64 samples/sec Loss 1.0967 Epoch: 17 Global Step: 173220 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:40:14,511-Speed 129.46 samples/sec Loss 1.1391 Epoch: 17 Global Step: 173225 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:40:34,194-Speed 130.07 samples/sec Loss 1.0344 Epoch: 17 Global Step: 173230 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:40:53,807-Speed 130.53 samples/sec Loss 1.0850 Epoch: 17 Global Step: 173235 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:41:13,423-Speed 130.51 samples/sec Loss 1.1129 Epoch: 17 Global Step: 173240 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:41:32,960-Speed 131.04 samples/sec Loss 1.1585 Epoch: 17 Global Step: 173245 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:41:52,573-Speed 130.53 samples/sec Loss 1.2512 Epoch: 17 Global Step: 173250 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:42:12,056-Speed 131.41 samples/sec Loss 1.2437 Epoch: 17 Global Step: 173255 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:42:31,688-Speed 130.40 samples/sec Loss 1.0077 Epoch: 17 Global Step: 173260 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:42:51,254-Speed 130.85 samples/sec Loss 1.0362 Epoch: 17 Global Step: 173265 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:43:11,172-Speed 128.54 samples/sec Loss 1.1686 Epoch: 17 Global Step: 173270 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:43:30,650-Speed 131.43 samples/sec Loss 0.9628 Epoch: 17 Global Step: 173275 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:43:50,247-Speed 130.64 samples/sec Loss 1.1895 Epoch: 17 Global Step: 173280 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:44:09,810-Speed 130.87 samples/sec Loss 1.1194 Epoch: 17 Global Step: 173285 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:44:29,341-Speed 131.08 samples/sec Loss 1.1374 Epoch: 17 Global Step: 173290 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:44:48,927-Speed 130.71 samples/sec Loss 1.0820 Epoch: 17 Global Step: 173295 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:45:08,378-Speed 131.62 samples/sec Loss 1.1676 Epoch: 17 Global Step: 173300 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:45:27,983-Speed 130.59 samples/sec Loss 1.0752 Epoch: 17 Global Step: 173305 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:45:47,453-Speed 131.49 samples/sec Loss 1.0874 Epoch: 17 Global Step: 173310 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:46:07,375-Speed 128.51 samples/sec Loss 1.1668 Epoch: 17 Global Step: 173315 Fp16 Grad Scale: 8192 Required: 90 hours Training: 2021-06-15 07:46:26,789-Speed 131.87 samples/sec Loss 1.0268 Epoch: 17 Global Step: 173320 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:46:46,566-Speed 129.45 samples/sec Loss 1.1317 Epoch: 17 Global Step: 173325 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:47:06,115-Speed 130.96 samples/sec Loss 1.0281 Epoch: 17 Global Step: 173330 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:47:25,585-Speed 131.49 samples/sec Loss 1.1413 Epoch: 17 Global Step: 173335 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:47:45,162-Speed 130.77 samples/sec Loss 1.0018 Epoch: 17 Global Step: 173340 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:48:04,745-Speed 130.73 samples/sec Loss 1.0448 Epoch: 17 Global Step: 173345 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:48:24,339-Speed 130.66 samples/sec Loss 1.1071 Epoch: 17 Global Step: 173350 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:48:43,840-Speed 131.28 samples/sec Loss 1.0888 Epoch: 17 Global Step: 173355 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:49:03,650-Speed 129.24 samples/sec Loss 1.2571 Epoch: 17 Global Step: 173360 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:49:23,087-Speed 131.71 samples/sec Loss 1.0687 Epoch: 17 Global Step: 173365 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:49:42,864-Speed 129.45 samples/sec Loss 0.9989 Epoch: 17 Global Step: 173370 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:50:02,503-Speed 130.36 samples/sec Loss 1.2100 Epoch: 17 Global Step: 173375 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:50:21,998-Speed 131.32 samples/sec Loss 1.1044 Epoch: 17 Global Step: 173380 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:50:41,191-Speed 133.39 samples/sec Loss 0.9897 Epoch: 17 Global Step: 173385 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:51:00,815-Speed 130.46 samples/sec Loss 1.1149 Epoch: 17 Global Step: 173390 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:51:20,434-Speed 130.50 samples/sec Loss 1.0194 Epoch: 17 Global Step: 173395 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:51:39,943-Speed 131.23 samples/sec Loss 1.1410 Epoch: 17 Global Step: 173400 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:51:59,737-Speed 129.34 samples/sec Loss 1.1112 Epoch: 17 Global Step: 173405 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:52:19,310-Speed 130.80 samples/sec Loss 1.0578 Epoch: 17 Global Step: 173410 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:52:39,062-Speed 129.61 samples/sec Loss 1.1137 Epoch: 17 Global Step: 173415 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:52:58,492-Speed 131.77 samples/sec Loss 1.1223 Epoch: 17 Global Step: 173420 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:53:18,107-Speed 130.51 samples/sec Loss 1.0463 Epoch: 17 Global Step: 173425 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:53:37,642-Speed 131.05 samples/sec Loss 1.1812 Epoch: 17 Global Step: 173430 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:53:57,194-Speed 130.95 samples/sec Loss 1.0807 Epoch: 17 Global Step: 173435 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:54:16,963-Speed 129.50 samples/sec Loss 1.0831 Epoch: 17 Global Step: 173440 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:54:36,588-Speed 130.45 samples/sec Loss 1.1854 Epoch: 17 Global Step: 173445 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:54:56,485-Speed 128.67 samples/sec Loss 1.0131 Epoch: 17 Global Step: 173450 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:55:16,206-Speed 129.82 samples/sec Loss 1.1360 Epoch: 17 Global Step: 173455 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:55:35,794-Speed 130.69 samples/sec Loss 0.9334 Epoch: 17 Global Step: 173460 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:55:55,319-Speed 131.12 samples/sec Loss 1.0927 Epoch: 17 Global Step: 173465 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:56:14,982-Speed 130.20 samples/sec Loss 1.0595 Epoch: 17 Global Step: 173470 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:56:34,601-Speed 130.49 samples/sec Loss 1.1345 Epoch: 17 Global Step: 173475 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:56:54,211-Speed 130.56 samples/sec Loss 1.1139 Epoch: 17 Global Step: 173480 Fp16 Grad Scale: 4096 Required: 89 hours Training: 2021-06-15 07:57:13,765-Speed 130.93 samples/sec Loss 1.1566 Epoch: 17 Global Step: 173485 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:57:33,332-Speed 130.84 samples/sec Loss 1.1566 Epoch: 17 Global Step: 173490 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:57:53,110-Speed 129.45 samples/sec Loss 1.1136 Epoch: 17 Global Step: 173495 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:58:12,868-Speed 129.58 samples/sec Loss 1.1359 Epoch: 17 Global Step: 173500 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:58:32,321-Speed 131.60 samples/sec Loss 1.0521 Epoch: 17 Global Step: 173505 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:58:51,837-Speed 131.19 samples/sec Loss 1.1015 Epoch: 17 Global Step: 173510 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:59:11,398-Speed 130.88 samples/sec Loss 1.1551 Epoch: 17 Global Step: 173515 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:59:30,959-Speed 130.88 samples/sec Loss 1.0958 Epoch: 17 Global Step: 173520 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 07:59:50,571-Speed 130.54 samples/sec Loss 1.0669 Epoch: 17 Global Step: 173525 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:00:09,994-Speed 131.81 samples/sec Loss 1.1418 Epoch: 17 Global Step: 173530 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:00:29,532-Speed 131.04 samples/sec Loss 1.0264 Epoch: 17 Global Step: 173535 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:00:49,326-Speed 129.34 samples/sec Loss 1.0414 Epoch: 17 Global Step: 173540 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:01:09,055-Speed 129.77 samples/sec Loss 1.1048 Epoch: 17 Global Step: 173545 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:01:28,532-Speed 131.44 samples/sec Loss 1.1417 Epoch: 17 Global Step: 173550 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:01:48,149-Speed 130.51 samples/sec Loss 1.0245 Epoch: 17 Global Step: 173555 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:02:07,769-Speed 130.49 samples/sec Loss 1.1531 Epoch: 17 Global Step: 173560 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:02:27,301-Speed 131.08 samples/sec Loss 1.1805 Epoch: 17 Global Step: 173565 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:02:46,882-Speed 130.74 samples/sec Loss 1.1414 Epoch: 17 Global Step: 173570 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:03:06,335-Speed 131.61 samples/sec Loss 1.0440 Epoch: 17 Global Step: 173575 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:03:25,948-Speed 130.54 samples/sec Loss 1.0088 Epoch: 17 Global Step: 173580 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:03:45,752-Speed 129.27 samples/sec Loss 0.9994 Epoch: 17 Global Step: 173585 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:04:05,388-Speed 130.38 samples/sec Loss 1.0660 Epoch: 17 Global Step: 173590 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:04:24,851-Speed 131.54 samples/sec Loss 1.1935 Epoch: 17 Global Step: 173595 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:04:44,475-Speed 130.46 samples/sec Loss 1.0117 Epoch: 17 Global Step: 173600 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:05:04,061-Speed 130.71 samples/sec Loss 1.1771 Epoch: 17 Global Step: 173605 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:05:23,918-Speed 128.93 samples/sec Loss 1.0686 Epoch: 17 Global Step: 173610 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:05:43,496-Speed 130.77 samples/sec Loss 1.1613 Epoch: 17 Global Step: 173615 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:06:03,034-Speed 131.03 samples/sec Loss 0.9897 Epoch: 17 Global Step: 173620 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:06:22,616-Speed 130.74 samples/sec Loss 1.0100 Epoch: 17 Global Step: 173625 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:06:42,393-Speed 129.45 samples/sec Loss 1.0889 Epoch: 17 Global Step: 173630 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:07:01,995-Speed 130.60 samples/sec Loss 1.0969 Epoch: 17 Global Step: 173635 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:07:21,533-Speed 131.03 samples/sec Loss 1.0917 Epoch: 17 Global Step: 173640 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:07:41,143-Speed 130.56 samples/sec Loss 1.0931 Epoch: 17 Global Step: 173645 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:08:00,800-Speed 130.24 samples/sec Loss 1.0770 Epoch: 17 Global Step: 173650 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:08:20,383-Speed 130.73 samples/sec Loss 0.9840 Epoch: 17 Global Step: 173655 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:08:39,804-Speed 131.82 samples/sec Loss 1.0821 Epoch: 17 Global Step: 173660 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:08:59,338-Speed 131.06 samples/sec Loss 1.0703 Epoch: 17 Global Step: 173665 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:09:19,002-Speed 130.20 samples/sec Loss 1.1228 Epoch: 17 Global Step: 173670 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:09:39,908-Speed 122.46 samples/sec Loss 1.0711 Epoch: 17 Global Step: 173675 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:09:59,350-Speed 131.68 samples/sec Loss 1.0338 Epoch: 17 Global Step: 173680 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:10:19,199-Speed 128.98 samples/sec Loss 1.0032 Epoch: 17 Global Step: 173685 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:10:38,753-Speed 130.92 samples/sec Loss 1.1611 Epoch: 17 Global Step: 173690 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:10:58,223-Speed 131.49 samples/sec Loss 1.1066 Epoch: 17 Global Step: 173695 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:11:17,826-Speed 130.60 samples/sec Loss 1.0903 Epoch: 17 Global Step: 173700 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:11:37,359-Speed 131.07 samples/sec Loss 1.0583 Epoch: 17 Global Step: 173705 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:11:56,968-Speed 130.56 samples/sec Loss 1.1619 Epoch: 17 Global Step: 173710 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:12:16,513-Speed 130.98 samples/sec Loss 1.0772 Epoch: 17 Global Step: 173715 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:12:36,310-Speed 129.32 samples/sec Loss 1.0970 Epoch: 17 Global Step: 173720 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:12:55,839-Speed 131.09 samples/sec Loss 1.0503 Epoch: 17 Global Step: 173725 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:13:15,539-Speed 129.96 samples/sec Loss 1.0208 Epoch: 17 Global Step: 173730 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:13:35,142-Speed 130.60 samples/sec Loss 1.1559 Epoch: 17 Global Step: 173735 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:13:54,689-Speed 130.98 samples/sec Loss 0.9894 Epoch: 17 Global Step: 173740 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:14:14,194-Speed 131.26 samples/sec Loss 1.0919 Epoch: 17 Global Step: 173745 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:14:33,766-Speed 130.81 samples/sec Loss 1.1180 Epoch: 17 Global Step: 173750 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:14:53,462-Speed 129.98 samples/sec Loss 1.2597 Epoch: 17 Global Step: 173755 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:15:12,897-Speed 131.73 samples/sec Loss 1.0709 Epoch: 17 Global Step: 173760 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:15:32,688-Speed 129.35 samples/sec Loss 0.9484 Epoch: 17 Global Step: 173765 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:15:52,231-Speed 131.00 samples/sec Loss 1.0848 Epoch: 17 Global Step: 173770 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:16:12,005-Speed 129.47 samples/sec Loss 1.1259 Epoch: 17 Global Step: 173775 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:16:31,440-Speed 131.73 samples/sec Loss 0.9311 Epoch: 17 Global Step: 173780 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:16:51,069-Speed 130.42 samples/sec Loss 1.1801 Epoch: 17 Global Step: 173785 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:17:10,620-Speed 130.95 samples/sec Loss 1.0450 Epoch: 17 Global Step: 173790 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:17:30,309-Speed 130.03 samples/sec Loss 1.1674 Epoch: 17 Global Step: 173795 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:17:49,860-Speed 130.94 samples/sec Loss 1.0554 Epoch: 17 Global Step: 173800 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:18:09,426-Speed 130.85 samples/sec Loss 1.1072 Epoch: 17 Global Step: 173805 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:18:29,219-Speed 129.35 samples/sec Loss 1.1063 Epoch: 17 Global Step: 173810 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:18:48,936-Speed 129.85 samples/sec Loss 1.1617 Epoch: 17 Global Step: 173815 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:19:08,671-Speed 129.73 samples/sec Loss 1.0895 Epoch: 17 Global Step: 173820 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:19:28,083-Speed 131.88 samples/sec Loss 1.1930 Epoch: 17 Global Step: 173825 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:19:47,676-Speed 130.66 samples/sec Loss 1.0226 Epoch: 17 Global Step: 173830 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:20:07,258-Speed 130.74 samples/sec Loss 1.1472 Epoch: 17 Global Step: 173835 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:20:26,781-Speed 131.13 samples/sec Loss 1.0089 Epoch: 17 Global Step: 173840 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:20:46,333-Speed 130.94 samples/sec Loss 1.1357 Epoch: 17 Global Step: 173845 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:21:05,778-Speed 131.66 samples/sec Loss 1.0569 Epoch: 17 Global Step: 173850 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:21:25,573-Speed 129.33 samples/sec Loss 1.1967 Epoch: 17 Global Step: 173855 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:21:45,447-Speed 128.82 samples/sec Loss 1.0784 Epoch: 17 Global Step: 173860 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:22:05,096-Speed 130.30 samples/sec Loss 1.1383 Epoch: 17 Global Step: 173865 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:22:24,575-Speed 131.43 samples/sec Loss 1.0844 Epoch: 17 Global Step: 173870 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:22:44,180-Speed 130.59 samples/sec Loss 1.1977 Epoch: 17 Global Step: 173875 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:23:03,897-Speed 129.84 samples/sec Loss 1.2101 Epoch: 17 Global Step: 173880 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:23:23,447-Speed 130.95 samples/sec Loss 1.0124 Epoch: 17 Global Step: 173885 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:23:43,011-Speed 130.86 samples/sec Loss 1.0958 Epoch: 17 Global Step: 173890 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:24:02,530-Speed 131.17 samples/sec Loss 1.2227 Epoch: 17 Global Step: 173895 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:24:22,344-Speed 129.21 samples/sec Loss 1.0399 Epoch: 17 Global Step: 173900 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:24:42,050-Speed 129.92 samples/sec Loss 1.0280 Epoch: 17 Global Step: 173905 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:25:01,546-Speed 131.32 samples/sec Loss 1.0095 Epoch: 17 Global Step: 173910 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:25:21,075-Speed 131.10 samples/sec Loss 1.1318 Epoch: 17 Global Step: 173915 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:25:40,653-Speed 130.76 samples/sec Loss 1.0501 Epoch: 17 Global Step: 173920 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:26:00,259-Speed 130.58 samples/sec Loss 1.2318 Epoch: 17 Global Step: 173925 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:26:19,852-Speed 130.67 samples/sec Loss 1.0575 Epoch: 17 Global Step: 173930 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:26:39,285-Speed 131.74 samples/sec Loss 1.1127 Epoch: 17 Global Step: 173935 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:26:58,789-Speed 131.27 samples/sec Loss 1.1202 Epoch: 17 Global Step: 173940 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:27:18,678-Speed 128.72 samples/sec Loss 1.1689 Epoch: 17 Global Step: 173945 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:27:38,377-Speed 129.96 samples/sec Loss 1.1684 Epoch: 17 Global Step: 173950 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:27:57,832-Speed 131.59 samples/sec Loss 1.1600 Epoch: 17 Global Step: 173955 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:28:17,429-Speed 130.64 samples/sec Loss 1.1078 Epoch: 17 Global Step: 173960 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:28:37,042-Speed 130.53 samples/sec Loss 1.1231 Epoch: 17 Global Step: 173965 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:28:56,590-Speed 130.97 samples/sec Loss 1.0855 Epoch: 17 Global Step: 173970 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:29:16,354-Speed 129.54 samples/sec Loss 1.0581 Epoch: 17 Global Step: 173975 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:29:35,901-Speed 130.98 samples/sec Loss 1.1100 Epoch: 17 Global Step: 173980 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:29:55,501-Speed 130.62 samples/sec Loss 1.1063 Epoch: 17 Global Step: 173985 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:30:15,685-Speed 126.84 samples/sec Loss 1.1370 Epoch: 17 Global Step: 173990 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:30:35,120-Speed 131.72 samples/sec Loss 1.0211 Epoch: 17 Global Step: 173995 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:30:54,663-Speed 131.00 samples/sec Loss 1.0195 Epoch: 17 Global Step: 174000 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:31:14,243-Speed 130.75 samples/sec Loss 1.0413 Epoch: 17 Global Step: 174005 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:31:33,815-Speed 130.81 samples/sec Loss 1.0616 Epoch: 17 Global Step: 174010 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:31:53,412-Speed 130.64 samples/sec Loss 1.0766 Epoch: 17 Global Step: 174015 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:32:12,865-Speed 131.61 samples/sec Loss 1.2338 Epoch: 17 Global Step: 174020 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:32:32,704-Speed 129.05 samples/sec Loss 0.9963 Epoch: 17 Global Step: 174025 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:32:52,355-Speed 130.28 samples/sec Loss 1.1386 Epoch: 17 Global Step: 174030 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:33:12,238-Speed 128.76 samples/sec Loss 1.1244 Epoch: 17 Global Step: 174035 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:33:31,661-Speed 131.81 samples/sec Loss 1.1089 Epoch: 17 Global Step: 174040 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:33:51,357-Speed 129.98 samples/sec Loss 1.0486 Epoch: 17 Global Step: 174045 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:34:10,893-Speed 131.05 samples/sec Loss 1.1200 Epoch: 17 Global Step: 174050 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:34:30,437-Speed 130.99 samples/sec Loss 1.0872 Epoch: 17 Global Step: 174055 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:34:50,005-Speed 130.84 samples/sec Loss 1.1183 Epoch: 17 Global Step: 174060 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:35:09,544-Speed 131.02 samples/sec Loss 1.1032 Epoch: 17 Global Step: 174065 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:35:29,233-Speed 130.03 samples/sec Loss 1.0562 Epoch: 17 Global Step: 174070 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:35:49,275-Speed 127.74 samples/sec Loss 1.0925 Epoch: 17 Global Step: 174075 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:36:09,198-Speed 128.50 samples/sec Loss 0.9866 Epoch: 17 Global Step: 174080 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:36:28,608-Speed 131.90 samples/sec Loss 1.0568 Epoch: 17 Global Step: 174085 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:36:48,280-Speed 130.14 samples/sec Loss 1.1962 Epoch: 17 Global Step: 174090 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:37:07,913-Speed 130.40 samples/sec Loss 1.1575 Epoch: 17 Global Step: 174095 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:37:27,427-Speed 131.20 samples/sec Loss 1.1778 Epoch: 17 Global Step: 174100 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:37:47,051-Speed 130.46 samples/sec Loss 1.0497 Epoch: 17 Global Step: 174105 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:38:06,678-Speed 130.44 samples/sec Loss 1.1161 Epoch: 17 Global Step: 174110 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:38:26,283-Speed 130.59 samples/sec Loss 1.0492 Epoch: 17 Global Step: 174115 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:38:45,856-Speed 130.80 samples/sec Loss 1.1194 Epoch: 17 Global Step: 174120 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:39:05,723-Speed 128.87 samples/sec Loss 1.1178 Epoch: 17 Global Step: 174125 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:39:25,146-Speed 131.81 samples/sec Loss 1.1211 Epoch: 17 Global Step: 174130 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:39:44,862-Speed 129.85 samples/sec Loss 1.0241 Epoch: 17 Global Step: 174135 Fp16 Grad Scale: 8192 Required: 89 hours Training: 2021-06-15 08:40:04,425-Speed 130.87 samples/sec Loss 1.2128 Epoch: 17 Global Step: 174140 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:40:24,056-Speed 130.41 samples/sec Loss 1.1419 Epoch: 17 Global Step: 174145 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:40:43,589-Speed 131.07 samples/sec Loss 1.0908 Epoch: 17 Global Step: 174150 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:41:03,127-Speed 131.04 samples/sec Loss 1.0603 Epoch: 17 Global Step: 174155 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:41:22,708-Speed 130.75 samples/sec Loss 1.2051 Epoch: 17 Global Step: 174160 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:41:42,198-Speed 131.36 samples/sec Loss 1.1715 Epoch: 17 Global Step: 174165 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:42:01,981-Speed 129.41 samples/sec Loss 1.1593 Epoch: 17 Global Step: 174170 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:42:21,741-Speed 129.56 samples/sec Loss 1.0867 Epoch: 17 Global Step: 174175 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:42:41,341-Speed 130.62 samples/sec Loss 0.9596 Epoch: 17 Global Step: 174180 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:43:00,781-Speed 131.69 samples/sec Loss 1.0244 Epoch: 17 Global Step: 174185 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:43:20,462-Speed 130.08 samples/sec Loss 0.9737 Epoch: 17 Global Step: 174190 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:43:39,985-Speed 131.14 samples/sec Loss 1.0734 Epoch: 17 Global Step: 174195 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:43:59,535-Speed 130.95 samples/sec Loss 1.0270 Epoch: 17 Global Step: 174200 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:44:19,128-Speed 130.66 samples/sec Loss 1.1628 Epoch: 17 Global Step: 174205 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:44:38,705-Speed 130.77 samples/sec Loss 1.1499 Epoch: 17 Global Step: 174210 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:44:58,505-Speed 129.31 samples/sec Loss 1.0260 Epoch: 17 Global Step: 174215 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:45:18,195-Speed 130.02 samples/sec Loss 1.1398 Epoch: 17 Global Step: 174220 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:45:37,730-Speed 131.05 samples/sec Loss 1.0038 Epoch: 17 Global Step: 174225 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:45:57,260-Speed 131.09 samples/sec Loss 1.0916 Epoch: 17 Global Step: 174230 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:46:16,868-Speed 130.57 samples/sec Loss 1.1419 Epoch: 17 Global Step: 174235 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:46:36,417-Speed 130.96 samples/sec Loss 1.0891 Epoch: 17 Global Step: 174240 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:46:56,011-Speed 130.66 samples/sec Loss 1.0698 Epoch: 17 Global Step: 174245 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:47:15,576-Speed 130.85 samples/sec Loss 1.1792 Epoch: 17 Global Step: 174250 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:47:35,234-Speed 130.23 samples/sec Loss 1.1584 Epoch: 17 Global Step: 174255 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:47:54,978-Speed 129.67 samples/sec Loss 0.9906 Epoch: 17 Global Step: 174260 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:48:14,715-Speed 129.71 samples/sec Loss 1.2775 Epoch: 17 Global Step: 174265 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:48:34,160-Speed 131.66 samples/sec Loss 1.0140 Epoch: 17 Global Step: 174270 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:48:53,676-Speed 131.18 samples/sec Loss 1.1311 Epoch: 17 Global Step: 174275 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:49:13,247-Speed 130.81 samples/sec Loss 1.0926 Epoch: 17 Global Step: 174280 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:49:32,775-Speed 131.10 samples/sec Loss 0.9912 Epoch: 17 Global Step: 174285 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:49:52,398-Speed 130.46 samples/sec Loss 1.0887 Epoch: 17 Global Step: 174290 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:50:11,906-Speed 131.24 samples/sec Loss 1.1067 Epoch: 17 Global Step: 174295 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:50:31,463-Speed 130.91 samples/sec Loss 0.8991 Epoch: 17 Global Step: 174300 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:50:51,285-Speed 129.16 samples/sec Loss 0.9943 Epoch: 17 Global Step: 174305 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:51:10,942-Speed 130.24 samples/sec Loss 1.0947 Epoch: 17 Global Step: 174310 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:51:30,400-Speed 131.57 samples/sec Loss 0.9225 Epoch: 17 Global Step: 174315 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:51:49,940-Speed 131.02 samples/sec Loss 0.9817 Epoch: 17 Global Step: 174320 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:52:09,487-Speed 130.98 samples/sec Loss 0.9652 Epoch: 17 Global Step: 174325 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:52:28,949-Speed 131.55 samples/sec Loss 1.1656 Epoch: 17 Global Step: 174330 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:52:48,557-Speed 130.57 samples/sec Loss 1.2181 Epoch: 17 Global Step: 174335 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:53:08,091-Speed 131.06 samples/sec Loss 1.2000 Epoch: 17 Global Step: 174340 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:53:27,633-Speed 131.00 samples/sec Loss 1.0790 Epoch: 17 Global Step: 174345 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:53:47,421-Speed 129.38 samples/sec Loss 0.9854 Epoch: 17 Global Step: 174350 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:54:07,646-Speed 126.58 samples/sec Loss 1.1154 Epoch: 17 Global Step: 174355 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:54:27,187-Speed 131.02 samples/sec Loss 1.0367 Epoch: 17 Global Step: 174360 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:54:46,793-Speed 130.58 samples/sec Loss 0.9508 Epoch: 17 Global Step: 174365 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:55:06,298-Speed 131.26 samples/sec Loss 1.1854 Epoch: 17 Global Step: 174370 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:55:26,109-Speed 129.23 samples/sec Loss 1.2749 Epoch: 17 Global Step: 174375 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:55:45,543-Speed 131.73 samples/sec Loss 1.0123 Epoch: 17 Global Step: 174380 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:56:05,846-Speed 126.10 samples/sec Loss 1.2686 Epoch: 17 Global Step: 174385 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:56:25,459-Speed 130.53 samples/sec Loss 1.1385 Epoch: 17 Global Step: 174390 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:56:45,291-Speed 129.09 samples/sec Loss 1.0540 Epoch: 17 Global Step: 174395 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:57:04,718-Speed 131.78 samples/sec Loss 1.1267 Epoch: 17 Global Step: 174400 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:57:24,324-Speed 130.58 samples/sec Loss 1.0507 Epoch: 17 Global Step: 174405 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:57:43,924-Speed 130.62 samples/sec Loss 1.1277 Epoch: 17 Global Step: 174410 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:58:03,344-Speed 131.83 samples/sec Loss 1.0022 Epoch: 17 Global Step: 174415 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:58:22,961-Speed 130.51 samples/sec Loss 0.9654 Epoch: 17 Global Step: 174420 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:58:42,488-Speed 131.11 samples/sec Loss 1.1391 Epoch: 17 Global Step: 174425 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:59:01,972-Speed 131.39 samples/sec Loss 1.1861 Epoch: 17 Global Step: 174430 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:59:21,555-Speed 130.73 samples/sec Loss 0.9924 Epoch: 17 Global Step: 174435 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 08:59:41,474-Speed 128.53 samples/sec Loss 1.0375 Epoch: 17 Global Step: 174440 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:00:00,911-Speed 131.71 samples/sec Loss 1.1427 Epoch: 17 Global Step: 174445 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:00:20,479-Speed 130.83 samples/sec Loss 1.1233 Epoch: 17 Global Step: 174450 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:00:40,049-Speed 130.82 samples/sec Loss 1.0546 Epoch: 17 Global Step: 174455 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:00:59,576-Speed 131.11 samples/sec Loss 1.0850 Epoch: 17 Global Step: 174460 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:01:19,183-Speed 130.57 samples/sec Loss 1.1762 Epoch: 17 Global Step: 174465 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:01:38,734-Speed 130.95 samples/sec Loss 1.1415 Epoch: 17 Global Step: 174470 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:01:58,274-Speed 131.02 samples/sec Loss 1.1831 Epoch: 17 Global Step: 174475 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:02:17,779-Speed 131.26 samples/sec Loss 1.0369 Epoch: 17 Global Step: 174480 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:02:37,653-Speed 128.82 samples/sec Loss 1.1105 Epoch: 17 Global Step: 174485 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:02:57,093-Speed 131.70 samples/sec Loss 1.1426 Epoch: 17 Global Step: 174490 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:03:16,734-Speed 130.35 samples/sec Loss 1.0104 Epoch: 17 Global Step: 174495 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:03:36,193-Speed 131.56 samples/sec Loss 1.0955 Epoch: 17 Global Step: 174500 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:03:55,666-Speed 131.47 samples/sec Loss 1.0284 Epoch: 17 Global Step: 174505 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:04:15,339-Speed 130.14 samples/sec Loss 1.0490 Epoch: 17 Global Step: 174510 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:04:34,779-Speed 131.69 samples/sec Loss 1.2307 Epoch: 17 Global Step: 174515 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:04:54,376-Speed 130.64 samples/sec Loss 1.0924 Epoch: 17 Global Step: 174520 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:05:13,942-Speed 130.84 samples/sec Loss 1.0703 Epoch: 17 Global Step: 174525 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:05:33,774-Speed 129.09 samples/sec Loss 1.2599 Epoch: 17 Global Step: 174530 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:05:53,209-Speed 131.73 samples/sec Loss 1.0456 Epoch: 17 Global Step: 174535 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:06:12,824-Speed 130.52 samples/sec Loss 1.0686 Epoch: 17 Global Step: 174540 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:06:32,365-Speed 131.01 samples/sec Loss 1.0826 Epoch: 17 Global Step: 174545 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:06:51,952-Speed 130.71 samples/sec Loss 1.0247 Epoch: 17 Global Step: 174550 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:07:11,480-Speed 131.10 samples/sec Loss 0.9365 Epoch: 17 Global Step: 174555 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:07:30,938-Speed 131.57 samples/sec Loss 1.0625 Epoch: 17 Global Step: 174560 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:07:50,523-Speed 130.72 samples/sec Loss 1.1050 Epoch: 17 Global Step: 174565 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:08:09,946-Speed 131.81 samples/sec Loss 1.2327 Epoch: 17 Global Step: 174570 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:08:29,738-Speed 129.35 samples/sec Loss 1.1502 Epoch: 17 Global Step: 174575 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:08:49,302-Speed 130.86 samples/sec Loss 1.0847 Epoch: 17 Global Step: 174580 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:09:08,805-Speed 131.27 samples/sec Loss 0.9947 Epoch: 17 Global Step: 174585 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:09:28,259-Speed 131.59 samples/sec Loss 1.0389 Epoch: 17 Global Step: 174590 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:09:47,919-Speed 130.23 samples/sec Loss 1.0769 Epoch: 17 Global Step: 174595 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:10:07,370-Speed 131.62 samples/sec Loss 1.0864 Epoch: 17 Global Step: 174600 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:10:27,004-Speed 130.39 samples/sec Loss 1.1979 Epoch: 17 Global Step: 174605 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:10:46,526-Speed 131.15 samples/sec Loss 1.0361 Epoch: 17 Global Step: 174610 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:11:06,128-Speed 130.61 samples/sec Loss 1.1855 Epoch: 17 Global Step: 174615 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:11:25,918-Speed 129.36 samples/sec Loss 1.1009 Epoch: 17 Global Step: 174620 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:11:45,516-Speed 130.64 samples/sec Loss 1.1396 Epoch: 17 Global Step: 174625 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:12:05,040-Speed 131.13 samples/sec Loss 1.1356 Epoch: 17 Global Step: 174630 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:12:24,650-Speed 130.55 samples/sec Loss 1.0376 Epoch: 17 Global Step: 174635 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:12:44,167-Speed 131.17 samples/sec Loss 1.0963 Epoch: 17 Global Step: 174640 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:13:03,594-Speed 131.79 samples/sec Loss 1.0805 Epoch: 17 Global Step: 174645 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:13:23,201-Speed 130.57 samples/sec Loss 0.9997 Epoch: 17 Global Step: 174650 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:13:42,756-Speed 130.92 samples/sec Loss 1.1506 Epoch: 17 Global Step: 174655 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:14:02,371-Speed 130.52 samples/sec Loss 1.0487 Epoch: 17 Global Step: 174660 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:14:22,149-Speed 129.44 samples/sec Loss 1.1765 Epoch: 17 Global Step: 174665 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:14:41,946-Speed 129.32 samples/sec Loss 1.1454 Epoch: 17 Global Step: 174670 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:15:01,474-Speed 131.10 samples/sec Loss 1.0709 Epoch: 17 Global Step: 174675 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:15:21,063-Speed 130.69 samples/sec Loss 1.0382 Epoch: 17 Global Step: 174680 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:15:40,720-Speed 130.24 samples/sec Loss 1.0522 Epoch: 17 Global Step: 174685 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:16:00,202-Speed 131.41 samples/sec Loss 1.1026 Epoch: 17 Global Step: 174690 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:16:19,812-Speed 130.55 samples/sec Loss 1.1496 Epoch: 17 Global Step: 174695 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:16:39,433-Speed 130.48 samples/sec Loss 1.1024 Epoch: 17 Global Step: 174700 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:16:59,600-Speed 126.95 samples/sec Loss 1.0285 Epoch: 17 Global Step: 174705 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:17:19,353-Speed 129.61 samples/sec Loss 1.1293 Epoch: 17 Global Step: 174710 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:17:38,985-Speed 130.40 samples/sec Loss 1.0442 Epoch: 17 Global Step: 174715 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:17:58,595-Speed 130.55 samples/sec Loss 1.1578 Epoch: 17 Global Step: 174720 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:18:18,195-Speed 130.62 samples/sec Loss 1.0053 Epoch: 17 Global Step: 174725 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:18:37,647-Speed 131.61 samples/sec Loss 1.0141 Epoch: 17 Global Step: 174730 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:18:57,269-Speed 130.48 samples/sec Loss 1.2456 Epoch: 17 Global Step: 174735 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:19:16,837-Speed 130.83 samples/sec Loss 1.0686 Epoch: 17 Global Step: 174740 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:19:36,446-Speed 130.56 samples/sec Loss 1.1970 Epoch: 17 Global Step: 174745 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:19:55,899-Speed 131.61 samples/sec Loss 1.0963 Epoch: 17 Global Step: 174750 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:20:15,823-Speed 128.49 samples/sec Loss 1.0246 Epoch: 17 Global Step: 174755 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:20:35,249-Speed 131.80 samples/sec Loss 1.1164 Epoch: 17 Global Step: 174760 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:20:54,960-Speed 129.88 samples/sec Loss 1.1117 Epoch: 17 Global Step: 174765 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:21:14,489-Speed 131.10 samples/sec Loss 1.0645 Epoch: 17 Global Step: 174770 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:21:33,957-Speed 131.50 samples/sec Loss 1.0301 Epoch: 17 Global Step: 174775 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:21:53,538-Speed 130.75 samples/sec Loss 0.9850 Epoch: 17 Global Step: 174780 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:22:13,335-Speed 129.32 samples/sec Loss 1.1209 Epoch: 17 Global Step: 174785 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:22:32,917-Speed 130.74 samples/sec Loss 1.0489 Epoch: 17 Global Step: 174790 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:22:52,441-Speed 131.13 samples/sec Loss 1.0415 Epoch: 17 Global Step: 174795 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:23:12,247-Speed 129.26 samples/sec Loss 0.9825 Epoch: 17 Global Step: 174800 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:23:31,807-Speed 130.89 samples/sec Loss 1.0288 Epoch: 17 Global Step: 174805 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:23:52,170-Speed 125.72 samples/sec Loss 1.0969 Epoch: 17 Global Step: 174810 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:24:11,727-Speed 130.90 samples/sec Loss 1.0895 Epoch: 17 Global Step: 174815 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:24:31,320-Speed 130.67 samples/sec Loss 1.0830 Epoch: 17 Global Step: 174820 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:24:50,850-Speed 131.09 samples/sec Loss 1.2106 Epoch: 17 Global Step: 174825 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:25:10,455-Speed 130.59 samples/sec Loss 1.0572 Epoch: 17 Global Step: 174830 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:25:30,312-Speed 128.93 samples/sec Loss 1.1111 Epoch: 17 Global Step: 174835 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:25:49,767-Speed 131.59 samples/sec Loss 1.0880 Epoch: 17 Global Step: 174840 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:26:10,600-Speed 122.89 samples/sec Loss 1.0071 Epoch: 17 Global Step: 174845 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:26:30,560-Speed 128.26 samples/sec Loss 1.0345 Epoch: 17 Global Step: 174850 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:26:50,098-Speed 131.03 samples/sec Loss 1.1419 Epoch: 17 Global Step: 174855 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:27:09,650-Speed 130.95 samples/sec Loss 1.1816 Epoch: 17 Global Step: 174860 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:27:29,214-Speed 130.85 samples/sec Loss 1.2446 Epoch: 17 Global Step: 174865 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:27:48,836-Speed 130.48 samples/sec Loss 1.0874 Epoch: 17 Global Step: 174870 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:28:08,450-Speed 130.53 samples/sec Loss 1.0478 Epoch: 17 Global Step: 174875 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:28:27,981-Speed 131.08 samples/sec Loss 0.9030 Epoch: 17 Global Step: 174880 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:28:47,526-Speed 130.99 samples/sec Loss 1.1275 Epoch: 17 Global Step: 174885 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:29:07,607-Speed 127.49 samples/sec Loss 1.0958 Epoch: 17 Global Step: 174890 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:29:27,301-Speed 130.00 samples/sec Loss 1.2465 Epoch: 17 Global Step: 174895 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:29:46,744-Speed 131.67 samples/sec Loss 1.0312 Epoch: 17 Global Step: 174900 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:30:06,303-Speed 130.90 samples/sec Loss 1.0496 Epoch: 17 Global Step: 174905 Fp16 Grad Scale: 8192 Required: 88 hours Training: 2021-06-15 09:30:25,476-Speed 133.53 samples/sec Loss 1.1077 Epoch: 17 Global Step: 174910 Fp16 Grad Scale: 4096 Required: 88 hours Training: 2021-06-15 09:30:44,950-Speed 131.46 samples/sec Loss 1.1896 Epoch: 17 Global Step: 174915 Fp16 Grad Scale: 4096 Required: 88 hours Training: 2021-06-15 09:31:04,514-Speed 130.86 samples/sec Loss 1.0721 Epoch: 17 Global Step: 174920 Fp16 Grad Scale: 4096 Required: 88 hours Training: 2021-06-15 09:31:24,085-Speed 130.81 samples/sec Loss 1.0221 Epoch: 17 Global Step: 174925 Fp16 Grad Scale: 4096 Required: 88 hours Training: 2021-06-15 09:31:43,687-Speed 130.61 samples/sec Loss 1.0439 Epoch: 17 Global Step: 174930 Fp16 Grad Scale: 4096 Required: 88 hours Training: 2021-06-15 09:32:03,299-Speed 130.54 samples/sec Loss 1.1568 Epoch: 17 Global Step: 174935 Fp16 Grad Scale: 4096 Required: 88 hours Training: 2021-06-15 09:32:22,861-Speed 130.88 samples/sec Loss 1.2070 Epoch: 17 Global Step: 174940 Fp16 Grad Scale: 4096 Required: 88 hours Training: 2021-06-15 09:32:42,391-Speed 131.09 samples/sec Loss 1.0698 Epoch: 17 Global Step: 174945 Fp16 Grad Scale: 4096 Required: 88 hours Training: 2021-06-15 09:33:01,928-Speed 131.04 samples/sec Loss 1.0645 Epoch: 17 Global Step: 174950 Fp16 Grad Scale: 4096 Required: 88 hours Training: 2021-06-15 09:33:21,484-Speed 130.91 samples/sec Loss 1.2382 Epoch: 17 Global Step: 174955 Fp16 Grad Scale: 4096 Required: 88 hours Training: 2021-06-15 09:33:41,450-Speed 128.22 samples/sec Loss 1.0022 Epoch: 17 Global Step: 174960 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:34:00,918-Speed 131.50 samples/sec Loss 1.0200 Epoch: 17 Global Step: 174965 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:34:20,586-Speed 130.17 samples/sec Loss 1.0933 Epoch: 17 Global Step: 174970 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:34:40,214-Speed 130.43 samples/sec Loss 1.1722 Epoch: 17 Global Step: 174975 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:34:59,681-Speed 131.51 samples/sec Loss 1.2042 Epoch: 17 Global Step: 174980 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:35:19,480-Speed 129.31 samples/sec Loss 1.0031 Epoch: 17 Global Step: 174985 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:35:39,044-Speed 130.86 samples/sec Loss 1.0885 Epoch: 17 Global Step: 174990 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:35:58,639-Speed 130.65 samples/sec Loss 1.1749 Epoch: 17 Global Step: 174995 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:36:18,236-Speed 130.64 samples/sec Loss 1.0936 Epoch: 17 Global Step: 175000 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:36:38,237-Speed 128.00 samples/sec Loss 1.0577 Epoch: 17 Global Step: 175005 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:36:57,705-Speed 131.51 samples/sec Loss 1.0528 Epoch: 17 Global Step: 175010 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:37:17,395-Speed 130.02 samples/sec Loss 1.1706 Epoch: 17 Global Step: 175015 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:37:37,060-Speed 130.19 samples/sec Loss 1.1697 Epoch: 17 Global Step: 175020 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:37:56,658-Speed 130.63 samples/sec Loss 1.0106 Epoch: 17 Global Step: 175025 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:38:16,145-Speed 131.38 samples/sec Loss 0.9531 Epoch: 17 Global Step: 175030 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:38:35,813-Speed 130.17 samples/sec Loss 1.1567 Epoch: 17 Global Step: 175035 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:38:55,431-Speed 130.50 samples/sec Loss 1.0737 Epoch: 17 Global Step: 175040 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:39:14,951-Speed 131.16 samples/sec Loss 1.1607 Epoch: 17 Global Step: 175045 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:39:34,892-Speed 128.39 samples/sec Loss 1.0316 Epoch: 17 Global Step: 175050 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:39:54,540-Speed 130.30 samples/sec Loss 1.0909 Epoch: 17 Global Step: 175055 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:40:14,128-Speed 130.70 samples/sec Loss 1.1834 Epoch: 17 Global Step: 175060 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:40:33,613-Speed 131.39 samples/sec Loss 1.1498 Epoch: 17 Global Step: 175065 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:40:53,214-Speed 130.62 samples/sec Loss 1.0524 Epoch: 17 Global Step: 175070 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:41:12,746-Speed 131.07 samples/sec Loss 1.1900 Epoch: 17 Global Step: 175075 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:41:32,285-Speed 131.03 samples/sec Loss 1.0633 Epoch: 17 Global Step: 175080 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:41:51,884-Speed 130.63 samples/sec Loss 1.0929 Epoch: 17 Global Step: 175085 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:42:11,353-Speed 131.50 samples/sec Loss 1.1267 Epoch: 17 Global Step: 175090 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:42:30,472-Speed 133.91 samples/sec Loss 1.0611 Epoch: 17 Global Step: 175095 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:42:50,338-Speed 128.87 samples/sec Loss 1.0430 Epoch: 17 Global Step: 175100 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:43:10,148-Speed 129.23 samples/sec Loss 1.1549 Epoch: 17 Global Step: 175105 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:43:29,762-Speed 130.53 samples/sec Loss 0.9679 Epoch: 17 Global Step: 175110 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:43:49,219-Speed 131.57 samples/sec Loss 1.0213 Epoch: 17 Global Step: 175115 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:44:08,741-Speed 131.14 samples/sec Loss 1.0267 Epoch: 17 Global Step: 175120 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:44:28,341-Speed 130.62 samples/sec Loss 1.0147 Epoch: 17 Global Step: 175125 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:44:47,887-Speed 130.98 samples/sec Loss 1.1145 Epoch: 17 Global Step: 175130 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:45:07,482-Speed 130.65 samples/sec Loss 1.1650 Epoch: 17 Global Step: 175135 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:45:27,188-Speed 129.92 samples/sec Loss 1.0037 Epoch: 17 Global Step: 175140 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:45:46,818-Speed 130.42 samples/sec Loss 1.0827 Epoch: 17 Global Step: 175145 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:46:06,423-Speed 130.59 samples/sec Loss 1.0833 Epoch: 17 Global Step: 175150 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:46:26,065-Speed 130.34 samples/sec Loss 1.1658 Epoch: 17 Global Step: 175155 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:46:45,596-Speed 131.09 samples/sec Loss 1.1961 Epoch: 17 Global Step: 175160 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:47:05,121-Speed 131.12 samples/sec Loss 1.0118 Epoch: 17 Global Step: 175165 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:47:24,720-Speed 130.63 samples/sec Loss 1.0906 Epoch: 17 Global Step: 175170 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:47:44,255-Speed 131.05 samples/sec Loss 1.0112 Epoch: 17 Global Step: 175175 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:48:03,793-Speed 131.04 samples/sec Loss 1.0228 Epoch: 17 Global Step: 175180 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:48:23,577-Speed 129.40 samples/sec Loss 1.1628 Epoch: 17 Global Step: 175185 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:48:43,204-Speed 130.44 samples/sec Loss 1.0834 Epoch: 17 Global Step: 175190 Fp16 Grad Scale: 4096 Required: 87 hours Training: 2021-06-15 09:49:02,703-Speed 131.29 samples/sec Loss 1.1600 Epoch: 17 Global Step: 175195 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:49:22,308-Speed 130.59 samples/sec Loss 1.0903 Epoch: 17 Global Step: 175200 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:49:41,996-Speed 130.04 samples/sec Loss 1.0682 Epoch: 17 Global Step: 175205 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:50:01,610-Speed 130.52 samples/sec Loss 1.0257 Epoch: 17 Global Step: 175210 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:50:21,206-Speed 130.65 samples/sec Loss 1.0647 Epoch: 17 Global Step: 175215 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:50:40,736-Speed 131.09 samples/sec Loss 1.0763 Epoch: 17 Global Step: 175220 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:51:00,288-Speed 130.94 samples/sec Loss 1.1118 Epoch: 17 Global Step: 175225 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:51:20,181-Speed 128.70 samples/sec Loss 1.0074 Epoch: 17 Global Step: 175230 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:51:39,917-Speed 129.71 samples/sec Loss 1.1086 Epoch: 17 Global Step: 175235 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:51:59,580-Speed 130.21 samples/sec Loss 1.3099 Epoch: 17 Global Step: 175240 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:52:19,366-Speed 129.39 samples/sec Loss 1.0497 Epoch: 17 Global Step: 175245 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:52:38,904-Speed 131.03 samples/sec Loss 1.1022 Epoch: 17 Global Step: 175250 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:52:58,521-Speed 130.51 samples/sec Loss 1.0359 Epoch: 17 Global Step: 175255 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:53:18,069-Speed 130.97 samples/sec Loss 1.0030 Epoch: 17 Global Step: 175260 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:53:37,672-Speed 130.60 samples/sec Loss 1.1016 Epoch: 17 Global Step: 175265 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:53:57,209-Speed 131.04 samples/sec Loss 1.0707 Epoch: 17 Global Step: 175270 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:54:17,100-Speed 128.71 samples/sec Loss 1.2002 Epoch: 17 Global Step: 175275 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:54:36,563-Speed 131.54 samples/sec Loss 1.2049 Epoch: 17 Global Step: 175280 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:54:56,239-Speed 130.11 samples/sec Loss 1.1004 Epoch: 17 Global Step: 175285 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:55:15,784-Speed 130.99 samples/sec Loss 1.1648 Epoch: 17 Global Step: 175290 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:55:35,316-Speed 131.08 samples/sec Loss 1.2234 Epoch: 17 Global Step: 175295 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:55:54,905-Speed 130.70 samples/sec Loss 1.0906 Epoch: 17 Global Step: 175300 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:56:14,374-Speed 131.49 samples/sec Loss 1.2484 Epoch: 17 Global Step: 175305 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:56:34,083-Speed 129.90 samples/sec Loss 1.0479 Epoch: 17 Global Step: 175310 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:56:53,605-Speed 131.14 samples/sec Loss 1.0944 Epoch: 17 Global Step: 175315 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:57:13,479-Speed 128.82 samples/sec Loss 1.0670 Epoch: 17 Global Step: 175320 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:57:32,965-Speed 131.38 samples/sec Loss 1.0890 Epoch: 17 Global Step: 175325 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:57:52,673-Speed 129.90 samples/sec Loss 1.1492 Epoch: 17 Global Step: 175330 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:58:12,190-Speed 131.17 samples/sec Loss 1.1209 Epoch: 17 Global Step: 175335 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:58:31,790-Speed 130.62 samples/sec Loss 1.0350 Epoch: 17 Global Step: 175340 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:58:51,347-Speed 130.91 samples/sec Loss 1.0599 Epoch: 17 Global Step: 175345 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:59:10,851-Speed 131.26 samples/sec Loss 1.0590 Epoch: 17 Global Step: 175350 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:59:30,447-Speed 130.64 samples/sec Loss 1.0349 Epoch: 17 Global Step: 175355 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 09:59:50,009-Speed 130.88 samples/sec Loss 1.0292 Epoch: 17 Global Step: 175360 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:00:09,917-Speed 128.60 samples/sec Loss 1.0997 Epoch: 17 Global Step: 175365 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:00:29,380-Speed 131.54 samples/sec Loss 1.1956 Epoch: 17 Global Step: 175370 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:00:49,007-Speed 130.44 samples/sec Loss 1.0210 Epoch: 17 Global Step: 175375 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:01:08,496-Speed 131.36 samples/sec Loss 1.0756 Epoch: 17 Global Step: 175380 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:01:28,065-Speed 130.83 samples/sec Loss 1.1050 Epoch: 17 Global Step: 175385 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:01:47,657-Speed 130.67 samples/sec Loss 1.1233 Epoch: 17 Global Step: 175390 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:02:07,190-Speed 131.07 samples/sec Loss 1.1615 Epoch: 17 Global Step: 175395 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:02:27,130-Speed 128.40 samples/sec Loss 1.2095 Epoch: 17 Global Step: 175400 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:02:46,606-Speed 131.45 samples/sec Loss 1.0514 Epoch: 17 Global Step: 175405 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:03:06,400-Speed 129.34 samples/sec Loss 1.0635 Epoch: 17 Global Step: 175410 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:03:25,966-Speed 130.85 samples/sec Loss 1.0985 Epoch: 17 Global Step: 175415 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:03:45,620-Speed 130.26 samples/sec Loss 1.1156 Epoch: 17 Global Step: 175420 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:04:05,277-Speed 130.24 samples/sec Loss 1.1390 Epoch: 17 Global Step: 175425 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:04:24,873-Speed 130.65 samples/sec Loss 1.1458 Epoch: 17 Global Step: 175430 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:04:44,403-Speed 131.09 samples/sec Loss 1.1656 Epoch: 17 Global Step: 175435 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:05:04,153-Speed 129.63 samples/sec Loss 1.0841 Epoch: 17 Global Step: 175440 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:05:23,736-Speed 130.74 samples/sec Loss 1.0470 Epoch: 17 Global Step: 175445 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:05:43,334-Speed 130.63 samples/sec Loss 1.0918 Epoch: 17 Global Step: 175450 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:06:03,131-Speed 129.32 samples/sec Loss 1.0593 Epoch: 17 Global Step: 175455 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:06:22,749-Speed 130.50 samples/sec Loss 1.1965 Epoch: 17 Global Step: 175460 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:06:42,315-Speed 130.85 samples/sec Loss 1.2500 Epoch: 17 Global Step: 175465 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:07:02,058-Speed 129.68 samples/sec Loss 1.1376 Epoch: 17 Global Step: 175470 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:07:21,636-Speed 130.76 samples/sec Loss 1.1204 Epoch: 17 Global Step: 175475 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:07:41,185-Speed 130.96 samples/sec Loss 1.0141 Epoch: 17 Global Step: 175480 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:08:00,794-Speed 130.56 samples/sec Loss 1.1099 Epoch: 17 Global Step: 175485 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:08:20,262-Speed 131.51 samples/sec Loss 1.0952 Epoch: 17 Global Step: 175490 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:08:39,823-Speed 130.88 samples/sec Loss 0.9821 Epoch: 17 Global Step: 175495 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:08:59,604-Speed 129.42 samples/sec Loss 1.0441 Epoch: 17 Global Step: 175500 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:09:19,577-Speed 128.18 samples/sec Loss 1.1080 Epoch: 17 Global Step: 175505 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:09:39,091-Speed 131.20 samples/sec Loss 1.2792 Epoch: 17 Global Step: 175510 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:09:58,628-Speed 131.04 samples/sec Loss 1.0154 Epoch: 17 Global Step: 175515 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:10:18,165-Speed 131.04 samples/sec Loss 1.1173 Epoch: 17 Global Step: 175520 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:10:37,724-Speed 130.89 samples/sec Loss 1.0122 Epoch: 17 Global Step: 175525 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:10:57,308-Speed 130.73 samples/sec Loss 0.9785 Epoch: 17 Global Step: 175530 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:11:16,905-Speed 130.64 samples/sec Loss 0.9709 Epoch: 17 Global Step: 175535 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:11:36,515-Speed 130.55 samples/sec Loss 1.0970 Epoch: 17 Global Step: 175540 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:11:56,408-Speed 128.69 samples/sec Loss 1.1202 Epoch: 17 Global Step: 175545 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:12:15,849-Speed 131.69 samples/sec Loss 1.0086 Epoch: 17 Global Step: 175550 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:12:35,543-Speed 129.99 samples/sec Loss 1.0436 Epoch: 17 Global Step: 175555 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:12:55,136-Speed 130.67 samples/sec Loss 0.9885 Epoch: 17 Global Step: 175560 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:13:14,671-Speed 131.05 samples/sec Loss 1.2043 Epoch: 17 Global Step: 175565 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:13:34,267-Speed 130.65 samples/sec Loss 1.0650 Epoch: 17 Global Step: 175570 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:13:53,823-Speed 130.92 samples/sec Loss 1.0409 Epoch: 17 Global Step: 175575 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:14:13,426-Speed 130.60 samples/sec Loss 1.2118 Epoch: 17 Global Step: 175580 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:14:32,993-Speed 130.84 samples/sec Loss 0.9758 Epoch: 17 Global Step: 175585 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:14:52,906-Speed 128.57 samples/sec Loss 1.0939 Epoch: 17 Global Step: 175590 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:15:12,310-Speed 131.93 samples/sec Loss 1.0244 Epoch: 17 Global Step: 175595 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:15:31,991-Speed 130.09 samples/sec Loss 1.1656 Epoch: 17 Global Step: 175600 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:15:51,755-Speed 129.54 samples/sec Loss 1.1180 Epoch: 17 Global Step: 175605 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:16:11,263-Speed 131.23 samples/sec Loss 1.0167 Epoch: 17 Global Step: 175610 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:16:30,866-Speed 130.60 samples/sec Loss 1.1061 Epoch: 17 Global Step: 175615 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:16:50,338-Speed 131.48 samples/sec Loss 1.0953 Epoch: 17 Global Step: 175620 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:17:09,933-Speed 130.66 samples/sec Loss 1.2014 Epoch: 17 Global Step: 175625 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:17:29,452-Speed 131.16 samples/sec Loss 1.1165 Epoch: 17 Global Step: 175630 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:17:50,010-Speed 124.53 samples/sec Loss 1.0736 Epoch: 17 Global Step: 175635 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:18:09,435-Speed 131.80 samples/sec Loss 1.1040 Epoch: 17 Global Step: 175640 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:18:29,196-Speed 129.56 samples/sec Loss 1.0362 Epoch: 17 Global Step: 175645 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:18:48,744-Speed 130.97 samples/sec Loss 1.2093 Epoch: 17 Global Step: 175650 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:19:08,288-Speed 130.99 samples/sec Loss 1.0084 Epoch: 17 Global Step: 175655 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:19:27,841-Speed 130.93 samples/sec Loss 1.1312 Epoch: 17 Global Step: 175660 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:19:47,455-Speed 130.53 samples/sec Loss 1.0358 Epoch: 17 Global Step: 175665 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:20:07,151-Speed 129.98 samples/sec Loss 1.1479 Epoch: 17 Global Step: 175670 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:20:26,707-Speed 130.91 samples/sec Loss 1.0648 Epoch: 17 Global Step: 175675 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:20:46,505-Speed 129.31 samples/sec Loss 1.1394 Epoch: 17 Global Step: 175680 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:21:06,063-Speed 130.90 samples/sec Loss 1.0917 Epoch: 17 Global Step: 175685 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:21:25,765-Speed 129.94 samples/sec Loss 1.1750 Epoch: 17 Global Step: 175690 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:21:45,833-Speed 127.57 samples/sec Loss 1.0651 Epoch: 17 Global Step: 175695 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:22:05,425-Speed 130.68 samples/sec Loss 1.0060 Epoch: 17 Global Step: 175700 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:22:25,026-Speed 130.62 samples/sec Loss 1.1373 Epoch: 17 Global Step: 175705 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:22:44,634-Speed 130.57 samples/sec Loss 1.1716 Epoch: 17 Global Step: 175710 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:23:04,302-Speed 130.17 samples/sec Loss 1.1284 Epoch: 17 Global Step: 175715 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:23:23,932-Speed 130.42 samples/sec Loss 1.0980 Epoch: 17 Global Step: 175720 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:23:44,037-Speed 127.34 samples/sec Loss 1.3285 Epoch: 17 Global Step: 175725 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:24:03,564-Speed 131.11 samples/sec Loss 1.1170 Epoch: 17 Global Step: 175730 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:24:23,209-Speed 130.32 samples/sec Loss 1.1583 Epoch: 17 Global Step: 175735 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:24:42,702-Speed 131.34 samples/sec Loss 1.1102 Epoch: 17 Global Step: 175740 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:25:02,311-Speed 130.56 samples/sec Loss 0.9766 Epoch: 17 Global Step: 175745 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:25:21,899-Speed 130.71 samples/sec Loss 1.1136 Epoch: 17 Global Step: 175750 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:25:41,436-Speed 131.04 samples/sec Loss 1.1732 Epoch: 17 Global Step: 175755 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:26:01,013-Speed 130.77 samples/sec Loss 1.0745 Epoch: 17 Global Step: 175760 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:26:20,448-Speed 131.73 samples/sec Loss 0.9622 Epoch: 17 Global Step: 175765 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:26:40,362-Speed 128.56 samples/sec Loss 1.0310 Epoch: 17 Global Step: 175770 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:27:00,095-Speed 129.74 samples/sec Loss 1.1774 Epoch: 17 Global Step: 175775 Fp16 Grad Scale: 8192 Required: 87 hours Training: 2021-06-15 10:27:19,635-Speed 131.02 samples/sec Loss 1.0887 Epoch: 17 Global Step: 175780 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:27:39,064-Speed 131.77 samples/sec Loss 1.1173 Epoch: 17 Global Step: 175785 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:27:58,660-Speed 130.64 samples/sec Loss 1.0712 Epoch: 17 Global Step: 175790 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:28:18,266-Speed 130.58 samples/sec Loss 1.0745 Epoch: 17 Global Step: 175795 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:28:37,830-Speed 130.86 samples/sec Loss 1.0217 Epoch: 17 Global Step: 175800 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:28:57,360-Speed 131.09 samples/sec Loss 1.0851 Epoch: 17 Global Step: 175805 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:29:17,355-Speed 128.04 samples/sec Loss 1.1775 Epoch: 17 Global Step: 175810 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:29:37,160-Speed 129.27 samples/sec Loss 1.1696 Epoch: 17 Global Step: 175815 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:29:56,937-Speed 129.45 samples/sec Loss 1.0918 Epoch: 17 Global Step: 175820 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:30:16,557-Speed 130.49 samples/sec Loss 1.1455 Epoch: 17 Global Step: 175825 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:30:36,258-Speed 129.95 samples/sec Loss 1.1369 Epoch: 17 Global Step: 175830 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:30:55,716-Speed 131.57 samples/sec Loss 1.0544 Epoch: 17 Global Step: 175835 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:31:15,284-Speed 130.83 samples/sec Loss 1.0677 Epoch: 17 Global Step: 175840 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:31:34,876-Speed 130.68 samples/sec Loss 1.2025 Epoch: 17 Global Step: 175845 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:31:54,508-Speed 130.40 samples/sec Loss 1.0647 Epoch: 17 Global Step: 175850 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:32:14,114-Speed 130.58 samples/sec Loss 1.1077 Epoch: 17 Global Step: 175855 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:32:33,978-Speed 128.89 samples/sec Loss 1.0333 Epoch: 17 Global Step: 175860 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:32:53,626-Speed 130.30 samples/sec Loss 1.1459 Epoch: 17 Global Step: 175865 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:33:12,610-Speed 134.86 samples/sec Loss 1.1457 Epoch: 17 Global Step: 175870 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:33:32,219-Speed 130.56 samples/sec Loss 1.0718 Epoch: 17 Global Step: 175875 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:33:51,809-Speed 130.68 samples/sec Loss 0.9844 Epoch: 17 Global Step: 175880 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:34:11,401-Speed 130.68 samples/sec Loss 1.0043 Epoch: 17 Global Step: 175885 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:34:30,980-Speed 130.76 samples/sec Loss 1.2193 Epoch: 17 Global Step: 175890 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:34:50,404-Speed 131.80 samples/sec Loss 1.0332 Epoch: 17 Global Step: 175895 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:35:10,004-Speed 130.62 samples/sec Loss 1.1385 Epoch: 17 Global Step: 175900 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:35:29,855-Speed 128.97 samples/sec Loss 1.0424 Epoch: 17 Global Step: 175905 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:35:49,485-Speed 130.42 samples/sec Loss 1.1245 Epoch: 17 Global Step: 175910 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:36:09,023-Speed 131.03 samples/sec Loss 1.0514 Epoch: 17 Global Step: 175915 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:36:28,608-Speed 130.72 samples/sec Loss 1.1528 Epoch: 17 Global Step: 175920 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:36:48,231-Speed 130.47 samples/sec Loss 1.1111 Epoch: 17 Global Step: 175925 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:37:07,830-Speed 130.64 samples/sec Loss 1.1042 Epoch: 17 Global Step: 175930 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:37:27,263-Speed 131.74 samples/sec Loss 1.2009 Epoch: 17 Global Step: 175935 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:37:46,787-Speed 131.13 samples/sec Loss 1.1406 Epoch: 17 Global Step: 175940 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:38:06,332-Speed 130.99 samples/sec Loss 1.0571 Epoch: 17 Global Step: 175945 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:38:26,323-Speed 128.07 samples/sec Loss 0.9673 Epoch: 17 Global Step: 175950 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:38:45,746-Speed 131.81 samples/sec Loss 1.0563 Epoch: 17 Global Step: 175955 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:39:05,374-Speed 130.43 samples/sec Loss 1.0562 Epoch: 17 Global Step: 175960 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:39:24,973-Speed 130.63 samples/sec Loss 1.1358 Epoch: 17 Global Step: 175965 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:39:44,531-Speed 130.90 samples/sec Loss 0.9676 Epoch: 17 Global Step: 175970 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:40:04,094-Speed 130.87 samples/sec Loss 1.1468 Epoch: 17 Global Step: 175975 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:40:23,532-Speed 131.70 samples/sec Loss 1.1733 Epoch: 17 Global Step: 175980 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:40:43,136-Speed 130.60 samples/sec Loss 1.0514 Epoch: 17 Global Step: 175985 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:41:02,758-Speed 130.47 samples/sec Loss 0.9935 Epoch: 17 Global Step: 175990 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:41:22,654-Speed 128.68 samples/sec Loss 1.1678 Epoch: 17 Global Step: 175995 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:41:42,136-Speed 131.41 samples/sec Loss 1.1497 Epoch: 17 Global Step: 176000 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:42:01,713-Speed 130.77 samples/sec Loss 1.0853 Epoch: 17 Global Step: 176005 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:42:21,330-Speed 130.51 samples/sec Loss 1.0712 Epoch: 17 Global Step: 176010 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:42:40,917-Speed 130.70 samples/sec Loss 1.0486 Epoch: 17 Global Step: 176015 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:43:00,355-Speed 131.72 samples/sec Loss 0.9519 Epoch: 17 Global Step: 176020 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:43:19,785-Speed 131.76 samples/sec Loss 0.9239 Epoch: 17 Global Step: 176025 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:43:39,393-Speed 130.57 samples/sec Loss 1.0347 Epoch: 17 Global Step: 176030 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:43:59,012-Speed 130.49 samples/sec Loss 1.0641 Epoch: 17 Global Step: 176035 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:44:18,838-Speed 129.13 samples/sec Loss 1.0028 Epoch: 17 Global Step: 176040 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:44:38,295-Speed 131.58 samples/sec Loss 1.1822 Epoch: 17 Global Step: 176045 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:44:57,873-Speed 130.77 samples/sec Loss 1.1329 Epoch: 17 Global Step: 176050 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:45:17,678-Speed 129.27 samples/sec Loss 1.0673 Epoch: 17 Global Step: 176055 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:45:37,384-Speed 129.92 samples/sec Loss 1.0025 Epoch: 17 Global Step: 176060 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:45:56,823-Speed 131.71 samples/sec Loss 1.0412 Epoch: 17 Global Step: 176065 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:46:16,327-Speed 131.26 samples/sec Loss 1.0252 Epoch: 17 Global Step: 176070 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:46:35,928-Speed 130.62 samples/sec Loss 1.1431 Epoch: 17 Global Step: 176075 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:46:55,461-Speed 131.07 samples/sec Loss 1.0175 Epoch: 17 Global Step: 176080 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:47:15,298-Speed 129.06 samples/sec Loss 1.1906 Epoch: 17 Global Step: 176085 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:47:34,916-Speed 130.50 samples/sec Loss 1.1380 Epoch: 17 Global Step: 176090 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:47:54,531-Speed 130.52 samples/sec Loss 1.1287 Epoch: 17 Global Step: 176095 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:48:14,085-Speed 130.92 samples/sec Loss 1.0137 Epoch: 17 Global Step: 176100 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:48:33,660-Speed 130.79 samples/sec Loss 1.0489 Epoch: 17 Global Step: 176105 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:48:53,103-Speed 131.68 samples/sec Loss 1.0953 Epoch: 17 Global Step: 176110 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:49:12,680-Speed 130.78 samples/sec Loss 1.1010 Epoch: 17 Global Step: 176115 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:49:32,237-Speed 130.90 samples/sec Loss 1.1697 Epoch: 17 Global Step: 176120 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:49:51,769-Speed 131.07 samples/sec Loss 1.1059 Epoch: 17 Global Step: 176125 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:50:11,530-Speed 129.56 samples/sec Loss 1.1144 Epoch: 17 Global Step: 176130 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:50:31,135-Speed 130.59 samples/sec Loss 0.9402 Epoch: 17 Global Step: 176135 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:50:50,662-Speed 131.11 samples/sec Loss 1.0267 Epoch: 17 Global Step: 176140 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:51:10,205-Speed 131.00 samples/sec Loss 0.9788 Epoch: 17 Global Step: 176145 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:51:29,791-Speed 130.71 samples/sec Loss 1.0232 Epoch: 17 Global Step: 176150 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:51:49,249-Speed 131.58 samples/sec Loss 1.0656 Epoch: 17 Global Step: 176155 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:52:08,860-Speed 130.55 samples/sec Loss 1.0998 Epoch: 17 Global Step: 176160 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:52:28,792-Speed 128.44 samples/sec Loss 1.1310 Epoch: 17 Global Step: 176165 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:52:48,270-Speed 131.44 samples/sec Loss 1.0493 Epoch: 17 Global Step: 176170 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:53:08,056-Speed 129.40 samples/sec Loss 0.9960 Epoch: 17 Global Step: 176175 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:53:27,771-Speed 129.86 samples/sec Loss 1.1456 Epoch: 17 Global Step: 176180 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:53:47,301-Speed 131.09 samples/sec Loss 0.9960 Epoch: 17 Global Step: 176185 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:54:06,894-Speed 130.66 samples/sec Loss 1.1731 Epoch: 17 Global Step: 176190 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:54:26,328-Speed 131.73 samples/sec Loss 1.1899 Epoch: 17 Global Step: 176195 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:54:45,934-Speed 130.58 samples/sec Loss 1.0438 Epoch: 17 Global Step: 176200 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 10:55:05,092-Speed 133.63 samples/sec Loss 1.1333 Epoch: 17 Global Step: 176205 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:55:24,630-Speed 131.04 samples/sec Loss 1.1002 Epoch: 17 Global Step: 176210 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:55:44,211-Speed 130.74 samples/sec Loss 1.1871 Epoch: 17 Global Step: 176215 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:56:03,655-Speed 131.67 samples/sec Loss 1.0502 Epoch: 17 Global Step: 176220 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:56:23,368-Speed 129.87 samples/sec Loss 1.0944 Epoch: 17 Global Step: 176225 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:56:43,141-Speed 129.48 samples/sec Loss 1.1608 Epoch: 17 Global Step: 176230 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:57:02,758-Speed 130.50 samples/sec Loss 1.2183 Epoch: 17 Global Step: 176235 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:57:22,214-Speed 131.58 samples/sec Loss 0.9797 Epoch: 17 Global Step: 176240 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:57:41,842-Speed 130.44 samples/sec Loss 0.9760 Epoch: 17 Global Step: 176245 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:58:01,377-Speed 131.06 samples/sec Loss 0.9562 Epoch: 17 Global Step: 176250 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:58:21,026-Speed 130.30 samples/sec Loss 0.8847 Epoch: 17 Global Step: 176255 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:58:40,673-Speed 130.30 samples/sec Loss 1.1915 Epoch: 17 Global Step: 176260 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:59:00,283-Speed 130.55 samples/sec Loss 1.1357 Epoch: 17 Global Step: 176265 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:59:19,877-Speed 130.66 samples/sec Loss 1.1854 Epoch: 17 Global Step: 176270 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:59:39,657-Speed 129.43 samples/sec Loss 1.1425 Epoch: 17 Global Step: 176275 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 10:59:59,226-Speed 130.83 samples/sec Loss 1.0699 Epoch: 17 Global Step: 176280 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 11:00:18,853-Speed 130.43 samples/sec Loss 1.0429 Epoch: 17 Global Step: 176285 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 11:00:38,478-Speed 130.45 samples/sec Loss 1.0981 Epoch: 17 Global Step: 176290 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 11:00:58,010-Speed 131.07 samples/sec Loss 0.9974 Epoch: 17 Global Step: 176295 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 11:01:17,501-Speed 131.35 samples/sec Loss 1.1340 Epoch: 17 Global Step: 176300 Fp16 Grad Scale: 4096 Required: 86 hours Training: 2021-06-15 11:01:37,116-Speed 130.52 samples/sec Loss 1.1035 Epoch: 17 Global Step: 176305 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:01:56,712-Speed 130.65 samples/sec Loss 1.1239 Epoch: 17 Global Step: 176310 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:02:16,292-Speed 130.75 samples/sec Loss 1.1888 Epoch: 17 Global Step: 176315 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:02:36,423-Speed 127.18 samples/sec Loss 1.0389 Epoch: 17 Global Step: 176320 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:02:55,977-Speed 130.93 samples/sec Loss 1.0076 Epoch: 17 Global Step: 176325 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:03:15,650-Speed 130.13 samples/sec Loss 1.1869 Epoch: 17 Global Step: 176330 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:03:35,255-Speed 130.58 samples/sec Loss 1.0996 Epoch: 17 Global Step: 176335 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:03:54,695-Speed 131.70 samples/sec Loss 1.2342 Epoch: 17 Global Step: 176340 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:04:14,199-Speed 131.26 samples/sec Loss 1.0221 Epoch: 17 Global Step: 176345 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:04:33,736-Speed 131.04 samples/sec Loss 1.1781 Epoch: 17 Global Step: 176350 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:04:53,300-Speed 130.86 samples/sec Loss 1.1133 Epoch: 17 Global Step: 176355 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:05:12,893-Speed 130.67 samples/sec Loss 1.1534 Epoch: 17 Global Step: 176360 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:05:32,703-Speed 129.23 samples/sec Loss 1.2119 Epoch: 17 Global Step: 176365 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:05:52,135-Speed 131.75 samples/sec Loss 0.9402 Epoch: 17 Global Step: 176370 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:06:11,840-Speed 129.92 samples/sec Loss 1.0967 Epoch: 17 Global Step: 176375 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:06:31,408-Speed 130.84 samples/sec Loss 1.1077 Epoch: 17 Global Step: 176380 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:06:50,843-Speed 131.73 samples/sec Loss 1.0872 Epoch: 17 Global Step: 176385 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:07:10,445-Speed 130.60 samples/sec Loss 0.9893 Epoch: 17 Global Step: 176390 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:07:30,059-Speed 130.53 samples/sec Loss 1.0909 Epoch: 17 Global Step: 176395 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:07:49,595-Speed 131.04 samples/sec Loss 1.1592 Epoch: 17 Global Step: 176400 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:08:09,161-Speed 130.85 samples/sec Loss 1.0618 Epoch: 17 Global Step: 176405 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:08:28,988-Speed 129.13 samples/sec Loss 1.1706 Epoch: 17 Global Step: 176410 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:08:48,445-Speed 131.58 samples/sec Loss 1.1441 Epoch: 17 Global Step: 176415 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:09:08,280-Speed 129.07 samples/sec Loss 1.0652 Epoch: 17 Global Step: 176420 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:09:27,863-Speed 130.73 samples/sec Loss 1.0717 Epoch: 17 Global Step: 176425 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:09:47,293-Speed 131.76 samples/sec Loss 1.1376 Epoch: 17 Global Step: 176430 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:10:07,020-Speed 129.78 samples/sec Loss 1.1368 Epoch: 17 Global Step: 176435 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:10:26,772-Speed 129.61 samples/sec Loss 1.0835 Epoch: 17 Global Step: 176440 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:10:46,370-Speed 130.64 samples/sec Loss 1.0181 Epoch: 17 Global Step: 176445 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:11:05,801-Speed 131.75 samples/sec Loss 1.0181 Epoch: 17 Global Step: 176450 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:11:25,663-Speed 128.89 samples/sec Loss 1.0690 Epoch: 17 Global Step: 176455 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:11:45,096-Speed 131.75 samples/sec Loss 1.0191 Epoch: 17 Global Step: 176460 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:12:04,812-Speed 129.85 samples/sec Loss 1.0762 Epoch: 17 Global Step: 176465 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:12:24,245-Speed 131.74 samples/sec Loss 1.0384 Epoch: 17 Global Step: 176470 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:12:43,826-Speed 130.75 samples/sec Loss 1.0340 Epoch: 17 Global Step: 176475 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:13:03,420-Speed 130.66 samples/sec Loss 1.2007 Epoch: 17 Global Step: 176480 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:13:22,908-Speed 131.37 samples/sec Loss 1.0145 Epoch: 17 Global Step: 176485 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:13:42,502-Speed 130.66 samples/sec Loss 1.0845 Epoch: 17 Global Step: 176490 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:14:01,943-Speed 131.69 samples/sec Loss 1.0940 Epoch: 17 Global Step: 176495 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:14:21,753-Speed 129.24 samples/sec Loss 1.0636 Epoch: 17 Global Step: 176500 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:14:41,378-Speed 130.45 samples/sec Loss 1.0902 Epoch: 17 Global Step: 176505 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:15:01,031-Speed 130.27 samples/sec Loss 1.1530 Epoch: 17 Global Step: 176510 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:15:20,557-Speed 131.12 samples/sec Loss 1.0252 Epoch: 17 Global Step: 176515 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:15:40,172-Speed 130.52 samples/sec Loss 1.1447 Epoch: 17 Global Step: 176520 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:15:59,781-Speed 130.56 samples/sec Loss 1.1999 Epoch: 17 Global Step: 176525 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:16:19,383-Speed 130.61 samples/sec Loss 1.1045 Epoch: 17 Global Step: 176530 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:16:38,811-Speed 131.78 samples/sec Loss 1.0415 Epoch: 17 Global Step: 176535 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:16:58,347-Speed 131.05 samples/sec Loss 1.0824 Epoch: 17 Global Step: 176540 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:17:18,126-Speed 129.44 samples/sec Loss 0.9807 Epoch: 17 Global Step: 176545 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:17:37,821-Speed 129.99 samples/sec Loss 1.1576 Epoch: 17 Global Step: 176550 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:17:57,252-Speed 131.75 samples/sec Loss 1.0883 Epoch: 17 Global Step: 176555 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:18:16,783-Speed 131.08 samples/sec Loss 1.1619 Epoch: 17 Global Step: 176560 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:18:36,326-Speed 131.00 samples/sec Loss 1.0594 Epoch: 17 Global Step: 176565 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:18:55,872-Speed 130.98 samples/sec Loss 1.1652 Epoch: 17 Global Step: 176570 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:19:15,461-Speed 130.70 samples/sec Loss 1.0195 Epoch: 17 Global Step: 176575 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:19:34,976-Speed 131.19 samples/sec Loss 1.1106 Epoch: 17 Global Step: 176580 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:19:54,506-Speed 131.09 samples/sec Loss 0.9999 Epoch: 17 Global Step: 176585 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:20:14,565-Speed 127.63 samples/sec Loss 0.9692 Epoch: 17 Global Step: 176590 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:20:34,270-Speed 129.92 samples/sec Loss 1.0834 Epoch: 17 Global Step: 176595 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:20:53,790-Speed 131.15 samples/sec Loss 1.2112 Epoch: 17 Global Step: 176600 Fp16 Grad Scale: 8192 Required: 86 hours Training: 2021-06-15 11:21:13,300-Speed 131.22 samples/sec Loss 1.0384 Epoch: 17 Global Step: 176605 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:21:32,877-Speed 130.78 samples/sec Loss 1.0076 Epoch: 17 Global Step: 176610 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:21:52,419-Speed 131.00 samples/sec Loss 1.0168 Epoch: 17 Global Step: 176615 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:22:12,621-Speed 126.73 samples/sec Loss 1.0718 Epoch: 17 Global Step: 176620 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:22:32,356-Speed 129.72 samples/sec Loss 1.1318 Epoch: 17 Global Step: 176625 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:22:51,903-Speed 130.97 samples/sec Loss 1.0199 Epoch: 17 Global Step: 176630 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:23:11,710-Speed 129.26 samples/sec Loss 1.1592 Epoch: 17 Global Step: 176635 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:23:31,372-Speed 130.21 samples/sec Loss 1.2899 Epoch: 17 Global Step: 176640 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:23:50,907-Speed 131.05 samples/sec Loss 1.0969 Epoch: 17 Global Step: 176645 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:24:10,460-Speed 130.94 samples/sec Loss 1.1487 Epoch: 17 Global Step: 176650 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:24:30,077-Speed 130.50 samples/sec Loss 1.0596 Epoch: 17 Global Step: 176655 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:24:49,532-Speed 131.59 samples/sec Loss 1.1386 Epoch: 17 Global Step: 176660 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:25:09,158-Speed 130.45 samples/sec Loss 1.0831 Epoch: 17 Global Step: 176665 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:25:28,736-Speed 130.77 samples/sec Loss 1.1379 Epoch: 17 Global Step: 176670 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:25:48,261-Speed 131.12 samples/sec Loss 1.1295 Epoch: 17 Global Step: 176675 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:26:08,076-Speed 129.20 samples/sec Loss 1.0528 Epoch: 17 Global Step: 176680 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:26:27,684-Speed 130.57 samples/sec Loss 1.1302 Epoch: 17 Global Step: 176685 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:26:47,108-Speed 131.80 samples/sec Loss 1.1476 Epoch: 17 Global Step: 176690 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:27:06,721-Speed 130.53 samples/sec Loss 1.0331 Epoch: 17 Global Step: 176695 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:27:26,372-Speed 130.28 samples/sec Loss 1.2000 Epoch: 17 Global Step: 176700 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:27:45,895-Speed 131.14 samples/sec Loss 1.0964 Epoch: 17 Global Step: 176705 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:28:05,464-Speed 130.82 samples/sec Loss 1.0407 Epoch: 17 Global Step: 176710 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:28:24,998-Speed 131.06 samples/sec Loss 1.0740 Epoch: 17 Global Step: 176715 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:28:44,563-Speed 130.86 samples/sec Loss 1.0872 Epoch: 17 Global Step: 176720 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:29:04,470-Speed 128.61 samples/sec Loss 0.9676 Epoch: 17 Global Step: 176725 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:29:23,909-Speed 131.70 samples/sec Loss 1.0756 Epoch: 17 Global Step: 176730 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:29:43,509-Speed 130.62 samples/sec Loss 1.1635 Epoch: 17 Global Step: 176735 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:30:03,044-Speed 131.05 samples/sec Loss 0.9937 Epoch: 17 Global Step: 176740 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:30:22,508-Speed 131.53 samples/sec Loss 1.1589 Epoch: 17 Global Step: 176745 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:30:42,111-Speed 130.60 samples/sec Loss 1.1080 Epoch: 17 Global Step: 176750 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:31:01,547-Speed 131.72 samples/sec Loss 1.0415 Epoch: 17 Global Step: 176755 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:31:21,136-Speed 130.69 samples/sec Loss 1.0454 Epoch: 17 Global Step: 176760 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:31:40,696-Speed 130.89 samples/sec Loss 1.1744 Epoch: 17 Global Step: 176765 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:32:00,515-Speed 129.17 samples/sec Loss 1.1722 Epoch: 17 Global Step: 176770 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:32:19,942-Speed 131.78 samples/sec Loss 1.1085 Epoch: 17 Global Step: 176775 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:32:39,806-Speed 128.89 samples/sec Loss 1.1442 Epoch: 17 Global Step: 176780 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:32:59,370-Speed 130.85 samples/sec Loss 1.1894 Epoch: 17 Global Step: 176785 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:33:18,888-Speed 131.17 samples/sec Loss 1.1268 Epoch: 17 Global Step: 176790 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:33:38,477-Speed 130.69 samples/sec Loss 1.1954 Epoch: 17 Global Step: 176795 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:33:58,025-Speed 130.97 samples/sec Loss 0.9667 Epoch: 17 Global Step: 176800 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:34:17,639-Speed 130.53 samples/sec Loss 1.0461 Epoch: 17 Global Step: 176805 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:34:37,197-Speed 130.90 samples/sec Loss 1.0836 Epoch: 17 Global Step: 176810 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:34:57,004-Speed 129.25 samples/sec Loss 1.1501 Epoch: 17 Global Step: 176815 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:35:16,436-Speed 131.76 samples/sec Loss 1.0588 Epoch: 17 Global Step: 176820 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:35:36,169-Speed 129.74 samples/sec Loss 1.1098 Epoch: 17 Global Step: 176825 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:35:55,683-Speed 131.20 samples/sec Loss 1.0811 Epoch: 17 Global Step: 176830 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:36:15,293-Speed 130.56 samples/sec Loss 1.1157 Epoch: 17 Global Step: 176835 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:36:34,787-Speed 131.33 samples/sec Loss 1.0618 Epoch: 17 Global Step: 176840 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:36:54,348-Speed 130.88 samples/sec Loss 1.1214 Epoch: 17 Global Step: 176845 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:37:13,916-Speed 130.83 samples/sec Loss 0.9931 Epoch: 17 Global Step: 176850 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:37:33,351-Speed 131.73 samples/sec Loss 1.1063 Epoch: 17 Global Step: 176855 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:37:53,154-Speed 129.29 samples/sec Loss 1.0621 Epoch: 17 Global Step: 176860 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:38:12,762-Speed 130.56 samples/sec Loss 1.0907 Epoch: 17 Global Step: 176865 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:38:32,376-Speed 130.52 samples/sec Loss 1.0864 Epoch: 17 Global Step: 176870 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:38:51,961-Speed 130.72 samples/sec Loss 1.0815 Epoch: 17 Global Step: 176875 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:39:11,546-Speed 130.72 samples/sec Loss 1.0224 Epoch: 17 Global Step: 176880 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:39:31,135-Speed 130.69 samples/sec Loss 1.1667 Epoch: 17 Global Step: 176885 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:39:50,741-Speed 130.58 samples/sec Loss 1.1410 Epoch: 17 Global Step: 176890 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:40:10,297-Speed 130.92 samples/sec Loss 1.1277 Epoch: 17 Global Step: 176895 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:40:30,027-Speed 129.76 samples/sec Loss 1.0866 Epoch: 17 Global Step: 176900 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:40:49,795-Speed 129.51 samples/sec Loss 1.0614 Epoch: 17 Global Step: 176905 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:41:09,485-Speed 130.02 samples/sec Loss 0.9784 Epoch: 17 Global Step: 176910 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:41:29,004-Speed 131.16 samples/sec Loss 0.9919 Epoch: 17 Global Step: 176915 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:41:48,561-Speed 130.91 samples/sec Loss 1.0566 Epoch: 17 Global Step: 176920 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:42:08,145-Speed 130.73 samples/sec Loss 1.1400 Epoch: 17 Global Step: 176925 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:42:27,860-Speed 129.86 samples/sec Loss 0.9853 Epoch: 17 Global Step: 176930 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:42:47,586-Speed 129.78 samples/sec Loss 1.1409 Epoch: 17 Global Step: 176935 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:43:07,123-Speed 131.04 samples/sec Loss 1.1325 Epoch: 17 Global Step: 176940 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:43:26,630-Speed 131.24 samples/sec Loss 1.1092 Epoch: 17 Global Step: 176945 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:43:46,362-Speed 129.75 samples/sec Loss 1.1352 Epoch: 17 Global Step: 176950 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:44:06,206-Speed 129.01 samples/sec Loss 1.0273 Epoch: 17 Global Step: 176955 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:44:25,809-Speed 130.60 samples/sec Loss 1.1457 Epoch: 17 Global Step: 176960 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:44:45,534-Speed 129.79 samples/sec Loss 1.1176 Epoch: 17 Global Step: 176965 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:45:04,982-Speed 131.64 samples/sec Loss 1.0981 Epoch: 17 Global Step: 176970 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:45:24,595-Speed 130.53 samples/sec Loss 0.9819 Epoch: 17 Global Step: 176975 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:45:44,170-Speed 130.78 samples/sec Loss 1.0026 Epoch: 17 Global Step: 176980 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:46:03,669-Speed 131.29 samples/sec Loss 1.1224 Epoch: 17 Global Step: 176985 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:46:23,192-Speed 131.14 samples/sec Loss 1.1472 Epoch: 17 Global Step: 176990 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:46:42,915-Speed 129.80 samples/sec Loss 1.1071 Epoch: 17 Global Step: 176995 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:47:02,697-Speed 129.42 samples/sec Loss 0.9347 Epoch: 17 Global Step: 177000 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:47:22,238-Speed 131.01 samples/sec Loss 1.1320 Epoch: 17 Global Step: 177005 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:47:41,862-Speed 130.46 samples/sec Loss 1.1083 Epoch: 17 Global Step: 177010 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:48:01,316-Speed 131.60 samples/sec Loss 1.0837 Epoch: 17 Global Step: 177015 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:48:20,934-Speed 130.50 samples/sec Loss 1.0996 Epoch: 17 Global Step: 177020 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:48:40,521-Speed 130.71 samples/sec Loss 1.1363 Epoch: 17 Global Step: 177025 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:49:00,051-Speed 131.09 samples/sec Loss 1.0376 Epoch: 17 Global Step: 177030 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:49:19,641-Speed 130.69 samples/sec Loss 1.0498 Epoch: 17 Global Step: 177035 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:49:39,319-Speed 130.10 samples/sec Loss 1.0551 Epoch: 17 Global Step: 177040 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:49:59,078-Speed 129.57 samples/sec Loss 1.1827 Epoch: 17 Global Step: 177045 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:50:18,533-Speed 131.59 samples/sec Loss 1.1034 Epoch: 17 Global Step: 177050 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:50:38,152-Speed 130.50 samples/sec Loss 1.1859 Epoch: 17 Global Step: 177055 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:50:57,689-Speed 131.04 samples/sec Loss 1.1233 Epoch: 17 Global Step: 177060 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:51:17,257-Speed 130.83 samples/sec Loss 1.1567 Epoch: 17 Global Step: 177065 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:51:36,852-Speed 130.65 samples/sec Loss 1.1563 Epoch: 17 Global Step: 177070 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:51:56,330-Speed 131.44 samples/sec Loss 1.0587 Epoch: 17 Global Step: 177075 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:52:15,953-Speed 130.47 samples/sec Loss 1.0179 Epoch: 17 Global Step: 177080 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:52:35,850-Speed 128.67 samples/sec Loss 1.1285 Epoch: 17 Global Step: 177085 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:52:55,339-Speed 131.36 samples/sec Loss 1.1697 Epoch: 17 Global Step: 177090 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 11:53:14,406-Speed 134.27 samples/sec Loss 1.2050 Epoch: 17 Global Step: 177095 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:53:34,083-Speed 130.11 samples/sec Loss 1.0664 Epoch: 17 Global Step: 177100 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:53:53,681-Speed 130.63 samples/sec Loss 1.0732 Epoch: 17 Global Step: 177105 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:54:13,297-Speed 130.51 samples/sec Loss 1.0246 Epoch: 17 Global Step: 177110 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:54:32,759-Speed 131.55 samples/sec Loss 1.0407 Epoch: 17 Global Step: 177115 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:54:52,344-Speed 130.72 samples/sec Loss 0.9994 Epoch: 17 Global Step: 177120 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:55:11,798-Speed 131.60 samples/sec Loss 1.1463 Epoch: 17 Global Step: 177125 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:55:31,804-Speed 127.97 samples/sec Loss 1.0989 Epoch: 17 Global Step: 177130 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:55:51,218-Speed 131.87 samples/sec Loss 0.9363 Epoch: 17 Global Step: 177135 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:56:10,846-Speed 130.44 samples/sec Loss 1.1390 Epoch: 17 Global Step: 177140 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:56:30,359-Speed 131.20 samples/sec Loss 1.0569 Epoch: 17 Global Step: 177145 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:56:50,018-Speed 130.23 samples/sec Loss 1.0443 Epoch: 17 Global Step: 177150 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:57:09,599-Speed 130.75 samples/sec Loss 1.1205 Epoch: 17 Global Step: 177155 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:57:29,030-Speed 131.76 samples/sec Loss 1.0564 Epoch: 17 Global Step: 177160 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:57:48,651-Speed 130.48 samples/sec Loss 0.9926 Epoch: 17 Global Step: 177165 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:58:08,155-Speed 131.26 samples/sec Loss 1.0299 Epoch: 17 Global Step: 177170 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:58:27,947-Speed 129.36 samples/sec Loss 1.0235 Epoch: 17 Global Step: 177175 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:58:47,583-Speed 130.38 samples/sec Loss 1.1822 Epoch: 17 Global Step: 177180 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:59:07,234-Speed 130.28 samples/sec Loss 1.1380 Epoch: 17 Global Step: 177185 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:59:26,832-Speed 130.64 samples/sec Loss 1.0407 Epoch: 17 Global Step: 177190 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 11:59:46,463-Speed 130.41 samples/sec Loss 1.1175 Epoch: 17 Global Step: 177195 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:00:05,893-Speed 131.76 samples/sec Loss 1.1304 Epoch: 17 Global Step: 177200 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:00:25,601-Speed 129.91 samples/sec Loss 1.0956 Epoch: 17 Global Step: 177205 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:00:45,930-Speed 125.93 samples/sec Loss 1.0114 Epoch: 17 Global Step: 177210 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:01:05,556-Speed 130.44 samples/sec Loss 1.1952 Epoch: 17 Global Step: 177215 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:01:24,907-Speed 132.30 samples/sec Loss 1.1159 Epoch: 17 Global Step: 177220 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:01:44,532-Speed 130.45 samples/sec Loss 1.0048 Epoch: 17 Global Step: 177225 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:02:03,964-Speed 131.75 samples/sec Loss 1.0724 Epoch: 17 Global Step: 177230 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:02:23,565-Speed 130.61 samples/sec Loss 1.0310 Epoch: 17 Global Step: 177235 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:02:43,124-Speed 130.89 samples/sec Loss 1.0945 Epoch: 17 Global Step: 177240 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:03:02,651-Speed 131.11 samples/sec Loss 1.0564 Epoch: 17 Global Step: 177245 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:03:22,474-Speed 129.15 samples/sec Loss 1.1132 Epoch: 17 Global Step: 177250 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:03:42,008-Speed 131.07 samples/sec Loss 1.0903 Epoch: 17 Global Step: 177255 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:04:01,642-Speed 130.39 samples/sec Loss 1.1076 Epoch: 17 Global Step: 177260 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:04:21,498-Speed 128.93 samples/sec Loss 1.1352 Epoch: 17 Global Step: 177265 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:04:41,232-Speed 129.73 samples/sec Loss 1.1756 Epoch: 17 Global Step: 177270 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:05:00,833-Speed 130.61 samples/sec Loss 0.9598 Epoch: 17 Global Step: 177275 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:05:20,589-Speed 129.59 samples/sec Loss 0.9867 Epoch: 17 Global Step: 177280 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:05:40,168-Speed 130.76 samples/sec Loss 1.0754 Epoch: 17 Global Step: 177285 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:05:59,697-Speed 131.09 samples/sec Loss 1.1871 Epoch: 17 Global Step: 177290 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:06:19,307-Speed 130.55 samples/sec Loss 1.0621 Epoch: 17 Global Step: 177295 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:06:38,930-Speed 130.47 samples/sec Loss 1.0805 Epoch: 17 Global Step: 177300 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:06:58,418-Speed 131.37 samples/sec Loss 1.1545 Epoch: 17 Global Step: 177305 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:07:18,180-Speed 129.55 samples/sec Loss 1.1059 Epoch: 17 Global Step: 177310 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:07:37,865-Speed 130.06 samples/sec Loss 1.0284 Epoch: 17 Global Step: 177315 Fp16 Grad Scale: 4096 Required: 85 hours Training: 2021-06-15 12:07:57,466-Speed 130.61 samples/sec Loss 1.1649 Epoch: 17 Global Step: 177320 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:08:17,139-Speed 130.13 samples/sec Loss 0.9730 Epoch: 17 Global Step: 177325 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:08:36,687-Speed 130.97 samples/sec Loss 1.0995 Epoch: 17 Global Step: 177330 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:08:56,112-Speed 131.80 samples/sec Loss 1.0471 Epoch: 17 Global Step: 177335 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:09:15,842-Speed 129.76 samples/sec Loss 1.1789 Epoch: 17 Global Step: 177340 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:09:35,435-Speed 130.67 samples/sec Loss 1.1643 Epoch: 17 Global Step: 177345 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:09:54,894-Speed 131.56 samples/sec Loss 1.0871 Epoch: 17 Global Step: 177350 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:10:14,652-Speed 129.58 samples/sec Loss 1.0486 Epoch: 17 Global Step: 177355 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:10:34,267-Speed 130.51 samples/sec Loss 1.0340 Epoch: 17 Global Step: 177360 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:10:53,796-Speed 131.09 samples/sec Loss 1.0402 Epoch: 17 Global Step: 177365 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:11:13,398-Speed 130.61 samples/sec Loss 1.0698 Epoch: 17 Global Step: 177370 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:11:32,946-Speed 130.96 samples/sec Loss 1.1523 Epoch: 17 Global Step: 177375 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:11:52,739-Speed 129.35 samples/sec Loss 1.0335 Epoch: 17 Global Step: 177380 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:12:12,358-Speed 130.49 samples/sec Loss 1.0945 Epoch: 17 Global Step: 177385 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:12:31,974-Speed 130.51 samples/sec Loss 1.0194 Epoch: 17 Global Step: 177390 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:12:51,588-Speed 130.53 samples/sec Loss 1.1072 Epoch: 17 Global Step: 177395 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:13:11,447-Speed 128.92 samples/sec Loss 0.9735 Epoch: 17 Global Step: 177400 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:13:30,877-Speed 131.76 samples/sec Loss 1.0193 Epoch: 17 Global Step: 177405 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:13:50,547-Speed 130.15 samples/sec Loss 1.2623 Epoch: 17 Global Step: 177410 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:14:10,196-Speed 130.29 samples/sec Loss 0.9521 Epoch: 17 Global Step: 177415 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:14:29,626-Speed 131.76 samples/sec Loss 0.9188 Epoch: 17 Global Step: 177420 Fp16 Grad Scale: 8192 Required: 85 hours Training: 2021-06-15 12:14:49,243-Speed 130.51 samples/sec Loss 1.0585 Epoch: 17 Global Step: 177425 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:15:08,785-Speed 131.01 samples/sec Loss 1.1617 Epoch: 17 Global Step: 177430 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:15:28,367-Speed 130.74 samples/sec Loss 1.0874 Epoch: 17 Global Step: 177435 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:15:47,924-Speed 130.91 samples/sec Loss 1.1861 Epoch: 17 Global Step: 177440 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:16:07,863-Speed 128.40 samples/sec Loss 1.0630 Epoch: 17 Global Step: 177445 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:16:27,369-Speed 131.27 samples/sec Loss 1.0593 Epoch: 17 Global Step: 177450 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:16:47,149-Speed 129.43 samples/sec Loss 1.0568 Epoch: 17 Global Step: 177455 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:17:06,987-Speed 129.06 samples/sec Loss 1.1371 Epoch: 17 Global Step: 177460 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:17:26,536-Speed 130.96 samples/sec Loss 1.0171 Epoch: 17 Global Step: 177465 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:17:46,135-Speed 130.62 samples/sec Loss 1.1441 Epoch: 17 Global Step: 177470 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:18:05,760-Speed 130.46 samples/sec Loss 1.1658 Epoch: 17 Global Step: 177475 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:18:25,419-Speed 130.23 samples/sec Loss 1.0342 Epoch: 17 Global Step: 177480 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:18:44,979-Speed 130.88 samples/sec Loss 1.0983 Epoch: 17 Global Step: 177485 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:19:04,770-Speed 129.36 samples/sec Loss 1.1501 Epoch: 17 Global Step: 177490 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:19:24,298-Speed 131.10 samples/sec Loss 1.0985 Epoch: 17 Global Step: 177495 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:19:44,031-Speed 129.74 samples/sec Loss 1.0928 Epoch: 17 Global Step: 177500 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:20:03,586-Speed 130.92 samples/sec Loss 1.0882 Epoch: 17 Global Step: 177505 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:20:23,180-Speed 130.66 samples/sec Loss 1.1857 Epoch: 17 Global Step: 177510 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:20:42,796-Speed 130.51 samples/sec Loss 1.0471 Epoch: 17 Global Step: 177515 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:21:02,335-Speed 131.03 samples/sec Loss 0.9632 Epoch: 17 Global Step: 177520 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:21:22,107-Speed 129.48 samples/sec Loss 1.0616 Epoch: 17 Global Step: 177525 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:21:41,644-Speed 131.04 samples/sec Loss 1.0571 Epoch: 17 Global Step: 177530 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:22:01,437-Speed 129.35 samples/sec Loss 1.1321 Epoch: 17 Global Step: 177535 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:22:21,071-Speed 130.39 samples/sec Loss 1.0848 Epoch: 17 Global Step: 177540 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:22:40,836-Speed 129.53 samples/sec Loss 1.0459 Epoch: 17 Global Step: 177545 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:23:00,465-Speed 130.43 samples/sec Loss 1.0385 Epoch: 17 Global Step: 177550 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:23:20,074-Speed 130.56 samples/sec Loss 1.0495 Epoch: 17 Global Step: 177555 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:23:39,822-Speed 129.64 samples/sec Loss 0.9838 Epoch: 17 Global Step: 177560 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:23:59,428-Speed 130.58 samples/sec Loss 1.1043 Epoch: 17 Global Step: 177565 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:24:18,981-Speed 130.94 samples/sec Loss 1.0031 Epoch: 17 Global Step: 177570 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:24:38,461-Speed 131.42 samples/sec Loss 1.0557 Epoch: 17 Global Step: 177575 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:24:58,250-Speed 129.38 samples/sec Loss 1.0637 Epoch: 17 Global Step: 177580 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:25:17,948-Speed 129.97 samples/sec Loss 1.0932 Epoch: 17 Global Step: 177585 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:25:37,456-Speed 131.24 samples/sec Loss 1.0093 Epoch: 17 Global Step: 177590 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:25:56,989-Speed 131.07 samples/sec Loss 1.0113 Epoch: 17 Global Step: 177595 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:26:16,626-Speed 130.37 samples/sec Loss 1.0515 Epoch: 17 Global Step: 177600 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:26:36,120-Speed 131.33 samples/sec Loss 1.1330 Epoch: 17 Global Step: 177605 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:26:55,690-Speed 130.82 samples/sec Loss 1.1660 Epoch: 17 Global Step: 177610 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:27:15,213-Speed 131.13 samples/sec Loss 1.1037 Epoch: 17 Global Step: 177615 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:27:34,810-Speed 130.64 samples/sec Loss 1.0706 Epoch: 17 Global Step: 177620 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:27:54,649-Speed 129.05 samples/sec Loss 1.1318 Epoch: 17 Global Step: 177625 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:28:14,218-Speed 130.82 samples/sec Loss 1.1420 Epoch: 17 Global Step: 177630 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:28:33,676-Speed 131.57 samples/sec Loss 1.0598 Epoch: 17 Global Step: 177635 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:28:53,303-Speed 130.44 samples/sec Loss 1.1551 Epoch: 17 Global Step: 177640 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:29:12,845-Speed 131.00 samples/sec Loss 1.1927 Epoch: 17 Global Step: 177645 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:29:32,388-Speed 131.00 samples/sec Loss 1.1569 Epoch: 17 Global Step: 177650 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:29:51,908-Speed 131.16 samples/sec Loss 1.1473 Epoch: 17 Global Step: 177655 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:30:11,363-Speed 131.59 samples/sec Loss 1.0314 Epoch: 17 Global Step: 177660 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:30:30,982-Speed 130.49 samples/sec Loss 1.1442 Epoch: 17 Global Step: 177665 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:30:50,751-Speed 129.50 samples/sec Loss 0.9861 Epoch: 17 Global Step: 177670 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:31:10,379-Speed 130.43 samples/sec Loss 1.0910 Epoch: 17 Global Step: 177675 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:31:30,027-Speed 130.30 samples/sec Loss 1.1324 Epoch: 17 Global Step: 177680 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:31:49,669-Speed 130.34 samples/sec Loss 1.0615 Epoch: 17 Global Step: 177685 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:32:09,120-Speed 131.62 samples/sec Loss 1.0987 Epoch: 17 Global Step: 177690 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:32:28,672-Speed 130.94 samples/sec Loss 1.2475 Epoch: 17 Global Step: 177695 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:32:48,164-Speed 131.35 samples/sec Loss 1.0870 Epoch: 17 Global Step: 177700 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:33:08,348-Speed 126.84 samples/sec Loss 1.0911 Epoch: 17 Global Step: 177705 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:33:27,998-Speed 130.28 samples/sec Loss 1.1242 Epoch: 17 Global Step: 177710 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:33:47,819-Speed 129.16 samples/sec Loss 1.2112 Epoch: 17 Global Step: 177715 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:34:07,244-Speed 131.80 samples/sec Loss 1.1233 Epoch: 17 Global Step: 177720 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:34:26,781-Speed 131.03 samples/sec Loss 1.0649 Epoch: 17 Global Step: 177725 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:34:46,406-Speed 130.46 samples/sec Loss 1.1773 Epoch: 17 Global Step: 177730 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:35:05,890-Speed 131.40 samples/sec Loss 1.0657 Epoch: 17 Global Step: 177735 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:35:25,045-Speed 133.65 samples/sec Loss 0.9763 Epoch: 17 Global Step: 177740 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:35:44,508-Speed 131.54 samples/sec Loss 1.0821 Epoch: 17 Global Step: 177745 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:36:04,110-Speed 130.61 samples/sec Loss 1.0204 Epoch: 17 Global Step: 177750 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:36:23,620-Speed 131.22 samples/sec Loss 1.1056 Epoch: 17 Global Step: 177755 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:36:43,236-Speed 130.52 samples/sec Loss 1.2499 Epoch: 17 Global Step: 177760 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:37:02,704-Speed 131.50 samples/sec Loss 1.1101 Epoch: 17 Global Step: 177765 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:37:22,408-Speed 129.93 samples/sec Loss 1.1023 Epoch: 17 Global Step: 177770 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:37:41,990-Speed 130.75 samples/sec Loss 1.1517 Epoch: 17 Global Step: 177775 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:38:01,445-Speed 131.59 samples/sec Loss 0.9724 Epoch: 17 Global Step: 177780 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:38:21,278-Speed 129.09 samples/sec Loss 1.0718 Epoch: 17 Global Step: 177785 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:38:40,880-Speed 130.61 samples/sec Loss 1.0850 Epoch: 17 Global Step: 177790 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:39:00,443-Speed 130.87 samples/sec Loss 1.1100 Epoch: 17 Global Step: 177795 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:39:20,289-Speed 129.00 samples/sec Loss 1.0311 Epoch: 17 Global Step: 177800 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:39:39,851-Speed 130.87 samples/sec Loss 1.0463 Epoch: 17 Global Step: 177805 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:39:59,390-Speed 131.03 samples/sec Loss 1.3214 Epoch: 17 Global Step: 177810 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:40:18,978-Speed 130.70 samples/sec Loss 1.0814 Epoch: 17 Global Step: 177815 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:40:38,498-Speed 131.16 samples/sec Loss 1.0934 Epoch: 17 Global Step: 177820 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:40:58,027-Speed 131.09 samples/sec Loss 1.1309 Epoch: 17 Global Step: 177825 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:41:17,802-Speed 129.46 samples/sec Loss 1.0742 Epoch: 17 Global Step: 177830 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:41:37,628-Speed 129.13 samples/sec Loss 1.1009 Epoch: 17 Global Step: 177835 Fp16 Grad Scale: 4096 Required: 84 hours Training: 2021-06-15 12:41:57,145-Speed 131.17 samples/sec Loss 1.0992 Epoch: 17 Global Step: 177840 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:42:16,704-Speed 130.90 samples/sec Loss 1.0322 Epoch: 17 Global Step: 177845 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:42:36,233-Speed 131.10 samples/sec Loss 1.0514 Epoch: 17 Global Step: 177850 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:42:55,790-Speed 130.91 samples/sec Loss 1.0777 Epoch: 17 Global Step: 177855 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:43:15,393-Speed 130.60 samples/sec Loss 1.0968 Epoch: 17 Global Step: 177860 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:43:34,936-Speed 131.00 samples/sec Loss 1.1368 Epoch: 17 Global Step: 177865 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:43:54,544-Speed 130.57 samples/sec Loss 1.0679 Epoch: 17 Global Step: 177870 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:44:14,312-Speed 129.51 samples/sec Loss 1.1081 Epoch: 17 Global Step: 177875 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:44:33,857-Speed 130.99 samples/sec Loss 1.0259 Epoch: 17 Global Step: 177880 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:44:53,470-Speed 130.53 samples/sec Loss 1.0346 Epoch: 17 Global Step: 177885 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:45:13,074-Speed 130.60 samples/sec Loss 1.1067 Epoch: 17 Global Step: 177890 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:45:32,605-Speed 131.08 samples/sec Loss 1.0410 Epoch: 17 Global Step: 177895 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:45:52,308-Speed 129.93 samples/sec Loss 1.1144 Epoch: 17 Global Step: 177900 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:46:12,056-Speed 129.64 samples/sec Loss 1.0906 Epoch: 17 Global Step: 177905 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:46:31,700-Speed 130.32 samples/sec Loss 1.1882 Epoch: 17 Global Step: 177910 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:46:51,192-Speed 131.34 samples/sec Loss 1.1318 Epoch: 17 Global Step: 177915 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:47:11,123-Speed 128.45 samples/sec Loss 1.1467 Epoch: 17 Global Step: 177920 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:47:30,758-Speed 130.39 samples/sec Loss 1.0351 Epoch: 17 Global Step: 177925 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:47:50,455-Speed 129.98 samples/sec Loss 1.0801 Epoch: 17 Global Step: 177930 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:48:09,917-Speed 131.54 samples/sec Loss 1.0919 Epoch: 17 Global Step: 177935 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:48:29,489-Speed 130.81 samples/sec Loss 1.0081 Epoch: 17 Global Step: 177940 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:48:49,107-Speed 130.50 samples/sec Loss 1.1860 Epoch: 17 Global Step: 177945 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:49:08,816-Speed 129.90 samples/sec Loss 1.0640 Epoch: 17 Global Step: 177950 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:49:28,406-Speed 130.68 samples/sec Loss 1.0722 Epoch: 17 Global Step: 177955 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:49:47,964-Speed 130.90 samples/sec Loss 1.1175 Epoch: 17 Global Step: 177960 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:50:07,901-Speed 128.41 samples/sec Loss 1.1574 Epoch: 17 Global Step: 177965 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:50:27,363-Speed 131.54 samples/sec Loss 1.0694 Epoch: 17 Global Step: 177970 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:50:47,151-Speed 129.38 samples/sec Loss 1.0538 Epoch: 17 Global Step: 177975 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:51:37,670-Speed 50.68 samples/sec Loss 1.1715 Epoch: 17 Global Step: 177980 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:52:28,228-Speed 50.64 samples/sec Loss 1.1306 Epoch: 17 Global Step: 177985 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:53:24,170-Speed 45.77 samples/sec Loss 1.1179 Epoch: 17 Global Step: 177990 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:54:24,412-Speed 42.51 samples/sec Loss 1.0417 Epoch: 17 Global Step: 177995 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:55:20,331-Speed 45.79 samples/sec Loss 1.0941 Epoch: 17 Global Step: 178000 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:56:18,217-Speed 44.23 samples/sec Loss 1.1506 Epoch: 17 Global Step: 178005 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:57:02,804-Speed 57.42 samples/sec Loss 1.1469 Epoch: 17 Global Step: 178010 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:57:22,440-Speed 130.38 samples/sec Loss 1.0388 Epoch: 17 Global Step: 178015 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:57:41,879-Speed 131.70 samples/sec Loss 1.0717 Epoch: 17 Global Step: 178020 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:58:02,268-Speed 125.56 samples/sec Loss 0.9626 Epoch: 17 Global Step: 178025 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:58:21,875-Speed 130.57 samples/sec Loss 1.0006 Epoch: 17 Global Step: 178030 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:58:41,495-Speed 130.48 samples/sec Loss 1.1206 Epoch: 17 Global Step: 178035 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:59:01,059-Speed 130.87 samples/sec Loss 1.2118 Epoch: 17 Global Step: 178040 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:59:20,616-Speed 130.91 samples/sec Loss 1.1569 Epoch: 17 Global Step: 178045 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 12:59:40,196-Speed 130.75 samples/sec Loss 1.0927 Epoch: 17 Global Step: 178050 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:00:00,179-Speed 128.12 samples/sec Loss 1.0467 Epoch: 17 Global Step: 178055 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:00:19,757-Speed 130.77 samples/sec Loss 1.0723 Epoch: 17 Global Step: 178060 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:00:39,329-Speed 130.81 samples/sec Loss 1.1765 Epoch: 17 Global Step: 178065 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:00:58,943-Speed 130.53 samples/sec Loss 1.0425 Epoch: 17 Global Step: 178070 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:01:18,604-Speed 130.22 samples/sec Loss 1.0831 Epoch: 17 Global Step: 178075 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:01:38,288-Speed 130.06 samples/sec Loss 1.1731 Epoch: 17 Global Step: 178080 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:01:57,829-Speed 131.01 samples/sec Loss 0.9832 Epoch: 17 Global Step: 178085 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:02:17,303-Speed 131.46 samples/sec Loss 1.0596 Epoch: 17 Global Step: 178090 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:02:37,511-Speed 126.69 samples/sec Loss 1.1309 Epoch: 17 Global Step: 178095 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:02:57,416-Speed 128.62 samples/sec Loss 1.0144 Epoch: 17 Global Step: 178100 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:03:17,540-Speed 127.21 samples/sec Loss 1.0882 Epoch: 17 Global Step: 178105 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:03:37,183-Speed 130.33 samples/sec Loss 1.1749 Epoch: 17 Global Step: 178110 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:03:56,806-Speed 130.46 samples/sec Loss 1.0392 Epoch: 17 Global Step: 178115 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:04:16,309-Speed 131.27 samples/sec Loss 1.1203 Epoch: 17 Global Step: 178120 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:04:35,958-Speed 130.29 samples/sec Loss 1.1138 Epoch: 17 Global Step: 178125 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:04:55,445-Speed 131.38 samples/sec Loss 1.0740 Epoch: 17 Global Step: 178130 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:05:15,063-Speed 130.50 samples/sec Loss 1.0065 Epoch: 17 Global Step: 178135 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:05:34,686-Speed 130.47 samples/sec Loss 1.0896 Epoch: 17 Global Step: 178140 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:05:54,567-Speed 128.77 samples/sec Loss 1.0777 Epoch: 17 Global Step: 178145 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:06:14,008-Speed 131.69 samples/sec Loss 1.1356 Epoch: 17 Global Step: 178150 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:06:33,529-Speed 131.15 samples/sec Loss 1.0522 Epoch: 17 Global Step: 178155 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:06:53,157-Speed 130.44 samples/sec Loss 1.0011 Epoch: 17 Global Step: 178160 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:07:12,712-Speed 130.92 samples/sec Loss 1.0456 Epoch: 17 Global Step: 178165 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:07:32,397-Speed 130.06 samples/sec Loss 1.1618 Epoch: 17 Global Step: 178170 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:07:51,834-Speed 131.71 samples/sec Loss 1.1767 Epoch: 17 Global Step: 178175 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:08:11,442-Speed 130.57 samples/sec Loss 1.2202 Epoch: 17 Global Step: 178180 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:08:31,008-Speed 130.85 samples/sec Loss 1.0245 Epoch: 17 Global Step: 178185 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:08:50,846-Speed 129.05 samples/sec Loss 1.0495 Epoch: 17 Global Step: 178190 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:09:10,284-Speed 131.70 samples/sec Loss 1.1175 Epoch: 17 Global Step: 178195 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:09:29,886-Speed 130.61 samples/sec Loss 1.1641 Epoch: 17 Global Step: 178200 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:09:49,483-Speed 130.64 samples/sec Loss 1.0833 Epoch: 17 Global Step: 178205 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:10:09,072-Speed 130.69 samples/sec Loss 1.1393 Epoch: 17 Global Step: 178210 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:10:28,684-Speed 130.54 samples/sec Loss 1.0553 Epoch: 17 Global Step: 178215 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:10:48,225-Speed 131.02 samples/sec Loss 1.0707 Epoch: 17 Global Step: 178220 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:11:07,763-Speed 131.03 samples/sec Loss 1.0551 Epoch: 17 Global Step: 178225 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:11:27,334-Speed 130.81 samples/sec Loss 1.0593 Epoch: 17 Global Step: 178230 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:11:47,252-Speed 128.53 samples/sec Loss 0.9307 Epoch: 17 Global Step: 178235 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:12:06,677-Speed 131.79 samples/sec Loss 1.0616 Epoch: 17 Global Step: 178240 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:12:26,385-Speed 129.90 samples/sec Loss 1.0474 Epoch: 17 Global Step: 178245 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:12:45,980-Speed 130.65 samples/sec Loss 0.9505 Epoch: 17 Global Step: 178250 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:13:05,986-Speed 127.97 samples/sec Loss 1.0316 Epoch: 17 Global Step: 178255 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:13:25,596-Speed 130.55 samples/sec Loss 1.1064 Epoch: 17 Global Step: 178260 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:13:45,138-Speed 131.00 samples/sec Loss 1.0582 Epoch: 17 Global Step: 178265 Fp16 Grad Scale: 8192 Required: 84 hours Training: 2021-06-15 13:14:04,764-Speed 130.45 samples/sec Loss 1.1438 Epoch: 17 Global Step: 178270 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:14:24,385-Speed 130.48 samples/sec Loss 1.1166 Epoch: 17 Global Step: 178275 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:14:44,412-Speed 127.83 samples/sec Loss 1.0129 Epoch: 17 Global Step: 178280 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:15:03,893-Speed 131.42 samples/sec Loss 1.0269 Epoch: 17 Global Step: 178285 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:16:39,283-Speed 26.84 samples/sec Loss 1.0422 Epoch: 17 Global Step: 178290 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:17:00,917-Speed 118.43 samples/sec Loss 1.1720 Epoch: 17 Global Step: 178295 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:17:20,482-Speed 130.85 samples/sec Loss 1.0111 Epoch: 17 Global Step: 178300 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:17:40,091-Speed 130.56 samples/sec Loss 0.9671 Epoch: 17 Global Step: 178305 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:17:59,636-Speed 130.99 samples/sec Loss 1.0521 Epoch: 17 Global Step: 178310 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:18:19,270-Speed 130.39 samples/sec Loss 1.1122 Epoch: 17 Global Step: 178315 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:18:38,728-Speed 131.58 samples/sec Loss 0.9869 Epoch: 17 Global Step: 178320 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:18:58,545-Speed 129.19 samples/sec Loss 1.0525 Epoch: 17 Global Step: 178325 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:19:17,975-Speed 131.77 samples/sec Loss 1.0848 Epoch: 17 Global Step: 178330 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:19:37,782-Speed 129.25 samples/sec Loss 1.0965 Epoch: 17 Global Step: 178335 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:19:56,794-Speed 134.66 samples/sec Loss 1.0623 Epoch: 17 Global Step: 178340 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:20:16,266-Speed 131.48 samples/sec Loss 1.1101 Epoch: 17 Global Step: 178345 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:20:35,846-Speed 130.75 samples/sec Loss 1.2038 Epoch: 17 Global Step: 178350 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:20:55,446-Speed 130.62 samples/sec Loss 1.0266 Epoch: 17 Global Step: 178355 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:21:15,076-Speed 130.42 samples/sec Loss 1.0276 Epoch: 17 Global Step: 178360 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:22:33,686-Speed 32.57 samples/sec Loss 1.1365 Epoch: 17 Global Step: 178365 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:23:11,724-Speed 67.33 samples/sec Loss 1.0512 Epoch: 17 Global Step: 178370 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:23:31,352-Speed 130.43 samples/sec Loss 1.1555 Epoch: 17 Global Step: 178375 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:23:50,810-Speed 131.57 samples/sec Loss 1.0815 Epoch: 17 Global Step: 178380 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:24:10,390-Speed 130.75 samples/sec Loss 1.0377 Epoch: 17 Global Step: 178385 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:24:29,937-Speed 130.97 samples/sec Loss 1.0825 Epoch: 17 Global Step: 178390 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:24:49,514-Speed 130.77 samples/sec Loss 1.0572 Epoch: 17 Global Step: 178395 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:25:09,228-Speed 129.87 samples/sec Loss 1.1061 Epoch: 17 Global Step: 178400 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:25:28,769-Speed 131.01 samples/sec Loss 1.0045 Epoch: 17 Global Step: 178405 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:25:48,355-Speed 130.71 samples/sec Loss 0.9392 Epoch: 17 Global Step: 178410 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:26:15,209-Speed 95.34 samples/sec Loss 1.0917 Epoch: 17 Global Step: 178415 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:26:44,289-Speed 88.04 samples/sec Loss 1.0958 Epoch: 17 Global Step: 178420 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:27:03,939-Speed 130.30 samples/sec Loss 1.1854 Epoch: 17 Global Step: 178425 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:27:23,679-Speed 129.70 samples/sec Loss 1.0789 Epoch: 17 Global Step: 178430 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:27:43,398-Speed 129.83 samples/sec Loss 1.1046 Epoch: 17 Global Step: 178435 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 13:28:03,047-Speed 130.29 samples/sec Loss 1.1508 Epoch: 17 Global Step: 178440 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:28:22,668-Speed 130.48 samples/sec Loss 1.1130 Epoch: 17 Global Step: 178445 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:28:42,192-Speed 131.13 samples/sec Loss 1.0658 Epoch: 17 Global Step: 178450 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:29:01,831-Speed 130.36 samples/sec Loss 0.9793 Epoch: 17 Global Step: 178455 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:29:21,829-Speed 128.02 samples/sec Loss 1.1711 Epoch: 17 Global Step: 178460 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:29:41,395-Speed 130.85 samples/sec Loss 1.1116 Epoch: 17 Global Step: 178465 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:30:01,301-Speed 128.61 samples/sec Loss 1.0937 Epoch: 17 Global Step: 178470 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:30:21,005-Speed 129.93 samples/sec Loss 0.9941 Epoch: 17 Global Step: 178475 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:30:40,799-Speed 129.34 samples/sec Loss 1.0395 Epoch: 17 Global Step: 178480 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:31:00,631-Speed 129.09 samples/sec Loss 1.0229 Epoch: 17 Global Step: 178485 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:31:20,325-Speed 130.00 samples/sec Loss 1.1287 Epoch: 17 Global Step: 178490 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:31:39,983-Speed 130.23 samples/sec Loss 1.0426 Epoch: 17 Global Step: 178495 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:31:59,664-Speed 130.09 samples/sec Loss 1.1073 Epoch: 17 Global Step: 178500 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:32:19,776-Speed 127.29 samples/sec Loss 1.1524 Epoch: 17 Global Step: 178505 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:32:39,438-Speed 130.21 samples/sec Loss 1.0564 Epoch: 17 Global Step: 178510 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:32:59,130-Speed 130.01 samples/sec Loss 1.0064 Epoch: 17 Global Step: 178515 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:33:18,841-Speed 129.89 samples/sec Loss 1.1390 Epoch: 17 Global Step: 178520 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:33:38,597-Speed 129.59 samples/sec Loss 1.0999 Epoch: 17 Global Step: 178525 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:33:58,293-Speed 129.98 samples/sec Loss 1.0442 Epoch: 17 Global Step: 178530 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:34:17,959-Speed 130.18 samples/sec Loss 1.0157 Epoch: 17 Global Step: 178535 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:34:37,629-Speed 130.16 samples/sec Loss 0.9954 Epoch: 17 Global Step: 178540 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:34:57,236-Speed 130.57 samples/sec Loss 1.1205 Epoch: 17 Global Step: 178545 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:35:17,148-Speed 128.58 samples/sec Loss 0.9818 Epoch: 17 Global Step: 178550 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:35:36,877-Speed 129.77 samples/sec Loss 1.1211 Epoch: 17 Global Step: 178555 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:35:56,518-Speed 130.35 samples/sec Loss 0.9858 Epoch: 17 Global Step: 178560 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:36:16,268-Speed 129.62 samples/sec Loss 1.0004 Epoch: 17 Global Step: 178565 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:36:35,969-Speed 129.95 samples/sec Loss 1.1059 Epoch: 17 Global Step: 178570 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:36:55,608-Speed 130.36 samples/sec Loss 1.0837 Epoch: 17 Global Step: 178575 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:37:15,349-Speed 129.68 samples/sec Loss 1.0654 Epoch: 17 Global Step: 178580 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:37:34,882-Speed 131.07 samples/sec Loss 1.1605 Epoch: 17 Global Step: 178585 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:37:54,427-Speed 130.98 samples/sec Loss 1.0255 Epoch: 17 Global Step: 178590 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:38:14,206-Speed 129.44 samples/sec Loss 1.1162 Epoch: 17 Global Step: 178595 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:38:33,900-Speed 130.00 samples/sec Loss 1.0358 Epoch: 17 Global Step: 178600 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:38:53,474-Speed 130.79 samples/sec Loss 1.1718 Epoch: 17 Global Step: 178605 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:39:13,019-Speed 130.99 samples/sec Loss 1.0466 Epoch: 17 Global Step: 178610 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:39:32,695-Speed 130.12 samples/sec Loss 1.0382 Epoch: 17 Global Step: 178615 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:39:52,222-Speed 131.10 samples/sec Loss 1.0660 Epoch: 17 Global Step: 178620 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:40:11,817-Speed 130.66 samples/sec Loss 0.9319 Epoch: 17 Global Step: 178625 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:40:31,354-Speed 131.04 samples/sec Loss 1.0210 Epoch: 17 Global Step: 178630 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:40:50,923-Speed 130.83 samples/sec Loss 1.0791 Epoch: 17 Global Step: 178635 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:41:10,697-Speed 129.47 samples/sec Loss 1.0874 Epoch: 17 Global Step: 178640 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:41:30,396-Speed 129.96 samples/sec Loss 1.0734 Epoch: 17 Global Step: 178645 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:41:49,841-Speed 131.66 samples/sec Loss 1.0622 Epoch: 17 Global Step: 178650 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:42:09,453-Speed 130.54 samples/sec Loss 1.1010 Epoch: 17 Global Step: 178655 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:42:28,951-Speed 131.30 samples/sec Loss 1.0898 Epoch: 17 Global Step: 178660 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:42:48,477-Speed 131.12 samples/sec Loss 1.0528 Epoch: 17 Global Step: 178665 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:43:08,047-Speed 130.82 samples/sec Loss 1.1158 Epoch: 17 Global Step: 178670 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:43:27,547-Speed 131.28 samples/sec Loss 1.1446 Epoch: 17 Global Step: 178675 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:43:51,876-Speed 105.23 samples/sec Loss 1.0911 Epoch: 17 Global Step: 178680 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:44:12,194-Speed 126.01 samples/sec Loss 1.0525 Epoch: 17 Global Step: 178685 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:44:31,701-Speed 131.24 samples/sec Loss 1.1143 Epoch: 17 Global Step: 178690 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:44:51,347-Speed 130.32 samples/sec Loss 1.1640 Epoch: 17 Global Step: 178695 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:45:10,961-Speed 130.53 samples/sec Loss 1.1167 Epoch: 17 Global Step: 178700 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:45:30,422-Speed 131.56 samples/sec Loss 1.0479 Epoch: 17 Global Step: 178705 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:45:50,041-Speed 130.49 samples/sec Loss 1.0203 Epoch: 17 Global Step: 178710 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:46:09,625-Speed 130.72 samples/sec Loss 0.9979 Epoch: 17 Global Step: 178715 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:46:29,185-Speed 130.89 samples/sec Loss 1.1221 Epoch: 17 Global Step: 178720 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:46:48,792-Speed 130.57 samples/sec Loss 1.1258 Epoch: 17 Global Step: 178725 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:47:08,532-Speed 129.69 samples/sec Loss 1.0851 Epoch: 17 Global Step: 178730 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:47:28,148-Speed 130.52 samples/sec Loss 1.0130 Epoch: 17 Global Step: 178735 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:47:47,725-Speed 130.77 samples/sec Loss 1.0505 Epoch: 17 Global Step: 178740 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:48:07,372-Speed 130.31 samples/sec Loss 1.1515 Epoch: 17 Global Step: 178745 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:48:26,805-Speed 131.74 samples/sec Loss 1.0340 Epoch: 17 Global Step: 178750 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:48:46,433-Speed 130.43 samples/sec Loss 1.0701 Epoch: 17 Global Step: 178755 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:49:05,987-Speed 130.93 samples/sec Loss 1.0215 Epoch: 17 Global Step: 178760 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:49:25,488-Speed 131.29 samples/sec Loss 1.1710 Epoch: 17 Global Step: 178765 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:49:45,115-Speed 130.44 samples/sec Loss 0.9446 Epoch: 17 Global Step: 178770 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:50:04,854-Speed 129.69 samples/sec Loss 1.0217 Epoch: 17 Global Step: 178775 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:50:24,398-Speed 131.00 samples/sec Loss 1.1715 Epoch: 17 Global Step: 178780 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:50:43,955-Speed 130.90 samples/sec Loss 1.0371 Epoch: 17 Global Step: 178785 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:51:03,550-Speed 130.65 samples/sec Loss 1.0576 Epoch: 17 Global Step: 178790 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:51:23,075-Speed 131.12 samples/sec Loss 1.0821 Epoch: 17 Global Step: 178795 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:51:42,611-Speed 131.04 samples/sec Loss 1.1901 Epoch: 17 Global Step: 178800 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:52:02,158-Speed 130.98 samples/sec Loss 0.9901 Epoch: 17 Global Step: 178805 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:52:21,683-Speed 131.12 samples/sec Loss 0.9826 Epoch: 17 Global Step: 178810 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:52:41,278-Speed 130.65 samples/sec Loss 1.1714 Epoch: 17 Global Step: 178815 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:53:01,036-Speed 129.57 samples/sec Loss 0.9814 Epoch: 17 Global Step: 178820 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:53:20,591-Speed 130.92 samples/sec Loss 1.0854 Epoch: 17 Global Step: 178825 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:53:40,233-Speed 130.34 samples/sec Loss 1.0337 Epoch: 17 Global Step: 178830 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:54:00,659-Speed 125.33 samples/sec Loss 1.1620 Epoch: 17 Global Step: 178835 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:54:25,007-Speed 105.15 samples/sec Loss 1.0199 Epoch: 17 Global Step: 178840 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:54:44,673-Speed 130.18 samples/sec Loss 1.0052 Epoch: 17 Global Step: 178845 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:55:04,308-Speed 130.38 samples/sec Loss 1.2277 Epoch: 17 Global Step: 178850 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:55:23,852-Speed 130.99 samples/sec Loss 0.9839 Epoch: 17 Global Step: 178855 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:55:43,439-Speed 130.71 samples/sec Loss 1.2006 Epoch: 17 Global Step: 178860 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:56:03,430-Speed 128.07 samples/sec Loss 1.0361 Epoch: 17 Global Step: 178865 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:56:22,918-Speed 131.37 samples/sec Loss 1.1679 Epoch: 17 Global Step: 178870 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:56:42,539-Speed 130.48 samples/sec Loss 1.0801 Epoch: 17 Global Step: 178875 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:57:02,110-Speed 130.81 samples/sec Loss 1.2196 Epoch: 17 Global Step: 178880 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:57:21,719-Speed 130.56 samples/sec Loss 1.0745 Epoch: 17 Global Step: 178885 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:57:41,350-Speed 130.41 samples/sec Loss 1.0015 Epoch: 17 Global Step: 178890 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:58:00,853-Speed 131.27 samples/sec Loss 1.1237 Epoch: 17 Global Step: 178895 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:58:20,328-Speed 131.46 samples/sec Loss 1.0037 Epoch: 17 Global Step: 178900 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:58:39,944-Speed 130.51 samples/sec Loss 0.9596 Epoch: 17 Global Step: 178905 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:58:59,932-Speed 128.08 samples/sec Loss 1.2217 Epoch: 17 Global Step: 178910 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:59:19,374-Speed 131.68 samples/sec Loss 1.1552 Epoch: 17 Global Step: 178915 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:59:38,966-Speed 130.67 samples/sec Loss 1.1254 Epoch: 17 Global Step: 178920 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 13:59:58,609-Speed 130.33 samples/sec Loss 1.0082 Epoch: 17 Global Step: 178925 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:00:18,234-Speed 130.45 samples/sec Loss 1.1767 Epoch: 17 Global Step: 178930 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:00:37,886-Speed 130.28 samples/sec Loss 0.9933 Epoch: 17 Global Step: 178935 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:00:57,428-Speed 131.00 samples/sec Loss 1.0419 Epoch: 17 Global Step: 178940 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:01:17,047-Speed 130.50 samples/sec Loss 1.0357 Epoch: 17 Global Step: 178945 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:01:36,700-Speed 130.27 samples/sec Loss 1.0256 Epoch: 17 Global Step: 178950 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:01:56,808-Speed 127.32 samples/sec Loss 1.0215 Epoch: 17 Global Step: 178955 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:02:16,245-Speed 131.72 samples/sec Loss 1.0908 Epoch: 17 Global Step: 178960 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:02:35,838-Speed 130.67 samples/sec Loss 1.0638 Epoch: 17 Global Step: 178965 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:02:55,414-Speed 130.78 samples/sec Loss 1.2018 Epoch: 17 Global Step: 178970 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:03:14,510-Speed 134.07 samples/sec Loss 1.0759 Epoch: 17 Global Step: 178975 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:03:34,116-Speed 130.58 samples/sec Loss 1.1709 Epoch: 17 Global Step: 178980 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:03:53,626-Speed 131.22 samples/sec Loss 1.1146 Epoch: 17 Global Step: 178985 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:04:13,208-Speed 130.74 samples/sec Loss 0.9525 Epoch: 17 Global Step: 178990 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:04:32,638-Speed 131.76 samples/sec Loss 1.0112 Epoch: 17 Global Step: 178995 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:04:52,539-Speed 128.65 samples/sec Loss 1.0202 Epoch: 17 Global Step: 179000 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:05:11,972-Speed 131.74 samples/sec Loss 1.1203 Epoch: 17 Global Step: 179005 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:05:31,712-Speed 129.69 samples/sec Loss 1.0220 Epoch: 17 Global Step: 179010 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:05:51,168-Speed 131.59 samples/sec Loss 1.2256 Epoch: 17 Global Step: 179015 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:06:10,813-Speed 130.33 samples/sec Loss 1.0287 Epoch: 17 Global Step: 179020 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:06:30,415-Speed 130.60 samples/sec Loss 0.9449 Epoch: 17 Global Step: 179025 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:06:49,978-Speed 130.87 samples/sec Loss 1.0916 Epoch: 17 Global Step: 179030 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:07:09,583-Speed 130.59 samples/sec Loss 1.1183 Epoch: 17 Global Step: 179035 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:07:29,014-Speed 131.75 samples/sec Loss 1.0849 Epoch: 17 Global Step: 179040 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:07:49,065-Speed 127.69 samples/sec Loss 1.1376 Epoch: 17 Global Step: 179045 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:08:08,506-Speed 131.69 samples/sec Loss 1.1036 Epoch: 17 Global Step: 179050 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:08:28,114-Speed 130.56 samples/sec Loss 1.1177 Epoch: 17 Global Step: 179055 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:08:47,636-Speed 131.15 samples/sec Loss 1.1449 Epoch: 17 Global Step: 179060 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:09:07,157-Speed 131.14 samples/sec Loss 0.9944 Epoch: 17 Global Step: 179065 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:09:26,761-Speed 130.60 samples/sec Loss 1.0093 Epoch: 17 Global Step: 179070 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:09:46,309-Speed 130.97 samples/sec Loss 1.1457 Epoch: 17 Global Step: 179075 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:10:05,914-Speed 130.59 samples/sec Loss 1.2230 Epoch: 17 Global Step: 179080 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:10:25,437-Speed 131.13 samples/sec Loss 1.0188 Epoch: 17 Global Step: 179085 Fp16 Grad Scale: 8192 Required: 83 hours Training: 2021-06-15 14:10:44,530-Speed 134.09 samples/sec Loss 1.0894 Epoch: 17 Global Step: 179090 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:11:04,112-Speed 130.75 samples/sec Loss 1.1159 Epoch: 17 Global Step: 179095 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:11:24,024-Speed 128.57 samples/sec Loss 1.0697 Epoch: 17 Global Step: 179100 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:11:43,468-Speed 131.66 samples/sec Loss 1.0781 Epoch: 17 Global Step: 179105 Fp16 Grad Scale: 4096 Required: 83 hours Training: 2021-06-15 14:12:03,064-Speed 130.65 samples/sec Loss 1.0972 Epoch: 17 Global Step: 179110 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:12:22,661-Speed 130.64 samples/sec Loss 1.1179 Epoch: 17 Global Step: 179115 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:12:42,162-Speed 131.28 samples/sec Loss 1.0488 Epoch: 17 Global Step: 179120 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:13:01,724-Speed 130.87 samples/sec Loss 1.0428 Epoch: 17 Global Step: 179125 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:13:21,399-Speed 130.12 samples/sec Loss 1.0475 Epoch: 17 Global Step: 179130 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:13:41,009-Speed 130.55 samples/sec Loss 1.0802 Epoch: 17 Global Step: 179135 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:14:00,595-Speed 130.71 samples/sec Loss 0.9411 Epoch: 17 Global Step: 179140 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:14:20,122-Speed 131.11 samples/sec Loss 1.2064 Epoch: 17 Global Step: 179145 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:14:39,696-Speed 130.79 samples/sec Loss 1.0297 Epoch: 17 Global Step: 179150 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:14:59,233-Speed 131.05 samples/sec Loss 1.0388 Epoch: 17 Global Step: 179155 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:15:19,013-Speed 129.43 samples/sec Loss 0.9850 Epoch: 17 Global Step: 179160 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:15:38,643-Speed 130.42 samples/sec Loss 0.9988 Epoch: 17 Global Step: 179165 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:15:58,344-Speed 129.95 samples/sec Loss 1.0144 Epoch: 17 Global Step: 179170 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:16:17,864-Speed 131.16 samples/sec Loss 1.1509 Epoch: 17 Global Step: 179175 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:16:37,400-Speed 131.05 samples/sec Loss 1.1872 Epoch: 17 Global Step: 179180 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:16:57,008-Speed 130.56 samples/sec Loss 1.0850 Epoch: 17 Global Step: 179185 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 14:17:16,565-Speed 130.91 samples/sec Loss 0.9683 Epoch: 17 Global Step: 179190 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:17:36,169-Speed 130.59 samples/sec Loss 1.1425 Epoch: 17 Global Step: 179195 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:17:55,858-Speed 130.03 samples/sec Loss 1.0534 Epoch: 17 Global Step: 179200 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:18:15,782-Speed 128.50 samples/sec Loss 1.1569 Epoch: 17 Global Step: 179205 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:18:35,416-Speed 130.39 samples/sec Loss 1.1187 Epoch: 17 Global Step: 179210 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:18:55,058-Speed 130.34 samples/sec Loss 1.0089 Epoch: 17 Global Step: 179215 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:19:14,582-Speed 131.13 samples/sec Loss 1.0238 Epoch: 17 Global Step: 179220 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:19:34,173-Speed 130.68 samples/sec Loss 1.1675 Epoch: 17 Global Step: 179225 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:19:53,949-Speed 129.46 samples/sec Loss 1.1154 Epoch: 17 Global Step: 179230 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:20:13,461-Speed 131.21 samples/sec Loss 1.1141 Epoch: 17 Global Step: 179235 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:20:33,040-Speed 130.76 samples/sec Loss 1.1320 Epoch: 17 Global Step: 179240 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:20:52,499-Speed 131.57 samples/sec Loss 1.1520 Epoch: 17 Global Step: 179245 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:21:12,296-Speed 129.31 samples/sec Loss 0.9765 Epoch: 17 Global Step: 179250 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:21:31,922-Speed 130.45 samples/sec Loss 1.0697 Epoch: 17 Global Step: 179255 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:21:51,657-Speed 129.73 samples/sec Loss 1.0264 Epoch: 17 Global Step: 179260 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:22:11,243-Speed 130.71 samples/sec Loss 1.0871 Epoch: 17 Global Step: 179265 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:22:30,727-Speed 131.40 samples/sec Loss 1.1290 Epoch: 17 Global Step: 179270 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:22:50,293-Speed 130.85 samples/sec Loss 1.0323 Epoch: 17 Global Step: 179275 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:23:09,810-Speed 131.18 samples/sec Loss 1.0126 Epoch: 17 Global Step: 179280 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:23:29,400-Speed 130.69 samples/sec Loss 1.1480 Epoch: 17 Global Step: 179285 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:23:48,937-Speed 131.04 samples/sec Loss 0.9307 Epoch: 17 Global Step: 179290 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:24:08,750-Speed 129.22 samples/sec Loss 1.0917 Epoch: 17 Global Step: 179295 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:24:28,577-Speed 129.12 samples/sec Loss 1.0631 Epoch: 17 Global Step: 179300 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:24:48,475-Speed 128.66 samples/sec Loss 1.1544 Epoch: 17 Global Step: 179305 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:25:07,935-Speed 131.56 samples/sec Loss 1.0577 Epoch: 17 Global Step: 179310 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:25:27,529-Speed 130.66 samples/sec Loss 1.1245 Epoch: 17 Global Step: 179315 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:25:47,265-Speed 129.71 samples/sec Loss 0.9849 Epoch: 17 Global Step: 179320 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:26:06,812-Speed 130.97 samples/sec Loss 1.0236 Epoch: 17 Global Step: 179325 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:26:26,405-Speed 130.67 samples/sec Loss 1.1639 Epoch: 17 Global Step: 179330 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:26:45,832-Speed 131.79 samples/sec Loss 1.1152 Epoch: 17 Global Step: 179335 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:27:05,648-Speed 129.20 samples/sec Loss 1.0362 Epoch: 17 Global Step: 179340 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:27:25,214-Speed 130.84 samples/sec Loss 1.0236 Epoch: 17 Global Step: 179345 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:27:44,880-Speed 130.18 samples/sec Loss 1.1651 Epoch: 17 Global Step: 179350 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:28:27,666-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-15 14:28:46,720-Speed 41.40 samples/sec Loss 0.8604 Epoch: 18 Global Step: 179355 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:29:06,455-Speed 129.73 samples/sec Loss 0.7564 Epoch: 18 Global Step: 179360 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:29:26,242-Speed 129.39 samples/sec Loss 0.8978 Epoch: 18 Global Step: 179365 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:29:46,135-Speed 128.70 samples/sec Loss 0.8844 Epoch: 18 Global Step: 179370 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:30:05,922-Speed 129.39 samples/sec Loss 0.8710 Epoch: 18 Global Step: 179375 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:30:25,645-Speed 129.80 samples/sec Loss 0.8246 Epoch: 18 Global Step: 179380 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:30:45,594-Speed 128.34 samples/sec Loss 0.8733 Epoch: 18 Global Step: 179385 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:31:05,187-Speed 130.67 samples/sec Loss 0.7870 Epoch: 18 Global Step: 179390 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:31:25,554-Speed 125.69 samples/sec Loss 0.7436 Epoch: 18 Global Step: 179395 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:31:45,328-Speed 129.47 samples/sec Loss 0.7299 Epoch: 18 Global Step: 179400 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:32:05,227-Speed 128.66 samples/sec Loss 0.7974 Epoch: 18 Global Step: 179405 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:32:24,849-Speed 130.47 samples/sec Loss 0.7556 Epoch: 18 Global Step: 179410 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:32:44,744-Speed 128.68 samples/sec Loss 0.7105 Epoch: 18 Global Step: 179415 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:33:04,468-Speed 129.80 samples/sec Loss 0.7622 Epoch: 18 Global Step: 179420 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:33:24,284-Speed 129.20 samples/sec Loss 0.7559 Epoch: 18 Global Step: 179425 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:33:44,143-Speed 128.92 samples/sec Loss 0.8105 Epoch: 18 Global Step: 179430 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:34:03,814-Speed 130.14 samples/sec Loss 0.7731 Epoch: 18 Global Step: 179435 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:34:24,208-Speed 125.54 samples/sec Loss 0.8068 Epoch: 18 Global Step: 179440 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:34:44,038-Speed 129.10 samples/sec Loss 0.6996 Epoch: 18 Global Step: 179445 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:35:03,870-Speed 129.09 samples/sec Loss 0.7094 Epoch: 18 Global Step: 179450 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:35:23,489-Speed 130.49 samples/sec Loss 0.7836 Epoch: 18 Global Step: 179455 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:35:43,395-Speed 128.61 samples/sec Loss 0.8281 Epoch: 18 Global Step: 179460 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:36:03,135-Speed 129.69 samples/sec Loss 0.8255 Epoch: 18 Global Step: 179465 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:36:23,023-Speed 128.73 samples/sec Loss 0.7989 Epoch: 18 Global Step: 179470 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:36:42,827-Speed 129.27 samples/sec Loss 0.7547 Epoch: 18 Global Step: 179475 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:37:02,463-Speed 130.37 samples/sec Loss 0.8443 Epoch: 18 Global Step: 179480 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:37:22,855-Speed 125.55 samples/sec Loss 0.8093 Epoch: 18 Global Step: 179485 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:37:42,737-Speed 128.76 samples/sec Loss 0.8246 Epoch: 18 Global Step: 179490 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:38:02,589-Speed 128.96 samples/sec Loss 0.6860 Epoch: 18 Global Step: 179495 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:38:22,205-Speed 130.52 samples/sec Loss 0.8273 Epoch: 18 Global Step: 179500 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:38:42,111-Speed 128.61 samples/sec Loss 0.7521 Epoch: 18 Global Step: 179505 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:39:01,823-Speed 129.88 samples/sec Loss 0.8522 Epoch: 18 Global Step: 179510 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:39:21,592-Speed 129.50 samples/sec Loss 0.7978 Epoch: 18 Global Step: 179515 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:39:41,477-Speed 128.75 samples/sec Loss 0.8873 Epoch: 18 Global Step: 179520 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:40:01,116-Speed 130.36 samples/sec Loss 0.7460 Epoch: 18 Global Step: 179525 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:40:21,678-Speed 124.51 samples/sec Loss 0.8074 Epoch: 18 Global Step: 179530 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:40:41,562-Speed 128.75 samples/sec Loss 0.8223 Epoch: 18 Global Step: 179535 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:41:01,307-Speed 129.66 samples/sec Loss 0.8443 Epoch: 18 Global Step: 179540 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:41:20,928-Speed 130.48 samples/sec Loss 0.6819 Epoch: 18 Global Step: 179545 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:41:40,823-Speed 128.68 samples/sec Loss 0.7998 Epoch: 18 Global Step: 179550 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:42:00,489-Speed 130.19 samples/sec Loss 0.8092 Epoch: 18 Global Step: 179555 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:42:20,398-Speed 128.59 samples/sec Loss 0.8550 Epoch: 18 Global Step: 179560 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:42:40,128-Speed 129.76 samples/sec Loss 0.7649 Epoch: 18 Global Step: 179565 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:42:59,793-Speed 130.19 samples/sec Loss 0.7395 Epoch: 18 Global Step: 179570 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:43:20,132-Speed 125.87 samples/sec Loss 0.8290 Epoch: 18 Global Step: 179575 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:43:40,347-Speed 126.64 samples/sec Loss 0.8257 Epoch: 18 Global Step: 179580 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:44:00,144-Speed 129.32 samples/sec Loss 0.7850 Epoch: 18 Global Step: 179585 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:44:19,885-Speed 129.68 samples/sec Loss 0.7966 Epoch: 18 Global Step: 179590 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:44:39,716-Speed 129.11 samples/sec Loss 0.8163 Epoch: 18 Global Step: 179595 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:44:59,367-Speed 130.28 samples/sec Loss 0.8050 Epoch: 18 Global Step: 179600 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:45:19,248-Speed 128.77 samples/sec Loss 0.6814 Epoch: 18 Global Step: 179605 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:45:38,981-Speed 129.74 samples/sec Loss 0.8530 Epoch: 18 Global Step: 179610 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:45:58,667-Speed 130.04 samples/sec Loss 0.8901 Epoch: 18 Global Step: 179615 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:46:18,902-Speed 126.52 samples/sec Loss 0.7709 Epoch: 18 Global Step: 179620 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:46:38,892-Speed 128.07 samples/sec Loss 0.8097 Epoch: 18 Global Step: 179625 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:46:58,608-Speed 129.85 samples/sec Loss 0.6815 Epoch: 18 Global Step: 179630 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:47:18,432-Speed 129.15 samples/sec Loss 0.8167 Epoch: 18 Global Step: 179635 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:47:38,176-Speed 129.67 samples/sec Loss 0.7251 Epoch: 18 Global Step: 179640 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:47:57,797-Speed 130.48 samples/sec Loss 0.7328 Epoch: 18 Global Step: 179645 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:48:17,697-Speed 128.65 samples/sec Loss 0.8620 Epoch: 18 Global Step: 179650 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:48:37,443-Speed 129.65 samples/sec Loss 0.7732 Epoch: 18 Global Step: 179655 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:48:57,103-Speed 130.22 samples/sec Loss 0.8533 Epoch: 18 Global Step: 179660 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:49:17,361-Speed 126.38 samples/sec Loss 0.8035 Epoch: 18 Global Step: 179665 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:49:37,279-Speed 128.53 samples/sec Loss 0.8605 Epoch: 18 Global Step: 179670 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:49:57,061-Speed 129.42 samples/sec Loss 0.8198 Epoch: 18 Global Step: 179675 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:50:16,963-Speed 128.64 samples/sec Loss 0.8806 Epoch: 18 Global Step: 179680 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:50:36,808-Speed 129.01 samples/sec Loss 0.7144 Epoch: 18 Global Step: 179685 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:50:56,463-Speed 130.25 samples/sec Loss 0.7775 Epoch: 18 Global Step: 179690 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:51:16,314-Speed 128.97 samples/sec Loss 0.8122 Epoch: 18 Global Step: 179695 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:51:36,126-Speed 129.22 samples/sec Loss 0.7428 Epoch: 18 Global Step: 179700 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:51:55,989-Speed 128.89 samples/sec Loss 0.7624 Epoch: 18 Global Step: 179705 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:52:16,102-Speed 127.29 samples/sec Loss 0.8184 Epoch: 18 Global Step: 179710 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:52:35,990-Speed 128.73 samples/sec Loss 0.7992 Epoch: 18 Global Step: 179715 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:52:55,788-Speed 129.31 samples/sec Loss 0.8602 Epoch: 18 Global Step: 179720 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:53:15,685-Speed 128.67 samples/sec Loss 0.7987 Epoch: 18 Global Step: 179725 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:53:35,462-Speed 129.45 samples/sec Loss 0.8316 Epoch: 18 Global Step: 179730 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:53:55,138-Speed 130.12 samples/sec Loss 0.7599 Epoch: 18 Global Step: 179735 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:54:14,943-Speed 129.27 samples/sec Loss 0.8088 Epoch: 18 Global Step: 179740 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:54:34,741-Speed 129.31 samples/sec Loss 0.7955 Epoch: 18 Global Step: 179745 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:54:54,654-Speed 128.56 samples/sec Loss 0.8139 Epoch: 18 Global Step: 179750 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:55:14,862-Speed 126.69 samples/sec Loss 0.8681 Epoch: 18 Global Step: 179755 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:55:34,650-Speed 129.37 samples/sec Loss 0.8271 Epoch: 18 Global Step: 179760 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:55:54,443-Speed 129.35 samples/sec Loss 0.7854 Epoch: 18 Global Step: 179765 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:56:14,355-Speed 128.57 samples/sec Loss 0.7733 Epoch: 18 Global Step: 179770 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:56:34,011-Speed 130.25 samples/sec Loss 0.8260 Epoch: 18 Global Step: 179775 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:56:53,834-Speed 129.16 samples/sec Loss 0.7487 Epoch: 18 Global Step: 179780 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:57:13,496-Speed 130.21 samples/sec Loss 0.7659 Epoch: 18 Global Step: 179785 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:57:33,333-Speed 129.06 samples/sec Loss 0.7812 Epoch: 18 Global Step: 179790 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:57:53,191-Speed 128.92 samples/sec Loss 0.8813 Epoch: 18 Global Step: 179795 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:58:13,484-Speed 126.16 samples/sec Loss 0.7828 Epoch: 18 Global Step: 179800 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:58:33,171-Speed 130.04 samples/sec Loss 0.8629 Epoch: 18 Global Step: 179805 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:58:52,915-Speed 129.66 samples/sec Loss 0.8024 Epoch: 18 Global Step: 179810 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:59:12,764-Speed 128.98 samples/sec Loss 0.8457 Epoch: 18 Global Step: 179815 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:59:32,479-Speed 129.86 samples/sec Loss 0.7871 Epoch: 18 Global Step: 179820 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 14:59:52,289-Speed 129.23 samples/sec Loss 0.7542 Epoch: 18 Global Step: 179825 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:00:11,919-Speed 130.42 samples/sec Loss 0.8198 Epoch: 18 Global Step: 179830 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:00:32,169-Speed 126.43 samples/sec Loss 0.8278 Epoch: 18 Global Step: 179835 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:01:13,298-Speed 62.25 samples/sec Loss 0.7950 Epoch: 18 Global Step: 179840 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:01:33,668-Speed 125.68 samples/sec Loss 0.8303 Epoch: 18 Global Step: 179845 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:01:53,412-Speed 129.67 samples/sec Loss 0.8397 Epoch: 18 Global Step: 179850 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:02:13,562-Speed 127.05 samples/sec Loss 0.7490 Epoch: 18 Global Step: 179855 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:02:33,293-Speed 129.75 samples/sec Loss 0.8242 Epoch: 18 Global Step: 179860 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:02:53,096-Speed 129.28 samples/sec Loss 0.7822 Epoch: 18 Global Step: 179865 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:03:12,761-Speed 130.19 samples/sec Loss 0.8269 Epoch: 18 Global Step: 179870 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:03:32,731-Speed 128.20 samples/sec Loss 0.8603 Epoch: 18 Global Step: 179875 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:03:52,647-Speed 128.55 samples/sec Loss 0.8134 Epoch: 18 Global Step: 179880 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:04:12,298-Speed 130.28 samples/sec Loss 0.7371 Epoch: 18 Global Step: 179885 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:04:32,459-Speed 126.98 samples/sec Loss 0.8303 Epoch: 18 Global Step: 179890 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:04:52,270-Speed 129.22 samples/sec Loss 0.8654 Epoch: 18 Global Step: 179895 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:05:12,146-Speed 128.80 samples/sec Loss 0.8229 Epoch: 18 Global Step: 179900 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:05:31,879-Speed 129.74 samples/sec Loss 0.8482 Epoch: 18 Global Step: 179905 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:05:51,681-Speed 129.29 samples/sec Loss 0.7296 Epoch: 18 Global Step: 179910 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:06:11,364-Speed 130.07 samples/sec Loss 0.7431 Epoch: 18 Global Step: 179915 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:06:31,223-Speed 128.92 samples/sec Loss 0.8401 Epoch: 18 Global Step: 179920 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:06:51,031-Speed 129.25 samples/sec Loss 0.8160 Epoch: 18 Global Step: 179925 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:07:10,773-Speed 129.68 samples/sec Loss 0.7686 Epoch: 18 Global Step: 179930 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:07:30,954-Speed 126.86 samples/sec Loss 0.7861 Epoch: 18 Global Step: 179935 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:07:50,800-Speed 129.00 samples/sec Loss 0.8476 Epoch: 18 Global Step: 179940 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 15:08:10,732-Speed 128.44 samples/sec Loss 0.6966 Epoch: 18 Global Step: 179945 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 15:08:30,430-Speed 129.97 samples/sec Loss 0.9418 Epoch: 18 Global Step: 179950 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 15:08:50,240-Speed 129.24 samples/sec Loss 0.8121 Epoch: 18 Global Step: 179955 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 15:09:09,960-Speed 129.83 samples/sec Loss 0.7361 Epoch: 18 Global Step: 179960 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 15:09:29,826-Speed 128.87 samples/sec Loss 0.7410 Epoch: 18 Global Step: 179965 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 15:09:49,622-Speed 129.32 samples/sec Loss 0.8581 Epoch: 18 Global Step: 179970 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 15:10:09,333-Speed 129.88 samples/sec Loss 0.8991 Epoch: 18 Global Step: 179975 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 15:10:29,553-Speed 126.61 samples/sec Loss 0.8539 Epoch: 18 Global Step: 179980 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 15:10:49,460-Speed 128.60 samples/sec Loss 0.7607 Epoch: 18 Global Step: 179985 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 15:11:09,239-Speed 129.44 samples/sec Loss 0.7849 Epoch: 18 Global Step: 179990 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 15:11:28,606-Speed 132.19 samples/sec Loss 0.7575 Epoch: 18 Global Step: 179995 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 15:11:48,295-Speed 130.03 samples/sec Loss 0.8362 Epoch: 18 Global Step: 180000 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 15:26:59,526-[lfw][180000]XNorm: 21.950780 Training: 2021-06-15 15:26:59,527-[lfw][180000]Accuracy-Flip: 0.99817+-0.00203 Training: 2021-06-15 15:26:59,527-[lfw][180000]Accuracy-Highest: 0.99833 Training: 2021-06-15 15:44:53,770-[cfp_fp][180000]XNorm: 23.399713 Training: 2021-06-15 15:44:53,770-[cfp_fp][180000]Accuracy-Flip: 0.99214+-0.00273 Training: 2021-06-15 15:44:53,770-[cfp_fp][180000]Accuracy-Highest: 0.99229 Training: 2021-06-15 16:01:35,131-[agedb_30][180000]XNorm: 22.950457 Training: 2021-06-15 16:01:35,131-[agedb_30][180000]Accuracy-Flip: 0.98533+-0.00640 Training: 2021-06-15 16:01:35,131-[agedb_30][180000]Accuracy-Highest: 0.98583 Training: 2021-06-15 16:01:55,146-Speed 0.85 samples/sec Loss 0.8132 Epoch: 18 Global Step: 180005 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:02:14,740-Speed 130.66 samples/sec Loss 0.8751 Epoch: 18 Global Step: 180010 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:02:34,542-Speed 129.29 samples/sec Loss 0.7484 Epoch: 18 Global Step: 180015 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:02:54,156-Speed 130.52 samples/sec Loss 0.8947 Epoch: 18 Global Step: 180020 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:03:18,246-Speed 106.27 samples/sec Loss 0.8864 Epoch: 18 Global Step: 180025 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:03:58,139-Speed 64.20 samples/sec Loss 0.8062 Epoch: 18 Global Step: 180030 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:04:17,923-Speed 129.41 samples/sec Loss 0.8828 Epoch: 18 Global Step: 180035 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:04:37,686-Speed 129.54 samples/sec Loss 0.8747 Epoch: 18 Global Step: 180040 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:04:57,468-Speed 129.42 samples/sec Loss 0.9300 Epoch: 18 Global Step: 180045 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:05:17,224-Speed 129.59 samples/sec Loss 0.8143 Epoch: 18 Global Step: 180050 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:05:37,042-Speed 129.18 samples/sec Loss 0.7653 Epoch: 18 Global Step: 180055 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:05:56,687-Speed 130.32 samples/sec Loss 0.7310 Epoch: 18 Global Step: 180060 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:06:16,651-Speed 128.24 samples/sec Loss 0.8205 Epoch: 18 Global Step: 180065 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:06:36,685-Speed 127.79 samples/sec Loss 0.8303 Epoch: 18 Global Step: 180070 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:06:56,672-Speed 128.09 samples/sec Loss 0.9414 Epoch: 18 Global Step: 180075 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:07:16,425-Speed 129.61 samples/sec Loss 0.7883 Epoch: 18 Global Step: 180080 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:07:36,256-Speed 129.10 samples/sec Loss 0.8405 Epoch: 18 Global Step: 180085 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:07:57,325-Speed 121.51 samples/sec Loss 0.8369 Epoch: 18 Global Step: 180090 Fp16 Grad Scale: 4096 Required: 82 hours Training: 2021-06-15 16:08:39,464-Speed 60.75 samples/sec Loss 0.8239 Epoch: 18 Global Step: 180095 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:08:59,410-Speed 128.35 samples/sec Loss 0.7635 Epoch: 18 Global Step: 180100 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:09:19,215-Speed 129.27 samples/sec Loss 0.7810 Epoch: 18 Global Step: 180105 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:09:49,147-Speed 85.53 samples/sec Loss 0.8022 Epoch: 18 Global Step: 180110 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:10:21,670-Speed 78.72 samples/sec Loss 0.8155 Epoch: 18 Global Step: 180115 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:10:41,829-Speed 127.00 samples/sec Loss 0.7257 Epoch: 18 Global Step: 180120 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:11:19,972-Speed 67.12 samples/sec Loss 0.6844 Epoch: 18 Global Step: 180125 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:11:44,140-Speed 105.93 samples/sec Loss 0.7766 Epoch: 18 Global Step: 180130 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:12:04,073-Speed 128.44 samples/sec Loss 0.8332 Epoch: 18 Global Step: 180135 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:12:36,801-Speed 78.22 samples/sec Loss 0.7848 Epoch: 18 Global Step: 180140 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:13:03,685-Speed 95.26 samples/sec Loss 0.7340 Epoch: 18 Global Step: 180145 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:13:24,846-Speed 120.99 samples/sec Loss 0.8965 Epoch: 18 Global Step: 180150 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:14:03,738-Speed 65.82 samples/sec Loss 0.8546 Epoch: 18 Global Step: 180155 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:14:25,129-Speed 119.68 samples/sec Loss 0.9412 Epoch: 18 Global Step: 180160 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:15:06,427-Speed 61.99 samples/sec Loss 0.7871 Epoch: 18 Global Step: 180165 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:15:26,500-Speed 127.54 samples/sec Loss 0.8455 Epoch: 18 Global Step: 180170 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:15:46,224-Speed 129.80 samples/sec Loss 0.7727 Epoch: 18 Global Step: 180175 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:16:26,616-Speed 63.38 samples/sec Loss 0.8998 Epoch: 18 Global Step: 180180 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:16:47,391-Speed 123.24 samples/sec Loss 0.8765 Epoch: 18 Global Step: 180185 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:17:07,314-Speed 128.50 samples/sec Loss 0.8651 Epoch: 18 Global Step: 180190 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:17:27,076-Speed 129.55 samples/sec Loss 0.8924 Epoch: 18 Global Step: 180195 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:17:47,058-Speed 128.12 samples/sec Loss 0.8016 Epoch: 18 Global Step: 180200 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:18:28,100-Speed 62.38 samples/sec Loss 0.8326 Epoch: 18 Global Step: 180205 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:19:06,417-Speed 66.81 samples/sec Loss 0.8127 Epoch: 18 Global Step: 180210 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:19:32,348-Speed 98.73 samples/sec Loss 0.8737 Epoch: 18 Global Step: 180215 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:19:52,325-Speed 128.16 samples/sec Loss 0.7435 Epoch: 18 Global Step: 180220 Fp16 Grad Scale: 8192 Required: 82 hours Training: 2021-06-15 16:20:12,073-Speed 129.64 samples/sec Loss 0.8168 Epoch: 18 Global Step: 180225 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:20:31,912-Speed 129.05 samples/sec Loss 0.7816 Epoch: 18 Global Step: 180230 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:21:11,685-Speed 64.37 samples/sec Loss 0.7263 Epoch: 18 Global Step: 180235 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:21:31,787-Speed 127.35 samples/sec Loss 0.7469 Epoch: 18 Global Step: 180240 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:21:51,573-Speed 129.40 samples/sec Loss 0.8582 Epoch: 18 Global Step: 180245 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:22:11,671-Speed 127.38 samples/sec Loss 0.7865 Epoch: 18 Global Step: 180250 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:22:31,846-Speed 126.90 samples/sec Loss 0.8481 Epoch: 18 Global Step: 180255 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:22:51,750-Speed 128.62 samples/sec Loss 0.7938 Epoch: 18 Global Step: 180260 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:23:11,423-Speed 130.13 samples/sec Loss 0.8692 Epoch: 18 Global Step: 180265 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:23:31,202-Speed 129.44 samples/sec Loss 0.8291 Epoch: 18 Global Step: 180270 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:23:51,147-Speed 128.36 samples/sec Loss 0.7745 Epoch: 18 Global Step: 180275 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:24:10,770-Speed 130.47 samples/sec Loss 0.8571 Epoch: 18 Global Step: 180280 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:24:30,681-Speed 128.58 samples/sec Loss 0.8510 Epoch: 18 Global Step: 180285 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:24:50,308-Speed 130.44 samples/sec Loss 0.7253 Epoch: 18 Global Step: 180290 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:25:10,451-Speed 127.10 samples/sec Loss 0.8307 Epoch: 18 Global Step: 180295 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:25:30,501-Speed 127.69 samples/sec Loss 0.8240 Epoch: 18 Global Step: 180300 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:25:50,307-Speed 129.26 samples/sec Loss 0.8778 Epoch: 18 Global Step: 180305 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:26:10,442-Speed 127.14 samples/sec Loss 0.8093 Epoch: 18 Global Step: 180310 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:26:40,859-Speed 84.17 samples/sec Loss 0.8916 Epoch: 18 Global Step: 180315 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:27:00,631-Speed 129.48 samples/sec Loss 0.7608 Epoch: 18 Global Step: 180320 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:27:20,347-Speed 129.85 samples/sec Loss 0.8706 Epoch: 18 Global Step: 180325 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:27:40,265-Speed 128.53 samples/sec Loss 0.7472 Epoch: 18 Global Step: 180330 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:27:59,887-Speed 130.47 samples/sec Loss 0.7883 Epoch: 18 Global Step: 180335 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:28:19,985-Speed 127.38 samples/sec Loss 0.8044 Epoch: 18 Global Step: 180340 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:28:40,002-Speed 127.90 samples/sec Loss 0.8003 Epoch: 18 Global Step: 180345 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:28:59,951-Speed 128.34 samples/sec Loss 0.8762 Epoch: 18 Global Step: 180350 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:29:19,690-Speed 129.70 samples/sec Loss 0.7738 Epoch: 18 Global Step: 180355 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:29:43,485-Speed 107.59 samples/sec Loss 0.7864 Epoch: 18 Global Step: 180360 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:30:16,760-Speed 76.94 samples/sec Loss 0.8537 Epoch: 18 Global Step: 180365 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:30:47,959-Speed 82.06 samples/sec Loss 0.8788 Epoch: 18 Global Step: 180370 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:31:20,432-Speed 78.84 samples/sec Loss 0.8360 Epoch: 18 Global Step: 180375 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:31:49,082-Speed 89.36 samples/sec Loss 0.8965 Epoch: 18 Global Step: 180380 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:32:12,611-Speed 108.81 samples/sec Loss 0.8920 Epoch: 18 Global Step: 180385 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:32:47,858-Speed 72.63 samples/sec Loss 0.7731 Epoch: 18 Global Step: 180390 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:33:19,804-Speed 80.14 samples/sec Loss 0.8467 Epoch: 18 Global Step: 180395 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:33:50,867-Speed 82.41 samples/sec Loss 0.7749 Epoch: 18 Global Step: 180400 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:34:19,810-Speed 88.46 samples/sec Loss 0.7710 Epoch: 18 Global Step: 180405 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:34:50,828-Speed 82.54 samples/sec Loss 0.8603 Epoch: 18 Global Step: 180410 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:35:25,084-Speed 74.73 samples/sec Loss 0.8186 Epoch: 18 Global Step: 180415 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:35:59,732-Speed 73.89 samples/sec Loss 0.7775 Epoch: 18 Global Step: 180420 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:36:31,371-Speed 80.92 samples/sec Loss 0.9024 Epoch: 18 Global Step: 180425 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:37:07,186-Speed 71.48 samples/sec Loss 0.8052 Epoch: 18 Global Step: 180430 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:37:36,401-Speed 87.63 samples/sec Loss 0.7921 Epoch: 18 Global Step: 180435 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:38:04,314-Speed 91.75 samples/sec Loss 0.8841 Epoch: 18 Global Step: 180440 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:38:36,173-Speed 80.36 samples/sec Loss 0.8111 Epoch: 18 Global Step: 180445 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:39:10,193-Speed 75.25 samples/sec Loss 0.7756 Epoch: 18 Global Step: 180450 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:39:39,343-Speed 87.83 samples/sec Loss 0.7311 Epoch: 18 Global Step: 180455 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:40:08,036-Speed 89.22 samples/sec Loss 0.7701 Epoch: 18 Global Step: 180460 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:40:40,028-Speed 80.03 samples/sec Loss 0.8922 Epoch: 18 Global Step: 180465 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:41:09,579-Speed 86.64 samples/sec Loss 0.7840 Epoch: 18 Global Step: 180470 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:41:42,632-Speed 77.46 samples/sec Loss 0.8862 Epoch: 18 Global Step: 180475 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:42:14,106-Speed 81.36 samples/sec Loss 0.8006 Epoch: 18 Global Step: 180480 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:42:43,695-Speed 86.52 samples/sec Loss 0.8508 Epoch: 18 Global Step: 180485 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:43:13,649-Speed 85.47 samples/sec Loss 0.8634 Epoch: 18 Global Step: 180490 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:43:41,929-Speed 90.53 samples/sec Loss 0.7268 Epoch: 18 Global Step: 180495 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:44:13,529-Speed 81.02 samples/sec Loss 0.8924 Epoch: 18 Global Step: 180500 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:44:45,456-Speed 80.19 samples/sec Loss 0.8949 Epoch: 18 Global Step: 180505 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:45:17,682-Speed 79.44 samples/sec Loss 0.8185 Epoch: 18 Global Step: 180510 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:45:49,598-Speed 80.22 samples/sec Loss 0.8321 Epoch: 18 Global Step: 180515 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:46:21,266-Speed 80.84 samples/sec Loss 0.7329 Epoch: 18 Global Step: 180520 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:46:42,814-Speed 118.82 samples/sec Loss 0.8651 Epoch: 18 Global Step: 180525 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:47:02,727-Speed 128.56 samples/sec Loss 0.8345 Epoch: 18 Global Step: 180530 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:47:30,224-Speed 93.10 samples/sec Loss 0.7925 Epoch: 18 Global Step: 180535 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:47:54,560-Speed 105.20 samples/sec Loss 0.8073 Epoch: 18 Global Step: 180540 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 16:48:21,420-Speed 95.32 samples/sec Loss 0.8354 Epoch: 18 Global Step: 180545 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:48:56,093-Speed 73.84 samples/sec Loss 0.7785 Epoch: 18 Global Step: 180550 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:49:21,838-Speed 99.45 samples/sec Loss 0.7847 Epoch: 18 Global Step: 180555 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:49:44,023-Speed 115.40 samples/sec Loss 0.8643 Epoch: 18 Global Step: 180560 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:50:30,846-Speed 54.69 samples/sec Loss 0.7824 Epoch: 18 Global Step: 180565 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:51:03,685-Speed 77.96 samples/sec Loss 0.9858 Epoch: 18 Global Step: 180570 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:51:23,551-Speed 128.87 samples/sec Loss 0.8564 Epoch: 18 Global Step: 180575 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:51:43,402-Speed 128.97 samples/sec Loss 0.7875 Epoch: 18 Global Step: 180580 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:52:02,979-Speed 130.78 samples/sec Loss 0.8400 Epoch: 18 Global Step: 180585 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:52:22,628-Speed 130.29 samples/sec Loss 0.7861 Epoch: 18 Global Step: 180590 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:52:44,606-Speed 116.49 samples/sec Loss 0.7325 Epoch: 18 Global Step: 180595 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:53:29,304-Speed 57.28 samples/sec Loss 0.8320 Epoch: 18 Global Step: 180600 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:54:09,705-Speed 63.37 samples/sec Loss 0.7926 Epoch: 18 Global Step: 180605 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:54:52,318-Speed 60.08 samples/sec Loss 0.8375 Epoch: 18 Global Step: 180610 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:55:22,037-Speed 86.14 samples/sec Loss 0.7927 Epoch: 18 Global Step: 180615 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:55:41,786-Speed 129.64 samples/sec Loss 0.7891 Epoch: 18 Global Step: 180620 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:56:26,041-Speed 57.85 samples/sec Loss 0.7060 Epoch: 18 Global Step: 180625 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:57:10,974-Speed 56.98 samples/sec Loss 0.7191 Epoch: 18 Global Step: 180630 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:57:56,642-Speed 56.06 samples/sec Loss 0.7995 Epoch: 18 Global Step: 180635 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:58:44,205-Speed 53.83 samples/sec Loss 0.8591 Epoch: 18 Global Step: 180640 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 16:59:27,740-Speed 58.82 samples/sec Loss 0.8086 Epoch: 18 Global Step: 180645 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:00:07,322-Speed 64.68 samples/sec Loss 0.9235 Epoch: 18 Global Step: 180650 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:00:36,642-Speed 87.32 samples/sec Loss 0.8206 Epoch: 18 Global Step: 180655 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:00:56,427-Speed 129.39 samples/sec Loss 0.9015 Epoch: 18 Global Step: 180660 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:01:21,535-Speed 101.97 samples/sec Loss 0.9107 Epoch: 18 Global Step: 180665 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:02:04,565-Speed 59.50 samples/sec Loss 0.8113 Epoch: 18 Global Step: 180670 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:02:49,379-Speed 57.13 samples/sec Loss 0.7614 Epoch: 18 Global Step: 180675 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:03:29,412-Speed 63.96 samples/sec Loss 0.8643 Epoch: 18 Global Step: 180680 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:04:15,664-Speed 55.35 samples/sec Loss 0.7091 Epoch: 18 Global Step: 180685 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:04:58,803-Speed 59.35 samples/sec Loss 0.8396 Epoch: 18 Global Step: 180690 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:05:38,188-Speed 65.00 samples/sec Loss 0.8813 Epoch: 18 Global Step: 180695 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:06:23,882-Speed 56.03 samples/sec Loss 0.7941 Epoch: 18 Global Step: 180700 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:07:08,654-Speed 57.18 samples/sec Loss 0.8522 Epoch: 18 Global Step: 180705 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:07:51,873-Speed 59.23 samples/sec Loss 0.7735 Epoch: 18 Global Step: 180710 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:08:38,682-Speed 54.69 samples/sec Loss 0.7428 Epoch: 18 Global Step: 180715 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:09:20,606-Speed 61.07 samples/sec Loss 0.8189 Epoch: 18 Global Step: 180720 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:10:03,871-Speed 59.17 samples/sec Loss 0.7859 Epoch: 18 Global Step: 180725 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:10:44,532-Speed 62.96 samples/sec Loss 0.8586 Epoch: 18 Global Step: 180730 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:11:06,955-Speed 114.18 samples/sec Loss 0.7366 Epoch: 18 Global Step: 180735 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:11:26,681-Speed 129.79 samples/sec Loss 0.7742 Epoch: 18 Global Step: 180740 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:11:46,618-Speed 128.41 samples/sec Loss 0.7870 Epoch: 18 Global Step: 180745 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:12:06,547-Speed 128.46 samples/sec Loss 0.7751 Epoch: 18 Global Step: 180750 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:12:26,121-Speed 130.79 samples/sec Loss 0.7644 Epoch: 18 Global Step: 180755 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:12:45,778-Speed 130.24 samples/sec Loss 0.8109 Epoch: 18 Global Step: 180760 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:13:05,506-Speed 129.77 samples/sec Loss 0.9415 Epoch: 18 Global Step: 180765 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:13:25,293-Speed 129.39 samples/sec Loss 0.8578 Epoch: 18 Global Step: 180770 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:13:45,093-Speed 129.30 samples/sec Loss 0.8782 Epoch: 18 Global Step: 180775 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:14:04,822-Speed 129.76 samples/sec Loss 0.8587 Epoch: 18 Global Step: 180780 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:14:24,312-Speed 131.35 samples/sec Loss 0.7683 Epoch: 18 Global Step: 180785 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:14:44,203-Speed 128.71 samples/sec Loss 0.7565 Epoch: 18 Global Step: 180790 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:15:04,055-Speed 128.96 samples/sec Loss 0.7890 Epoch: 18 Global Step: 180795 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:15:28,143-Speed 106.28 samples/sec Loss 0.7599 Epoch: 18 Global Step: 180800 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:16:09,593-Speed 61.76 samples/sec Loss 0.8942 Epoch: 18 Global Step: 180805 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:16:49,788-Speed 63.69 samples/sec Loss 0.8570 Epoch: 18 Global Step: 180810 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:17:33,292-Speed 58.85 samples/sec Loss 0.9367 Epoch: 18 Global Step: 180815 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:18:04,231-Speed 82.75 samples/sec Loss 0.8598 Epoch: 18 Global Step: 180820 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:18:53,076-Speed 52.41 samples/sec Loss 0.9093 Epoch: 18 Global Step: 180825 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:19:22,546-Speed 86.88 samples/sec Loss 0.8678 Epoch: 18 Global Step: 180830 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:19:42,486-Speed 128.39 samples/sec Loss 0.8570 Epoch: 18 Global Step: 180835 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:20:02,369-Speed 128.76 samples/sec Loss 0.7536 Epoch: 18 Global Step: 180840 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:20:21,899-Speed 131.09 samples/sec Loss 0.8171 Epoch: 18 Global Step: 180845 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:20:41,771-Speed 128.83 samples/sec Loss 0.9689 Epoch: 18 Global Step: 180850 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:21:01,314-Speed 131.00 samples/sec Loss 0.9119 Epoch: 18 Global Step: 180855 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:21:21,155-Speed 129.03 samples/sec Loss 0.8789 Epoch: 18 Global Step: 180860 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:21:40,913-Speed 129.57 samples/sec Loss 0.8731 Epoch: 18 Global Step: 180865 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:22:00,826-Speed 128.56 samples/sec Loss 0.8107 Epoch: 18 Global Step: 180870 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:22:22,965-Speed 115.64 samples/sec Loss 0.8249 Epoch: 18 Global Step: 180875 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:22:48,130-Speed 101.79 samples/sec Loss 0.8748 Epoch: 18 Global Step: 180880 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:23:07,932-Speed 129.29 samples/sec Loss 0.7992 Epoch: 18 Global Step: 180885 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:23:27,469-Speed 131.04 samples/sec Loss 1.0126 Epoch: 18 Global Step: 180890 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:23:48,493-Speed 121.77 samples/sec Loss 0.7479 Epoch: 18 Global Step: 180895 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:24:31,153-Speed 60.02 samples/sec Loss 0.8600 Epoch: 18 Global Step: 180900 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:25:13,538-Speed 60.40 samples/sec Loss 0.8018 Epoch: 18 Global Step: 180905 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:25:59,354-Speed 55.88 samples/sec Loss 0.8382 Epoch: 18 Global Step: 180910 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:26:43,027-Speed 58.63 samples/sec Loss 0.8800 Epoch: 18 Global Step: 180915 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:27:22,898-Speed 64.21 samples/sec Loss 0.8496 Epoch: 18 Global Step: 180920 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:28:11,762-Speed 52.39 samples/sec Loss 0.8407 Epoch: 18 Global Step: 180925 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:28:54,224-Speed 60.30 samples/sec Loss 0.8198 Epoch: 18 Global Step: 180930 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:29:38,911-Speed 57.29 samples/sec Loss 0.6927 Epoch: 18 Global Step: 180935 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:30:21,855-Speed 59.61 samples/sec Loss 0.8138 Epoch: 18 Global Step: 180940 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:31:02,028-Speed 63.76 samples/sec Loss 0.8017 Epoch: 18 Global Step: 180945 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:31:44,241-Speed 60.65 samples/sec Loss 0.8895 Epoch: 18 Global Step: 180950 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:32:27,356-Speed 59.38 samples/sec Loss 0.8745 Epoch: 18 Global Step: 180955 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:33:09,296-Speed 61.05 samples/sec Loss 0.7980 Epoch: 18 Global Step: 180960 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:33:46,814-Speed 68.24 samples/sec Loss 0.8376 Epoch: 18 Global Step: 180965 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:34:31,329-Speed 57.51 samples/sec Loss 0.8721 Epoch: 18 Global Step: 180970 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:35:15,031-Speed 58.58 samples/sec Loss 0.7667 Epoch: 18 Global Step: 180975 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:35:55,759-Speed 62.86 samples/sec Loss 0.7796 Epoch: 18 Global Step: 180980 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:36:37,006-Speed 62.08 samples/sec Loss 0.9176 Epoch: 18 Global Step: 180985 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:37:18,512-Speed 61.68 samples/sec Loss 0.8713 Epoch: 18 Global Step: 180990 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:37:58,732-Speed 63.65 samples/sec Loss 0.8380 Epoch: 18 Global Step: 180995 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:38:41,270-Speed 60.18 samples/sec Loss 0.8470 Epoch: 18 Global Step: 181000 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:39:27,672-Speed 55.17 samples/sec Loss 0.7518 Epoch: 18 Global Step: 181005 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:40:10,264-Speed 60.11 samples/sec Loss 0.7429 Epoch: 18 Global Step: 181010 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:40:54,772-Speed 57.53 samples/sec Loss 0.8271 Epoch: 18 Global Step: 181015 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:41:37,827-Speed 59.49 samples/sec Loss 0.8317 Epoch: 18 Global Step: 181020 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:42:19,619-Speed 61.26 samples/sec Loss 0.8094 Epoch: 18 Global Step: 181025 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:43:01,733-Speed 60.81 samples/sec Loss 0.9227 Epoch: 18 Global Step: 181030 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:43:22,027-Speed 126.17 samples/sec Loss 0.8538 Epoch: 18 Global Step: 181035 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:43:41,815-Speed 129.38 samples/sec Loss 0.9057 Epoch: 18 Global Step: 181040 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:44:01,476-Speed 130.22 samples/sec Loss 0.9193 Epoch: 18 Global Step: 181045 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:44:21,354-Speed 128.79 samples/sec Loss 0.7488 Epoch: 18 Global Step: 181050 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:44:40,814-Speed 131.55 samples/sec Loss 0.8758 Epoch: 18 Global Step: 181055 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:45:00,730-Speed 128.55 samples/sec Loss 0.7990 Epoch: 18 Global Step: 181060 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:45:20,327-Speed 130.64 samples/sec Loss 0.8261 Epoch: 18 Global Step: 181065 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:45:40,045-Speed 129.84 samples/sec Loss 0.8101 Epoch: 18 Global Step: 181070 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:45:59,998-Speed 128.31 samples/sec Loss 0.9113 Epoch: 18 Global Step: 181075 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:46:19,431-Speed 131.74 samples/sec Loss 0.9204 Epoch: 18 Global Step: 181080 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:46:39,156-Speed 129.79 samples/sec Loss 0.8755 Epoch: 18 Global Step: 181085 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:47:04,495-Speed 101.04 samples/sec Loss 0.8345 Epoch: 18 Global Step: 181090 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:47:39,361-Speed 73.43 samples/sec Loss 0.9359 Epoch: 18 Global Step: 181095 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:48:13,138-Speed 75.80 samples/sec Loss 0.7030 Epoch: 18 Global Step: 181100 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:48:51,271-Speed 67.14 samples/sec Loss 0.7919 Epoch: 18 Global Step: 181105 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:49:22,398-Speed 82.25 samples/sec Loss 0.8558 Epoch: 18 Global Step: 181110 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:49:56,034-Speed 76.11 samples/sec Loss 0.7869 Epoch: 18 Global Step: 181115 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:50:32,359-Speed 70.48 samples/sec Loss 0.8662 Epoch: 18 Global Step: 181120 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 17:51:04,391-Speed 79.92 samples/sec Loss 0.8889 Epoch: 18 Global Step: 181125 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:51:32,733-Speed 90.33 samples/sec Loss 0.8451 Epoch: 18 Global Step: 181130 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:52:05,343-Speed 78.51 samples/sec Loss 0.7739 Epoch: 18 Global Step: 181135 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:52:37,142-Speed 80.51 samples/sec Loss 0.8426 Epoch: 18 Global Step: 181140 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:53:07,041-Speed 85.63 samples/sec Loss 0.8459 Epoch: 18 Global Step: 181145 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:53:37,081-Speed 85.23 samples/sec Loss 0.7835 Epoch: 18 Global Step: 181150 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:54:11,704-Speed 73.94 samples/sec Loss 0.9171 Epoch: 18 Global Step: 181155 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:54:44,925-Speed 77.06 samples/sec Loss 0.7290 Epoch: 18 Global Step: 181160 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:55:17,472-Speed 78.66 samples/sec Loss 0.7700 Epoch: 18 Global Step: 181165 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:55:45,629-Speed 90.93 samples/sec Loss 0.6813 Epoch: 18 Global Step: 181170 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:56:15,244-Speed 86.45 samples/sec Loss 0.8310 Epoch: 18 Global Step: 181175 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:56:47,570-Speed 79.20 samples/sec Loss 0.8552 Epoch: 18 Global Step: 181180 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:57:20,221-Speed 78.41 samples/sec Loss 0.7473 Epoch: 18 Global Step: 181185 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:57:51,379-Speed 82.17 samples/sec Loss 0.7738 Epoch: 18 Global Step: 181190 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:58:22,790-Speed 81.50 samples/sec Loss 0.9088 Epoch: 18 Global Step: 181195 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:58:54,128-Speed 81.69 samples/sec Loss 0.9765 Epoch: 18 Global Step: 181200 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:59:24,013-Speed 85.67 samples/sec Loss 0.8193 Epoch: 18 Global Step: 181205 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 17:59:54,564-Speed 83.80 samples/sec Loss 0.7216 Epoch: 18 Global Step: 181210 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 18:00:30,206-Speed 71.83 samples/sec Loss 0.8667 Epoch: 18 Global Step: 181215 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 18:01:01,451-Speed 81.94 samples/sec Loss 0.8363 Epoch: 18 Global Step: 181220 Fp16 Grad Scale: 4096 Required: 81 hours Training: 2021-06-15 18:01:29,971-Speed 89.76 samples/sec Loss 0.8689 Epoch: 18 Global Step: 181225 Fp16 Grad Scale: 8192 Required: 81 hours Training: 2021-06-15 18:01:58,456-Speed 89.88 samples/sec Loss 0.8509 Epoch: 18 Global Step: 181230 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:02:31,215-Speed 78.15 samples/sec Loss 0.8406 Epoch: 18 Global Step: 181235 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:03:02,639-Speed 81.47 samples/sec Loss 0.8027 Epoch: 18 Global Step: 181240 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:03:32,370-Speed 86.12 samples/sec Loss 0.7964 Epoch: 18 Global Step: 181245 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:04:00,425-Speed 91.25 samples/sec Loss 0.7070 Epoch: 18 Global Step: 181250 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:04:29,840-Speed 87.03 samples/sec Loss 0.7413 Epoch: 18 Global Step: 181255 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:05:03,675-Speed 75.67 samples/sec Loss 0.8908 Epoch: 18 Global Step: 181260 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:05:36,570-Speed 77.83 samples/sec Loss 0.8592 Epoch: 18 Global Step: 181265 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:06:08,922-Speed 79.13 samples/sec Loss 0.7986 Epoch: 18 Global Step: 181270 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:06:41,455-Speed 78.69 samples/sec Loss 0.8771 Epoch: 18 Global Step: 181275 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:07:15,508-Speed 75.18 samples/sec Loss 0.7421 Epoch: 18 Global Step: 181280 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:07:46,725-Speed 82.01 samples/sec Loss 0.8727 Epoch: 18 Global Step: 181285 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:08:19,511-Speed 78.09 samples/sec Loss 0.8377 Epoch: 18 Global Step: 181290 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:08:54,542-Speed 73.08 samples/sec Loss 0.9995 Epoch: 18 Global Step: 181295 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:09:25,771-Speed 81.98 samples/sec Loss 0.7608 Epoch: 18 Global Step: 181300 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:10:02,263-Speed 70.16 samples/sec Loss 0.8580 Epoch: 18 Global Step: 181305 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:10:33,552-Speed 81.82 samples/sec Loss 0.8005 Epoch: 18 Global Step: 181310 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:11:07,468-Speed 75.48 samples/sec Loss 0.8940 Epoch: 18 Global Step: 181315 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:11:38,725-Speed 81.91 samples/sec Loss 0.9425 Epoch: 18 Global Step: 181320 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:12:07,898-Speed 87.75 samples/sec Loss 0.8978 Epoch: 18 Global Step: 181325 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:12:38,690-Speed 83.14 samples/sec Loss 0.8044 Epoch: 18 Global Step: 181330 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:13:10,055-Speed 81.63 samples/sec Loss 0.8797 Epoch: 18 Global Step: 181335 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:13:42,329-Speed 79.33 samples/sec Loss 0.7504 Epoch: 18 Global Step: 181340 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:14:13,118-Speed 83.15 samples/sec Loss 0.8536 Epoch: 18 Global Step: 181345 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:14:44,332-Speed 82.02 samples/sec Loss 0.8000 Epoch: 18 Global Step: 181350 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:15:18,247-Speed 75.49 samples/sec Loss 0.8382 Epoch: 18 Global Step: 181355 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:15:48,732-Speed 83.98 samples/sec Loss 0.7146 Epoch: 18 Global Step: 181360 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:16:18,596-Speed 85.73 samples/sec Loss 0.8465 Epoch: 18 Global Step: 181365 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:16:51,045-Speed 78.90 samples/sec Loss 0.8014 Epoch: 18 Global Step: 181370 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:17:21,597-Speed 83.79 samples/sec Loss 0.7470 Epoch: 18 Global Step: 181375 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:17:57,061-Speed 72.19 samples/sec Loss 0.7599 Epoch: 18 Global Step: 181380 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:18:26,467-Speed 87.06 samples/sec Loss 0.7620 Epoch: 18 Global Step: 181385 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:18:57,768-Speed 81.79 samples/sec Loss 0.8131 Epoch: 18 Global Step: 181390 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:19:27,717-Speed 85.48 samples/sec Loss 0.8825 Epoch: 18 Global Step: 181395 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:19:58,682-Speed 82.68 samples/sec Loss 0.7802 Epoch: 18 Global Step: 181400 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:20:29,218-Speed 83.84 samples/sec Loss 0.8000 Epoch: 18 Global Step: 181405 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:20:57,208-Speed 91.50 samples/sec Loss 0.7441 Epoch: 18 Global Step: 181410 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:21:28,716-Speed 81.26 samples/sec Loss 0.9184 Epoch: 18 Global Step: 181415 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:21:59,118-Speed 84.21 samples/sec Loss 0.7492 Epoch: 18 Global Step: 181420 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:22:51,142-Speed 49.21 samples/sec Loss 0.6886 Epoch: 18 Global Step: 181425 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:23:26,602-Speed 72.20 samples/sec Loss 0.8025 Epoch: 18 Global Step: 181430 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:23:58,123-Speed 81.22 samples/sec Loss 0.8682 Epoch: 18 Global Step: 181435 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:24:32,362-Speed 74.77 samples/sec Loss 0.8856 Epoch: 18 Global Step: 181440 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:25:06,246-Speed 75.55 samples/sec Loss 0.8004 Epoch: 18 Global Step: 181445 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:25:38,887-Speed 78.46 samples/sec Loss 0.8126 Epoch: 18 Global Step: 181450 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:26:10,445-Speed 81.12 samples/sec Loss 0.8566 Epoch: 18 Global Step: 181455 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:26:38,907-Speed 89.95 samples/sec Loss 0.9005 Epoch: 18 Global Step: 181460 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:27:14,353-Speed 72.22 samples/sec Loss 0.7928 Epoch: 18 Global Step: 181465 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:27:45,908-Speed 81.13 samples/sec Loss 0.7984 Epoch: 18 Global Step: 181470 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:28:19,034-Speed 77.29 samples/sec Loss 0.7780 Epoch: 18 Global Step: 181475 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:28:50,743-Speed 80.74 samples/sec Loss 0.8617 Epoch: 18 Global Step: 181480 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:29:20,515-Speed 85.99 samples/sec Loss 0.7464 Epoch: 18 Global Step: 181485 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:29:53,310-Speed 78.07 samples/sec Loss 0.8284 Epoch: 18 Global Step: 181490 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:30:29,294-Speed 71.16 samples/sec Loss 0.8706 Epoch: 18 Global Step: 181495 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:31:01,795-Speed 78.77 samples/sec Loss 0.7030 Epoch: 18 Global Step: 181500 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:31:31,174-Speed 87.14 samples/sec Loss 0.7772 Epoch: 18 Global Step: 181505 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:32:01,378-Speed 84.76 samples/sec Loss 0.8369 Epoch: 18 Global Step: 181510 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:32:28,704-Speed 93.69 samples/sec Loss 0.7057 Epoch: 18 Global Step: 181515 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 18:33:02,884-Speed 74.90 samples/sec Loss 0.8548 Epoch: 18 Global Step: 181520 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:33:32,589-Speed 86.19 samples/sec Loss 0.7509 Epoch: 18 Global Step: 181525 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:34:02,753-Speed 84.88 samples/sec Loss 0.8607 Epoch: 18 Global Step: 181530 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:34:31,779-Speed 88.20 samples/sec Loss 0.7941 Epoch: 18 Global Step: 181535 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:35:03,707-Speed 80.19 samples/sec Loss 0.9323 Epoch: 18 Global Step: 181540 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:35:36,787-Speed 77.39 samples/sec Loss 0.8012 Epoch: 18 Global Step: 181545 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:36:06,825-Speed 85.23 samples/sec Loss 0.7845 Epoch: 18 Global Step: 181550 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:36:38,179-Speed 81.65 samples/sec Loss 0.9366 Epoch: 18 Global Step: 181555 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:37:07,816-Speed 86.38 samples/sec Loss 0.6958 Epoch: 18 Global Step: 181560 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:37:35,007-Speed 94.16 samples/sec Loss 0.8229 Epoch: 18 Global Step: 181565 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:38:05,717-Speed 83.37 samples/sec Loss 0.7927 Epoch: 18 Global Step: 181570 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:38:38,456-Speed 78.22 samples/sec Loss 0.8672 Epoch: 18 Global Step: 181575 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:39:12,534-Speed 75.13 samples/sec Loss 0.9425 Epoch: 18 Global Step: 181580 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:39:44,486-Speed 80.12 samples/sec Loss 0.8686 Epoch: 18 Global Step: 181585 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:40:15,495-Speed 82.56 samples/sec Loss 0.8075 Epoch: 18 Global Step: 181590 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:40:50,735-Speed 72.65 samples/sec Loss 0.8909 Epoch: 18 Global Step: 181595 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:41:25,949-Speed 72.70 samples/sec Loss 0.7741 Epoch: 18 Global Step: 181600 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:41:59,998-Speed 75.19 samples/sec Loss 0.8507 Epoch: 18 Global Step: 181605 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:42:32,526-Speed 78.71 samples/sec Loss 0.8524 Epoch: 18 Global Step: 181610 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:43:05,075-Speed 78.65 samples/sec Loss 0.8266 Epoch: 18 Global Step: 181615 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:43:33,391-Speed 90.42 samples/sec Loss 0.8853 Epoch: 18 Global Step: 181620 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:44:02,791-Speed 87.08 samples/sec Loss 0.8103 Epoch: 18 Global Step: 181625 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:44:38,238-Speed 72.22 samples/sec Loss 0.8325 Epoch: 18 Global Step: 181630 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:45:09,096-Speed 82.96 samples/sec Loss 0.8216 Epoch: 18 Global Step: 181635 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:45:40,741-Speed 80.90 samples/sec Loss 0.9662 Epoch: 18 Global Step: 181640 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:46:10,968-Speed 84.70 samples/sec Loss 0.7874 Epoch: 18 Global Step: 181645 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:46:42,407-Speed 81.43 samples/sec Loss 0.8448 Epoch: 18 Global Step: 181650 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:47:22,616-Speed 63.67 samples/sec Loss 0.7618 Epoch: 18 Global Step: 181655 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:47:57,958-Speed 72.44 samples/sec Loss 0.7133 Epoch: 18 Global Step: 181660 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:48:31,758-Speed 75.74 samples/sec Loss 0.8141 Epoch: 18 Global Step: 181665 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:49:06,763-Speed 73.14 samples/sec Loss 0.8483 Epoch: 18 Global Step: 181670 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:49:40,534-Speed 75.81 samples/sec Loss 0.9225 Epoch: 18 Global Step: 181675 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:50:12,341-Speed 80.49 samples/sec Loss 0.8630 Epoch: 18 Global Step: 181680 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:50:47,946-Speed 71.90 samples/sec Loss 0.8353 Epoch: 18 Global Step: 181685 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:51:21,308-Speed 76.74 samples/sec Loss 0.8354 Epoch: 18 Global Step: 181690 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:51:52,413-Speed 82.31 samples/sec Loss 0.8372 Epoch: 18 Global Step: 181695 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:52:22,880-Speed 84.03 samples/sec Loss 0.8775 Epoch: 18 Global Step: 181700 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:52:52,252-Speed 87.16 samples/sec Loss 0.8319 Epoch: 18 Global Step: 181705 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:53:23,130-Speed 82.91 samples/sec Loss 0.8511 Epoch: 18 Global Step: 181710 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:53:56,015-Speed 77.85 samples/sec Loss 0.8050 Epoch: 18 Global Step: 181715 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:54:28,106-Speed 79.78 samples/sec Loss 0.7527 Epoch: 18 Global Step: 181720 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:55:00,633-Speed 78.71 samples/sec Loss 0.8005 Epoch: 18 Global Step: 181725 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:55:37,418-Speed 69.60 samples/sec Loss 0.8664 Epoch: 18 Global Step: 181730 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:56:09,567-Speed 79.63 samples/sec Loss 0.7855 Epoch: 18 Global Step: 181735 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:56:42,193-Speed 78.47 samples/sec Loss 0.8514 Epoch: 18 Global Step: 181740 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:57:12,787-Speed 83.68 samples/sec Loss 0.8242 Epoch: 18 Global Step: 181745 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:57:44,348-Speed 81.12 samples/sec Loss 0.8465 Epoch: 18 Global Step: 181750 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:58:15,873-Speed 81.21 samples/sec Loss 0.8504 Epoch: 18 Global Step: 181755 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:58:49,715-Speed 75.65 samples/sec Loss 0.7916 Epoch: 18 Global Step: 181760 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:59:23,467-Speed 75.85 samples/sec Loss 0.7428 Epoch: 18 Global Step: 181765 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 18:59:56,161-Speed 78.30 samples/sec Loss 0.7953 Epoch: 18 Global Step: 181770 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:00:25,627-Speed 86.89 samples/sec Loss 0.8967 Epoch: 18 Global Step: 181775 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:00:55,318-Speed 86.23 samples/sec Loss 0.9218 Epoch: 18 Global Step: 181780 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:01:26,312-Speed 82.60 samples/sec Loss 0.9395 Epoch: 18 Global Step: 181785 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:01:58,851-Speed 78.68 samples/sec Loss 0.8915 Epoch: 18 Global Step: 181790 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:02:28,753-Speed 85.62 samples/sec Loss 0.8508 Epoch: 18 Global Step: 181795 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:02:57,642-Speed 88.62 samples/sec Loss 0.7976 Epoch: 18 Global Step: 181800 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:03:31,237-Speed 76.21 samples/sec Loss 0.7580 Epoch: 18 Global Step: 181805 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:04:05,118-Speed 75.56 samples/sec Loss 0.8527 Epoch: 18 Global Step: 181810 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:04:35,672-Speed 83.79 samples/sec Loss 0.8288 Epoch: 18 Global Step: 181815 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:05:04,974-Speed 87.37 samples/sec Loss 0.7822 Epoch: 18 Global Step: 181820 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:05:34,481-Speed 86.76 samples/sec Loss 0.9294 Epoch: 18 Global Step: 181825 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:06:06,276-Speed 80.52 samples/sec Loss 0.9468 Epoch: 18 Global Step: 181830 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:06:34,785-Speed 89.80 samples/sec Loss 0.8295 Epoch: 18 Global Step: 181835 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:07:05,618-Speed 83.03 samples/sec Loss 0.7776 Epoch: 18 Global Step: 181840 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:07:35,256-Speed 86.38 samples/sec Loss 0.8230 Epoch: 18 Global Step: 181845 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:08:05,925-Speed 83.48 samples/sec Loss 0.8464 Epoch: 18 Global Step: 181850 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:08:34,692-Speed 89.00 samples/sec Loss 0.8456 Epoch: 18 Global Step: 181855 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:09:03,949-Speed 87.50 samples/sec Loss 0.7755 Epoch: 18 Global Step: 181860 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:09:35,284-Speed 81.70 samples/sec Loss 0.8120 Epoch: 18 Global Step: 181865 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:10:07,769-Speed 78.81 samples/sec Loss 0.8187 Epoch: 18 Global Step: 181870 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:10:40,052-Speed 79.30 samples/sec Loss 0.8426 Epoch: 18 Global Step: 181875 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:11:09,878-Speed 85.84 samples/sec Loss 0.7819 Epoch: 18 Global Step: 181880 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:11:40,743-Speed 82.95 samples/sec Loss 0.8031 Epoch: 18 Global Step: 181885 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:12:10,467-Speed 86.13 samples/sec Loss 0.8300 Epoch: 18 Global Step: 181890 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:12:43,366-Speed 77.82 samples/sec Loss 0.8622 Epoch: 18 Global Step: 181895 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:13:17,781-Speed 74.39 samples/sec Loss 0.8473 Epoch: 18 Global Step: 181900 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:13:49,155-Speed 81.60 samples/sec Loss 0.8556 Epoch: 18 Global Step: 181905 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:14:22,291-Speed 77.26 samples/sec Loss 0.9431 Epoch: 18 Global Step: 181910 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:14:52,567-Speed 84.56 samples/sec Loss 0.7973 Epoch: 18 Global Step: 181915 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:15:24,861-Speed 79.28 samples/sec Loss 0.7978 Epoch: 18 Global Step: 181920 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:15:58,199-Speed 76.80 samples/sec Loss 0.8707 Epoch: 18 Global Step: 181925 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:16:29,236-Speed 82.48 samples/sec Loss 0.7265 Epoch: 18 Global Step: 181930 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:17:01,297-Speed 79.85 samples/sec Loss 0.8998 Epoch: 18 Global Step: 181935 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:17:34,783-Speed 76.45 samples/sec Loss 0.8590 Epoch: 18 Global Step: 181940 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:18:17,857-Speed 59.44 samples/sec Loss 0.8688 Epoch: 18 Global Step: 181945 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:18:50,858-Speed 77.58 samples/sec Loss 0.8874 Epoch: 18 Global Step: 181950 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:19:21,445-Speed 83.70 samples/sec Loss 0.8013 Epoch: 18 Global Step: 181955 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:19:50,592-Speed 87.84 samples/sec Loss 0.8927 Epoch: 18 Global Step: 181960 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:20:20,347-Speed 86.04 samples/sec Loss 0.7707 Epoch: 18 Global Step: 181965 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:20:53,429-Speed 77.39 samples/sec Loss 0.9203 Epoch: 18 Global Step: 181970 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:21:24,239-Speed 83.10 samples/sec Loss 0.8735 Epoch: 18 Global Step: 181975 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:22:08,127-Speed 58.33 samples/sec Loss 0.8945 Epoch: 18 Global Step: 181980 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:23:08,917-Speed 42.12 samples/sec Loss 0.8715 Epoch: 18 Global Step: 181985 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:23:39,406-Speed 83.97 samples/sec Loss 0.7449 Epoch: 18 Global Step: 181990 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:24:09,758-Speed 84.35 samples/sec Loss 0.7503 Epoch: 18 Global Step: 181995 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:24:45,194-Speed 72.25 samples/sec Loss 0.8036 Epoch: 18 Global Step: 182000 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:25:13,746-Speed 89.67 samples/sec Loss 0.8416 Epoch: 18 Global Step: 182005 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:25:51,795-Speed 67.28 samples/sec Loss 0.7982 Epoch: 18 Global Step: 182010 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:26:36,084-Speed 57.82 samples/sec Loss 0.8500 Epoch: 18 Global Step: 182015 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:27:07,243-Speed 82.17 samples/sec Loss 0.8527 Epoch: 18 Global Step: 182020 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:27:44,248-Speed 69.18 samples/sec Loss 0.8247 Epoch: 18 Global Step: 182025 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:28:21,171-Speed 69.34 samples/sec Loss 0.8278 Epoch: 18 Global Step: 182030 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:29:20,162-Speed 43.40 samples/sec Loss 0.7888 Epoch: 18 Global Step: 182035 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:30:22,489-Speed 41.07 samples/sec Loss 0.8592 Epoch: 18 Global Step: 182040 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:31:28,337-Speed 38.88 samples/sec Loss 0.8468 Epoch: 18 Global Step: 182045 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:32:39,707-Speed 35.87 samples/sec Loss 0.8796 Epoch: 18 Global Step: 182050 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:33:47,708-Speed 37.66 samples/sec Loss 0.9199 Epoch: 18 Global Step: 182055 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:34:52,093-Speed 39.76 samples/sec Loss 0.8215 Epoch: 18 Global Step: 182060 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:35:53,966-Speed 41.38 samples/sec Loss 0.8610 Epoch: 18 Global Step: 182065 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:37:00,580-Speed 38.43 samples/sec Loss 0.8233 Epoch: 18 Global Step: 182070 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:38:08,722-Speed 37.57 samples/sec Loss 0.8550 Epoch: 18 Global Step: 182075 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:39:17,021-Speed 37.48 samples/sec Loss 0.7917 Epoch: 18 Global Step: 182080 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:40:29,966-Speed 35.10 samples/sec Loss 0.8153 Epoch: 18 Global Step: 182085 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:41:34,489-Speed 39.68 samples/sec Loss 0.9373 Epoch: 18 Global Step: 182090 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:42:33,260-Speed 43.56 samples/sec Loss 0.7908 Epoch: 18 Global Step: 182095 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:43:40,914-Speed 37.84 samples/sec Loss 0.8421 Epoch: 18 Global Step: 182100 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:44:47,259-Speed 38.59 samples/sec Loss 0.7636 Epoch: 18 Global Step: 182105 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:45:56,319-Speed 37.07 samples/sec Loss 0.8037 Epoch: 18 Global Step: 182110 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:47:03,786-Speed 37.95 samples/sec Loss 0.8622 Epoch: 18 Global Step: 182115 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:48:05,214-Speed 41.68 samples/sec Loss 0.8408 Epoch: 18 Global Step: 182120 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:49:07,943-Speed 40.82 samples/sec Loss 0.7199 Epoch: 18 Global Step: 182125 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:50:14,535-Speed 38.45 samples/sec Loss 0.9799 Epoch: 18 Global Step: 182130 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:51:15,968-Speed 41.68 samples/sec Loss 0.8728 Epoch: 18 Global Step: 182135 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:52:25,761-Speed 36.68 samples/sec Loss 0.9163 Epoch: 18 Global Step: 182140 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:53:33,353-Speed 37.88 samples/sec Loss 0.9975 Epoch: 18 Global Step: 182145 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 19:54:23,032-Speed 51.53 samples/sec Loss 0.8569 Epoch: 18 Global Step: 182150 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:54:57,679-Speed 73.89 samples/sec Loss 0.9507 Epoch: 18 Global Step: 182155 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:55:25,115-Speed 93.32 samples/sec Loss 0.8515 Epoch: 18 Global Step: 182160 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:55:55,917-Speed 83.12 samples/sec Loss 0.8449 Epoch: 18 Global Step: 182165 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:56:25,776-Speed 85.74 samples/sec Loss 0.8960 Epoch: 18 Global Step: 182170 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:56:56,205-Speed 84.13 samples/sec Loss 0.7561 Epoch: 18 Global Step: 182175 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:57:24,804-Speed 89.53 samples/sec Loss 0.8662 Epoch: 18 Global Step: 182180 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:57:55,148-Speed 84.37 samples/sec Loss 0.8112 Epoch: 18 Global Step: 182185 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:58:25,342-Speed 84.79 samples/sec Loss 0.8304 Epoch: 18 Global Step: 182190 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:58:54,965-Speed 86.43 samples/sec Loss 0.7728 Epoch: 18 Global Step: 182195 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:59:23,294-Speed 90.37 samples/sec Loss 0.8393 Epoch: 18 Global Step: 182200 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 19:59:53,522-Speed 84.69 samples/sec Loss 0.8625 Epoch: 18 Global Step: 182205 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 20:00:24,273-Speed 83.26 samples/sec Loss 0.8578 Epoch: 18 Global Step: 182210 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 20:00:54,968-Speed 83.43 samples/sec Loss 0.8985 Epoch: 18 Global Step: 182215 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 20:01:26,716-Speed 80.64 samples/sec Loss 0.8226 Epoch: 18 Global Step: 182220 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 20:01:59,575-Speed 77.91 samples/sec Loss 0.8459 Epoch: 18 Global Step: 182225 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 20:02:35,197-Speed 71.87 samples/sec Loss 0.7861 Epoch: 18 Global Step: 182230 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 20:03:06,740-Speed 81.16 samples/sec Loss 0.8980 Epoch: 18 Global Step: 182235 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 20:03:41,735-Speed 73.16 samples/sec Loss 0.8353 Epoch: 18 Global Step: 182240 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 20:04:13,353-Speed 80.97 samples/sec Loss 0.8857 Epoch: 18 Global Step: 182245 Fp16 Grad Scale: 4096 Required: 80 hours Training: 2021-06-15 20:04:44,916-Speed 81.11 samples/sec Loss 0.8835 Epoch: 18 Global Step: 182250 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:05:20,020-Speed 72.93 samples/sec Loss 0.7234 Epoch: 18 Global Step: 182255 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:05:52,375-Speed 79.13 samples/sec Loss 0.8948 Epoch: 18 Global Step: 182260 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:06:24,991-Speed 78.52 samples/sec Loss 0.8142 Epoch: 18 Global Step: 182265 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:06:55,184-Speed 84.79 samples/sec Loss 0.8003 Epoch: 18 Global Step: 182270 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:07:26,593-Speed 81.51 samples/sec Loss 0.8364 Epoch: 18 Global Step: 182275 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:08:00,847-Speed 74.74 samples/sec Loss 0.7712 Epoch: 18 Global Step: 182280 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:08:33,681-Speed 77.97 samples/sec Loss 0.7824 Epoch: 18 Global Step: 182285 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:09:07,025-Speed 76.78 samples/sec Loss 0.9016 Epoch: 18 Global Step: 182290 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:09:35,303-Speed 90.54 samples/sec Loss 0.8936 Epoch: 18 Global Step: 182295 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:10:06,481-Speed 82.11 samples/sec Loss 0.8557 Epoch: 18 Global Step: 182300 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:10:36,929-Speed 84.08 samples/sec Loss 0.8823 Epoch: 18 Global Step: 182305 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:11:08,106-Speed 82.12 samples/sec Loss 0.8546 Epoch: 18 Global Step: 182310 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:11:41,199-Speed 77.36 samples/sec Loss 0.7719 Epoch: 18 Global Step: 182315 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:12:12,685-Speed 81.31 samples/sec Loss 0.7731 Epoch: 18 Global Step: 182320 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:12:43,031-Speed 84.37 samples/sec Loss 0.8861 Epoch: 18 Global Step: 182325 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:13:24,021-Speed 62.46 samples/sec Loss 0.8240 Epoch: 18 Global Step: 182330 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:13:55,938-Speed 80.21 samples/sec Loss 0.8556 Epoch: 18 Global Step: 182335 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:14:26,697-Speed 83.23 samples/sec Loss 0.8745 Epoch: 18 Global Step: 182340 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:15:01,948-Speed 72.63 samples/sec Loss 0.8918 Epoch: 18 Global Step: 182345 Fp16 Grad Scale: 8192 Required: 80 hours Training: 2021-06-15 20:15:32,875-Speed 82.78 samples/sec Loss 0.8594 Epoch: 18 Global Step: 182350 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:16:01,837-Speed 88.40 samples/sec Loss 0.8562 Epoch: 18 Global Step: 182355 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:16:31,112-Speed 87.45 samples/sec Loss 0.8895 Epoch: 18 Global Step: 182360 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:17:05,293-Speed 74.90 samples/sec Loss 0.7935 Epoch: 18 Global Step: 182365 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:17:38,484-Speed 77.13 samples/sec Loss 0.8811 Epoch: 18 Global Step: 182370 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:18:06,196-Speed 92.38 samples/sec Loss 0.8935 Epoch: 18 Global Step: 182375 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:18:37,042-Speed 83.00 samples/sec Loss 0.8458 Epoch: 18 Global Step: 182380 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:19:07,633-Speed 83.69 samples/sec Loss 0.7824 Epoch: 18 Global Step: 182385 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:19:39,759-Speed 79.69 samples/sec Loss 0.9164 Epoch: 18 Global Step: 182390 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:20:13,230-Speed 76.49 samples/sec Loss 0.8352 Epoch: 18 Global Step: 182395 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:20:44,299-Speed 82.40 samples/sec Loss 0.8919 Epoch: 18 Global Step: 182400 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:21:16,580-Speed 79.31 samples/sec Loss 0.8653 Epoch: 18 Global Step: 182405 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:21:48,095-Speed 81.24 samples/sec Loss 0.7044 Epoch: 18 Global Step: 182410 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:22:18,594-Speed 83.94 samples/sec Loss 0.8781 Epoch: 18 Global Step: 182415 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:22:53,414-Speed 73.52 samples/sec Loss 0.8234 Epoch: 18 Global Step: 182420 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:23:26,802-Speed 76.68 samples/sec Loss 0.8466 Epoch: 18 Global Step: 182425 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:23:59,764-Speed 77.67 samples/sec Loss 0.8047 Epoch: 18 Global Step: 182430 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:24:31,004-Speed 81.95 samples/sec Loss 0.7703 Epoch: 18 Global Step: 182435 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:25:04,205-Speed 77.11 samples/sec Loss 0.8039 Epoch: 18 Global Step: 182440 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:25:37,859-Speed 76.07 samples/sec Loss 0.7830 Epoch: 18 Global Step: 182445 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:26:10,861-Speed 77.58 samples/sec Loss 0.7610 Epoch: 18 Global Step: 182450 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:26:43,238-Speed 79.07 samples/sec Loss 0.8595 Epoch: 18 Global Step: 182455 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:27:15,320-Speed 79.80 samples/sec Loss 0.8420 Epoch: 18 Global Step: 182460 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:27:50,057-Speed 73.70 samples/sec Loss 0.7583 Epoch: 18 Global Step: 182465 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:28:17,650-Speed 92.78 samples/sec Loss 0.8607 Epoch: 18 Global Step: 182470 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:28:55,850-Speed 67.02 samples/sec Loss 0.8234 Epoch: 18 Global Step: 182475 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:29:26,207-Speed 84.33 samples/sec Loss 0.8729 Epoch: 18 Global Step: 182480 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:29:56,161-Speed 85.47 samples/sec Loss 0.8742 Epoch: 18 Global Step: 182485 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:30:29,363-Speed 77.11 samples/sec Loss 0.8354 Epoch: 18 Global Step: 182490 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:31:00,942-Speed 81.07 samples/sec Loss 0.9583 Epoch: 18 Global Step: 182495 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:31:34,939-Speed 75.31 samples/sec Loss 0.7731 Epoch: 18 Global Step: 182500 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:32:03,413-Speed 89.92 samples/sec Loss 0.8100 Epoch: 18 Global Step: 182505 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:32:35,324-Speed 80.23 samples/sec Loss 0.8441 Epoch: 18 Global Step: 182510 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:33:09,127-Speed 75.74 samples/sec Loss 0.7974 Epoch: 18 Global Step: 182515 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:33:37,525-Speed 90.15 samples/sec Loss 0.9369 Epoch: 18 Global Step: 182520 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:34:07,988-Speed 84.04 samples/sec Loss 0.8991 Epoch: 18 Global Step: 182525 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:34:36,283-Speed 90.48 samples/sec Loss 0.7500 Epoch: 18 Global Step: 182530 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:35:08,034-Speed 80.64 samples/sec Loss 0.7824 Epoch: 18 Global Step: 182535 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:35:43,861-Speed 71.46 samples/sec Loss 0.8824 Epoch: 18 Global Step: 182540 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:36:19,483-Speed 71.87 samples/sec Loss 0.8211 Epoch: 18 Global Step: 182545 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:36:49,716-Speed 84.68 samples/sec Loss 0.8266 Epoch: 18 Global Step: 182550 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:37:18,851-Speed 87.87 samples/sec Loss 0.9151 Epoch: 18 Global Step: 182555 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:37:49,391-Speed 83.83 samples/sec Loss 0.8277 Epoch: 18 Global Step: 182560 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:38:21,752-Speed 79.11 samples/sec Loss 0.8873 Epoch: 18 Global Step: 182565 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:38:57,849-Speed 70.92 samples/sec Loss 0.8127 Epoch: 18 Global Step: 182570 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:39:28,951-Speed 82.31 samples/sec Loss 0.8858 Epoch: 18 Global Step: 182575 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:39:59,163-Speed 84.74 samples/sec Loss 0.8789 Epoch: 18 Global Step: 182580 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:40:37,478-Speed 66.82 samples/sec Loss 0.9168 Epoch: 18 Global Step: 182585 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:41:08,080-Speed 83.66 samples/sec Loss 0.8097 Epoch: 18 Global Step: 182590 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:41:39,419-Speed 81.69 samples/sec Loss 0.8290 Epoch: 18 Global Step: 182595 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:42:10,723-Speed 81.78 samples/sec Loss 0.8707 Epoch: 18 Global Step: 182600 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:42:41,620-Speed 82.88 samples/sec Loss 0.8426 Epoch: 18 Global Step: 182605 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:43:16,107-Speed 74.24 samples/sec Loss 0.8746 Epoch: 18 Global Step: 182610 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:43:52,895-Speed 69.59 samples/sec Loss 0.8729 Epoch: 18 Global Step: 182615 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:44:26,502-Speed 76.18 samples/sec Loss 0.8304 Epoch: 18 Global Step: 182620 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:44:58,972-Speed 78.85 samples/sec Loss 0.8951 Epoch: 18 Global Step: 182625 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:45:31,867-Speed 77.83 samples/sec Loss 0.8703 Epoch: 18 Global Step: 182630 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:46:01,777-Speed 85.59 samples/sec Loss 0.8062 Epoch: 18 Global Step: 182635 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:46:32,707-Speed 82.77 samples/sec Loss 0.7956 Epoch: 18 Global Step: 182640 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:47:04,621-Speed 80.22 samples/sec Loss 0.8213 Epoch: 18 Global Step: 182645 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:47:41,054-Speed 70.27 samples/sec Loss 0.9436 Epoch: 18 Global Step: 182650 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:48:14,389-Speed 76.80 samples/sec Loss 0.7632 Epoch: 18 Global Step: 182655 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:48:42,892-Speed 89.83 samples/sec Loss 0.8512 Epoch: 18 Global Step: 182660 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:49:13,146-Speed 84.62 samples/sec Loss 0.9149 Epoch: 18 Global Step: 182665 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:49:48,703-Speed 72.00 samples/sec Loss 0.8739 Epoch: 18 Global Step: 182670 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:50:24,003-Speed 72.53 samples/sec Loss 0.8410 Epoch: 18 Global Step: 182675 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:50:55,976-Speed 80.07 samples/sec Loss 0.9028 Epoch: 18 Global Step: 182680 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:51:25,182-Speed 87.66 samples/sec Loss 0.9435 Epoch: 18 Global Step: 182685 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:51:53,740-Speed 89.65 samples/sec Loss 0.8332 Epoch: 18 Global Step: 182690 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:52:25,392-Speed 80.88 samples/sec Loss 0.7886 Epoch: 18 Global Step: 182695 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:52:57,247-Speed 80.37 samples/sec Loss 0.8022 Epoch: 18 Global Step: 182700 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:53:29,292-Speed 79.89 samples/sec Loss 0.8651 Epoch: 18 Global Step: 182705 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:54:01,826-Speed 78.69 samples/sec Loss 1.0187 Epoch: 18 Global Step: 182710 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:54:33,958-Speed 79.68 samples/sec Loss 0.7723 Epoch: 18 Global Step: 182715 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:55:03,711-Speed 86.05 samples/sec Loss 0.8497 Epoch: 18 Global Step: 182720 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:55:34,357-Speed 83.54 samples/sec Loss 0.8175 Epoch: 18 Global Step: 182725 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:56:04,123-Speed 86.01 samples/sec Loss 0.8917 Epoch: 18 Global Step: 182730 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:56:37,462-Speed 76.79 samples/sec Loss 0.9027 Epoch: 18 Global Step: 182735 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:57:06,535-Speed 88.06 samples/sec Loss 0.7750 Epoch: 18 Global Step: 182740 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:57:37,646-Speed 82.29 samples/sec Loss 0.7682 Epoch: 18 Global Step: 182745 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:58:07,146-Speed 86.78 samples/sec Loss 0.8403 Epoch: 18 Global Step: 182750 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:58:37,060-Speed 85.59 samples/sec Loss 0.8706 Epoch: 18 Global Step: 182755 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:59:04,255-Speed 94.14 samples/sec Loss 0.8418 Epoch: 18 Global Step: 182760 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 20:59:34,933-Speed 83.45 samples/sec Loss 0.8779 Epoch: 18 Global Step: 182765 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:00:03,391-Speed 89.96 samples/sec Loss 0.8259 Epoch: 18 Global Step: 182770 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:00:32,709-Speed 87.32 samples/sec Loss 0.8320 Epoch: 18 Global Step: 182775 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:01:04,391-Speed 80.80 samples/sec Loss 0.8155 Epoch: 18 Global Step: 182780 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:01:34,559-Speed 84.89 samples/sec Loss 0.7848 Epoch: 18 Global Step: 182785 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:02:04,776-Speed 84.73 samples/sec Loss 0.7055 Epoch: 18 Global Step: 182790 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:02:35,393-Speed 83.62 samples/sec Loss 0.8902 Epoch: 18 Global Step: 182795 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:03:06,496-Speed 82.31 samples/sec Loss 0.7764 Epoch: 18 Global Step: 182800 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:03:34,612-Speed 91.06 samples/sec Loss 0.8021 Epoch: 18 Global Step: 182805 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:04:08,527-Speed 75.49 samples/sec Loss 0.8409 Epoch: 18 Global Step: 182810 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:04:44,896-Speed 70.39 samples/sec Loss 0.8744 Epoch: 18 Global Step: 182815 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:05:16,868-Speed 80.07 samples/sec Loss 0.7816 Epoch: 18 Global Step: 182820 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:05:48,116-Speed 81.93 samples/sec Loss 0.7966 Epoch: 18 Global Step: 182825 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:06:21,569-Speed 76.54 samples/sec Loss 0.8310 Epoch: 18 Global Step: 182830 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:06:51,402-Speed 85.82 samples/sec Loss 0.7853 Epoch: 18 Global Step: 182835 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:07:23,490-Speed 79.79 samples/sec Loss 0.8475 Epoch: 18 Global Step: 182840 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:07:53,246-Speed 86.04 samples/sec Loss 0.7424 Epoch: 18 Global Step: 182845 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:08:24,542-Speed 81.80 samples/sec Loss 0.8004 Epoch: 18 Global Step: 182850 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:08:55,768-Speed 81.99 samples/sec Loss 0.8307 Epoch: 18 Global Step: 182855 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:09:25,149-Speed 87.13 samples/sec Loss 0.7298 Epoch: 18 Global Step: 182860 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:09:56,294-Speed 82.20 samples/sec Loss 0.8887 Epoch: 18 Global Step: 182865 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:10:28,062-Speed 80.59 samples/sec Loss 0.8362 Epoch: 18 Global Step: 182870 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:10:56,733-Speed 89.30 samples/sec Loss 0.8318 Epoch: 18 Global Step: 182875 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:11:27,851-Speed 82.27 samples/sec Loss 0.8423 Epoch: 18 Global Step: 182880 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:11:59,832-Speed 80.05 samples/sec Loss 0.8586 Epoch: 18 Global Step: 182885 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:12:29,831-Speed 85.34 samples/sec Loss 0.7891 Epoch: 18 Global Step: 182890 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:13:02,592-Speed 78.15 samples/sec Loss 0.9586 Epoch: 18 Global Step: 182895 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:13:33,847-Speed 81.91 samples/sec Loss 0.8484 Epoch: 18 Global Step: 182900 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:14:02,349-Speed 89.82 samples/sec Loss 0.7545 Epoch: 18 Global Step: 182905 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:14:34,769-Speed 78.97 samples/sec Loss 0.8543 Epoch: 18 Global Step: 182910 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:15:05,084-Speed 84.45 samples/sec Loss 0.8326 Epoch: 18 Global Step: 182915 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:15:37,170-Speed 79.79 samples/sec Loss 0.7605 Epoch: 18 Global Step: 182920 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:16:07,712-Speed 83.82 samples/sec Loss 0.8686 Epoch: 18 Global Step: 182925 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:16:41,689-Speed 75.35 samples/sec Loss 0.9506 Epoch: 18 Global Step: 182930 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:17:14,691-Speed 77.57 samples/sec Loss 0.8623 Epoch: 18 Global Step: 182935 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:17:45,868-Speed 82.12 samples/sec Loss 0.7588 Epoch: 18 Global Step: 182940 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:18:17,550-Speed 80.81 samples/sec Loss 0.8075 Epoch: 18 Global Step: 182945 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:18:50,882-Speed 76.81 samples/sec Loss 0.8418 Epoch: 18 Global Step: 182950 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:19:21,317-Speed 84.12 samples/sec Loss 0.8974 Epoch: 18 Global Step: 182955 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:19:54,225-Speed 77.80 samples/sec Loss 0.8753 Epoch: 18 Global Step: 182960 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:20:22,731-Speed 89.81 samples/sec Loss 0.8064 Epoch: 18 Global Step: 182965 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:20:56,145-Speed 76.62 samples/sec Loss 0.8843 Epoch: 18 Global Step: 182970 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:21:29,436-Speed 76.90 samples/sec Loss 0.9234 Epoch: 18 Global Step: 182975 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:22:01,996-Speed 78.63 samples/sec Loss 0.8846 Epoch: 18 Global Step: 182980 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:22:30,844-Speed 88.74 samples/sec Loss 0.8491 Epoch: 18 Global Step: 182985 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:23:04,847-Speed 75.29 samples/sec Loss 1.0039 Epoch: 18 Global Step: 182990 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:23:38,904-Speed 75.17 samples/sec Loss 0.7562 Epoch: 18 Global Step: 182995 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:24:10,634-Speed 80.68 samples/sec Loss 0.8905 Epoch: 18 Global Step: 183000 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:24:45,350-Speed 73.75 samples/sec Loss 0.8390 Epoch: 18 Global Step: 183005 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:25:19,592-Speed 74.77 samples/sec Loss 0.9704 Epoch: 18 Global Step: 183010 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:25:53,411-Speed 75.70 samples/sec Loss 0.8716 Epoch: 18 Global Step: 183015 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:26:22,418-Speed 88.26 samples/sec Loss 0.8293 Epoch: 18 Global Step: 183020 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:26:53,776-Speed 81.64 samples/sec Loss 0.8480 Epoch: 18 Global Step: 183025 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:27:24,044-Speed 84.58 samples/sec Loss 0.8130 Epoch: 18 Global Step: 183030 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:27:56,557-Speed 78.74 samples/sec Loss 0.9556 Epoch: 18 Global Step: 183035 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:28:26,975-Speed 84.17 samples/sec Loss 0.8824 Epoch: 18 Global Step: 183040 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:28:57,207-Speed 84.68 samples/sec Loss 0.8660 Epoch: 18 Global Step: 183045 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:29:26,291-Speed 88.03 samples/sec Loss 0.8483 Epoch: 18 Global Step: 183050 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:29:59,614-Speed 76.83 samples/sec Loss 0.7661 Epoch: 18 Global Step: 183055 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:30:31,371-Speed 80.62 samples/sec Loss 0.8055 Epoch: 18 Global Step: 183060 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:31:01,290-Speed 85.57 samples/sec Loss 0.9048 Epoch: 18 Global Step: 183065 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:31:33,529-Speed 79.41 samples/sec Loss 0.8162 Epoch: 18 Global Step: 183070 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:32:03,456-Speed 85.55 samples/sec Loss 0.7877 Epoch: 18 Global Step: 183075 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:32:34,395-Speed 82.75 samples/sec Loss 0.8491 Epoch: 18 Global Step: 183080 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:33:06,131-Speed 80.67 samples/sec Loss 0.8704 Epoch: 18 Global Step: 183085 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:33:36,702-Speed 83.77 samples/sec Loss 0.8758 Epoch: 18 Global Step: 183090 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:34:05,945-Speed 87.55 samples/sec Loss 0.8645 Epoch: 18 Global Step: 183095 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:34:37,518-Speed 81.09 samples/sec Loss 0.8725 Epoch: 18 Global Step: 183100 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:35:10,601-Speed 77.39 samples/sec Loss 0.8123 Epoch: 18 Global Step: 183105 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:35:44,232-Speed 76.12 samples/sec Loss 0.8427 Epoch: 18 Global Step: 183110 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:36:19,488-Speed 72.62 samples/sec Loss 0.9669 Epoch: 18 Global Step: 183115 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:36:50,774-Speed 81.83 samples/sec Loss 0.8104 Epoch: 18 Global Step: 183120 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:37:21,749-Speed 82.65 samples/sec Loss 0.9123 Epoch: 18 Global Step: 183125 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:37:52,293-Speed 83.82 samples/sec Loss 0.9262 Epoch: 18 Global Step: 183130 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:38:22,240-Speed 85.49 samples/sec Loss 0.8488 Epoch: 18 Global Step: 183135 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:38:51,904-Speed 86.31 samples/sec Loss 0.7836 Epoch: 18 Global Step: 183140 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:39:24,427-Speed 78.72 samples/sec Loss 0.8512 Epoch: 18 Global Step: 183145 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:39:56,069-Speed 80.91 samples/sec Loss 0.8064 Epoch: 18 Global Step: 183150 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:40:29,723-Speed 76.07 samples/sec Loss 0.8953 Epoch: 18 Global Step: 183155 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:40:58,772-Speed 88.13 samples/sec Loss 0.8445 Epoch: 18 Global Step: 183160 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:41:32,731-Speed 75.39 samples/sec Loss 0.8192 Epoch: 18 Global Step: 183165 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:42:00,432-Speed 92.42 samples/sec Loss 0.8887 Epoch: 18 Global Step: 183170 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:42:35,858-Speed 72.27 samples/sec Loss 0.8554 Epoch: 18 Global Step: 183175 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:43:05,828-Speed 85.43 samples/sec Loss 0.7973 Epoch: 18 Global Step: 183180 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:43:37,904-Speed 79.81 samples/sec Loss 0.7665 Epoch: 18 Global Step: 183185 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:44:12,437-Speed 74.14 samples/sec Loss 0.9082 Epoch: 18 Global Step: 183190 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:44:44,385-Speed 80.13 samples/sec Loss 0.7992 Epoch: 18 Global Step: 183195 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:45:12,097-Speed 92.38 samples/sec Loss 0.8811 Epoch: 18 Global Step: 183200 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:45:42,481-Speed 84.26 samples/sec Loss 0.7920 Epoch: 18 Global Step: 183205 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:46:13,579-Speed 82.33 samples/sec Loss 0.8453 Epoch: 18 Global Step: 183210 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:46:43,749-Speed 84.86 samples/sec Loss 0.8607 Epoch: 18 Global Step: 183215 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:47:13,880-Speed 84.97 samples/sec Loss 0.8702 Epoch: 18 Global Step: 183220 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:47:42,279-Speed 90.15 samples/sec Loss 0.8727 Epoch: 18 Global Step: 183225 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:48:12,252-Speed 85.42 samples/sec Loss 0.7930 Epoch: 18 Global Step: 183230 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:48:44,439-Speed 79.54 samples/sec Loss 0.8073 Epoch: 18 Global Step: 183235 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:49:15,568-Speed 82.25 samples/sec Loss 0.8454 Epoch: 18 Global Step: 183240 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:49:45,513-Speed 85.50 samples/sec Loss 0.9300 Epoch: 18 Global Step: 183245 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:50:16,913-Speed 81.53 samples/sec Loss 0.8358 Epoch: 18 Global Step: 183250 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:50:46,859-Speed 85.49 samples/sec Loss 0.9418 Epoch: 18 Global Step: 183255 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:51:19,019-Speed 79.61 samples/sec Loss 0.9297 Epoch: 18 Global Step: 183260 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:51:51,567-Speed 78.67 samples/sec Loss 0.7798 Epoch: 18 Global Step: 183265 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:52:22,147-Speed 83.72 samples/sec Loss 0.8172 Epoch: 18 Global Step: 183270 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:52:54,044-Speed 80.26 samples/sec Loss 0.8330 Epoch: 18 Global Step: 183275 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:53:28,327-Speed 74.68 samples/sec Loss 0.8071 Epoch: 18 Global Step: 183280 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:53:58,577-Speed 84.63 samples/sec Loss 0.8755 Epoch: 18 Global Step: 183285 Fp16 Grad Scale: 4096 Required: 79 hours Training: 2021-06-15 21:54:29,867-Speed 81.82 samples/sec Loss 0.7957 Epoch: 18 Global Step: 183290 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:54:57,684-Speed 92.06 samples/sec Loss 0.7878 Epoch: 18 Global Step: 183295 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:55:29,488-Speed 80.50 samples/sec Loss 0.8884 Epoch: 18 Global Step: 183300 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:56:01,112-Speed 80.95 samples/sec Loss 0.8220 Epoch: 18 Global Step: 183305 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:56:33,358-Speed 79.40 samples/sec Loss 0.8253 Epoch: 18 Global Step: 183310 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:57:03,665-Speed 84.47 samples/sec Loss 0.8497 Epoch: 18 Global Step: 183315 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:57:36,830-Speed 77.19 samples/sec Loss 0.9901 Epoch: 18 Global Step: 183320 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:58:10,412-Speed 76.24 samples/sec Loss 0.8584 Epoch: 18 Global Step: 183325 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:58:48,216-Speed 67.72 samples/sec Loss 0.8940 Epoch: 18 Global Step: 183330 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:59:18,778-Speed 83.77 samples/sec Loss 0.7827 Epoch: 18 Global Step: 183335 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 21:59:49,255-Speed 84.00 samples/sec Loss 0.7264 Epoch: 18 Global Step: 183340 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 22:00:23,565-Speed 74.62 samples/sec Loss 0.9328 Epoch: 18 Global Step: 183345 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 22:00:56,552-Speed 77.61 samples/sec Loss 0.8511 Epoch: 18 Global Step: 183350 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 22:01:29,656-Speed 77.34 samples/sec Loss 0.8914 Epoch: 18 Global Step: 183355 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 22:02:06,013-Speed 70.42 samples/sec Loss 0.6471 Epoch: 18 Global Step: 183360 Fp16 Grad Scale: 8192 Required: 79 hours Training: 2021-06-15 22:02:39,238-Speed 77.05 samples/sec Loss 0.7971 Epoch: 18 Global Step: 183365 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:03:20,294-Speed 62.36 samples/sec Loss 0.8166 Epoch: 18 Global Step: 183370 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:04:21,202-Speed 42.04 samples/sec Loss 0.9180 Epoch: 18 Global Step: 183375 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:05:02,477-Speed 62.03 samples/sec Loss 0.8897 Epoch: 18 Global Step: 183380 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:05:35,623-Speed 77.24 samples/sec Loss 0.8698 Epoch: 18 Global Step: 183385 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:06:06,963-Speed 81.69 samples/sec Loss 0.8745 Epoch: 18 Global Step: 183390 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:06:41,572-Speed 73.97 samples/sec Loss 0.8822 Epoch: 18 Global Step: 183395 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:07:37,185-Speed 46.03 samples/sec Loss 0.8654 Epoch: 18 Global Step: 183400 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:08:09,748-Speed 78.64 samples/sec Loss 0.7955 Epoch: 18 Global Step: 183405 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:08:42,676-Speed 77.75 samples/sec Loss 0.9241 Epoch: 18 Global Step: 183410 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:09:13,231-Speed 83.79 samples/sec Loss 0.9296 Epoch: 18 Global Step: 183415 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:09:46,795-Speed 76.28 samples/sec Loss 0.9578 Epoch: 18 Global Step: 183420 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:10:16,314-Speed 86.73 samples/sec Loss 0.8114 Epoch: 18 Global Step: 183425 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:10:48,943-Speed 78.47 samples/sec Loss 0.8725 Epoch: 18 Global Step: 183430 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:11:20,720-Speed 80.56 samples/sec Loss 0.8083 Epoch: 18 Global Step: 183435 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:11:52,072-Speed 81.66 samples/sec Loss 0.7828 Epoch: 18 Global Step: 183440 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:12:21,313-Speed 87.55 samples/sec Loss 0.8470 Epoch: 18 Global Step: 183445 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:12:51,574-Speed 84.60 samples/sec Loss 0.8763 Epoch: 18 Global Step: 183450 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:13:21,312-Speed 86.09 samples/sec Loss 0.7552 Epoch: 18 Global Step: 183455 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:13:52,788-Speed 81.34 samples/sec Loss 0.8170 Epoch: 18 Global Step: 183460 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:14:24,506-Speed 80.72 samples/sec Loss 0.7286 Epoch: 18 Global Step: 183465 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:14:54,864-Speed 84.34 samples/sec Loss 0.8080 Epoch: 18 Global Step: 183470 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:15:25,128-Speed 84.59 samples/sec Loss 0.8130 Epoch: 18 Global Step: 183475 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:15:56,301-Speed 82.13 samples/sec Loss 0.8448 Epoch: 18 Global Step: 183480 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:16:25,769-Speed 86.89 samples/sec Loss 0.8251 Epoch: 18 Global Step: 183485 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:16:57,824-Speed 79.87 samples/sec Loss 0.8948 Epoch: 18 Global Step: 183490 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:17:31,728-Speed 75.51 samples/sec Loss 0.7265 Epoch: 18 Global Step: 183495 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:18:05,941-Speed 74.83 samples/sec Loss 0.8427 Epoch: 18 Global Step: 183500 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:18:34,386-Speed 90.01 samples/sec Loss 0.8437 Epoch: 18 Global Step: 183505 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:19:03,756-Speed 87.17 samples/sec Loss 0.7594 Epoch: 18 Global Step: 183510 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:19:34,175-Speed 84.16 samples/sec Loss 0.8097 Epoch: 18 Global Step: 183515 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:20:05,893-Speed 80.72 samples/sec Loss 0.8338 Epoch: 18 Global Step: 183520 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:20:36,785-Speed 82.88 samples/sec Loss 0.8184 Epoch: 18 Global Step: 183525 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:21:08,397-Speed 80.99 samples/sec Loss 0.8227 Epoch: 18 Global Step: 183530 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:21:38,797-Speed 84.21 samples/sec Loss 0.9024 Epoch: 18 Global Step: 183535 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:22:10,694-Speed 80.26 samples/sec Loss 0.8575 Epoch: 18 Global Step: 183540 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:22:41,384-Speed 83.42 samples/sec Loss 0.9088 Epoch: 18 Global Step: 183545 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:23:12,230-Speed 83.00 samples/sec Loss 0.8844 Epoch: 18 Global Step: 183550 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:23:44,482-Speed 79.38 samples/sec Loss 0.8779 Epoch: 18 Global Step: 183555 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:24:13,236-Speed 89.04 samples/sec Loss 0.8320 Epoch: 18 Global Step: 183560 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:24:42,083-Speed 88.75 samples/sec Loss 0.8704 Epoch: 18 Global Step: 183565 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:25:12,100-Speed 85.29 samples/sec Loss 0.8416 Epoch: 18 Global Step: 183570 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:25:46,135-Speed 75.22 samples/sec Loss 0.9036 Epoch: 18 Global Step: 183575 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:26:21,521-Speed 72.35 samples/sec Loss 0.7780 Epoch: 18 Global Step: 183580 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:26:53,823-Speed 79.26 samples/sec Loss 0.7698 Epoch: 18 Global Step: 183585 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:27:28,266-Speed 74.33 samples/sec Loss 0.8686 Epoch: 18 Global Step: 183590 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:28:02,562-Speed 74.65 samples/sec Loss 0.7900 Epoch: 18 Global Step: 183595 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:28:37,317-Speed 73.66 samples/sec Loss 0.7998 Epoch: 18 Global Step: 183600 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:29:06,949-Speed 86.40 samples/sec Loss 0.8140 Epoch: 18 Global Step: 183605 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:29:41,169-Speed 74.81 samples/sec Loss 0.8821 Epoch: 18 Global Step: 183610 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:30:15,764-Speed 74.00 samples/sec Loss 0.8804 Epoch: 18 Global Step: 183615 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:30:46,190-Speed 84.14 samples/sec Loss 0.8328 Epoch: 18 Global Step: 183620 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:31:17,023-Speed 83.03 samples/sec Loss 0.8679 Epoch: 18 Global Step: 183625 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:31:47,341-Speed 84.44 samples/sec Loss 0.8845 Epoch: 18 Global Step: 183630 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:32:25,665-Speed 66.80 samples/sec Loss 0.7563 Epoch: 18 Global Step: 183635 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:32:58,526-Speed 77.91 samples/sec Loss 0.8587 Epoch: 18 Global Step: 183640 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:33:30,740-Speed 79.47 samples/sec Loss 0.8815 Epoch: 18 Global Step: 183645 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:34:03,401-Speed 78.39 samples/sec Loss 0.8716 Epoch: 18 Global Step: 183650 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:34:32,403-Speed 88.28 samples/sec Loss 0.9761 Epoch: 18 Global Step: 183655 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:35:03,206-Speed 83.11 samples/sec Loss 0.7811 Epoch: 18 Global Step: 183660 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:35:33,680-Speed 84.01 samples/sec Loss 0.8232 Epoch: 18 Global Step: 183665 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:36:03,405-Speed 86.13 samples/sec Loss 0.8847 Epoch: 18 Global Step: 183670 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:36:33,324-Speed 85.57 samples/sec Loss 0.8420 Epoch: 18 Global Step: 183675 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:37:03,946-Speed 83.60 samples/sec Loss 0.8787 Epoch: 18 Global Step: 183680 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:37:36,573-Speed 78.47 samples/sec Loss 0.8182 Epoch: 18 Global Step: 183685 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:38:05,578-Speed 88.27 samples/sec Loss 0.8106 Epoch: 18 Global Step: 183690 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:38:41,391-Speed 71.49 samples/sec Loss 0.8081 Epoch: 18 Global Step: 183695 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:39:12,621-Speed 82.00 samples/sec Loss 0.8138 Epoch: 18 Global Step: 183700 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:39:44,721-Speed 79.75 samples/sec Loss 0.7546 Epoch: 18 Global Step: 183705 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:40:17,482-Speed 78.15 samples/sec Loss 0.8348 Epoch: 18 Global Step: 183710 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:40:48,649-Speed 82.15 samples/sec Loss 0.9266 Epoch: 18 Global Step: 183715 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:41:21,478-Speed 77.98 samples/sec Loss 0.8984 Epoch: 18 Global Step: 183720 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:41:50,060-Speed 89.58 samples/sec Loss 0.7415 Epoch: 18 Global Step: 183725 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:42:20,000-Speed 85.51 samples/sec Loss 0.8244 Epoch: 18 Global Step: 183730 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:42:49,361-Speed 87.19 samples/sec Loss 0.8264 Epoch: 18 Global Step: 183735 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:43:20,097-Speed 83.29 samples/sec Loss 0.9534 Epoch: 18 Global Step: 183740 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:44:35,248-Speed 34.07 samples/sec Loss 0.8352 Epoch: 18 Global Step: 183745 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:45:54,733-Speed 32.21 samples/sec Loss 0.7908 Epoch: 18 Global Step: 183750 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:46:59,415-Speed 39.58 samples/sec Loss 0.8972 Epoch: 18 Global Step: 183755 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:48:02,477-Speed 40.60 samples/sec Loss 0.7763 Epoch: 18 Global Step: 183760 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:49:08,899-Speed 38.54 samples/sec Loss 0.8909 Epoch: 18 Global Step: 183765 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:49:38,456-Speed 86.62 samples/sec Loss 0.9221 Epoch: 18 Global Step: 183770 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:50:12,334-Speed 75.57 samples/sec Loss 0.8912 Epoch: 18 Global Step: 183775 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:50:47,800-Speed 72.19 samples/sec Loss 0.8671 Epoch: 18 Global Step: 183780 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 22:51:32,359-Speed 57.45 samples/sec Loss 0.8666 Epoch: 18 Global Step: 183785 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 22:52:00,221-Speed 91.89 samples/sec Loss 0.9062 Epoch: 18 Global Step: 183790 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 22:52:34,453-Speed 74.79 samples/sec Loss 0.8259 Epoch: 18 Global Step: 183795 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 22:53:08,835-Speed 74.46 samples/sec Loss 0.8610 Epoch: 18 Global Step: 183800 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 22:53:39,805-Speed 82.66 samples/sec Loss 0.8689 Epoch: 18 Global Step: 183805 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 22:54:11,259-Speed 81.39 samples/sec Loss 0.9231 Epoch: 18 Global Step: 183810 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 22:54:41,999-Speed 83.35 samples/sec Loss 0.7454 Epoch: 18 Global Step: 183815 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 22:55:54,586-Speed 35.27 samples/sec Loss 0.8192 Epoch: 18 Global Step: 183820 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 22:57:24,566-Speed 28.46 samples/sec Loss 0.8682 Epoch: 18 Global Step: 183825 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 22:58:57,475-Speed 27.56 samples/sec Loss 0.8043 Epoch: 18 Global Step: 183830 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:00:34,777-Speed 26.32 samples/sec Loss 0.9238 Epoch: 18 Global Step: 183835 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:02:11,665-Speed 26.42 samples/sec Loss 0.8563 Epoch: 18 Global Step: 183840 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:03:50,136-Speed 26.00 samples/sec Loss 0.7607 Epoch: 18 Global Step: 183845 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:05:22,825-Speed 27.63 samples/sec Loss 0.9229 Epoch: 18 Global Step: 183850 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:06:55,886-Speed 27.51 samples/sec Loss 0.8500 Epoch: 18 Global Step: 183855 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:08:31,823-Speed 26.69 samples/sec Loss 0.8778 Epoch: 18 Global Step: 183860 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:10:08,225-Speed 26.56 samples/sec Loss 0.8641 Epoch: 18 Global Step: 183865 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:11:41,425-Speed 27.47 samples/sec Loss 0.8158 Epoch: 18 Global Step: 183870 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:13:30,810-Speed 23.41 samples/sec Loss 0.9490 Epoch: 18 Global Step: 183875 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:15:17,725-Speed 23.95 samples/sec Loss 0.8193 Epoch: 18 Global Step: 183880 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-15 23:17:19,826-Speed 20.97 samples/sec Loss 0.9307 Epoch: 18 Global Step: 183885 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:19:19,193-Speed 21.45 samples/sec Loss 0.9226 Epoch: 18 Global Step: 183890 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:21:23,077-Speed 20.67 samples/sec Loss 0.8417 Epoch: 18 Global Step: 183895 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:23:26,262-Speed 20.78 samples/sec Loss 0.7890 Epoch: 18 Global Step: 183900 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:25:22,258-Speed 22.07 samples/sec Loss 0.8306 Epoch: 18 Global Step: 183905 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:27:24,350-Speed 20.97 samples/sec Loss 0.8206 Epoch: 18 Global Step: 183910 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:29:23,145-Speed 21.56 samples/sec Loss 0.8408 Epoch: 18 Global Step: 183915 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:31:06,603-Speed 24.74 samples/sec Loss 0.8387 Epoch: 18 Global Step: 183920 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:32:45,025-Speed 26.01 samples/sec Loss 0.8985 Epoch: 18 Global Step: 183925 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:34:17,268-Speed 27.75 samples/sec Loss 0.7603 Epoch: 18 Global Step: 183930 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:35:49,928-Speed 27.63 samples/sec Loss 0.8278 Epoch: 18 Global Step: 183935 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:37:22,311-Speed 27.72 samples/sec Loss 0.8472 Epoch: 18 Global Step: 183940 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:39:01,594-Speed 25.79 samples/sec Loss 0.9547 Epoch: 18 Global Step: 183945 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:40:35,942-Speed 27.14 samples/sec Loss 0.8374 Epoch: 18 Global Step: 183950 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:42:20,655-Speed 24.45 samples/sec Loss 0.8767 Epoch: 18 Global Step: 183955 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:44:11,147-Speed 23.17 samples/sec Loss 0.8836 Epoch: 18 Global Step: 183960 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:45:41,800-Speed 28.25 samples/sec Loss 0.7765 Epoch: 18 Global Step: 183965 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:47:24,741-Speed 24.87 samples/sec Loss 0.8617 Epoch: 18 Global Step: 183970 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:48:59,206-Speed 27.10 samples/sec Loss 0.9010 Epoch: 18 Global Step: 183975 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:50:32,128-Speed 27.55 samples/sec Loss 0.7868 Epoch: 18 Global Step: 183980 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:51:50,264-Speed 32.77 samples/sec Loss 0.8266 Epoch: 18 Global Step: 183985 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:53:05,787-Speed 33.90 samples/sec Loss 0.8277 Epoch: 18 Global Step: 183990 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:54:27,670-Speed 31.27 samples/sec Loss 0.7842 Epoch: 18 Global Step: 183995 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:55:42,332-Speed 34.30 samples/sec Loss 0.8146 Epoch: 18 Global Step: 184000 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:57:02,663-Speed 31.87 samples/sec Loss 0.8001 Epoch: 18 Global Step: 184005 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:58:20,850-Speed 32.75 samples/sec Loss 0.9268 Epoch: 18 Global Step: 184010 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-15 23:59:33,983-Speed 35.01 samples/sec Loss 0.8427 Epoch: 18 Global Step: 184015 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:00:48,396-Speed 34.41 samples/sec Loss 0.7420 Epoch: 18 Global Step: 184020 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:02:03,228-Speed 34.21 samples/sec Loss 0.8027 Epoch: 18 Global Step: 184025 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:02:47,906-Speed 57.30 samples/sec Loss 0.7660 Epoch: 18 Global Step: 184030 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:03:31,493-Speed 58.74 samples/sec Loss 0.9989 Epoch: 18 Global Step: 184035 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:04:07,848-Speed 70.42 samples/sec Loss 0.7590 Epoch: 18 Global Step: 184040 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:04:50,426-Speed 60.13 samples/sec Loss 0.8101 Epoch: 18 Global Step: 184045 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:05:35,389-Speed 56.94 samples/sec Loss 0.8746 Epoch: 18 Global Step: 184050 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:06:18,404-Speed 59.52 samples/sec Loss 0.8745 Epoch: 18 Global Step: 184055 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:06:59,894-Speed 61.70 samples/sec Loss 0.8688 Epoch: 18 Global Step: 184060 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:07:43,787-Speed 58.33 samples/sec Loss 0.8625 Epoch: 18 Global Step: 184065 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:08:22,879-Speed 65.49 samples/sec Loss 0.8396 Epoch: 18 Global Step: 184070 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:09:06,480-Speed 58.72 samples/sec Loss 0.8562 Epoch: 18 Global Step: 184075 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:09:49,629-Speed 59.33 samples/sec Loss 0.9057 Epoch: 18 Global Step: 184080 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:10:33,112-Speed 58.88 samples/sec Loss 0.7916 Epoch: 18 Global Step: 184085 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:11:13,469-Speed 63.44 samples/sec Loss 0.8029 Epoch: 18 Global Step: 184090 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:11:55,589-Speed 60.78 samples/sec Loss 0.9097 Epoch: 18 Global Step: 184095 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:12:36,177-Speed 63.07 samples/sec Loss 0.9005 Epoch: 18 Global Step: 184100 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:13:14,519-Speed 66.77 samples/sec Loss 0.8446 Epoch: 18 Global Step: 184105 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:13:39,130-Speed 104.02 samples/sec Loss 0.8460 Epoch: 18 Global Step: 184110 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:13:59,003-Speed 128.82 samples/sec Loss 0.8456 Epoch: 18 Global Step: 184115 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:14:18,936-Speed 128.44 samples/sec Loss 0.9104 Epoch: 18 Global Step: 184120 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:14:38,911-Speed 128.17 samples/sec Loss 0.7692 Epoch: 18 Global Step: 184125 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:14:58,873-Speed 128.25 samples/sec Loss 0.8848 Epoch: 18 Global Step: 184130 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:15:18,490-Speed 130.51 samples/sec Loss 0.8946 Epoch: 18 Global Step: 184135 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:15:38,603-Speed 127.29 samples/sec Loss 0.7828 Epoch: 18 Global Step: 184140 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:15:58,330-Speed 129.77 samples/sec Loss 0.7820 Epoch: 18 Global Step: 184145 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:16:18,231-Speed 128.64 samples/sec Loss 0.8586 Epoch: 18 Global Step: 184150 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:16:38,028-Speed 129.32 samples/sec Loss 0.8364 Epoch: 18 Global Step: 184155 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:16:57,793-Speed 129.53 samples/sec Loss 0.8225 Epoch: 18 Global Step: 184160 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:17:17,640-Speed 129.00 samples/sec Loss 0.8904 Epoch: 18 Global Step: 184165 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:17:37,564-Speed 128.49 samples/sec Loss 0.7393 Epoch: 18 Global Step: 184170 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:17:57,451-Speed 128.73 samples/sec Loss 0.7891 Epoch: 18 Global Step: 184175 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:18:17,186-Speed 129.73 samples/sec Loss 0.8546 Epoch: 18 Global Step: 184180 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:18:37,301-Speed 127.27 samples/sec Loss 1.0187 Epoch: 18 Global Step: 184185 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:18:56,918-Speed 130.51 samples/sec Loss 0.8837 Epoch: 18 Global Step: 184190 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:19:16,808-Speed 128.71 samples/sec Loss 0.8527 Epoch: 18 Global Step: 184195 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:19:36,154-Speed 132.34 samples/sec Loss 0.8309 Epoch: 18 Global Step: 184200 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:19:55,959-Speed 129.26 samples/sec Loss 0.8558 Epoch: 18 Global Step: 184205 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:20:15,819-Speed 128.92 samples/sec Loss 0.8318 Epoch: 18 Global Step: 184210 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:20:35,462-Speed 130.33 samples/sec Loss 0.8078 Epoch: 18 Global Step: 184215 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:20:55,366-Speed 128.63 samples/sec Loss 0.8280 Epoch: 18 Global Step: 184220 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:21:15,429-Speed 127.61 samples/sec Loss 0.8944 Epoch: 18 Global Step: 184225 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:21:35,498-Speed 127.56 samples/sec Loss 0.8672 Epoch: 18 Global Step: 184230 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:21:55,115-Speed 130.51 samples/sec Loss 0.7856 Epoch: 18 Global Step: 184235 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:22:15,042-Speed 128.48 samples/sec Loss 0.8279 Epoch: 18 Global Step: 184240 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:22:34,756-Speed 129.86 samples/sec Loss 0.9311 Epoch: 18 Global Step: 184245 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:22:54,493-Speed 129.71 samples/sec Loss 0.8806 Epoch: 18 Global Step: 184250 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:23:14,372-Speed 128.79 samples/sec Loss 0.8423 Epoch: 18 Global Step: 184255 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:23:34,015-Speed 130.33 samples/sec Loss 0.8418 Epoch: 18 Global Step: 184260 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:23:53,895-Speed 128.78 samples/sec Loss 0.9147 Epoch: 18 Global Step: 184265 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:24:13,979-Speed 127.47 samples/sec Loss 0.8151 Epoch: 18 Global Step: 184270 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:24:34,046-Speed 127.58 samples/sec Loss 0.8781 Epoch: 18 Global Step: 184275 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:24:53,641-Speed 130.65 samples/sec Loss 0.8218 Epoch: 18 Global Step: 184280 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:25:13,733-Speed 127.42 samples/sec Loss 0.7832 Epoch: 18 Global Step: 184285 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:25:33,405-Speed 130.15 samples/sec Loss 0.8782 Epoch: 18 Global Step: 184290 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:25:53,245-Speed 129.04 samples/sec Loss 0.7744 Epoch: 18 Global Step: 184295 Fp16 Grad Scale: 4096 Required: 78 hours Training: 2021-06-16 00:26:13,022-Speed 129.45 samples/sec Loss 0.8646 Epoch: 18 Global Step: 184300 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:26:32,689-Speed 130.17 samples/sec Loss 0.7615 Epoch: 18 Global Step: 184305 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:26:52,597-Speed 128.60 samples/sec Loss 0.9230 Epoch: 18 Global Step: 184310 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:27:12,607-Speed 127.94 samples/sec Loss 0.7934 Epoch: 18 Global Step: 184315 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:27:32,724-Speed 127.26 samples/sec Loss 0.8166 Epoch: 18 Global Step: 184320 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:27:52,366-Speed 130.34 samples/sec Loss 0.7179 Epoch: 18 Global Step: 184325 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:28:12,359-Speed 128.05 samples/sec Loss 0.8859 Epoch: 18 Global Step: 184330 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:28:32,072-Speed 129.87 samples/sec Loss 0.8313 Epoch: 18 Global Step: 184335 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:28:52,034-Speed 128.25 samples/sec Loss 0.8563 Epoch: 18 Global Step: 184340 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:29:11,711-Speed 130.11 samples/sec Loss 0.8893 Epoch: 18 Global Step: 184345 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:29:31,414-Speed 129.94 samples/sec Loss 0.8475 Epoch: 18 Global Step: 184350 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:29:51,275-Speed 128.90 samples/sec Loss 0.8984 Epoch: 18 Global Step: 184355 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:30:11,278-Speed 127.98 samples/sec Loss 0.9681 Epoch: 18 Global Step: 184360 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:30:31,428-Speed 127.06 samples/sec Loss 0.8580 Epoch: 18 Global Step: 184365 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:30:51,063-Speed 130.38 samples/sec Loss 0.8229 Epoch: 18 Global Step: 184370 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:31:10,928-Speed 128.87 samples/sec Loss 0.8228 Epoch: 18 Global Step: 184375 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:31:30,624-Speed 129.98 samples/sec Loss 0.8000 Epoch: 18 Global Step: 184380 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:31:50,527-Speed 128.63 samples/sec Loss 0.7635 Epoch: 18 Global Step: 184385 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:32:10,297-Speed 129.50 samples/sec Loss 0.8532 Epoch: 18 Global Step: 184390 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:32:30,020-Speed 129.80 samples/sec Loss 0.8921 Epoch: 18 Global Step: 184395 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:32:49,886-Speed 128.87 samples/sec Loss 0.8464 Epoch: 18 Global Step: 184400 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:33:09,889-Speed 127.99 samples/sec Loss 0.7501 Epoch: 18 Global Step: 184405 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:33:30,025-Speed 127.14 samples/sec Loss 0.9418 Epoch: 18 Global Step: 184410 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:33:49,861-Speed 129.07 samples/sec Loss 0.8056 Epoch: 18 Global Step: 184415 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:34:09,590-Speed 129.77 samples/sec Loss 0.8030 Epoch: 18 Global Step: 184420 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:34:29,336-Speed 129.65 samples/sec Loss 0.9323 Epoch: 18 Global Step: 184425 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:34:49,191-Speed 128.94 samples/sec Loss 0.8812 Epoch: 18 Global Step: 184430 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:35:08,954-Speed 129.54 samples/sec Loss 0.8350 Epoch: 18 Global Step: 184435 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:35:28,914-Speed 128.26 samples/sec Loss 0.7507 Epoch: 18 Global Step: 184440 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:35:48,632-Speed 129.83 samples/sec Loss 0.9025 Epoch: 18 Global Step: 184445 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:36:08,678-Speed 127.71 samples/sec Loss 0.8608 Epoch: 18 Global Step: 184450 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:36:28,782-Speed 127.35 samples/sec Loss 0.8462 Epoch: 18 Global Step: 184455 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:36:48,701-Speed 128.53 samples/sec Loss 0.8756 Epoch: 18 Global Step: 184460 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:37:08,411-Speed 129.89 samples/sec Loss 0.8403 Epoch: 18 Global Step: 184465 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:37:28,273-Speed 128.90 samples/sec Loss 0.8225 Epoch: 18 Global Step: 184470 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:37:48,039-Speed 129.52 samples/sec Loss 0.9755 Epoch: 18 Global Step: 184475 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:38:07,833-Speed 129.34 samples/sec Loss 0.8861 Epoch: 18 Global Step: 184480 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:38:27,743-Speed 128.58 samples/sec Loss 0.9331 Epoch: 18 Global Step: 184485 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:38:47,370-Speed 130.44 samples/sec Loss 0.9695 Epoch: 18 Global Step: 184490 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:39:07,324-Speed 128.30 samples/sec Loss 0.8834 Epoch: 18 Global Step: 184495 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:39:27,441-Speed 127.26 samples/sec Loss 0.9351 Epoch: 18 Global Step: 184500 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:39:48,200-Speed 123.33 samples/sec Loss 0.8227 Epoch: 18 Global Step: 184505 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:40:17,702-Speed 86.78 samples/sec Loss 0.7865 Epoch: 18 Global Step: 184510 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:40:37,643-Speed 128.38 samples/sec Loss 1.0027 Epoch: 18 Global Step: 184515 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:40:57,543-Speed 128.65 samples/sec Loss 0.7142 Epoch: 18 Global Step: 184520 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:41:17,481-Speed 128.41 samples/sec Loss 0.9017 Epoch: 18 Global Step: 184525 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:41:37,254-Speed 129.47 samples/sec Loss 0.7558 Epoch: 18 Global Step: 184530 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:41:56,990-Speed 129.72 samples/sec Loss 0.8814 Epoch: 18 Global Step: 184535 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:42:17,103-Speed 127.29 samples/sec Loss 0.9144 Epoch: 18 Global Step: 184540 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:42:37,033-Speed 128.45 samples/sec Loss 1.0597 Epoch: 18 Global Step: 184545 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:42:56,893-Speed 128.91 samples/sec Loss 0.9123 Epoch: 18 Global Step: 184550 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:43:16,653-Speed 129.56 samples/sec Loss 0.8201 Epoch: 18 Global Step: 184555 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:43:36,466-Speed 129.21 samples/sec Loss 0.7735 Epoch: 18 Global Step: 184560 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:43:56,183-Speed 129.85 samples/sec Loss 0.9985 Epoch: 18 Global Step: 184565 Fp16 Grad Scale: 8192 Required: 78 hours Training: 2021-06-16 00:44:16,084-Speed 128.64 samples/sec Loss 0.8211 Epoch: 18 Global Step: 184570 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:44:35,846-Speed 129.55 samples/sec Loss 0.9057 Epoch: 18 Global Step: 184575 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:44:55,614-Speed 129.51 samples/sec Loss 0.9070 Epoch: 18 Global Step: 184580 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:45:15,676-Speed 127.61 samples/sec Loss 0.8355 Epoch: 18 Global Step: 184585 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:45:35,778-Speed 127.36 samples/sec Loss 0.9393 Epoch: 18 Global Step: 184590 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:45:55,532-Speed 129.60 samples/sec Loss 0.8499 Epoch: 18 Global Step: 184595 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:46:15,315-Speed 129.41 samples/sec Loss 0.8932 Epoch: 18 Global Step: 184600 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:46:35,072-Speed 129.59 samples/sec Loss 0.8379 Epoch: 18 Global Step: 184605 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:46:54,875-Speed 129.28 samples/sec Loss 0.8663 Epoch: 18 Global Step: 184610 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:47:14,785-Speed 128.59 samples/sec Loss 0.8564 Epoch: 18 Global Step: 184615 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:47:34,542-Speed 129.58 samples/sec Loss 0.8384 Epoch: 18 Global Step: 184620 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:47:54,316-Speed 129.47 samples/sec Loss 0.8312 Epoch: 18 Global Step: 184625 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:48:14,367-Speed 127.69 samples/sec Loss 0.8617 Epoch: 18 Global Step: 184630 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:48:34,679-Speed 126.04 samples/sec Loss 0.7841 Epoch: 18 Global Step: 184635 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:48:54,384-Speed 129.92 samples/sec Loss 0.8976 Epoch: 18 Global Step: 184640 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:49:14,150-Speed 129.53 samples/sec Loss 0.8224 Epoch: 18 Global Step: 184645 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:49:33,913-Speed 129.54 samples/sec Loss 0.8511 Epoch: 18 Global Step: 184650 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:49:53,713-Speed 129.30 samples/sec Loss 0.9046 Epoch: 18 Global Step: 184655 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:50:13,610-Speed 128.67 samples/sec Loss 0.8098 Epoch: 18 Global Step: 184660 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:50:33,361-Speed 129.62 samples/sec Loss 0.9607 Epoch: 18 Global Step: 184665 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:50:53,121-Speed 129.56 samples/sec Loss 0.8998 Epoch: 18 Global Step: 184670 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:51:13,196-Speed 127.53 samples/sec Loss 0.9097 Epoch: 18 Global Step: 184675 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:51:33,810-Speed 124.20 samples/sec Loss 0.6891 Epoch: 18 Global Step: 184680 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:51:53,557-Speed 129.65 samples/sec Loss 0.8928 Epoch: 18 Global Step: 184685 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:52:13,453-Speed 128.68 samples/sec Loss 0.9101 Epoch: 18 Global Step: 184690 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:52:33,092-Speed 130.36 samples/sec Loss 0.7902 Epoch: 18 Global Step: 184695 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:52:52,910-Speed 129.19 samples/sec Loss 0.8786 Epoch: 18 Global Step: 184700 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:53:12,744-Speed 129.07 samples/sec Loss 0.8120 Epoch: 18 Global Step: 184705 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:53:32,533-Speed 129.38 samples/sec Loss 0.8445 Epoch: 18 Global Step: 184710 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:53:52,405-Speed 128.83 samples/sec Loss 0.7835 Epoch: 18 Global Step: 184715 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:54:12,318-Speed 128.56 samples/sec Loss 0.8258 Epoch: 18 Global Step: 184720 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:54:32,412-Speed 127.41 samples/sec Loss 0.9713 Epoch: 18 Global Step: 184725 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:54:52,194-Speed 129.42 samples/sec Loss 0.8028 Epoch: 18 Global Step: 184730 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:55:12,015-Speed 129.16 samples/sec Loss 0.7522 Epoch: 18 Global Step: 184735 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:55:31,654-Speed 130.38 samples/sec Loss 0.8055 Epoch: 18 Global Step: 184740 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:55:51,376-Speed 129.81 samples/sec Loss 1.0710 Epoch: 18 Global Step: 184745 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:56:11,357-Speed 128.13 samples/sec Loss 0.8788 Epoch: 18 Global Step: 184750 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:56:31,127-Speed 129.50 samples/sec Loss 0.7860 Epoch: 18 Global Step: 184755 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:56:50,983-Speed 128.94 samples/sec Loss 0.9615 Epoch: 18 Global Step: 184760 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:57:10,977-Speed 128.05 samples/sec Loss 0.8954 Epoch: 18 Global Step: 184765 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:57:31,213-Speed 126.51 samples/sec Loss 0.8669 Epoch: 18 Global Step: 184770 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:57:50,955-Speed 129.68 samples/sec Loss 0.8674 Epoch: 18 Global Step: 184775 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:58:10,783-Speed 129.12 samples/sec Loss 1.0375 Epoch: 18 Global Step: 184780 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:58:30,510-Speed 129.77 samples/sec Loss 0.8624 Epoch: 18 Global Step: 184785 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 00:58:49,839-Speed 132.45 samples/sec Loss 0.8723 Epoch: 18 Global Step: 184790 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 00:59:09,781-Speed 128.38 samples/sec Loss 0.8471 Epoch: 18 Global Step: 184795 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 00:59:29,592-Speed 129.23 samples/sec Loss 0.9014 Epoch: 18 Global Step: 184800 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 00:59:49,504-Speed 128.57 samples/sec Loss 0.8979 Epoch: 18 Global Step: 184805 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:00:09,438-Speed 128.43 samples/sec Loss 1.0569 Epoch: 18 Global Step: 184810 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:00:29,589-Speed 127.05 samples/sec Loss 0.8711 Epoch: 18 Global Step: 184815 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:00:49,277-Speed 130.03 samples/sec Loss 0.8350 Epoch: 18 Global Step: 184820 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:01:09,199-Speed 128.51 samples/sec Loss 0.7786 Epoch: 18 Global Step: 184825 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:01:28,868-Speed 130.16 samples/sec Loss 0.9508 Epoch: 18 Global Step: 184830 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:01:48,626-Speed 129.57 samples/sec Loss 0.8218 Epoch: 18 Global Step: 184835 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:02:08,521-Speed 128.68 samples/sec Loss 0.8523 Epoch: 18 Global Step: 184840 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:02:28,224-Speed 129.94 samples/sec Loss 0.9283 Epoch: 18 Global Step: 184845 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:02:48,048-Speed 129.14 samples/sec Loss 0.9384 Epoch: 18 Global Step: 184850 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:03:08,014-Speed 128.23 samples/sec Loss 0.9164 Epoch: 18 Global Step: 184855 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:03:28,219-Speed 126.71 samples/sec Loss 0.9316 Epoch: 18 Global Step: 184860 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:03:47,926-Speed 129.90 samples/sec Loss 0.8042 Epoch: 18 Global Step: 184865 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:04:07,696-Speed 129.50 samples/sec Loss 0.9031 Epoch: 18 Global Step: 184870 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:04:27,458-Speed 129.55 samples/sec Loss 0.9113 Epoch: 18 Global Step: 184875 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:04:47,252-Speed 129.34 samples/sec Loss 0.8867 Epoch: 18 Global Step: 184880 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:05:07,154-Speed 128.64 samples/sec Loss 0.8846 Epoch: 18 Global Step: 184885 Fp16 Grad Scale: 4096 Required: 77 hours Training: 2021-06-16 01:05:26,870-Speed 129.85 samples/sec Loss 0.8330 Epoch: 18 Global Step: 184890 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:05:46,641-Speed 129.49 samples/sec Loss 0.8968 Epoch: 18 Global Step: 184895 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:06:06,775-Speed 127.16 samples/sec Loss 0.8915 Epoch: 18 Global Step: 184900 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:06:27,052-Speed 126.26 samples/sec Loss 0.9136 Epoch: 18 Global Step: 184905 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:06:46,769-Speed 129.84 samples/sec Loss 0.9446 Epoch: 18 Global Step: 184910 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:07:06,589-Speed 129.17 samples/sec Loss 0.8490 Epoch: 18 Global Step: 184915 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:07:26,348-Speed 129.57 samples/sec Loss 0.8500 Epoch: 18 Global Step: 184920 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:07:46,135-Speed 129.39 samples/sec Loss 0.8976 Epoch: 18 Global Step: 184925 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:08:06,041-Speed 128.62 samples/sec Loss 0.8344 Epoch: 18 Global Step: 184930 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:08:25,756-Speed 129.86 samples/sec Loss 0.8730 Epoch: 18 Global Step: 184935 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:08:45,554-Speed 129.31 samples/sec Loss 0.8442 Epoch: 18 Global Step: 184940 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:09:05,599-Speed 127.72 samples/sec Loss 0.8684 Epoch: 18 Global Step: 184945 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:09:25,694-Speed 127.40 samples/sec Loss 0.8798 Epoch: 18 Global Step: 184950 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:09:45,416-Speed 129.81 samples/sec Loss 0.8405 Epoch: 18 Global Step: 184955 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:10:05,068-Speed 130.28 samples/sec Loss 0.9598 Epoch: 18 Global Step: 184960 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:10:24,864-Speed 129.32 samples/sec Loss 0.8755 Epoch: 18 Global Step: 184965 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:10:44,623-Speed 129.57 samples/sec Loss 0.8060 Epoch: 18 Global Step: 184970 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:11:04,539-Speed 128.54 samples/sec Loss 0.8360 Epoch: 18 Global Step: 184975 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:11:24,265-Speed 129.79 samples/sec Loss 0.9069 Epoch: 18 Global Step: 184980 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:11:43,934-Speed 130.16 samples/sec Loss 0.9498 Epoch: 18 Global Step: 184985 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:12:04,152-Speed 126.63 samples/sec Loss 0.8710 Epoch: 18 Global Step: 184990 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:12:24,248-Speed 127.39 samples/sec Loss 0.8934 Epoch: 18 Global Step: 184995 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:12:43,979-Speed 129.75 samples/sec Loss 0.8208 Epoch: 18 Global Step: 185000 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:13:03,668-Speed 130.03 samples/sec Loss 0.8333 Epoch: 18 Global Step: 185005 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:13:23,464-Speed 129.33 samples/sec Loss 0.8280 Epoch: 18 Global Step: 185010 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:13:43,338-Speed 128.81 samples/sec Loss 0.8714 Epoch: 18 Global Step: 185015 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:14:03,149-Speed 129.23 samples/sec Loss 0.7684 Epoch: 18 Global Step: 185020 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:14:22,860-Speed 129.88 samples/sec Loss 0.8702 Epoch: 18 Global Step: 185025 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:14:42,697-Speed 129.06 samples/sec Loss 0.8369 Epoch: 18 Global Step: 185030 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:15:02,782-Speed 127.46 samples/sec Loss 0.8559 Epoch: 18 Global Step: 185035 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:15:23,432-Speed 123.98 samples/sec Loss 0.8868 Epoch: 18 Global Step: 185040 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:15:43,164-Speed 129.75 samples/sec Loss 0.8431 Epoch: 18 Global Step: 185045 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:16:02,852-Speed 130.03 samples/sec Loss 0.7988 Epoch: 18 Global Step: 185050 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:16:22,710-Speed 128.92 samples/sec Loss 0.9242 Epoch: 18 Global Step: 185055 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:16:42,518-Speed 129.25 samples/sec Loss 0.8165 Epoch: 18 Global Step: 185060 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:17:02,344-Speed 129.14 samples/sec Loss 0.8178 Epoch: 18 Global Step: 185065 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:17:22,060-Speed 129.85 samples/sec Loss 0.7516 Epoch: 18 Global Step: 185070 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:17:41,828-Speed 129.51 samples/sec Loss 0.9625 Epoch: 18 Global Step: 185075 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:18:01,816-Speed 128.08 samples/sec Loss 0.8302 Epoch: 18 Global Step: 185080 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:18:21,811-Speed 128.04 samples/sec Loss 0.8360 Epoch: 18 Global Step: 185085 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:18:41,538-Speed 129.78 samples/sec Loss 0.8643 Epoch: 18 Global Step: 185090 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:19:01,264-Speed 129.78 samples/sec Loss 0.8910 Epoch: 18 Global Step: 185095 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:19:21,413-Speed 127.06 samples/sec Loss 0.8627 Epoch: 18 Global Step: 185100 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:19:41,214-Speed 129.29 samples/sec Loss 0.7720 Epoch: 18 Global Step: 185105 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:20:01,029-Speed 129.20 samples/sec Loss 0.8334 Epoch: 18 Global Step: 185110 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:20:20,773-Speed 129.67 samples/sec Loss 0.8837 Epoch: 18 Global Step: 185115 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:20:40,496-Speed 129.81 samples/sec Loss 0.7998 Epoch: 18 Global Step: 185120 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:21:00,655-Speed 127.00 samples/sec Loss 0.9235 Epoch: 18 Global Step: 185125 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:21:20,647-Speed 128.06 samples/sec Loss 0.8547 Epoch: 18 Global Step: 185130 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:21:40,424-Speed 129.45 samples/sec Loss 0.8570 Epoch: 18 Global Step: 185135 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:22:00,244-Speed 129.17 samples/sec Loss 0.8987 Epoch: 18 Global Step: 185140 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:22:20,001-Speed 129.58 samples/sec Loss 0.7736 Epoch: 18 Global Step: 185145 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:22:39,811-Speed 129.24 samples/sec Loss 0.8233 Epoch: 18 Global Step: 185150 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:22:59,650-Speed 129.05 samples/sec Loss 0.8204 Epoch: 18 Global Step: 185155 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:23:19,303-Speed 130.27 samples/sec Loss 0.8049 Epoch: 18 Global Step: 185160 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:23:39,156-Speed 128.96 samples/sec Loss 0.8855 Epoch: 18 Global Step: 185165 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:23:59,221-Speed 127.59 samples/sec Loss 0.9513 Epoch: 18 Global Step: 185170 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:24:19,299-Speed 127.51 samples/sec Loss 0.8498 Epoch: 18 Global Step: 185175 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:24:38,995-Speed 129.98 samples/sec Loss 0.9433 Epoch: 18 Global Step: 185180 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:24:58,905-Speed 128.59 samples/sec Loss 0.7330 Epoch: 18 Global Step: 185185 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:25:18,614-Speed 129.89 samples/sec Loss 0.9192 Epoch: 18 Global Step: 185190 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:25:38,427-Speed 129.21 samples/sec Loss 0.7881 Epoch: 18 Global Step: 185195 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:25:58,241-Speed 129.21 samples/sec Loss 0.8008 Epoch: 18 Global Step: 185200 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:26:18,099-Speed 128.92 samples/sec Loss 0.9063 Epoch: 18 Global Step: 185205 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:26:37,992-Speed 128.70 samples/sec Loss 0.8337 Epoch: 18 Global Step: 185210 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:26:58,057-Speed 127.59 samples/sec Loss 0.8128 Epoch: 18 Global Step: 185215 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:27:18,129-Speed 127.55 samples/sec Loss 0.8068 Epoch: 18 Global Step: 185220 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:27:37,746-Speed 130.50 samples/sec Loss 0.9127 Epoch: 18 Global Step: 185225 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:27:57,626-Speed 128.77 samples/sec Loss 0.8674 Epoch: 18 Global Step: 185230 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:28:17,546-Speed 128.52 samples/sec Loss 0.8553 Epoch: 18 Global Step: 185235 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:28:37,475-Speed 128.47 samples/sec Loss 0.7970 Epoch: 18 Global Step: 185240 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:28:57,249-Speed 129.47 samples/sec Loss 0.9036 Epoch: 18 Global Step: 185245 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:29:16,957-Speed 129.90 samples/sec Loss 0.8225 Epoch: 18 Global Step: 185250 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:29:36,865-Speed 128.60 samples/sec Loss 1.0178 Epoch: 18 Global Step: 185255 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:29:57,011-Speed 127.07 samples/sec Loss 0.7829 Epoch: 18 Global Step: 185260 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:30:17,137-Speed 127.21 samples/sec Loss 0.9928 Epoch: 18 Global Step: 185265 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:30:36,915-Speed 129.44 samples/sec Loss 0.8487 Epoch: 18 Global Step: 185270 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:30:56,751-Speed 129.06 samples/sec Loss 0.8096 Epoch: 18 Global Step: 185275 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:31:16,531-Speed 129.43 samples/sec Loss 0.9022 Epoch: 18 Global Step: 185280 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:31:36,523-Speed 128.06 samples/sec Loss 0.8775 Epoch: 18 Global Step: 185285 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:31:56,287-Speed 129.53 samples/sec Loss 0.7696 Epoch: 18 Global Step: 185290 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:32:16,040-Speed 129.61 samples/sec Loss 0.7599 Epoch: 18 Global Step: 185295 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:32:35,885-Speed 129.01 samples/sec Loss 0.8414 Epoch: 18 Global Step: 185300 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:32:56,038-Speed 127.03 samples/sec Loss 0.8885 Epoch: 18 Global Step: 185305 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:33:16,188-Speed 127.05 samples/sec Loss 0.8749 Epoch: 18 Global Step: 185310 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:33:35,950-Speed 129.55 samples/sec Loss 0.9466 Epoch: 18 Global Step: 185315 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:33:55,713-Speed 129.54 samples/sec Loss 0.7848 Epoch: 18 Global Step: 185320 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:34:15,396-Speed 130.07 samples/sec Loss 1.0254 Epoch: 18 Global Step: 185325 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:34:35,344-Speed 128.34 samples/sec Loss 0.8816 Epoch: 18 Global Step: 185330 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:34:55,027-Speed 130.07 samples/sec Loss 0.8007 Epoch: 18 Global Step: 185335 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:35:14,757-Speed 129.76 samples/sec Loss 0.8335 Epoch: 18 Global Step: 185340 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:35:34,554-Speed 129.32 samples/sec Loss 0.7986 Epoch: 18 Global Step: 185345 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:35:54,534-Speed 128.13 samples/sec Loss 0.9107 Epoch: 18 Global Step: 185350 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:36:14,615-Speed 127.49 samples/sec Loss 0.8473 Epoch: 18 Global Step: 185355 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:36:34,364-Speed 129.63 samples/sec Loss 0.8638 Epoch: 18 Global Step: 185360 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:36:54,220-Speed 128.94 samples/sec Loss 0.7714 Epoch: 18 Global Step: 185365 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:37:13,923-Speed 129.94 samples/sec Loss 0.8730 Epoch: 18 Global Step: 185370 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:37:33,663-Speed 129.69 samples/sec Loss 0.8683 Epoch: 18 Global Step: 185375 Fp16 Grad Scale: 8192 Required: 77 hours Training: 2021-06-16 01:37:53,420-Speed 129.58 samples/sec Loss 0.7559 Epoch: 18 Global Step: 185380 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:38:13,157-Speed 129.71 samples/sec Loss 0.9642 Epoch: 18 Global Step: 185385 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:38:33,000-Speed 129.02 samples/sec Loss 0.8709 Epoch: 18 Global Step: 185390 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:38:53,066-Speed 127.59 samples/sec Loss 0.8129 Epoch: 18 Global Step: 185395 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:39:13,138-Speed 127.55 samples/sec Loss 0.8861 Epoch: 18 Global Step: 185400 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:39:32,864-Speed 129.79 samples/sec Loss 0.8157 Epoch: 18 Global Step: 185405 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:39:52,713-Speed 128.98 samples/sec Loss 0.8099 Epoch: 18 Global Step: 185410 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:40:12,428-Speed 129.86 samples/sec Loss 0.9245 Epoch: 18 Global Step: 185415 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:40:32,410-Speed 128.12 samples/sec Loss 0.8475 Epoch: 18 Global Step: 185420 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:40:52,204-Speed 129.34 samples/sec Loss 0.8343 Epoch: 18 Global Step: 185425 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:41:11,952-Speed 129.64 samples/sec Loss 0.8856 Epoch: 18 Global Step: 185430 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:41:31,790-Speed 129.05 samples/sec Loss 0.8539 Epoch: 18 Global Step: 185435 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:41:51,795-Speed 127.98 samples/sec Loss 0.8174 Epoch: 18 Global Step: 185440 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:42:11,875-Speed 127.50 samples/sec Loss 0.9052 Epoch: 18 Global Step: 185445 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:42:31,561-Speed 130.04 samples/sec Loss 1.0061 Epoch: 18 Global Step: 185450 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:42:51,350-Speed 129.37 samples/sec Loss 0.9783 Epoch: 18 Global Step: 185455 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:43:10,961-Speed 130.55 samples/sec Loss 0.8394 Epoch: 18 Global Step: 185460 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:43:30,870-Speed 128.59 samples/sec Loss 0.9748 Epoch: 18 Global Step: 185465 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:43:50,567-Speed 129.97 samples/sec Loss 0.8091 Epoch: 18 Global Step: 185470 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:44:10,315-Speed 129.64 samples/sec Loss 0.8602 Epoch: 18 Global Step: 185475 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:44:30,159-Speed 129.01 samples/sec Loss 1.0143 Epoch: 18 Global Step: 185480 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:44:50,514-Speed 125.77 samples/sec Loss 0.9440 Epoch: 18 Global Step: 185485 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:45:10,572-Speed 127.64 samples/sec Loss 0.8683 Epoch: 18 Global Step: 185490 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:45:30,331-Speed 129.57 samples/sec Loss 0.9139 Epoch: 18 Global Step: 185495 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:45:50,173-Speed 129.03 samples/sec Loss 0.8670 Epoch: 18 Global Step: 185500 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:46:09,899-Speed 129.78 samples/sec Loss 1.0460 Epoch: 18 Global Step: 185505 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:46:29,858-Speed 128.27 samples/sec Loss 0.8508 Epoch: 18 Global Step: 185510 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:46:49,530-Speed 130.14 samples/sec Loss 0.7946 Epoch: 18 Global Step: 185515 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:47:09,307-Speed 129.45 samples/sec Loss 0.9070 Epoch: 18 Global Step: 185520 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:47:29,664-Speed 125.76 samples/sec Loss 0.9022 Epoch: 18 Global Step: 185525 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:47:49,599-Speed 128.42 samples/sec Loss 0.8300 Epoch: 18 Global Step: 185530 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:48:09,720-Speed 127.24 samples/sec Loss 0.7564 Epoch: 18 Global Step: 185535 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:48:29,604-Speed 128.75 samples/sec Loss 0.8622 Epoch: 18 Global Step: 185540 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:48:49,314-Speed 129.89 samples/sec Loss 0.9554 Epoch: 18 Global Step: 185545 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:49:09,005-Speed 130.02 samples/sec Loss 0.9197 Epoch: 18 Global Step: 185550 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:49:28,931-Speed 128.48 samples/sec Loss 1.0234 Epoch: 18 Global Step: 185555 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:49:48,646-Speed 129.86 samples/sec Loss 0.8562 Epoch: 18 Global Step: 185560 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:50:08,576-Speed 128.46 samples/sec Loss 0.8638 Epoch: 18 Global Step: 185565 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:50:28,229-Speed 130.26 samples/sec Loss 1.0000 Epoch: 18 Global Step: 185570 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:50:48,254-Speed 127.85 samples/sec Loss 0.9927 Epoch: 18 Global Step: 185575 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:51:08,313-Speed 127.64 samples/sec Loss 0.8436 Epoch: 18 Global Step: 185580 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:51:28,252-Speed 128.40 samples/sec Loss 0.8648 Epoch: 18 Global Step: 185585 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:51:47,970-Speed 129.83 samples/sec Loss 0.9167 Epoch: 18 Global Step: 185590 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:52:07,828-Speed 128.93 samples/sec Loss 0.7996 Epoch: 18 Global Step: 185595 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:52:27,678-Speed 128.97 samples/sec Loss 0.8531 Epoch: 18 Global Step: 185600 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:52:47,415-Speed 129.72 samples/sec Loss 0.8480 Epoch: 18 Global Step: 185605 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:53:07,315-Speed 128.65 samples/sec Loss 0.8590 Epoch: 18 Global Step: 185610 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:53:27,025-Speed 129.89 samples/sec Loss 0.8970 Epoch: 18 Global Step: 185615 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:53:47,057-Speed 127.80 samples/sec Loss 0.9451 Epoch: 18 Global Step: 185620 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:54:07,061-Speed 127.98 samples/sec Loss 0.8990 Epoch: 18 Global Step: 185625 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:54:26,945-Speed 128.75 samples/sec Loss 0.8821 Epoch: 18 Global Step: 185630 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:54:46,641-Speed 129.98 samples/sec Loss 0.8230 Epoch: 18 Global Step: 185635 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:55:06,369-Speed 129.77 samples/sec Loss 0.8889 Epoch: 18 Global Step: 185640 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:55:26,224-Speed 128.94 samples/sec Loss 0.8360 Epoch: 18 Global Step: 185645 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:55:45,988-Speed 129.53 samples/sec Loss 0.8036 Epoch: 18 Global Step: 185650 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:56:05,901-Speed 128.57 samples/sec Loss 0.9042 Epoch: 18 Global Step: 185655 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:56:25,598-Speed 129.97 samples/sec Loss 0.7853 Epoch: 18 Global Step: 185660 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:56:45,550-Speed 128.32 samples/sec Loss 0.8471 Epoch: 18 Global Step: 185665 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:57:05,583-Speed 127.79 samples/sec Loss 0.8787 Epoch: 18 Global Step: 185670 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:57:25,476-Speed 128.69 samples/sec Loss 0.9155 Epoch: 18 Global Step: 185675 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:57:45,180-Speed 129.93 samples/sec Loss 0.8798 Epoch: 18 Global Step: 185680 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:58:04,979-Speed 129.30 samples/sec Loss 0.9060 Epoch: 18 Global Step: 185685 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:58:24,782-Speed 129.28 samples/sec Loss 0.8427 Epoch: 18 Global Step: 185690 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:58:44,478-Speed 129.98 samples/sec Loss 0.8068 Epoch: 18 Global Step: 185695 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:59:04,384-Speed 128.61 samples/sec Loss 0.8325 Epoch: 18 Global Step: 185700 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:59:24,073-Speed 130.03 samples/sec Loss 0.7626 Epoch: 18 Global Step: 185705 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 01:59:44,188-Speed 127.27 samples/sec Loss 0.8896 Epoch: 18 Global Step: 185710 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:00:04,172-Speed 128.11 samples/sec Loss 0.9058 Epoch: 18 Global Step: 185715 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:00:24,023-Speed 128.97 samples/sec Loss 0.8476 Epoch: 18 Global Step: 185720 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:00:43,654-Speed 130.41 samples/sec Loss 0.8600 Epoch: 18 Global Step: 185725 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:01:03,601-Speed 128.35 samples/sec Loss 0.9010 Epoch: 18 Global Step: 185730 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:01:23,389-Speed 129.38 samples/sec Loss 0.8155 Epoch: 18 Global Step: 185735 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:01:43,049-Speed 130.22 samples/sec Loss 0.9120 Epoch: 18 Global Step: 185740 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:02:02,938-Speed 128.72 samples/sec Loss 0.9054 Epoch: 18 Global Step: 185745 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:02:22,657-Speed 129.83 samples/sec Loss 1.0321 Epoch: 18 Global Step: 185750 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:02:42,822-Speed 126.96 samples/sec Loss 0.8410 Epoch: 18 Global Step: 185755 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:03:02,752-Speed 128.46 samples/sec Loss 0.8709 Epoch: 18 Global Step: 185760 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:03:22,518-Speed 129.52 samples/sec Loss 0.7998 Epoch: 18 Global Step: 185765 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:03:42,215-Speed 129.98 samples/sec Loss 0.9250 Epoch: 18 Global Step: 185770 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:04:02,081-Speed 128.87 samples/sec Loss 0.7953 Epoch: 18 Global Step: 185775 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:04:21,798-Speed 129.85 samples/sec Loss 0.8815 Epoch: 18 Global Step: 185780 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:04:41,646-Speed 128.98 samples/sec Loss 0.8288 Epoch: 18 Global Step: 185785 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:05:01,534-Speed 128.73 samples/sec Loss 0.8597 Epoch: 18 Global Step: 185790 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:05:21,168-Speed 130.39 samples/sec Loss 0.8524 Epoch: 18 Global Step: 185795 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:05:41,259-Speed 127.42 samples/sec Loss 0.8189 Epoch: 18 Global Step: 185800 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:06:01,247-Speed 128.09 samples/sec Loss 0.8631 Epoch: 18 Global Step: 185805 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:06:21,053-Speed 129.26 samples/sec Loss 0.7912 Epoch: 18 Global Step: 185810 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:06:40,748-Speed 129.99 samples/sec Loss 0.7707 Epoch: 18 Global Step: 185815 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:07:00,545-Speed 129.32 samples/sec Loss 0.8486 Epoch: 18 Global Step: 185820 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:07:20,270-Speed 129.79 samples/sec Loss 0.7236 Epoch: 18 Global Step: 185825 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:07:40,398-Speed 127.19 samples/sec Loss 0.8433 Epoch: 18 Global Step: 185830 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:08:00,370-Speed 128.19 samples/sec Loss 0.8416 Epoch: 18 Global Step: 185835 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:08:20,144-Speed 129.47 samples/sec Loss 0.8620 Epoch: 18 Global Step: 185840 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:08:40,206-Speed 127.61 samples/sec Loss 0.8317 Epoch: 18 Global Step: 185845 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:09:00,391-Speed 126.84 samples/sec Loss 0.7742 Epoch: 18 Global Step: 185850 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:09:20,046-Speed 130.25 samples/sec Loss 0.8255 Epoch: 18 Global Step: 185855 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:09:39,825-Speed 129.43 samples/sec Loss 0.7921 Epoch: 18 Global Step: 185860 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:09:59,628-Speed 129.28 samples/sec Loss 0.9238 Epoch: 18 Global Step: 185865 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:10:19,344-Speed 129.85 samples/sec Loss 0.8926 Epoch: 18 Global Step: 185870 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:10:39,249-Speed 128.61 samples/sec Loss 0.9177 Epoch: 18 Global Step: 185875 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:10:58,940-Speed 130.02 samples/sec Loss 0.8376 Epoch: 18 Global Step: 185880 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:11:18,704-Speed 129.54 samples/sec Loss 0.9614 Epoch: 18 Global Step: 185885 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:11:38,763-Speed 127.63 samples/sec Loss 0.9701 Epoch: 18 Global Step: 185890 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:11:58,824-Speed 127.62 samples/sec Loss 0.7579 Epoch: 18 Global Step: 185895 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:12:18,504-Speed 130.08 samples/sec Loss 0.8784 Epoch: 18 Global Step: 185900 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:12:38,272-Speed 129.50 samples/sec Loss 0.7970 Epoch: 18 Global Step: 185905 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:12:58,122-Speed 128.98 samples/sec Loss 0.7961 Epoch: 18 Global Step: 185910 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:13:17,832-Speed 129.89 samples/sec Loss 0.8328 Epoch: 18 Global Step: 185915 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:13:38,163-Speed 125.92 samples/sec Loss 0.9195 Epoch: 18 Global Step: 185920 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:13:57,790-Speed 130.44 samples/sec Loss 0.7413 Epoch: 18 Global Step: 185925 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:14:17,595-Speed 129.27 samples/sec Loss 0.9213 Epoch: 18 Global Step: 185930 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:14:37,299-Speed 129.93 samples/sec Loss 0.8499 Epoch: 18 Global Step: 185935 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:14:57,033-Speed 129.73 samples/sec Loss 0.9697 Epoch: 18 Global Step: 185940 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:15:16,773-Speed 129.69 samples/sec Loss 0.9008 Epoch: 18 Global Step: 185945 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:15:36,846-Speed 127.54 samples/sec Loss 0.7911 Epoch: 18 Global Step: 185950 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:15:56,564-Speed 129.84 samples/sec Loss 0.8597 Epoch: 18 Global Step: 185955 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:16:16,336-Speed 129.48 samples/sec Loss 0.8750 Epoch: 18 Global Step: 185960 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:16:36,271-Speed 128.42 samples/sec Loss 0.8888 Epoch: 18 Global Step: 185965 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:16:55,888-Speed 130.51 samples/sec Loss 0.8595 Epoch: 18 Global Step: 185970 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:17:15,810-Speed 128.51 samples/sec Loss 0.9158 Epoch: 18 Global Step: 185975 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:17:36,234-Speed 125.35 samples/sec Loss 0.8090 Epoch: 18 Global Step: 185980 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:17:56,200-Speed 128.23 samples/sec Loss 0.8399 Epoch: 18 Global Step: 185985 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:18:15,808-Speed 130.57 samples/sec Loss 0.8525 Epoch: 18 Global Step: 185990 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:18:35,897-Speed 127.44 samples/sec Loss 0.8514 Epoch: 18 Global Step: 185995 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:18:55,735-Speed 129.05 samples/sec Loss 0.7800 Epoch: 18 Global Step: 186000 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:19:15,497-Speed 129.55 samples/sec Loss 0.8266 Epoch: 18 Global Step: 186005 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:19:35,416-Speed 128.53 samples/sec Loss 0.8458 Epoch: 18 Global Step: 186010 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:19:55,041-Speed 130.45 samples/sec Loss 0.8606 Epoch: 18 Global Step: 186015 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:20:14,938-Speed 128.67 samples/sec Loss 0.8684 Epoch: 18 Global Step: 186020 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:20:34,946-Speed 127.95 samples/sec Loss 0.8359 Epoch: 18 Global Step: 186025 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:20:54,775-Speed 129.12 samples/sec Loss 0.9457 Epoch: 18 Global Step: 186030 Fp16 Grad Scale: 4096 Required: 76 hours Training: 2021-06-16 02:21:14,520-Speed 129.66 samples/sec Loss 0.7905 Epoch: 18 Global Step: 186035 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:21:34,588-Speed 127.57 samples/sec Loss 0.8757 Epoch: 18 Global Step: 186040 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:21:54,207-Speed 130.49 samples/sec Loss 0.8830 Epoch: 18 Global Step: 186045 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:22:13,978-Speed 129.49 samples/sec Loss 0.7674 Epoch: 18 Global Step: 186050 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:22:33,906-Speed 128.47 samples/sec Loss 0.7865 Epoch: 18 Global Step: 186055 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:22:53,557-Speed 130.28 samples/sec Loss 0.9603 Epoch: 18 Global Step: 186060 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:23:13,457-Speed 128.65 samples/sec Loss 0.8316 Epoch: 18 Global Step: 186065 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:23:33,409-Speed 128.31 samples/sec Loss 0.8382 Epoch: 18 Global Step: 186070 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:23:53,301-Speed 128.70 samples/sec Loss 0.8585 Epoch: 18 Global Step: 186075 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:24:13,134-Speed 129.09 samples/sec Loss 0.8978 Epoch: 18 Global Step: 186080 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:24:33,151-Speed 127.90 samples/sec Loss 0.8898 Epoch: 18 Global Step: 186085 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:24:52,874-Speed 129.80 samples/sec Loss 0.9032 Epoch: 18 Global Step: 186090 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:25:12,610-Speed 129.72 samples/sec Loss 0.7904 Epoch: 18 Global Step: 186095 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:25:32,432-Speed 129.15 samples/sec Loss 0.8658 Epoch: 18 Global Step: 186100 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:25:52,449-Speed 127.90 samples/sec Loss 0.8404 Epoch: 18 Global Step: 186105 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:26:12,389-Speed 128.39 samples/sec Loss 0.9470 Epoch: 18 Global Step: 186110 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:26:32,319-Speed 128.46 samples/sec Loss 0.8892 Epoch: 18 Global Step: 186115 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:26:52,011-Speed 130.02 samples/sec Loss 0.9533 Epoch: 18 Global Step: 186120 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:27:11,919-Speed 128.60 samples/sec Loss 0.9953 Epoch: 18 Global Step: 186125 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:27:31,992-Speed 127.54 samples/sec Loss 0.9122 Epoch: 18 Global Step: 186130 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:27:51,621-Speed 130.42 samples/sec Loss 0.8338 Epoch: 18 Global Step: 186135 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:28:11,395-Speed 129.47 samples/sec Loss 0.8734 Epoch: 18 Global Step: 186140 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:28:31,170-Speed 129.46 samples/sec Loss 0.8743 Epoch: 18 Global Step: 186145 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:28:50,957-Speed 129.39 samples/sec Loss 0.9353 Epoch: 18 Global Step: 186150 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:29:10,840-Speed 128.76 samples/sec Loss 0.9402 Epoch: 18 Global Step: 186155 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:29:30,844-Speed 127.98 samples/sec Loss 0.8794 Epoch: 18 Global Step: 186160 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:29:50,569-Speed 129.79 samples/sec Loss 0.8373 Epoch: 18 Global Step: 186165 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:30:10,441-Speed 128.83 samples/sec Loss 0.9790 Epoch: 18 Global Step: 186170 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:30:30,539-Speed 127.39 samples/sec Loss 0.8729 Epoch: 18 Global Step: 186175 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:30:50,227-Speed 130.03 samples/sec Loss 0.7537 Epoch: 18 Global Step: 186180 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:31:09,983-Speed 129.59 samples/sec Loss 0.9744 Epoch: 18 Global Step: 186185 Fp16 Grad Scale: 8192 Required: 76 hours Training: 2021-06-16 02:31:29,788-Speed 129.27 samples/sec Loss 0.9405 Epoch: 18 Global Step: 186190 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:31:49,581-Speed 129.35 samples/sec Loss 0.8731 Epoch: 18 Global Step: 186195 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:32:09,373-Speed 129.35 samples/sec Loss 0.9148 Epoch: 18 Global Step: 186200 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:32:29,345-Speed 128.18 samples/sec Loss 0.9028 Epoch: 18 Global Step: 186205 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:32:49,182-Speed 129.06 samples/sec Loss 0.7689 Epoch: 18 Global Step: 186210 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:33:08,979-Speed 129.32 samples/sec Loss 0.7842 Epoch: 18 Global Step: 186215 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:33:29,056-Speed 127.52 samples/sec Loss 0.8399 Epoch: 18 Global Step: 186220 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:33:48,759-Speed 129.94 samples/sec Loss 0.8939 Epoch: 18 Global Step: 186225 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:34:08,469-Speed 129.88 samples/sec Loss 0.9137 Epoch: 18 Global Step: 186230 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:34:28,230-Speed 129.56 samples/sec Loss 0.7612 Epoch: 18 Global Step: 186235 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:34:48,019-Speed 129.37 samples/sec Loss 0.9243 Epoch: 18 Global Step: 186240 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:35:07,814-Speed 129.33 samples/sec Loss 0.9246 Epoch: 18 Global Step: 186245 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:35:27,721-Speed 128.60 samples/sec Loss 0.9029 Epoch: 18 Global Step: 186250 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:35:47,565-Speed 129.01 samples/sec Loss 0.8530 Epoch: 18 Global Step: 186255 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:36:07,365-Speed 129.30 samples/sec Loss 0.9183 Epoch: 18 Global Step: 186260 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:36:27,453-Speed 127.44 samples/sec Loss 0.9138 Epoch: 18 Global Step: 186265 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:36:47,155-Speed 129.94 samples/sec Loss 0.8398 Epoch: 18 Global Step: 186270 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:37:06,819-Speed 130.19 samples/sec Loss 0.9658 Epoch: 18 Global Step: 186275 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:37:26,694-Speed 128.81 samples/sec Loss 0.9610 Epoch: 18 Global Step: 186280 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:37:46,412-Speed 129.84 samples/sec Loss 0.9689 Epoch: 18 Global Step: 186285 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:38:06,378-Speed 128.22 samples/sec Loss 0.7939 Epoch: 18 Global Step: 186290 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:38:26,270-Speed 128.70 samples/sec Loss 0.7652 Epoch: 18 Global Step: 186295 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:38:46,223-Speed 128.31 samples/sec Loss 0.8010 Epoch: 18 Global Step: 186300 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:39:06,114-Speed 128.71 samples/sec Loss 1.0072 Epoch: 18 Global Step: 186305 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:39:26,273-Speed 126.99 samples/sec Loss 0.9025 Epoch: 18 Global Step: 186310 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:39:45,982-Speed 129.90 samples/sec Loss 0.7188 Epoch: 18 Global Step: 186315 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:40:05,741-Speed 129.56 samples/sec Loss 0.8038 Epoch: 18 Global Step: 186320 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:40:25,613-Speed 128.83 samples/sec Loss 0.9051 Epoch: 18 Global Step: 186325 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:40:45,340-Speed 129.78 samples/sec Loss 0.9604 Epoch: 18 Global Step: 186330 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:41:05,279-Speed 128.40 samples/sec Loss 0.8818 Epoch: 18 Global Step: 186335 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:41:25,193-Speed 128.56 samples/sec Loss 0.8866 Epoch: 18 Global Step: 186340 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:41:44,968-Speed 129.47 samples/sec Loss 0.9041 Epoch: 18 Global Step: 186345 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:42:05,001-Speed 127.79 samples/sec Loss 1.0421 Epoch: 18 Global Step: 186350 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:42:25,152-Speed 127.05 samples/sec Loss 0.8654 Epoch: 18 Global Step: 186355 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:42:44,875-Speed 129.80 samples/sec Loss 0.8502 Epoch: 18 Global Step: 186360 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:43:05,282-Speed 125.45 samples/sec Loss 0.9249 Epoch: 18 Global Step: 186365 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:43:25,170-Speed 128.73 samples/sec Loss 0.9176 Epoch: 18 Global Step: 186370 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:43:44,823-Speed 130.27 samples/sec Loss 0.9170 Epoch: 18 Global Step: 186375 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:44:04,762-Speed 128.40 samples/sec Loss 0.7836 Epoch: 18 Global Step: 186380 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:44:24,746-Speed 128.11 samples/sec Loss 0.8796 Epoch: 18 Global Step: 186385 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:44:44,566-Speed 129.17 samples/sec Loss 0.8384 Epoch: 18 Global Step: 186390 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:45:04,273-Speed 129.91 samples/sec Loss 0.8406 Epoch: 18 Global Step: 186395 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:45:24,419-Speed 127.08 samples/sec Loss 0.8484 Epoch: 18 Global Step: 186400 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:45:44,158-Speed 129.70 samples/sec Loss 0.9878 Epoch: 18 Global Step: 186405 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:46:03,907-Speed 129.64 samples/sec Loss 0.8946 Epoch: 18 Global Step: 186410 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:46:23,760-Speed 128.95 samples/sec Loss 0.7496 Epoch: 18 Global Step: 186415 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:46:43,474-Speed 129.87 samples/sec Loss 0.9819 Epoch: 18 Global Step: 186420 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:47:03,420-Speed 128.35 samples/sec Loss 0.8746 Epoch: 18 Global Step: 186425 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:47:23,459-Speed 127.76 samples/sec Loss 0.8526 Epoch: 18 Global Step: 186430 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:47:43,384-Speed 128.49 samples/sec Loss 0.9874 Epoch: 18 Global Step: 186435 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:48:03,029-Speed 130.32 samples/sec Loss 0.8364 Epoch: 18 Global Step: 186440 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:48:23,142-Speed 127.29 samples/sec Loss 0.8740 Epoch: 18 Global Step: 186445 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:48:42,876-Speed 129.73 samples/sec Loss 0.8636 Epoch: 18 Global Step: 186450 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:49:03,016-Speed 127.11 samples/sec Loss 0.8884 Epoch: 18 Global Step: 186455 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:49:22,728-Speed 129.87 samples/sec Loss 0.9039 Epoch: 18 Global Step: 186460 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:49:42,563-Speed 129.07 samples/sec Loss 0.9550 Epoch: 18 Global Step: 186465 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:50:02,322-Speed 129.57 samples/sec Loss 0.7286 Epoch: 18 Global Step: 186470 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:50:22,278-Speed 128.29 samples/sec Loss 0.8263 Epoch: 18 Global Step: 186475 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:50:42,194-Speed 128.55 samples/sec Loss 0.8322 Epoch: 18 Global Step: 186480 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:51:01,863-Speed 130.16 samples/sec Loss 0.8435 Epoch: 18 Global Step: 186485 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:51:21,941-Speed 127.51 samples/sec Loss 0.8600 Epoch: 18 Global Step: 186490 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:51:41,597-Speed 130.25 samples/sec Loss 0.9883 Epoch: 18 Global Step: 186495 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:52:01,496-Speed 128.66 samples/sec Loss 0.9022 Epoch: 18 Global Step: 186500 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:52:21,205-Speed 129.89 samples/sec Loss 0.8600 Epoch: 18 Global Step: 186505 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:52:40,961-Speed 129.59 samples/sec Loss 0.9134 Epoch: 18 Global Step: 186510 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:53:00,789-Speed 129.12 samples/sec Loss 0.8852 Epoch: 18 Global Step: 186515 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:53:20,803-Speed 127.92 samples/sec Loss 0.8278 Epoch: 18 Global Step: 186520 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:53:40,850-Speed 127.72 samples/sec Loss 0.8986 Epoch: 18 Global Step: 186525 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:54:00,537-Speed 130.04 samples/sec Loss 0.8935 Epoch: 18 Global Step: 186530 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:54:20,666-Speed 127.18 samples/sec Loss 0.8824 Epoch: 18 Global Step: 186535 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:54:40,397-Speed 129.75 samples/sec Loss 0.8672 Epoch: 18 Global Step: 186540 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:55:00,289-Speed 128.70 samples/sec Loss 0.9180 Epoch: 18 Global Step: 186545 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:55:20,011-Speed 129.81 samples/sec Loss 0.8686 Epoch: 18 Global Step: 186550 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:55:39,796-Speed 129.40 samples/sec Loss 0.9236 Epoch: 18 Global Step: 186555 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:55:59,645-Speed 128.98 samples/sec Loss 0.7981 Epoch: 18 Global Step: 186560 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:56:19,579-Speed 128.43 samples/sec Loss 0.8352 Epoch: 18 Global Step: 186565 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:56:39,531-Speed 128.32 samples/sec Loss 0.8262 Epoch: 18 Global Step: 186570 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:56:59,259-Speed 129.77 samples/sec Loss 0.9198 Epoch: 18 Global Step: 186575 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:57:19,330-Speed 127.55 samples/sec Loss 0.8174 Epoch: 18 Global Step: 186580 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:57:39,097-Speed 129.51 samples/sec Loss 0.8995 Epoch: 18 Global Step: 186585 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:57:59,349-Speed 126.41 samples/sec Loss 0.8486 Epoch: 18 Global Step: 186590 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:58:19,076-Speed 129.78 samples/sec Loss 0.8127 Epoch: 18 Global Step: 186595 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:58:39,036-Speed 128.26 samples/sec Loss 1.0883 Epoch: 18 Global Step: 186600 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:58:58,658-Speed 130.47 samples/sec Loss 0.8342 Epoch: 18 Global Step: 186605 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:59:18,660-Speed 128.00 samples/sec Loss 0.9157 Epoch: 18 Global Step: 186610 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:59:38,553-Speed 128.70 samples/sec Loss 0.9241 Epoch: 18 Global Step: 186615 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 02:59:58,201-Speed 130.30 samples/sec Loss 0.8525 Epoch: 18 Global Step: 186620 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:00:18,293-Speed 127.42 samples/sec Loss 0.8329 Epoch: 18 Global Step: 186625 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:00:38,028-Speed 129.72 samples/sec Loss 0.9574 Epoch: 18 Global Step: 186630 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:00:57,500-Speed 131.48 samples/sec Loss 0.8199 Epoch: 18 Global Step: 186635 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:01:17,224-Speed 129.80 samples/sec Loss 0.8537 Epoch: 18 Global Step: 186640 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:01:37,138-Speed 128.57 samples/sec Loss 0.8483 Epoch: 18 Global Step: 186645 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:01:56,921-Speed 129.41 samples/sec Loss 0.8003 Epoch: 18 Global Step: 186650 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:02:16,696-Speed 129.46 samples/sec Loss 0.8790 Epoch: 18 Global Step: 186655 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:02:36,597-Speed 128.64 samples/sec Loss 0.8634 Epoch: 18 Global Step: 186660 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:02:56,657-Speed 127.62 samples/sec Loss 0.9057 Epoch: 18 Global Step: 186665 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:03:16,707-Speed 127.69 samples/sec Loss 0.8003 Epoch: 18 Global Step: 186670 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:03:36,444-Speed 129.71 samples/sec Loss 1.0148 Epoch: 18 Global Step: 186675 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:03:56,317-Speed 128.82 samples/sec Loss 0.8189 Epoch: 18 Global Step: 186680 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:04:16,016-Speed 129.97 samples/sec Loss 0.8471 Epoch: 18 Global Step: 186685 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:04:35,799-Speed 129.41 samples/sec Loss 0.8818 Epoch: 18 Global Step: 186690 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:04:55,612-Speed 129.22 samples/sec Loss 0.7702 Epoch: 18 Global Step: 186695 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:05:15,352-Speed 129.69 samples/sec Loss 0.9912 Epoch: 18 Global Step: 186700 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:05:35,231-Speed 128.78 samples/sec Loss 0.8813 Epoch: 18 Global Step: 186705 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:05:55,444-Speed 126.66 samples/sec Loss 0.8597 Epoch: 18 Global Step: 186710 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:06:15,624-Speed 126.87 samples/sec Loss 0.8654 Epoch: 18 Global Step: 186715 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:06:35,408-Speed 129.40 samples/sec Loss 0.9543 Epoch: 18 Global Step: 186720 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:06:55,583-Speed 126.90 samples/sec Loss 0.9288 Epoch: 18 Global Step: 186725 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:07:15,281-Speed 129.96 samples/sec Loss 0.8571 Epoch: 18 Global Step: 186730 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:07:35,171-Speed 128.72 samples/sec Loss 0.8231 Epoch: 18 Global Step: 186735 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:07:54,866-Speed 129.99 samples/sec Loss 0.8763 Epoch: 18 Global Step: 186740 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:08:14,192-Speed 132.47 samples/sec Loss 0.8356 Epoch: 18 Global Step: 186745 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:08:34,094-Speed 128.64 samples/sec Loss 0.9271 Epoch: 18 Global Step: 186750 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:08:54,103-Speed 127.95 samples/sec Loss 0.7807 Epoch: 18 Global Step: 186755 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:09:14,219-Speed 127.27 samples/sec Loss 1.0181 Epoch: 18 Global Step: 186760 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:09:33,968-Speed 129.63 samples/sec Loss 0.8400 Epoch: 18 Global Step: 186765 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:09:53,827-Speed 128.91 samples/sec Loss 0.8416 Epoch: 18 Global Step: 186770 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:10:13,626-Speed 129.31 samples/sec Loss 0.7953 Epoch: 18 Global Step: 186775 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:10:33,527-Speed 128.64 samples/sec Loss 0.9069 Epoch: 18 Global Step: 186780 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:10:53,332-Speed 129.27 samples/sec Loss 0.8504 Epoch: 18 Global Step: 186785 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:11:13,212-Speed 128.78 samples/sec Loss 1.0014 Epoch: 18 Global Step: 186790 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:11:32,975-Speed 129.54 samples/sec Loss 0.8869 Epoch: 18 Global Step: 186795 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:11:52,959-Speed 128.11 samples/sec Loss 0.8087 Epoch: 18 Global Step: 186800 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:12:13,114-Speed 127.02 samples/sec Loss 0.9247 Epoch: 18 Global Step: 186805 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:12:32,986-Speed 128.83 samples/sec Loss 0.9439 Epoch: 18 Global Step: 186810 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:12:52,626-Speed 130.35 samples/sec Loss 0.9169 Epoch: 18 Global Step: 186815 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:13:12,300-Speed 130.13 samples/sec Loss 0.7967 Epoch: 18 Global Step: 186820 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:13:32,251-Speed 128.32 samples/sec Loss 0.8125 Epoch: 18 Global Step: 186825 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:13:51,948-Speed 129.98 samples/sec Loss 0.8878 Epoch: 18 Global Step: 186830 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:14:11,758-Speed 129.24 samples/sec Loss 0.8754 Epoch: 18 Global Step: 186835 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:14:31,554-Speed 129.32 samples/sec Loss 0.7276 Epoch: 18 Global Step: 186840 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:14:51,683-Speed 127.19 samples/sec Loss 0.8777 Epoch: 18 Global Step: 186845 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:15:11,769-Speed 127.46 samples/sec Loss 0.7466 Epoch: 18 Global Step: 186850 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:15:31,541-Speed 129.48 samples/sec Loss 0.9002 Epoch: 18 Global Step: 186855 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:15:51,365-Speed 129.14 samples/sec Loss 0.8158 Epoch: 18 Global Step: 186860 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:16:10,636-Speed 132.86 samples/sec Loss 0.9413 Epoch: 18 Global Step: 186865 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:16:30,590-Speed 128.30 samples/sec Loss 0.8684 Epoch: 18 Global Step: 186870 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:16:50,267-Speed 130.11 samples/sec Loss 0.9153 Epoch: 18 Global Step: 186875 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:17:10,182-Speed 128.55 samples/sec Loss 0.7976 Epoch: 18 Global Step: 186880 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:17:29,784-Speed 130.60 samples/sec Loss 0.8532 Epoch: 18 Global Step: 186885 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:17:49,923-Speed 127.12 samples/sec Loss 1.0746 Epoch: 18 Global Step: 186890 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:18:09,771-Speed 128.99 samples/sec Loss 0.7644 Epoch: 18 Global Step: 186895 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:18:29,347-Speed 130.78 samples/sec Loss 0.9314 Epoch: 18 Global Step: 186900 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:18:49,158-Speed 129.23 samples/sec Loss 0.9428 Epoch: 18 Global Step: 186905 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:19:08,630-Speed 131.47 samples/sec Loss 0.8541 Epoch: 18 Global Step: 186910 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:19:28,600-Speed 128.20 samples/sec Loss 0.8872 Epoch: 18 Global Step: 186915 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:19:48,238-Speed 130.37 samples/sec Loss 0.9062 Epoch: 18 Global Step: 186920 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:20:08,145-Speed 128.60 samples/sec Loss 0.9600 Epoch: 18 Global Step: 186925 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:20:27,611-Speed 131.52 samples/sec Loss 0.8560 Epoch: 18 Global Step: 186930 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:20:47,440-Speed 129.11 samples/sec Loss 0.8843 Epoch: 18 Global Step: 186935 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:21:07,145-Speed 129.92 samples/sec Loss 0.8854 Epoch: 18 Global Step: 186940 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:21:26,658-Speed 131.20 samples/sec Loss 0.7948 Epoch: 18 Global Step: 186945 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:21:46,561-Speed 128.63 samples/sec Loss 0.7923 Epoch: 18 Global Step: 186950 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:22:06,087-Speed 131.11 samples/sec Loss 0.9090 Epoch: 18 Global Step: 186955 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:22:25,918-Speed 129.10 samples/sec Loss 0.8850 Epoch: 18 Global Step: 186960 Fp16 Grad Scale: 4096 Required: 75 hours Training: 2021-06-16 03:22:45,690-Speed 129.48 samples/sec Loss 0.8318 Epoch: 18 Global Step: 186965 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:23:05,509-Speed 129.18 samples/sec Loss 0.9706 Epoch: 18 Global Step: 186970 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:23:24,942-Speed 131.74 samples/sec Loss 1.0194 Epoch: 18 Global Step: 186975 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:23:44,843-Speed 128.64 samples/sec Loss 0.9683 Epoch: 18 Global Step: 186980 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:24:04,602-Speed 129.57 samples/sec Loss 0.8314 Epoch: 18 Global Step: 186985 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:24:24,047-Speed 131.67 samples/sec Loss 0.8004 Epoch: 18 Global Step: 186990 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:24:43,971-Speed 128.49 samples/sec Loss 0.9198 Epoch: 18 Global Step: 186995 Fp16 Grad Scale: 8192 Required: 75 hours Training: 2021-06-16 03:25:03,515-Speed 131.00 samples/sec Loss 0.9001 Epoch: 18 Global Step: 187000 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:25:23,338-Speed 129.15 samples/sec Loss 0.9905 Epoch: 18 Global Step: 187005 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:25:43,090-Speed 129.62 samples/sec Loss 0.8359 Epoch: 18 Global Step: 187010 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:26:03,029-Speed 128.40 samples/sec Loss 0.8422 Epoch: 18 Global Step: 187015 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:26:22,438-Speed 131.90 samples/sec Loss 0.9052 Epoch: 18 Global Step: 187020 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:26:42,320-Speed 128.77 samples/sec Loss 0.8910 Epoch: 18 Global Step: 187025 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:27:02,048-Speed 129.77 samples/sec Loss 0.8744 Epoch: 18 Global Step: 187030 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:27:21,620-Speed 130.81 samples/sec Loss 0.9181 Epoch: 18 Global Step: 187035 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:27:41,524-Speed 128.62 samples/sec Loss 0.7737 Epoch: 18 Global Step: 187040 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:28:01,017-Speed 131.34 samples/sec Loss 0.7625 Epoch: 18 Global Step: 187045 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:28:20,807-Speed 129.37 samples/sec Loss 0.9327 Epoch: 18 Global Step: 187050 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:28:40,561-Speed 129.60 samples/sec Loss 0.8577 Epoch: 18 Global Step: 187055 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:29:00,458-Speed 128.67 samples/sec Loss 0.6802 Epoch: 18 Global Step: 187060 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:29:19,880-Speed 131.82 samples/sec Loss 0.8012 Epoch: 18 Global Step: 187065 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:29:39,744-Speed 128.88 samples/sec Loss 0.8321 Epoch: 18 Global Step: 187070 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:29:59,513-Speed 129.50 samples/sec Loss 0.8160 Epoch: 18 Global Step: 187075 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:30:19,031-Speed 131.17 samples/sec Loss 0.9888 Epoch: 18 Global Step: 187080 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:30:38,943-Speed 128.58 samples/sec Loss 0.8491 Epoch: 18 Global Step: 187085 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:30:58,428-Speed 131.39 samples/sec Loss 0.8526 Epoch: 18 Global Step: 187090 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:31:18,252-Speed 129.14 samples/sec Loss 0.9527 Epoch: 18 Global Step: 187095 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:31:38,000-Speed 129.64 samples/sec Loss 0.9276 Epoch: 18 Global Step: 187100 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:31:57,910-Speed 128.58 samples/sec Loss 0.8716 Epoch: 18 Global Step: 187105 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:32:17,358-Speed 131.64 samples/sec Loss 0.8493 Epoch: 18 Global Step: 187110 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:32:37,319-Speed 128.26 samples/sec Loss 0.8772 Epoch: 18 Global Step: 187115 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:32:57,076-Speed 129.58 samples/sec Loss 0.7993 Epoch: 18 Global Step: 187120 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:33:16,613-Speed 131.04 samples/sec Loss 0.8588 Epoch: 18 Global Step: 187125 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:33:36,430-Speed 129.19 samples/sec Loss 0.8561 Epoch: 18 Global Step: 187130 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:33:55,954-Speed 131.13 samples/sec Loss 0.8577 Epoch: 18 Global Step: 187135 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:34:15,822-Speed 128.86 samples/sec Loss 0.9733 Epoch: 18 Global Step: 187140 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:34:35,566-Speed 129.67 samples/sec Loss 0.8492 Epoch: 18 Global Step: 187145 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:34:55,491-Speed 128.49 samples/sec Loss 0.8654 Epoch: 18 Global Step: 187150 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:35:14,910-Speed 131.84 samples/sec Loss 0.9441 Epoch: 18 Global Step: 187155 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:35:34,724-Speed 129.21 samples/sec Loss 0.8173 Epoch: 18 Global Step: 187160 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:35:54,614-Speed 128.72 samples/sec Loss 0.8563 Epoch: 18 Global Step: 187165 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:36:14,198-Speed 130.73 samples/sec Loss 0.9279 Epoch: 18 Global Step: 187170 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:36:34,111-Speed 128.57 samples/sec Loss 0.8948 Epoch: 18 Global Step: 187175 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:36:53,629-Speed 131.17 samples/sec Loss 0.8204 Epoch: 18 Global Step: 187180 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:37:13,293-Speed 130.20 samples/sec Loss 0.7882 Epoch: 18 Global Step: 187185 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:37:33,071-Speed 129.45 samples/sec Loss 0.8812 Epoch: 18 Global Step: 187190 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:37:53,145-Speed 127.53 samples/sec Loss 0.9064 Epoch: 18 Global Step: 187195 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:38:12,553-Speed 131.91 samples/sec Loss 0.8552 Epoch: 18 Global Step: 187200 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:38:32,541-Speed 128.08 samples/sec Loss 0.8999 Epoch: 18 Global Step: 187205 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:38:52,367-Speed 129.13 samples/sec Loss 0.9723 Epoch: 18 Global Step: 187210 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:39:11,900-Speed 131.07 samples/sec Loss 0.9268 Epoch: 18 Global Step: 187215 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:39:31,812-Speed 128.57 samples/sec Loss 0.8042 Epoch: 18 Global Step: 187220 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:39:51,334-Speed 131.14 samples/sec Loss 0.9236 Epoch: 18 Global Step: 187225 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:40:10,987-Speed 130.27 samples/sec Loss 0.9193 Epoch: 18 Global Step: 187230 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:40:30,725-Speed 129.72 samples/sec Loss 0.8200 Epoch: 18 Global Step: 187235 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:40:50,641-Speed 128.55 samples/sec Loss 0.8581 Epoch: 18 Global Step: 187240 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:41:10,083-Speed 131.68 samples/sec Loss 0.6940 Epoch: 18 Global Step: 187245 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:41:29,836-Speed 129.61 samples/sec Loss 0.8178 Epoch: 18 Global Step: 187250 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:41:49,596-Speed 129.56 samples/sec Loss 0.8934 Epoch: 18 Global Step: 187255 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:42:09,220-Speed 130.46 samples/sec Loss 0.9078 Epoch: 18 Global Step: 187260 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:42:29,172-Speed 128.32 samples/sec Loss 0.9246 Epoch: 18 Global Step: 187265 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:42:48,699-Speed 131.11 samples/sec Loss 0.8636 Epoch: 18 Global Step: 187270 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:43:08,355-Speed 130.25 samples/sec Loss 0.8518 Epoch: 18 Global Step: 187275 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:43:28,098-Speed 129.67 samples/sec Loss 0.9871 Epoch: 18 Global Step: 187280 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:43:48,085-Speed 128.09 samples/sec Loss 0.7551 Epoch: 18 Global Step: 187285 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:44:07,621-Speed 131.05 samples/sec Loss 0.8535 Epoch: 18 Global Step: 187290 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:44:27,447-Speed 129.13 samples/sec Loss 0.7964 Epoch: 18 Global Step: 187295 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:44:47,119-Speed 130.14 samples/sec Loss 0.9130 Epoch: 18 Global Step: 187300 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:45:06,659-Speed 131.02 samples/sec Loss 0.9788 Epoch: 18 Global Step: 187305 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:45:26,573-Speed 128.56 samples/sec Loss 0.9637 Epoch: 18 Global Step: 187310 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:45:46,097-Speed 131.13 samples/sec Loss 0.8597 Epoch: 18 Global Step: 187315 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:46:05,739-Speed 130.34 samples/sec Loss 0.8562 Epoch: 18 Global Step: 187320 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:46:25,497-Speed 129.58 samples/sec Loss 0.9565 Epoch: 18 Global Step: 187325 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:46:45,305-Speed 129.25 samples/sec Loss 0.9688 Epoch: 18 Global Step: 187330 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:47:04,737-Speed 131.75 samples/sec Loss 0.7996 Epoch: 18 Global Step: 187335 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:47:24,635-Speed 128.66 samples/sec Loss 0.8537 Epoch: 18 Global Step: 187340 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:47:44,272-Speed 130.37 samples/sec Loss 0.8403 Epoch: 18 Global Step: 187345 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:48:03,749-Speed 131.44 samples/sec Loss 0.9636 Epoch: 18 Global Step: 187350 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:48:23,681-Speed 128.45 samples/sec Loss 0.8306 Epoch: 18 Global Step: 187355 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:48:43,165-Speed 131.39 samples/sec Loss 0.8595 Epoch: 18 Global Step: 187360 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:49:03,048-Speed 128.76 samples/sec Loss 0.7727 Epoch: 18 Global Step: 187365 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:49:22,705-Speed 130.24 samples/sec Loss 0.9363 Epoch: 18 Global Step: 187370 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:49:42,570-Speed 128.88 samples/sec Loss 0.8659 Epoch: 18 Global Step: 187375 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:50:01,998-Speed 131.78 samples/sec Loss 0.8388 Epoch: 18 Global Step: 187380 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:50:21,903-Speed 128.61 samples/sec Loss 0.9939 Epoch: 18 Global Step: 187385 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:50:41,508-Speed 130.59 samples/sec Loss 0.8985 Epoch: 18 Global Step: 187390 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:51:01,099-Speed 130.68 samples/sec Loss 0.9114 Epoch: 18 Global Step: 187395 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:51:21,061-Speed 128.25 samples/sec Loss 0.8268 Epoch: 18 Global Step: 187400 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:51:40,598-Speed 131.04 samples/sec Loss 0.8444 Epoch: 18 Global Step: 187405 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:52:00,475-Speed 128.80 samples/sec Loss 0.8310 Epoch: 18 Global Step: 187410 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:52:20,496-Speed 127.88 samples/sec Loss 0.8174 Epoch: 18 Global Step: 187415 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:52:40,136-Speed 130.35 samples/sec Loss 0.8884 Epoch: 18 Global Step: 187420 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:52:59,702-Speed 130.84 samples/sec Loss 0.9220 Epoch: 18 Global Step: 187425 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:53:19,633-Speed 128.45 samples/sec Loss 0.8996 Epoch: 18 Global Step: 187430 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:53:39,238-Speed 130.58 samples/sec Loss 0.9009 Epoch: 18 Global Step: 187435 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:53:58,782-Speed 130.99 samples/sec Loss 0.8141 Epoch: 18 Global Step: 187440 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:54:18,575-Speed 129.35 samples/sec Loss 0.9510 Epoch: 18 Global Step: 187445 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:54:38,199-Speed 130.46 samples/sec Loss 0.8742 Epoch: 18 Global Step: 187450 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:54:58,088-Speed 128.72 samples/sec Loss 0.8518 Epoch: 18 Global Step: 187455 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:55:17,854-Speed 129.52 samples/sec Loss 0.8544 Epoch: 18 Global Step: 187460 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:55:37,606-Speed 129.62 samples/sec Loss 0.8396 Epoch: 18 Global Step: 187465 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:55:57,058-Speed 131.61 samples/sec Loss 0.8792 Epoch: 18 Global Step: 187470 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:56:17,020-Speed 128.25 samples/sec Loss 0.9102 Epoch: 18 Global Step: 187475 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:56:36,630-Speed 130.55 samples/sec Loss 0.9408 Epoch: 18 Global Step: 187480 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 03:56:55,802-Speed 133.53 samples/sec Loss 0.8470 Epoch: 18 Global Step: 187485 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 03:57:15,529-Speed 129.78 samples/sec Loss 0.9465 Epoch: 18 Global Step: 187490 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 03:57:35,307-Speed 129.44 samples/sec Loss 0.8236 Epoch: 18 Global Step: 187495 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 03:57:55,124-Speed 129.19 samples/sec Loss 0.9641 Epoch: 18 Global Step: 187500 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 03:58:15,035-Speed 128.58 samples/sec Loss 0.8304 Epoch: 18 Global Step: 187505 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 03:58:34,826-Speed 129.35 samples/sec Loss 0.9074 Epoch: 18 Global Step: 187510 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 03:58:54,562-Speed 129.72 samples/sec Loss 0.7594 Epoch: 18 Global Step: 187515 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 03:59:14,585-Speed 127.86 samples/sec Loss 0.8997 Epoch: 18 Global Step: 187520 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 03:59:34,182-Speed 130.64 samples/sec Loss 0.8404 Epoch: 18 Global Step: 187525 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 03:59:53,973-Speed 129.36 samples/sec Loss 0.8163 Epoch: 18 Global Step: 187530 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:00:13,697-Speed 129.80 samples/sec Loss 0.7713 Epoch: 18 Global Step: 187535 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:00:33,471-Speed 129.47 samples/sec Loss 0.8764 Epoch: 18 Global Step: 187540 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:00:53,582-Speed 127.30 samples/sec Loss 0.8783 Epoch: 18 Global Step: 187545 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:01:13,578-Speed 128.03 samples/sec Loss 0.9349 Epoch: 18 Global Step: 187550 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:01:33,293-Speed 129.86 samples/sec Loss 0.7748 Epoch: 18 Global Step: 187555 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:01:53,060-Speed 129.52 samples/sec Loss 0.7946 Epoch: 18 Global Step: 187560 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:02:12,915-Speed 128.94 samples/sec Loss 0.9241 Epoch: 18 Global Step: 187565 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:02:32,600-Speed 130.05 samples/sec Loss 0.9561 Epoch: 18 Global Step: 187570 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:02:52,389-Speed 129.37 samples/sec Loss 0.9442 Epoch: 18 Global Step: 187575 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:03:12,122-Speed 129.74 samples/sec Loss 0.9440 Epoch: 18 Global Step: 187580 Fp16 Grad Scale: 4096 Required: 74 hours Training: 2021-06-16 04:03:31,866-Speed 129.67 samples/sec Loss 0.9433 Epoch: 18 Global Step: 187585 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:03:52,003-Speed 127.13 samples/sec Loss 1.0137 Epoch: 18 Global Step: 187590 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:04:11,967-Speed 128.23 samples/sec Loss 0.9638 Epoch: 18 Global Step: 187595 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:04:31,693-Speed 129.79 samples/sec Loss 0.8484 Epoch: 18 Global Step: 187600 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:04:51,593-Speed 128.64 samples/sec Loss 0.9504 Epoch: 18 Global Step: 187605 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:05:11,377-Speed 129.40 samples/sec Loss 0.8360 Epoch: 18 Global Step: 187610 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:05:31,148-Speed 129.49 samples/sec Loss 0.9236 Epoch: 18 Global Step: 187615 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:05:50,944-Speed 129.33 samples/sec Loss 0.8833 Epoch: 18 Global Step: 187620 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:06:10,664-Speed 129.82 samples/sec Loss 0.8585 Epoch: 18 Global Step: 187625 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:06:30,496-Speed 129.09 samples/sec Loss 0.8315 Epoch: 18 Global Step: 187630 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:06:50,557-Speed 127.61 samples/sec Loss 0.9427 Epoch: 18 Global Step: 187635 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:07:10,542-Speed 128.11 samples/sec Loss 0.8094 Epoch: 18 Global Step: 187640 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:07:30,249-Speed 129.91 samples/sec Loss 0.8304 Epoch: 18 Global Step: 187645 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:07:50,124-Speed 128.81 samples/sec Loss 0.8628 Epoch: 18 Global Step: 187650 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:08:09,844-Speed 129.83 samples/sec Loss 0.9778 Epoch: 18 Global Step: 187655 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:08:29,609-Speed 129.52 samples/sec Loss 0.8866 Epoch: 18 Global Step: 187660 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:08:49,415-Speed 129.27 samples/sec Loss 0.8140 Epoch: 18 Global Step: 187665 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:09:09,217-Speed 129.29 samples/sec Loss 0.8922 Epoch: 18 Global Step: 187670 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:09:29,162-Speed 128.36 samples/sec Loss 0.9647 Epoch: 18 Global Step: 187675 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:09:49,264-Speed 127.36 samples/sec Loss 0.8610 Epoch: 18 Global Step: 187680 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:10:09,286-Speed 127.87 samples/sec Loss 0.9074 Epoch: 18 Global Step: 187685 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:10:29,024-Speed 129.70 samples/sec Loss 0.9206 Epoch: 18 Global Step: 187690 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:10:48,904-Speed 128.78 samples/sec Loss 0.9271 Epoch: 18 Global Step: 187695 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:11:08,692-Speed 129.38 samples/sec Loss 0.9047 Epoch: 18 Global Step: 187700 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:11:28,453-Speed 129.55 samples/sec Loss 0.9069 Epoch: 18 Global Step: 187705 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:11:48,322-Speed 128.85 samples/sec Loss 0.9293 Epoch: 18 Global Step: 187710 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:12:08,114-Speed 129.35 samples/sec Loss 0.9641 Epoch: 18 Global Step: 187715 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:12:27,946-Speed 129.09 samples/sec Loss 0.9767 Epoch: 18 Global Step: 187720 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:12:48,011-Speed 127.59 samples/sec Loss 0.9439 Epoch: 18 Global Step: 187725 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:13:08,028-Speed 127.90 samples/sec Loss 0.8469 Epoch: 18 Global Step: 187730 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:13:28,528-Speed 124.88 samples/sec Loss 0.8621 Epoch: 18 Global Step: 187735 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:13:48,427-Speed 128.66 samples/sec Loss 1.0946 Epoch: 18 Global Step: 187740 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:14:08,178-Speed 129.62 samples/sec Loss 0.9224 Epoch: 18 Global Step: 187745 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:14:28,078-Speed 128.65 samples/sec Loss 0.8969 Epoch: 18 Global Step: 187750 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:14:47,782-Speed 129.93 samples/sec Loss 0.8992 Epoch: 18 Global Step: 187755 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:15:07,520-Speed 129.71 samples/sec Loss 0.8548 Epoch: 18 Global Step: 187760 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:15:27,334-Speed 129.21 samples/sec Loss 0.9220 Epoch: 18 Global Step: 187765 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:15:47,327-Speed 128.05 samples/sec Loss 0.8454 Epoch: 18 Global Step: 187770 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:16:07,390-Speed 127.61 samples/sec Loss 0.8396 Epoch: 18 Global Step: 187775 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:16:27,121-Speed 129.75 samples/sec Loss 0.8667 Epoch: 18 Global Step: 187780 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:16:47,030-Speed 128.60 samples/sec Loss 0.8158 Epoch: 18 Global Step: 187785 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:17:06,758-Speed 129.77 samples/sec Loss 0.8424 Epoch: 18 Global Step: 187790 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:17:26,485-Speed 129.77 samples/sec Loss 0.9284 Epoch: 18 Global Step: 187795 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:17:46,342-Speed 128.93 samples/sec Loss 0.8142 Epoch: 18 Global Step: 187800 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:18:06,082-Speed 129.69 samples/sec Loss 0.8373 Epoch: 18 Global Step: 187805 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:18:25,943-Speed 128.90 samples/sec Loss 0.8740 Epoch: 18 Global Step: 187810 Fp16 Grad Scale: 8192 Required: 74 hours Training: 2021-06-16 04:18:45,991-Speed 127.70 samples/sec Loss 0.8155 Epoch: 18 Global Step: 187815 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:19:06,072-Speed 127.49 samples/sec Loss 0.7804 Epoch: 18 Global Step: 187820 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:19:25,398-Speed 132.47 samples/sec Loss 0.8053 Epoch: 18 Global Step: 187825 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:19:45,258-Speed 128.91 samples/sec Loss 0.9009 Epoch: 18 Global Step: 187830 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:20:04,859-Speed 130.61 samples/sec Loss 0.9250 Epoch: 18 Global Step: 187835 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:20:24,682-Speed 129.15 samples/sec Loss 0.8275 Epoch: 18 Global Step: 187840 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:20:44,539-Speed 128.93 samples/sec Loss 0.8903 Epoch: 18 Global Step: 187845 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:21:04,259-Speed 129.82 samples/sec Loss 0.8248 Epoch: 18 Global Step: 187850 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:21:24,105-Speed 129.00 samples/sec Loss 0.8615 Epoch: 18 Global Step: 187855 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:21:44,114-Speed 127.95 samples/sec Loss 0.9315 Epoch: 18 Global Step: 187860 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:22:04,233-Speed 127.25 samples/sec Loss 0.7964 Epoch: 18 Global Step: 187865 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:22:23,824-Speed 130.67 samples/sec Loss 0.8423 Epoch: 18 Global Step: 187870 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:22:43,675-Speed 128.97 samples/sec Loss 0.9265 Epoch: 18 Global Step: 187875 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:23:03,371-Speed 129.98 samples/sec Loss 0.9836 Epoch: 18 Global Step: 187880 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:23:23,243-Speed 128.83 samples/sec Loss 0.8028 Epoch: 18 Global Step: 187885 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:23:43,046-Speed 129.28 samples/sec Loss 0.8693 Epoch: 18 Global Step: 187890 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:24:02,678-Speed 130.41 samples/sec Loss 0.8190 Epoch: 18 Global Step: 187895 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:24:22,526-Speed 128.99 samples/sec Loss 0.9188 Epoch: 18 Global Step: 187900 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:24:42,394-Speed 128.85 samples/sec Loss 0.8088 Epoch: 18 Global Step: 187905 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:25:02,542-Speed 127.07 samples/sec Loss 0.9506 Epoch: 18 Global Step: 187910 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:25:22,198-Speed 130.25 samples/sec Loss 0.8644 Epoch: 18 Global Step: 187915 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:25:42,071-Speed 128.82 samples/sec Loss 0.9247 Epoch: 18 Global Step: 187920 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:26:01,688-Speed 130.51 samples/sec Loss 0.8714 Epoch: 18 Global Step: 187925 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:26:21,607-Speed 128.53 samples/sec Loss 0.9426 Epoch: 18 Global Step: 187930 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:26:41,319-Speed 129.87 samples/sec Loss 0.9087 Epoch: 18 Global Step: 187935 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:27:00,987-Speed 130.17 samples/sec Loss 0.8773 Epoch: 18 Global Step: 187940 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:27:20,339-Speed 132.29 samples/sec Loss 0.9841 Epoch: 18 Global Step: 187945 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:27:40,248-Speed 128.59 samples/sec Loss 0.9131 Epoch: 18 Global Step: 187950 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:28:00,231-Speed 128.12 samples/sec Loss 0.9824 Epoch: 18 Global Step: 187955 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:28:19,864-Speed 130.40 samples/sec Loss 0.9296 Epoch: 18 Global Step: 187960 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:28:39,916-Speed 127.68 samples/sec Loss 0.7860 Epoch: 18 Global Step: 187965 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:28:59,519-Speed 130.60 samples/sec Loss 0.8922 Epoch: 18 Global Step: 187970 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:29:19,411-Speed 128.70 samples/sec Loss 0.8281 Epoch: 18 Global Step: 187975 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:29:39,189-Speed 129.45 samples/sec Loss 0.8867 Epoch: 18 Global Step: 187980 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:29:59,051-Speed 128.89 samples/sec Loss 0.8766 Epoch: 18 Global Step: 187985 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:30:18,727-Speed 130.12 samples/sec Loss 0.9452 Epoch: 18 Global Step: 187990 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:30:38,675-Speed 128.34 samples/sec Loss 0.8460 Epoch: 18 Global Step: 187995 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:30:58,533-Speed 128.92 samples/sec Loss 0.8229 Epoch: 18 Global Step: 188000 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:31:18,257-Speed 129.81 samples/sec Loss 0.8384 Epoch: 18 Global Step: 188005 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:31:38,217-Speed 128.26 samples/sec Loss 0.9275 Epoch: 18 Global Step: 188010 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:31:57,867-Speed 130.28 samples/sec Loss 0.9394 Epoch: 18 Global Step: 188015 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:32:17,623-Speed 129.59 samples/sec Loss 0.9338 Epoch: 18 Global Step: 188020 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:32:37,422-Speed 129.31 samples/sec Loss 0.8592 Epoch: 18 Global Step: 188025 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:32:57,325-Speed 128.63 samples/sec Loss 0.8448 Epoch: 18 Global Step: 188030 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:33:16,979-Speed 130.26 samples/sec Loss 0.8438 Epoch: 18 Global Step: 188035 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:33:36,912-Speed 128.44 samples/sec Loss 0.7110 Epoch: 18 Global Step: 188040 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:33:56,786-Speed 128.82 samples/sec Loss 0.9585 Epoch: 18 Global Step: 188045 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:34:16,504-Speed 129.84 samples/sec Loss 0.9083 Epoch: 18 Global Step: 188050 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:34:36,615-Speed 127.30 samples/sec Loss 0.9385 Epoch: 18 Global Step: 188055 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:34:56,289-Speed 130.13 samples/sec Loss 1.0150 Epoch: 18 Global Step: 188060 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:35:16,131-Speed 129.03 samples/sec Loss 0.8860 Epoch: 18 Global Step: 188065 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:35:35,857-Speed 129.78 samples/sec Loss 0.8343 Epoch: 18 Global Step: 188070 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:35:55,838-Speed 128.13 samples/sec Loss 0.9421 Epoch: 18 Global Step: 188075 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:36:15,549-Speed 129.88 samples/sec Loss 0.9447 Epoch: 18 Global Step: 188080 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:36:35,502-Speed 128.31 samples/sec Loss 0.9451 Epoch: 18 Global Step: 188085 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:36:54,628-Speed 133.86 samples/sec Loss 0.8725 Epoch: 18 Global Step: 188090 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:37:14,717-Speed 127.44 samples/sec Loss 0.9237 Epoch: 18 Global Step: 188095 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:37:34,765-Speed 127.70 samples/sec Loss 0.9737 Epoch: 18 Global Step: 188100 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:37:54,426-Speed 130.21 samples/sec Loss 0.8811 Epoch: 18 Global Step: 188105 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:38:14,291-Speed 128.87 samples/sec Loss 0.7846 Epoch: 18 Global Step: 188110 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:38:34,005-Speed 129.87 samples/sec Loss 0.7745 Epoch: 18 Global Step: 188115 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:38:53,614-Speed 130.55 samples/sec Loss 0.9387 Epoch: 18 Global Step: 188120 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:39:13,038-Speed 131.81 samples/sec Loss 0.8772 Epoch: 18 Global Step: 188125 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:39:32,589-Speed 130.94 samples/sec Loss 0.8769 Epoch: 18 Global Step: 188130 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:39:52,303-Speed 129.86 samples/sec Loss 0.8693 Epoch: 18 Global Step: 188135 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:40:12,438-Speed 127.15 samples/sec Loss 0.8302 Epoch: 18 Global Step: 188140 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:40:32,304-Speed 128.87 samples/sec Loss 0.8544 Epoch: 18 Global Step: 188145 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:40:51,862-Speed 130.90 samples/sec Loss 0.8472 Epoch: 18 Global Step: 188150 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:41:11,391-Speed 131.10 samples/sec Loss 0.8331 Epoch: 18 Global Step: 188155 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:41:30,933-Speed 131.00 samples/sec Loss 0.8579 Epoch: 18 Global Step: 188160 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:41:50,552-Speed 130.49 samples/sec Loss 0.8864 Epoch: 18 Global Step: 188165 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:42:10,133-Speed 130.75 samples/sec Loss 0.9606 Epoch: 18 Global Step: 188170 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:42:29,738-Speed 130.58 samples/sec Loss 0.8519 Epoch: 18 Global Step: 188175 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:42:49,301-Speed 130.87 samples/sec Loss 0.8701 Epoch: 18 Global Step: 188180 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:43:08,760-Speed 131.56 samples/sec Loss 0.9328 Epoch: 18 Global Step: 188185 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:43:28,555-Speed 129.33 samples/sec Loss 0.8735 Epoch: 18 Global Step: 188190 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:43:48,123-Speed 130.83 samples/sec Loss 0.8188 Epoch: 18 Global Step: 188195 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:44:07,333-Speed 133.28 samples/sec Loss 0.8982 Epoch: 18 Global Step: 188200 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:44:26,877-Speed 130.99 samples/sec Loss 0.9218 Epoch: 18 Global Step: 188205 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:44:46,384-Speed 131.24 samples/sec Loss 0.8602 Epoch: 18 Global Step: 188210 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:45:05,943-Speed 130.90 samples/sec Loss 0.8714 Epoch: 18 Global Step: 188215 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:45:25,698-Speed 129.59 samples/sec Loss 0.9434 Epoch: 18 Global Step: 188220 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:45:45,303-Speed 130.59 samples/sec Loss 1.0095 Epoch: 18 Global Step: 188225 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:46:04,844-Speed 131.01 samples/sec Loss 1.0077 Epoch: 18 Global Step: 188230 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:46:24,645-Speed 129.30 samples/sec Loss 0.9304 Epoch: 18 Global Step: 188235 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:46:44,460-Speed 129.20 samples/sec Loss 0.9374 Epoch: 18 Global Step: 188240 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:47:04,099-Speed 130.36 samples/sec Loss 0.9300 Epoch: 18 Global Step: 188245 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:47:23,657-Speed 130.90 samples/sec Loss 0.8703 Epoch: 18 Global Step: 188250 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:47:43,168-Speed 131.21 samples/sec Loss 0.8629 Epoch: 18 Global Step: 188255 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:48:02,807-Speed 130.36 samples/sec Loss 0.8936 Epoch: 18 Global Step: 188260 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:48:22,357-Speed 130.96 samples/sec Loss 0.9705 Epoch: 18 Global Step: 188265 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:48:41,959-Speed 130.60 samples/sec Loss 0.8683 Epoch: 18 Global Step: 188270 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:49:01,753-Speed 129.34 samples/sec Loss 0.8611 Epoch: 18 Global Step: 188275 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:49:21,559-Speed 129.26 samples/sec Loss 0.8765 Epoch: 18 Global Step: 188280 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:49:41,067-Speed 131.23 samples/sec Loss 0.8252 Epoch: 18 Global Step: 188285 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:50:00,770-Speed 129.94 samples/sec Loss 0.9240 Epoch: 18 Global Step: 188290 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:50:20,304-Speed 131.07 samples/sec Loss 0.9830 Epoch: 18 Global Step: 188295 Fp16 Grad Scale: 4096 Required: 73 hours Training: 2021-06-16 04:50:39,807-Speed 131.27 samples/sec Loss 1.0119 Epoch: 18 Global Step: 188300 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:50:59,411-Speed 130.59 samples/sec Loss 0.7917 Epoch: 18 Global Step: 188305 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:51:18,933-Speed 131.14 samples/sec Loss 0.8720 Epoch: 18 Global Step: 188310 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:51:38,540-Speed 130.57 samples/sec Loss 0.8553 Epoch: 18 Global Step: 188315 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:51:58,082-Speed 131.00 samples/sec Loss 0.9424 Epoch: 18 Global Step: 188320 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:52:17,941-Speed 128.92 samples/sec Loss 0.8232 Epoch: 18 Global Step: 188325 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:52:37,393-Speed 131.61 samples/sec Loss 0.9553 Epoch: 18 Global Step: 188330 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:52:57,050-Speed 130.24 samples/sec Loss 0.9318 Epoch: 18 Global Step: 188335 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:53:16,563-Speed 131.20 samples/sec Loss 0.8050 Epoch: 18 Global Step: 188340 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:53:36,105-Speed 131.01 samples/sec Loss 0.9021 Epoch: 18 Global Step: 188345 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:53:55,718-Speed 130.53 samples/sec Loss 0.8749 Epoch: 18 Global Step: 188350 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:54:15,320-Speed 130.61 samples/sec Loss 0.7687 Epoch: 18 Global Step: 188355 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:54:34,942-Speed 130.47 samples/sec Loss 0.8549 Epoch: 18 Global Step: 188360 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:54:54,382-Speed 131.69 samples/sec Loss 0.8956 Epoch: 18 Global Step: 188365 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:55:14,262-Speed 128.78 samples/sec Loss 0.8544 Epoch: 18 Global Step: 188370 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:55:33,691-Speed 131.77 samples/sec Loss 0.9022 Epoch: 18 Global Step: 188375 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:55:53,402-Speed 129.89 samples/sec Loss 0.8095 Epoch: 18 Global Step: 188380 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:56:12,834-Speed 131.75 samples/sec Loss 0.7752 Epoch: 18 Global Step: 188385 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:56:32,294-Speed 131.56 samples/sec Loss 0.9065 Epoch: 18 Global Step: 188390 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:56:51,903-Speed 130.56 samples/sec Loss 0.9285 Epoch: 18 Global Step: 188395 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:57:11,364-Speed 131.56 samples/sec Loss 0.7826 Epoch: 18 Global Step: 188400 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:57:30,971-Speed 130.57 samples/sec Loss 0.9265 Epoch: 18 Global Step: 188405 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:57:50,402-Speed 131.75 samples/sec Loss 0.8698 Epoch: 18 Global Step: 188410 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:58:10,227-Speed 129.14 samples/sec Loss 0.8617 Epoch: 18 Global Step: 188415 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:58:29,691-Speed 131.53 samples/sec Loss 0.9289 Epoch: 18 Global Step: 188420 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:58:49,491-Speed 129.30 samples/sec Loss 0.8868 Epoch: 18 Global Step: 188425 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:59:09,029-Speed 131.03 samples/sec Loss 0.7975 Epoch: 18 Global Step: 188430 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:59:28,484-Speed 131.60 samples/sec Loss 0.8393 Epoch: 18 Global Step: 188435 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 04:59:48,063-Speed 130.76 samples/sec Loss 0.9734 Epoch: 18 Global Step: 188440 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:00:07,612-Speed 130.96 samples/sec Loss 0.8670 Epoch: 18 Global Step: 188445 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:00:27,231-Speed 130.49 samples/sec Loss 0.8425 Epoch: 18 Global Step: 188450 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:00:46,784-Speed 130.93 samples/sec Loss 0.9399 Epoch: 18 Global Step: 188455 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:01:06,881-Speed 127.39 samples/sec Loss 0.7801 Epoch: 18 Global Step: 188460 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:01:26,454-Speed 130.80 samples/sec Loss 0.9523 Epoch: 18 Global Step: 188465 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:01:46,178-Speed 129.80 samples/sec Loss 0.8466 Epoch: 18 Global Step: 188470 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:02:05,717-Speed 131.03 samples/sec Loss 0.8834 Epoch: 18 Global Step: 188475 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:02:25,349-Speed 130.41 samples/sec Loss 0.9090 Epoch: 18 Global Step: 188480 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:02:44,944-Speed 130.65 samples/sec Loss 0.8813 Epoch: 18 Global Step: 188485 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:03:04,510-Speed 130.85 samples/sec Loss 0.8503 Epoch: 18 Global Step: 188490 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:03:24,196-Speed 130.05 samples/sec Loss 0.8517 Epoch: 18 Global Step: 188495 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:03:43,648-Speed 131.62 samples/sec Loss 0.9341 Epoch: 18 Global Step: 188500 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:04:03,468-Speed 129.17 samples/sec Loss 1.0507 Epoch: 18 Global Step: 188505 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:04:23,351-Speed 128.76 samples/sec Loss 0.8285 Epoch: 18 Global Step: 188510 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:04:42,897-Speed 130.98 samples/sec Loss 0.7867 Epoch: 18 Global Step: 188515 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:05:02,347-Speed 131.63 samples/sec Loss 0.8866 Epoch: 18 Global Step: 188520 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:05:21,964-Speed 130.51 samples/sec Loss 0.9586 Epoch: 18 Global Step: 188525 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:05:41,434-Speed 131.49 samples/sec Loss 0.9335 Epoch: 18 Global Step: 188530 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:06:01,119-Speed 130.06 samples/sec Loss 0.9264 Epoch: 18 Global Step: 188535 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:06:20,703-Speed 130.73 samples/sec Loss 0.9098 Epoch: 18 Global Step: 188540 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:06:40,145-Speed 131.69 samples/sec Loss 0.8466 Epoch: 18 Global Step: 188545 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:06:59,936-Speed 129.36 samples/sec Loss 0.9145 Epoch: 18 Global Step: 188550 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:07:19,400-Speed 131.53 samples/sec Loss 0.8708 Epoch: 18 Global Step: 188555 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:07:39,119-Speed 129.83 samples/sec Loss 0.9280 Epoch: 18 Global Step: 188560 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:07:58,556-Speed 131.72 samples/sec Loss 0.7739 Epoch: 18 Global Step: 188565 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:08:18,099-Speed 131.00 samples/sec Loss 0.9600 Epoch: 18 Global Step: 188570 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:08:37,607-Speed 131.24 samples/sec Loss 0.8281 Epoch: 18 Global Step: 188575 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:08:57,141-Speed 131.06 samples/sec Loss 0.9054 Epoch: 18 Global Step: 188580 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:09:16,762-Speed 130.48 samples/sec Loss 0.7892 Epoch: 18 Global Step: 188585 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:09:36,189-Speed 131.78 samples/sec Loss 1.0216 Epoch: 18 Global Step: 188590 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:09:55,987-Speed 129.32 samples/sec Loss 0.7624 Epoch: 18 Global Step: 188595 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:10:15,434-Speed 131.65 samples/sec Loss 0.9271 Epoch: 18 Global Step: 188600 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:10:35,111-Speed 130.10 samples/sec Loss 0.8474 Epoch: 18 Global Step: 188605 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:10:54,563-Speed 131.62 samples/sec Loss 0.8436 Epoch: 18 Global Step: 188610 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:11:14,460-Speed 128.67 samples/sec Loss 0.9099 Epoch: 18 Global Step: 188615 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:11:33,907-Speed 131.66 samples/sec Loss 0.9401 Epoch: 18 Global Step: 188620 Fp16 Grad Scale: 8192 Required: 73 hours Training: 2021-06-16 05:11:53,556-Speed 130.29 samples/sec Loss 0.8112 Epoch: 18 Global Step: 188625 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:12:13,172-Speed 130.51 samples/sec Loss 0.9043 Epoch: 18 Global Step: 188630 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:12:32,630-Speed 131.58 samples/sec Loss 0.7991 Epoch: 18 Global Step: 188635 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:12:52,447-Speed 129.19 samples/sec Loss 0.9049 Epoch: 18 Global Step: 188640 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:13:11,922-Speed 131.46 samples/sec Loss 0.8346 Epoch: 18 Global Step: 188645 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:13:31,408-Speed 131.39 samples/sec Loss 0.8280 Epoch: 18 Global Step: 188650 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:13:51,173-Speed 129.53 samples/sec Loss 0.8246 Epoch: 18 Global Step: 188655 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:14:11,010-Speed 129.06 samples/sec Loss 0.8549 Epoch: 18 Global Step: 188660 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:14:30,710-Speed 129.96 samples/sec Loss 0.8818 Epoch: 18 Global Step: 188665 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:14:50,661-Speed 128.32 samples/sec Loss 0.8271 Epoch: 18 Global Step: 188670 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:15:10,420-Speed 129.56 samples/sec Loss 0.9444 Epoch: 18 Global Step: 188675 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:15:30,203-Speed 129.41 samples/sec Loss 0.9203 Epoch: 18 Global Step: 188680 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:15:49,929-Speed 129.79 samples/sec Loss 0.8919 Epoch: 18 Global Step: 188685 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:16:09,668-Speed 129.70 samples/sec Loss 0.7688 Epoch: 18 Global Step: 188690 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:16:29,950-Speed 126.23 samples/sec Loss 0.9112 Epoch: 18 Global Step: 188695 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:16:49,611-Speed 130.22 samples/sec Loss 0.8174 Epoch: 18 Global Step: 188700 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:17:09,391-Speed 129.43 samples/sec Loss 0.7484 Epoch: 18 Global Step: 188705 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:17:29,309-Speed 128.53 samples/sec Loss 0.8891 Epoch: 18 Global Step: 188710 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:17:49,178-Speed 128.85 samples/sec Loss 0.9025 Epoch: 18 Global Step: 188715 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:18:08,984-Speed 129.25 samples/sec Loss 0.8101 Epoch: 18 Global Step: 188720 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:18:28,858-Speed 128.82 samples/sec Loss 0.8357 Epoch: 18 Global Step: 188725 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:18:48,466-Speed 130.56 samples/sec Loss 0.8825 Epoch: 18 Global Step: 188730 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:19:08,201-Speed 129.73 samples/sec Loss 0.8230 Epoch: 18 Global Step: 188735 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:19:28,393-Speed 126.79 samples/sec Loss 0.8779 Epoch: 18 Global Step: 188740 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:19:48,113-Speed 129.82 samples/sec Loss 0.7993 Epoch: 18 Global Step: 188745 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:20:07,890-Speed 129.45 samples/sec Loss 0.8474 Epoch: 18 Global Step: 188750 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:20:27,838-Speed 128.34 samples/sec Loss 0.8696 Epoch: 18 Global Step: 188755 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:20:47,668-Speed 129.10 samples/sec Loss 0.8945 Epoch: 18 Global Step: 188760 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:21:07,451-Speed 129.42 samples/sec Loss 0.9576 Epoch: 18 Global Step: 188765 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:21:27,263-Speed 129.22 samples/sec Loss 0.8864 Epoch: 18 Global Step: 188770 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:21:47,091-Speed 129.11 samples/sec Loss 0.8934 Epoch: 18 Global Step: 188775 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:22:06,915-Speed 129.14 samples/sec Loss 0.8697 Epoch: 18 Global Step: 188780 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:22:27,117-Speed 126.73 samples/sec Loss 0.9343 Epoch: 18 Global Step: 188785 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:22:46,738-Speed 130.49 samples/sec Loss 0.9177 Epoch: 18 Global Step: 188790 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:23:06,527-Speed 129.37 samples/sec Loss 0.8625 Epoch: 18 Global Step: 188795 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:23:26,457-Speed 128.45 samples/sec Loss 0.7417 Epoch: 18 Global Step: 188800 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:23:46,286-Speed 129.11 samples/sec Loss 0.7783 Epoch: 18 Global Step: 188805 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:24:06,011-Speed 129.79 samples/sec Loss 1.0199 Epoch: 18 Global Step: 188810 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:24:25,925-Speed 128.56 samples/sec Loss 0.9262 Epoch: 18 Global Step: 188815 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:24:45,552-Speed 130.44 samples/sec Loss 0.8356 Epoch: 18 Global Step: 188820 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:25:05,407-Speed 128.94 samples/sec Loss 0.8988 Epoch: 18 Global Step: 188825 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:25:25,337-Speed 128.45 samples/sec Loss 0.8913 Epoch: 18 Global Step: 188830 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:25:45,189-Speed 128.97 samples/sec Loss 0.9026 Epoch: 18 Global Step: 188835 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:26:04,948-Speed 129.57 samples/sec Loss 0.9208 Epoch: 18 Global Step: 188840 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:26:24,984-Speed 127.78 samples/sec Loss 0.8603 Epoch: 18 Global Step: 188845 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:26:44,745-Speed 129.55 samples/sec Loss 1.0050 Epoch: 18 Global Step: 188850 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:27:04,509-Speed 129.53 samples/sec Loss 0.9737 Epoch: 18 Global Step: 188855 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:27:24,469-Speed 128.27 samples/sec Loss 0.8066 Epoch: 18 Global Step: 188860 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:27:44,112-Speed 130.33 samples/sec Loss 0.8758 Epoch: 18 Global Step: 188865 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:28:03,908-Speed 129.32 samples/sec Loss 0.8946 Epoch: 18 Global Step: 188870 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:28:23,927-Speed 127.89 samples/sec Loss 0.8324 Epoch: 18 Global Step: 188875 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:28:43,693-Speed 129.53 samples/sec Loss 0.7453 Epoch: 18 Global Step: 188880 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:29:03,468-Speed 129.46 samples/sec Loss 0.8339 Epoch: 18 Global Step: 188885 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:29:23,388-Speed 128.52 samples/sec Loss 0.9438 Epoch: 18 Global Step: 188890 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:29:43,350-Speed 128.25 samples/sec Loss 0.8706 Epoch: 18 Global Step: 188895 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:30:03,149-Speed 129.31 samples/sec Loss 0.9179 Epoch: 18 Global Step: 188900 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:30:22,961-Speed 129.22 samples/sec Loss 0.9262 Epoch: 18 Global Step: 188905 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:30:42,662-Speed 129.95 samples/sec Loss 0.8772 Epoch: 18 Global Step: 188910 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:31:02,455-Speed 129.35 samples/sec Loss 0.9597 Epoch: 18 Global Step: 188915 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:31:22,613-Speed 127.00 samples/sec Loss 0.8150 Epoch: 18 Global Step: 188920 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:31:42,330-Speed 129.85 samples/sec Loss 1.0021 Epoch: 18 Global Step: 188925 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:32:02,219-Speed 128.72 samples/sec Loss 0.9984 Epoch: 18 Global Step: 188930 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:32:22,047-Speed 129.12 samples/sec Loss 0.8672 Epoch: 18 Global Step: 188935 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:32:41,928-Speed 128.77 samples/sec Loss 0.8597 Epoch: 18 Global Step: 188940 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:33:01,674-Speed 129.65 samples/sec Loss 0.9187 Epoch: 18 Global Step: 188945 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:33:21,567-Speed 128.70 samples/sec Loss 0.8545 Epoch: 18 Global Step: 188950 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:33:41,305-Speed 129.70 samples/sec Loss 0.9256 Epoch: 18 Global Step: 188955 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:34:01,079-Speed 129.47 samples/sec Loss 0.8750 Epoch: 18 Global Step: 188960 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:34:21,052-Speed 128.18 samples/sec Loss 0.8463 Epoch: 18 Global Step: 188965 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:34:40,839-Speed 129.38 samples/sec Loss 0.9678 Epoch: 18 Global Step: 188970 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:35:00,709-Speed 128.85 samples/sec Loss 0.9836 Epoch: 18 Global Step: 188975 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:35:20,728-Speed 127.88 samples/sec Loss 0.8497 Epoch: 18 Global Step: 188980 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:35:40,468-Speed 129.69 samples/sec Loss 0.9817 Epoch: 18 Global Step: 188985 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:36:00,249-Speed 129.43 samples/sec Loss 0.8927 Epoch: 18 Global Step: 188990 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:36:20,175-Speed 128.48 samples/sec Loss 0.8235 Epoch: 18 Global Step: 188995 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:36:39,889-Speed 129.87 samples/sec Loss 0.8850 Epoch: 18 Global Step: 189000 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:36:59,662-Speed 129.47 samples/sec Loss 0.8516 Epoch: 18 Global Step: 189005 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:37:19,923-Speed 126.36 samples/sec Loss 0.9726 Epoch: 18 Global Step: 189010 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:37:39,805-Speed 128.77 samples/sec Loss 1.0281 Epoch: 18 Global Step: 189015 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:37:59,621-Speed 129.21 samples/sec Loss 0.8653 Epoch: 18 Global Step: 189020 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:38:19,588-Speed 128.22 samples/sec Loss 0.9398 Epoch: 18 Global Step: 189025 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:38:39,323-Speed 129.73 samples/sec Loss 0.9130 Epoch: 18 Global Step: 189030 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:38:58,618-Speed 132.69 samples/sec Loss 0.8712 Epoch: 18 Global Step: 189035 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:39:18,577-Speed 128.27 samples/sec Loss 0.8076 Epoch: 18 Global Step: 189040 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:39:38,164-Speed 130.70 samples/sec Loss 0.9017 Epoch: 18 Global Step: 189045 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:39:57,973-Speed 129.24 samples/sec Loss 0.8713 Epoch: 18 Global Step: 189050 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:40:17,680-Speed 129.92 samples/sec Loss 0.8627 Epoch: 18 Global Step: 189055 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:40:37,345-Speed 130.18 samples/sec Loss 0.8422 Epoch: 18 Global Step: 189060 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:40:57,481-Speed 127.14 samples/sec Loss 0.9368 Epoch: 18 Global Step: 189065 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:41:17,544-Speed 127.60 samples/sec Loss 0.8911 Epoch: 18 Global Step: 189070 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:41:37,239-Speed 129.99 samples/sec Loss 0.9230 Epoch: 18 Global Step: 189075 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:41:56,963-Speed 129.80 samples/sec Loss 0.8488 Epoch: 18 Global Step: 189080 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:42:16,848-Speed 128.74 samples/sec Loss 0.8317 Epoch: 18 Global Step: 189085 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:42:36,503-Speed 130.26 samples/sec Loss 0.9711 Epoch: 18 Global Step: 189090 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:42:56,461-Speed 128.28 samples/sec Loss 0.8727 Epoch: 18 Global Step: 189095 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:43:16,159-Speed 129.97 samples/sec Loss 0.8601 Epoch: 18 Global Step: 189100 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:43:35,914-Speed 129.59 samples/sec Loss 0.8596 Epoch: 18 Global Step: 189105 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:43:55,991-Speed 127.52 samples/sec Loss 0.8880 Epoch: 18 Global Step: 189110 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:44:16,077-Speed 127.46 samples/sec Loss 0.8110 Epoch: 18 Global Step: 189115 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:44:35,819-Speed 129.68 samples/sec Loss 0.9038 Epoch: 18 Global Step: 189120 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:44:55,560-Speed 129.68 samples/sec Loss 0.9296 Epoch: 18 Global Step: 189125 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:45:15,512-Speed 128.32 samples/sec Loss 0.9748 Epoch: 18 Global Step: 189130 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 05:45:35,171-Speed 130.23 samples/sec Loss 0.8360 Epoch: 18 Global Step: 189135 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:45:55,122-Speed 128.32 samples/sec Loss 0.7800 Epoch: 18 Global Step: 189140 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:46:14,857-Speed 129.72 samples/sec Loss 0.9803 Epoch: 18 Global Step: 189145 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:46:34,607-Speed 129.63 samples/sec Loss 0.9325 Epoch: 18 Global Step: 189150 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:46:54,728-Speed 127.24 samples/sec Loss 0.8301 Epoch: 18 Global Step: 189155 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:47:14,817-Speed 127.44 samples/sec Loss 0.8585 Epoch: 18 Global Step: 189160 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:47:34,549-Speed 129.74 samples/sec Loss 0.8890 Epoch: 18 Global Step: 189165 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:47:54,269-Speed 129.82 samples/sec Loss 0.8589 Epoch: 18 Global Step: 189170 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:48:14,162-Speed 128.69 samples/sec Loss 0.9413 Epoch: 18 Global Step: 189175 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:48:33,842-Speed 130.09 samples/sec Loss 0.8547 Epoch: 18 Global Step: 189180 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:48:53,647-Speed 129.26 samples/sec Loss 0.9802 Epoch: 18 Global Step: 189185 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:49:13,394-Speed 129.65 samples/sec Loss 0.8385 Epoch: 18 Global Step: 189190 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:49:33,038-Speed 130.33 samples/sec Loss 0.9324 Epoch: 18 Global Step: 189195 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:49:53,549-Speed 124.81 samples/sec Loss 0.7652 Epoch: 18 Global Step: 189200 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:50:13,538-Speed 128.08 samples/sec Loss 0.9915 Epoch: 18 Global Step: 189205 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:50:33,239-Speed 129.95 samples/sec Loss 0.9436 Epoch: 18 Global Step: 189210 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:50:53,009-Speed 129.49 samples/sec Loss 0.9562 Epoch: 18 Global Step: 189215 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:51:12,933-Speed 128.50 samples/sec Loss 0.8841 Epoch: 18 Global Step: 189220 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:51:32,745-Speed 129.22 samples/sec Loss 0.8845 Epoch: 18 Global Step: 189225 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:51:52,537-Speed 129.35 samples/sec Loss 0.9079 Epoch: 18 Global Step: 189230 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:52:12,243-Speed 129.92 samples/sec Loss 0.8785 Epoch: 18 Global Step: 189235 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:52:32,002-Speed 129.57 samples/sec Loss 0.9910 Epoch: 18 Global Step: 189240 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:52:52,237-Speed 126.52 samples/sec Loss 0.8989 Epoch: 18 Global Step: 189245 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:53:12,201-Speed 128.24 samples/sec Loss 0.9605 Epoch: 18 Global Step: 189250 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:53:31,903-Speed 129.94 samples/sec Loss 0.8316 Epoch: 18 Global Step: 189255 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:53:51,670-Speed 129.51 samples/sec Loss 0.7247 Epoch: 18 Global Step: 189260 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:54:11,536-Speed 128.87 samples/sec Loss 0.8775 Epoch: 18 Global Step: 189265 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:54:31,292-Speed 129.58 samples/sec Loss 0.8663 Epoch: 18 Global Step: 189270 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:54:51,092-Speed 129.30 samples/sec Loss 1.0016 Epoch: 18 Global Step: 189275 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:55:10,804-Speed 129.88 samples/sec Loss 1.0445 Epoch: 18 Global Step: 189280 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:55:30,719-Speed 128.56 samples/sec Loss 0.7812 Epoch: 18 Global Step: 189285 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:55:50,807-Speed 127.44 samples/sec Loss 0.8573 Epoch: 18 Global Step: 189290 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:56:10,875-Speed 127.57 samples/sec Loss 0.8368 Epoch: 18 Global Step: 189295 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:56:30,574-Speed 129.96 samples/sec Loss 0.8940 Epoch: 18 Global Step: 189300 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:56:50,437-Speed 128.89 samples/sec Loss 0.8270 Epoch: 18 Global Step: 189305 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:57:10,611-Speed 126.90 samples/sec Loss 0.8771 Epoch: 18 Global Step: 189310 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:57:30,245-Speed 130.39 samples/sec Loss 0.8879 Epoch: 18 Global Step: 189315 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:58:09,750-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-16 05:58:32,407-Speed 41.18 samples/sec Loss 0.5940 Epoch: 19 Global Step: 189320 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:58:52,458-Speed 127.71 samples/sec Loss 0.6963 Epoch: 19 Global Step: 189325 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:59:12,289-Speed 129.09 samples/sec Loss 0.6283 Epoch: 19 Global Step: 189330 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:59:32,676-Speed 125.57 samples/sec Loss 0.5902 Epoch: 19 Global Step: 189335 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 05:59:52,471-Speed 129.33 samples/sec Loss 0.6668 Epoch: 19 Global Step: 189340 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 06:00:12,373-Speed 128.64 samples/sec Loss 0.5225 Epoch: 19 Global Step: 189345 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 06:00:32,002-Speed 130.43 samples/sec Loss 0.5834 Epoch: 19 Global Step: 189350 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 06:00:51,709-Speed 129.91 samples/sec Loss 0.5896 Epoch: 19 Global Step: 189355 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 06:01:11,618-Speed 128.59 samples/sec Loss 0.6677 Epoch: 19 Global Step: 189360 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 06:01:31,403-Speed 129.39 samples/sec Loss 0.6425 Epoch: 19 Global Step: 189365 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 06:01:51,302-Speed 128.66 samples/sec Loss 0.6356 Epoch: 19 Global Step: 189370 Fp16 Grad Scale: 8192 Required: 72 hours Training: 2021-06-16 06:02:10,480-Speed 133.49 samples/sec Loss 0.5418 Epoch: 19 Global Step: 189375 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:02:30,871-Speed 125.55 samples/sec Loss 0.6511 Epoch: 19 Global Step: 189380 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:02:50,649-Speed 129.44 samples/sec Loss 0.6907 Epoch: 19 Global Step: 189385 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:03:10,496-Speed 128.99 samples/sec Loss 0.6647 Epoch: 19 Global Step: 189390 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:03:30,103-Speed 130.58 samples/sec Loss 0.5410 Epoch: 19 Global Step: 189395 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:03:49,780-Speed 130.11 samples/sec Loss 0.6010 Epoch: 19 Global Step: 189400 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:04:09,708-Speed 128.47 samples/sec Loss 0.6432 Epoch: 19 Global Step: 189405 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:04:29,553-Speed 129.01 samples/sec Loss 0.6794 Epoch: 19 Global Step: 189410 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:04:49,497-Speed 128.37 samples/sec Loss 0.6653 Epoch: 19 Global Step: 189415 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:05:09,327-Speed 129.10 samples/sec Loss 0.6875 Epoch: 19 Global Step: 189420 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:05:29,486-Speed 127.00 samples/sec Loss 0.6777 Epoch: 19 Global Step: 189425 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:05:49,246-Speed 129.56 samples/sec Loss 0.6600 Epoch: 19 Global Step: 189430 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:06:09,038-Speed 129.35 samples/sec Loss 0.6728 Epoch: 19 Global Step: 189435 Fp16 Grad Scale: 4096 Required: 72 hours Training: 2021-06-16 06:06:28,629-Speed 130.67 samples/sec Loss 0.7052 Epoch: 19 Global Step: 189440 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:06:48,375-Speed 129.66 samples/sec Loss 0.7024 Epoch: 19 Global Step: 189445 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:07:08,206-Speed 129.10 samples/sec Loss 0.6852 Epoch: 19 Global Step: 189450 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:07:28,035-Speed 129.11 samples/sec Loss 0.6917 Epoch: 19 Global Step: 189455 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:07:47,800-Speed 129.53 samples/sec Loss 0.6264 Epoch: 19 Global Step: 189460 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:08:07,412-Speed 130.54 samples/sec Loss 0.6652 Epoch: 19 Global Step: 189465 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:08:27,782-Speed 125.68 samples/sec Loss 0.7582 Epoch: 19 Global Step: 189470 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:08:47,597-Speed 129.20 samples/sec Loss 0.5835 Epoch: 19 Global Step: 189475 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:09:07,360-Speed 129.54 samples/sec Loss 0.6480 Epoch: 19 Global Step: 189480 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:09:27,105-Speed 129.66 samples/sec Loss 0.6665 Epoch: 19 Global Step: 189485 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:09:46,996-Speed 128.71 samples/sec Loss 0.6045 Epoch: 19 Global Step: 189490 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:10:06,731-Speed 129.73 samples/sec Loss 0.7370 Epoch: 19 Global Step: 189495 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:10:26,478-Speed 129.65 samples/sec Loss 0.7184 Epoch: 19 Global Step: 189500 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:10:46,276-Speed 129.31 samples/sec Loss 0.6003 Epoch: 19 Global Step: 189505 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:11:05,899-Speed 130.46 samples/sec Loss 0.7095 Epoch: 19 Global Step: 189510 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:11:26,256-Speed 125.76 samples/sec Loss 0.6809 Epoch: 19 Global Step: 189515 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:11:46,079-Speed 129.14 samples/sec Loss 0.5934 Epoch: 19 Global Step: 189520 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:12:05,834-Speed 129.59 samples/sec Loss 0.6494 Epoch: 19 Global Step: 189525 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:12:25,428-Speed 130.66 samples/sec Loss 0.6392 Epoch: 19 Global Step: 189530 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:12:45,336-Speed 128.60 samples/sec Loss 0.6170 Epoch: 19 Global Step: 189535 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:13:05,068-Speed 129.75 samples/sec Loss 0.6968 Epoch: 19 Global Step: 189540 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:13:24,993-Speed 128.49 samples/sec Loss 0.5430 Epoch: 19 Global Step: 189545 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:13:44,619-Speed 130.45 samples/sec Loss 0.6288 Epoch: 19 Global Step: 189550 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:14:04,264-Speed 130.31 samples/sec Loss 0.7073 Epoch: 19 Global Step: 189555 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:14:24,535-Speed 126.30 samples/sec Loss 0.6055 Epoch: 19 Global Step: 189560 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:14:44,508-Speed 128.18 samples/sec Loss 0.6416 Epoch: 19 Global Step: 189565 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:15:04,409-Speed 128.64 samples/sec Loss 0.6547 Epoch: 19 Global Step: 189570 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:15:24,293-Speed 128.76 samples/sec Loss 0.6363 Epoch: 19 Global Step: 189575 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:15:44,005-Speed 129.87 samples/sec Loss 0.5429 Epoch: 19 Global Step: 189580 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:16:03,767-Speed 129.55 samples/sec Loss 0.5883 Epoch: 19 Global Step: 189585 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:16:23,531-Speed 129.53 samples/sec Loss 0.6477 Epoch: 19 Global Step: 189590 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:16:43,342-Speed 129.23 samples/sec Loss 0.6914 Epoch: 19 Global Step: 189595 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:17:03,186-Speed 129.01 samples/sec Loss 0.6794 Epoch: 19 Global Step: 189600 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:17:22,964-Speed 129.45 samples/sec Loss 0.7166 Epoch: 19 Global Step: 189605 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:17:42,723-Speed 129.56 samples/sec Loss 0.6444 Epoch: 19 Global Step: 189610 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:18:02,511-Speed 129.38 samples/sec Loss 0.6574 Epoch: 19 Global Step: 189615 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:18:22,400-Speed 128.72 samples/sec Loss 0.5915 Epoch: 19 Global Step: 189620 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:18:42,113-Speed 129.87 samples/sec Loss 0.6488 Epoch: 19 Global Step: 189625 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:19:02,032-Speed 128.53 samples/sec Loss 0.5491 Epoch: 19 Global Step: 189630 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:19:21,683-Speed 130.28 samples/sec Loss 0.5683 Epoch: 19 Global Step: 189635 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:19:41,507-Speed 129.14 samples/sec Loss 0.6175 Epoch: 19 Global Step: 189640 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:20:01,440-Speed 128.44 samples/sec Loss 0.6875 Epoch: 19 Global Step: 189645 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:20:21,838-Speed 125.51 samples/sec Loss 0.6894 Epoch: 19 Global Step: 189650 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:20:41,443-Speed 130.58 samples/sec Loss 0.6599 Epoch: 19 Global Step: 189655 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:21:01,290-Speed 129.00 samples/sec Loss 0.5898 Epoch: 19 Global Step: 189660 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:21:21,035-Speed 129.66 samples/sec Loss 0.6544 Epoch: 19 Global Step: 189665 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:21:40,754-Speed 129.83 samples/sec Loss 0.6233 Epoch: 19 Global Step: 189670 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:22:00,664-Speed 128.59 samples/sec Loss 0.6103 Epoch: 19 Global Step: 189675 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:22:20,363-Speed 129.96 samples/sec Loss 0.6646 Epoch: 19 Global Step: 189680 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:22:40,324-Speed 128.26 samples/sec Loss 0.6612 Epoch: 19 Global Step: 189685 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:22:59,928-Speed 130.60 samples/sec Loss 0.7267 Epoch: 19 Global Step: 189690 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:23:20,339-Speed 125.43 samples/sec Loss 0.5223 Epoch: 19 Global Step: 189695 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:23:40,105-Speed 129.52 samples/sec Loss 0.6992 Epoch: 19 Global Step: 189700 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 06:24:00,048-Speed 128.37 samples/sec Loss 0.6411 Epoch: 19 Global Step: 189705 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:24:19,805-Speed 129.58 samples/sec Loss 0.6058 Epoch: 19 Global Step: 189710 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:24:39,569-Speed 129.54 samples/sec Loss 0.7386 Epoch: 19 Global Step: 189715 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:24:59,442-Speed 128.83 samples/sec Loss 0.6290 Epoch: 19 Global Step: 189720 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:25:19,193-Speed 129.62 samples/sec Loss 0.6325 Epoch: 19 Global Step: 189725 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:25:39,054-Speed 128.90 samples/sec Loss 0.7415 Epoch: 19 Global Step: 189730 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:25:58,648-Speed 130.66 samples/sec Loss 0.6731 Epoch: 19 Global Step: 189735 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:26:19,033-Speed 125.59 samples/sec Loss 0.7030 Epoch: 19 Global Step: 189740 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:26:38,738-Speed 129.92 samples/sec Loss 0.6357 Epoch: 19 Global Step: 189745 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:26:58,651-Speed 128.57 samples/sec Loss 0.5759 Epoch: 19 Global Step: 189750 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:27:18,251-Speed 130.61 samples/sec Loss 0.5531 Epoch: 19 Global Step: 189755 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:27:38,020-Speed 129.50 samples/sec Loss 0.6807 Epoch: 19 Global Step: 189760 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:27:57,870-Speed 128.98 samples/sec Loss 0.7344 Epoch: 19 Global Step: 189765 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:28:17,601-Speed 129.75 samples/sec Loss 0.7737 Epoch: 19 Global Step: 189770 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:28:37,474-Speed 128.82 samples/sec Loss 0.6569 Epoch: 19 Global Step: 189775 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:28:57,567-Speed 127.41 samples/sec Loss 0.6606 Epoch: 19 Global Step: 189780 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:29:17,916-Speed 125.81 samples/sec Loss 0.5787 Epoch: 19 Global Step: 189785 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:29:37,858-Speed 128.38 samples/sec Loss 0.6503 Epoch: 19 Global Step: 189790 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:29:57,684-Speed 129.12 samples/sec Loss 0.7155 Epoch: 19 Global Step: 189795 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:30:17,425-Speed 129.69 samples/sec Loss 0.6989 Epoch: 19 Global Step: 189800 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:30:37,289-Speed 128.88 samples/sec Loss 0.5914 Epoch: 19 Global Step: 189805 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:30:57,021-Speed 129.74 samples/sec Loss 0.7352 Epoch: 19 Global Step: 189810 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:31:16,747-Speed 129.79 samples/sec Loss 0.6128 Epoch: 19 Global Step: 189815 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:31:36,601-Speed 128.95 samples/sec Loss 0.6103 Epoch: 19 Global Step: 189820 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:31:56,338-Speed 129.71 samples/sec Loss 0.7013 Epoch: 19 Global Step: 189825 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:32:16,716-Speed 125.63 samples/sec Loss 0.7526 Epoch: 19 Global Step: 189830 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:32:36,615-Speed 128.65 samples/sec Loss 0.6923 Epoch: 19 Global Step: 189835 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:32:56,302-Speed 130.04 samples/sec Loss 0.6838 Epoch: 19 Global Step: 189840 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:33:16,100-Speed 129.31 samples/sec Loss 0.7220 Epoch: 19 Global Step: 189845 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:33:35,849-Speed 129.64 samples/sec Loss 0.6122 Epoch: 19 Global Step: 189850 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:33:55,584-Speed 129.73 samples/sec Loss 0.5944 Epoch: 19 Global Step: 189855 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:34:15,376-Speed 129.35 samples/sec Loss 0.5775 Epoch: 19 Global Step: 189860 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:34:35,171-Speed 129.33 samples/sec Loss 0.6864 Epoch: 19 Global Step: 189865 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:34:54,938-Speed 129.52 samples/sec Loss 0.6592 Epoch: 19 Global Step: 189870 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:35:15,285-Speed 125.82 samples/sec Loss 0.6596 Epoch: 19 Global Step: 189875 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:35:35,156-Speed 128.84 samples/sec Loss 0.6524 Epoch: 19 Global Step: 189880 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:35:54,993-Speed 129.06 samples/sec Loss 0.6845 Epoch: 19 Global Step: 189885 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:36:14,939-Speed 128.35 samples/sec Loss 0.7269 Epoch: 19 Global Step: 189890 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:36:34,603-Speed 130.20 samples/sec Loss 0.6429 Epoch: 19 Global Step: 189895 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:36:54,355-Speed 129.62 samples/sec Loss 0.6422 Epoch: 19 Global Step: 189900 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:37:14,102-Speed 129.64 samples/sec Loss 0.6514 Epoch: 19 Global Step: 189905 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:37:33,902-Speed 129.30 samples/sec Loss 0.5612 Epoch: 19 Global Step: 189910 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:37:53,699-Speed 129.32 samples/sec Loss 0.6036 Epoch: 19 Global Step: 189915 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:38:13,957-Speed 126.37 samples/sec Loss 0.6835 Epoch: 19 Global Step: 189920 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:38:33,705-Speed 129.64 samples/sec Loss 0.6007 Epoch: 19 Global Step: 189925 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:38:53,870-Speed 126.96 samples/sec Loss 0.6052 Epoch: 19 Global Step: 189930 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:39:13,728-Speed 128.92 samples/sec Loss 0.6284 Epoch: 19 Global Step: 189935 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:39:33,455-Speed 129.78 samples/sec Loss 0.5290 Epoch: 19 Global Step: 189940 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:39:53,222-Speed 129.52 samples/sec Loss 0.6679 Epoch: 19 Global Step: 189945 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:40:12,991-Speed 129.50 samples/sec Loss 0.7019 Epoch: 19 Global Step: 189950 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:40:32,917-Speed 128.49 samples/sec Loss 0.5691 Epoch: 19 Global Step: 189955 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:40:52,750-Speed 129.08 samples/sec Loss 0.5859 Epoch: 19 Global Step: 189960 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:41:12,971-Speed 126.61 samples/sec Loss 0.6282 Epoch: 19 Global Step: 189965 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:41:32,740-Speed 129.50 samples/sec Loss 0.6651 Epoch: 19 Global Step: 189970 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:41:52,510-Speed 129.50 samples/sec Loss 0.6152 Epoch: 19 Global Step: 189975 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:42:12,333-Speed 129.15 samples/sec Loss 0.6220 Epoch: 19 Global Step: 189980 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:42:32,049-Speed 129.85 samples/sec Loss 0.6280 Epoch: 19 Global Step: 189985 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:42:51,918-Speed 128.85 samples/sec Loss 0.6637 Epoch: 19 Global Step: 189990 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:43:11,623-Speed 129.93 samples/sec Loss 0.6630 Epoch: 19 Global Step: 189995 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:43:31,405-Speed 129.42 samples/sec Loss 0.6093 Epoch: 19 Global Step: 190000 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 06:58:19,302-[lfw][190000]XNorm: 21.840959 Training: 2021-06-16 06:58:19,302-[lfw][190000]Accuracy-Flip: 0.99800+-0.00194 Training: 2021-06-16 06:58:19,302-[lfw][190000]Accuracy-Highest: 0.99833 Training: 2021-06-16 07:15:34,221-[cfp_fp][190000]XNorm: 23.406103 Training: 2021-06-16 07:15:34,221-[cfp_fp][190000]Accuracy-Flip: 0.99157+-0.00243 Training: 2021-06-16 07:15:34,221-[cfp_fp][190000]Accuracy-Highest: 0.99229 Training: 2021-06-16 07:30:21,348-[agedb_30][190000]XNorm: 22.804657 Training: 2021-06-16 07:30:21,348-[agedb_30][190000]Accuracy-Flip: 0.98450+-0.00582 Training: 2021-06-16 07:30:21,348-[agedb_30][190000]Accuracy-Highest: 0.98583 Training: 2021-06-16 07:30:41,000-Speed 0.90 samples/sec Loss 0.6892 Epoch: 19 Global Step: 190005 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:31:01,153-Speed 127.04 samples/sec Loss 0.7162 Epoch: 19 Global Step: 190010 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:31:20,664-Speed 131.21 samples/sec Loss 0.6271 Epoch: 19 Global Step: 190015 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:31:40,286-Speed 130.47 samples/sec Loss 0.6197 Epoch: 19 Global Step: 190020 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:31:59,701-Speed 131.86 samples/sec Loss 0.6537 Epoch: 19 Global Step: 190025 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:32:19,598-Speed 128.67 samples/sec Loss 0.6130 Epoch: 19 Global Step: 190030 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:32:39,075-Speed 131.44 samples/sec Loss 0.7026 Epoch: 19 Global Step: 190035 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:32:58,683-Speed 130.57 samples/sec Loss 0.6454 Epoch: 19 Global Step: 190040 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:33:18,530-Speed 128.99 samples/sec Loss 0.5997 Epoch: 19 Global Step: 190045 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:33:38,239-Speed 129.90 samples/sec Loss 0.6814 Epoch: 19 Global Step: 190050 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:33:58,287-Speed 127.70 samples/sec Loss 0.6358 Epoch: 19 Global Step: 190055 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:34:18,023-Speed 129.72 samples/sec Loss 0.6671 Epoch: 19 Global Step: 190060 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:34:37,927-Speed 128.62 samples/sec Loss 0.6715 Epoch: 19 Global Step: 190065 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:34:57,546-Speed 130.50 samples/sec Loss 0.6572 Epoch: 19 Global Step: 190070 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:35:17,426-Speed 128.78 samples/sec Loss 0.6880 Epoch: 19 Global Step: 190075 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:35:37,295-Speed 128.85 samples/sec Loss 0.6302 Epoch: 19 Global Step: 190080 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:35:57,174-Speed 128.79 samples/sec Loss 0.6582 Epoch: 19 Global Step: 190085 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:36:16,928-Speed 129.61 samples/sec Loss 0.6643 Epoch: 19 Global Step: 190090 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:36:36,647-Speed 129.83 samples/sec Loss 0.7086 Epoch: 19 Global Step: 190095 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:36:56,613-Speed 128.22 samples/sec Loss 0.6923 Epoch: 19 Global Step: 190100 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:37:16,498-Speed 128.75 samples/sec Loss 0.6882 Epoch: 19 Global Step: 190105 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:37:36,223-Speed 129.79 samples/sec Loss 0.5980 Epoch: 19 Global Step: 190110 Fp16 Grad Scale: 4096 Required: 71 hours Training: 2021-06-16 07:37:55,952-Speed 129.77 samples/sec Loss 0.6293 Epoch: 19 Global Step: 190115 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:38:15,836-Speed 128.75 samples/sec Loss 0.6113 Epoch: 19 Global Step: 190120 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:38:35,727-Speed 128.70 samples/sec Loss 0.6670 Epoch: 19 Global Step: 190125 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:38:55,499-Speed 129.48 samples/sec Loss 0.6235 Epoch: 19 Global Step: 190130 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:39:15,409-Speed 128.59 samples/sec Loss 0.5480 Epoch: 19 Global Step: 190135 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:39:35,143-Speed 129.73 samples/sec Loss 0.6007 Epoch: 19 Global Step: 190140 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:39:55,105-Speed 128.25 samples/sec Loss 0.7315 Epoch: 19 Global Step: 190145 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:40:15,016-Speed 128.58 samples/sec Loss 0.6844 Epoch: 19 Global Step: 190150 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:40:34,756-Speed 129.69 samples/sec Loss 0.5897 Epoch: 19 Global Step: 190155 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:40:54,555-Speed 129.30 samples/sec Loss 0.7061 Epoch: 19 Global Step: 190160 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:41:14,421-Speed 128.88 samples/sec Loss 0.6559 Epoch: 19 Global Step: 190165 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:41:34,361-Speed 128.39 samples/sec Loss 0.5723 Epoch: 19 Global Step: 190170 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:41:54,377-Speed 127.90 samples/sec Loss 0.6123 Epoch: 19 Global Step: 190175 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:42:14,159-Speed 129.42 samples/sec Loss 0.6238 Epoch: 19 Global Step: 190180 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:42:33,874-Speed 129.86 samples/sec Loss 0.6805 Epoch: 19 Global Step: 190185 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:42:53,831-Speed 128.28 samples/sec Loss 0.5639 Epoch: 19 Global Step: 190190 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:43:13,765-Speed 128.42 samples/sec Loss 0.6567 Epoch: 19 Global Step: 190195 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:43:33,468-Speed 129.94 samples/sec Loss 0.6422 Epoch: 19 Global Step: 190200 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:43:53,316-Speed 128.99 samples/sec Loss 0.6854 Epoch: 19 Global Step: 190205 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:44:13,240-Speed 128.49 samples/sec Loss 0.6911 Epoch: 19 Global Step: 190210 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:44:33,181-Speed 128.38 samples/sec Loss 0.6382 Epoch: 19 Global Step: 190215 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:44:52,991-Speed 129.23 samples/sec Loss 0.6960 Epoch: 19 Global Step: 190220 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:45:12,835-Speed 129.02 samples/sec Loss 0.6103 Epoch: 19 Global Step: 190225 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:45:32,591-Speed 129.59 samples/sec Loss 0.6191 Epoch: 19 Global Step: 190230 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:45:52,607-Speed 127.90 samples/sec Loss 0.6230 Epoch: 19 Global Step: 190235 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:46:12,520-Speed 128.56 samples/sec Loss 0.7238 Epoch: 19 Global Step: 190240 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:46:32,174-Speed 130.26 samples/sec Loss 0.5984 Epoch: 19 Global Step: 190245 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:46:52,061-Speed 128.73 samples/sec Loss 0.7205 Epoch: 19 Global Step: 190250 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:47:11,931-Speed 128.85 samples/sec Loss 0.7221 Epoch: 19 Global Step: 190255 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:47:32,524-Speed 124.32 samples/sec Loss 0.5401 Epoch: 19 Global Step: 190260 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:47:53,122-Speed 124.29 samples/sec Loss 0.6137 Epoch: 19 Global Step: 190265 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:48:12,910-Speed 129.38 samples/sec Loss 0.6019 Epoch: 19 Global Step: 190270 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:48:32,784-Speed 128.82 samples/sec Loss 0.7309 Epoch: 19 Global Step: 190275 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:48:52,594-Speed 129.24 samples/sec Loss 0.5799 Epoch: 19 Global Step: 190280 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:49:12,535-Speed 128.38 samples/sec Loss 0.5686 Epoch: 19 Global Step: 190285 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:49:32,314-Speed 129.44 samples/sec Loss 0.7702 Epoch: 19 Global Step: 190290 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:49:52,286-Speed 128.19 samples/sec Loss 0.6527 Epoch: 19 Global Step: 190295 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:50:12,019-Speed 129.74 samples/sec Loss 0.7046 Epoch: 19 Global Step: 190300 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:50:32,139-Speed 127.24 samples/sec Loss 0.5782 Epoch: 19 Global Step: 190305 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:50:51,937-Speed 129.31 samples/sec Loss 0.5740 Epoch: 19 Global Step: 190310 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:51:11,840-Speed 128.63 samples/sec Loss 0.6607 Epoch: 19 Global Step: 190315 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:51:31,720-Speed 128.78 samples/sec Loss 0.5738 Epoch: 19 Global Step: 190320 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:51:51,696-Speed 128.16 samples/sec Loss 0.6259 Epoch: 19 Global Step: 190325 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:52:11,400-Speed 129.93 samples/sec Loss 0.7240 Epoch: 19 Global Step: 190330 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:52:31,162-Speed 129.55 samples/sec Loss 0.6280 Epoch: 19 Global Step: 190335 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:52:51,064-Speed 128.64 samples/sec Loss 0.6129 Epoch: 19 Global Step: 190340 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:53:10,791-Speed 129.78 samples/sec Loss 0.6636 Epoch: 19 Global Step: 190345 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:53:30,965-Speed 126.90 samples/sec Loss 0.5887 Epoch: 19 Global Step: 190350 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:53:50,767-Speed 129.29 samples/sec Loss 0.6101 Epoch: 19 Global Step: 190355 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:54:10,551-Speed 129.40 samples/sec Loss 0.6476 Epoch: 19 Global Step: 190360 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:54:30,375-Speed 129.14 samples/sec Loss 0.6379 Epoch: 19 Global Step: 190365 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:54:50,482-Speed 127.33 samples/sec Loss 0.5788 Epoch: 19 Global Step: 190370 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:55:10,224-Speed 129.68 samples/sec Loss 0.6508 Epoch: 19 Global Step: 190375 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:55:30,100-Speed 128.80 samples/sec Loss 0.6882 Epoch: 19 Global Step: 190380 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:55:49,908-Speed 129.25 samples/sec Loss 0.7053 Epoch: 19 Global Step: 190385 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:56:09,621-Speed 129.87 samples/sec Loss 0.6281 Epoch: 19 Global Step: 190390 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:56:29,719-Speed 127.38 samples/sec Loss 0.6299 Epoch: 19 Global Step: 190395 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:56:49,506-Speed 129.38 samples/sec Loss 0.6814 Epoch: 19 Global Step: 190400 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:57:09,403-Speed 128.67 samples/sec Loss 0.6837 Epoch: 19 Global Step: 190405 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:57:29,115-Speed 129.88 samples/sec Loss 0.7018 Epoch: 19 Global Step: 190410 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:57:49,138-Speed 127.87 samples/sec Loss 0.6979 Epoch: 19 Global Step: 190415 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:58:08,902-Speed 129.53 samples/sec Loss 0.6355 Epoch: 19 Global Step: 190420 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:58:28,872-Speed 128.20 samples/sec Loss 0.6786 Epoch: 19 Global Step: 190425 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:58:48,589-Speed 129.84 samples/sec Loss 0.6983 Epoch: 19 Global Step: 190430 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:59:08,255-Speed 130.18 samples/sec Loss 0.6754 Epoch: 19 Global Step: 190435 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:59:28,348-Speed 127.41 samples/sec Loss 0.6657 Epoch: 19 Global Step: 190440 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 07:59:48,148-Speed 129.30 samples/sec Loss 0.7201 Epoch: 19 Global Step: 190445 Fp16 Grad Scale: 8192 Required: 71 hours Training: 2021-06-16 08:00:08,038-Speed 128.71 samples/sec Loss 0.7623 Epoch: 19 Global Step: 190450 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:00:27,659-Speed 130.48 samples/sec Loss 0.6376 Epoch: 19 Global Step: 190455 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:00:47,723-Speed 127.60 samples/sec Loss 0.7222 Epoch: 19 Global Step: 190460 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:01:07,537-Speed 129.21 samples/sec Loss 0.6855 Epoch: 19 Global Step: 190465 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:01:27,493-Speed 128.28 samples/sec Loss 0.6293 Epoch: 19 Global Step: 190470 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:01:47,227-Speed 129.73 samples/sec Loss 0.6151 Epoch: 19 Global Step: 190475 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:02:07,004-Speed 129.45 samples/sec Loss 0.6645 Epoch: 19 Global Step: 190480 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:02:26,845-Speed 129.03 samples/sec Loss 0.7154 Epoch: 19 Global Step: 190485 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:02:46,707-Speed 128.89 samples/sec Loss 0.6246 Epoch: 19 Global Step: 190490 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:03:06,572-Speed 128.88 samples/sec Loss 0.6660 Epoch: 19 Global Step: 190495 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:03:26,179-Speed 130.57 samples/sec Loss 0.6498 Epoch: 19 Global Step: 190500 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:03:46,364-Speed 126.83 samples/sec Loss 0.7147 Epoch: 19 Global Step: 190505 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:04:06,256-Speed 128.70 samples/sec Loss 0.7011 Epoch: 19 Global Step: 190510 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:04:26,038-Speed 129.42 samples/sec Loss 0.6645 Epoch: 19 Global Step: 190515 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:04:45,867-Speed 129.11 samples/sec Loss 0.6909 Epoch: 19 Global Step: 190520 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:05:05,549-Speed 130.07 samples/sec Loss 0.6579 Epoch: 19 Global Step: 190525 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:05:25,589-Speed 127.75 samples/sec Loss 0.6391 Epoch: 19 Global Step: 190530 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:05:45,533-Speed 128.37 samples/sec Loss 0.6152 Epoch: 19 Global Step: 190535 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:06:05,254-Speed 129.82 samples/sec Loss 0.6952 Epoch: 19 Global Step: 190540 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:06:25,032-Speed 129.45 samples/sec Loss 0.6789 Epoch: 19 Global Step: 190545 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:06:44,989-Speed 128.28 samples/sec Loss 0.6307 Epoch: 19 Global Step: 190550 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:07:04,849-Speed 128.90 samples/sec Loss 0.6213 Epoch: 19 Global Step: 190555 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:07:24,571-Speed 129.81 samples/sec Loss 0.5436 Epoch: 19 Global Step: 190560 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:07:44,591-Speed 127.88 samples/sec Loss 0.6310 Epoch: 19 Global Step: 190565 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:08:04,206-Speed 130.52 samples/sec Loss 0.7523 Epoch: 19 Global Step: 190570 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:08:24,263-Speed 127.64 samples/sec Loss 0.6621 Epoch: 19 Global Step: 190575 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:08:44,133-Speed 128.84 samples/sec Loss 0.6065 Epoch: 19 Global Step: 190580 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:09:03,905-Speed 129.49 samples/sec Loss 0.7358 Epoch: 19 Global Step: 190585 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:09:23,878-Speed 128.18 samples/sec Loss 0.6402 Epoch: 19 Global Step: 190590 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:09:43,830-Speed 128.31 samples/sec Loss 0.6714 Epoch: 19 Global Step: 190595 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:10:03,697-Speed 128.87 samples/sec Loss 0.7002 Epoch: 19 Global Step: 190600 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:10:23,377-Speed 130.08 samples/sec Loss 0.6874 Epoch: 19 Global Step: 190605 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:10:43,364-Speed 128.09 samples/sec Loss 0.7273 Epoch: 19 Global Step: 190610 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:11:02,966-Speed 130.61 samples/sec Loss 0.6945 Epoch: 19 Global Step: 190615 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:11:23,099-Speed 127.16 samples/sec Loss 0.6593 Epoch: 19 Global Step: 190620 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:11:42,739-Speed 130.36 samples/sec Loss 0.5678 Epoch: 19 Global Step: 190625 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:12:02,547-Speed 129.25 samples/sec Loss 0.7295 Epoch: 19 Global Step: 190630 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:12:22,405-Speed 128.92 samples/sec Loss 0.6520 Epoch: 19 Global Step: 190635 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:12:42,470-Speed 127.60 samples/sec Loss 0.6447 Epoch: 19 Global Step: 190640 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:13:02,209-Speed 129.69 samples/sec Loss 0.6079 Epoch: 19 Global Step: 190645 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:13:21,883-Speed 130.13 samples/sec Loss 0.5912 Epoch: 19 Global Step: 190650 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:13:41,817-Speed 128.43 samples/sec Loss 0.6721 Epoch: 19 Global Step: 190655 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:14:01,463-Speed 130.31 samples/sec Loss 0.6975 Epoch: 19 Global Step: 190660 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:14:21,670-Speed 126.69 samples/sec Loss 0.6725 Epoch: 19 Global Step: 190665 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:14:41,329-Speed 130.23 samples/sec Loss 0.6280 Epoch: 19 Global Step: 190670 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:15:01,202-Speed 128.83 samples/sec Loss 0.5806 Epoch: 19 Global Step: 190675 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:15:20,884-Speed 130.08 samples/sec Loss 0.6270 Epoch: 19 Global Step: 190680 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:15:40,957-Speed 127.54 samples/sec Loss 0.5888 Epoch: 19 Global Step: 190685 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:16:00,654-Speed 129.97 samples/sec Loss 0.7107 Epoch: 19 Global Step: 190690 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:16:19,993-Speed 132.39 samples/sec Loss 0.6670 Epoch: 19 Global Step: 190695 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:16:39,841-Speed 128.99 samples/sec Loss 0.6478 Epoch: 19 Global Step: 190700 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:16:59,501-Speed 130.22 samples/sec Loss 0.7059 Epoch: 19 Global Step: 190705 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:17:19,581-Speed 127.50 samples/sec Loss 0.6962 Epoch: 19 Global Step: 190710 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:17:39,395-Speed 129.21 samples/sec Loss 0.7334 Epoch: 19 Global Step: 190715 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:17:59,312-Speed 128.54 samples/sec Loss 0.6246 Epoch: 19 Global Step: 190720 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:18:19,117-Speed 129.27 samples/sec Loss 0.6773 Epoch: 19 Global Step: 190725 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:18:39,204-Speed 127.46 samples/sec Loss 0.5999 Epoch: 19 Global Step: 190730 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:18:58,860-Speed 130.24 samples/sec Loss 0.6234 Epoch: 19 Global Step: 190735 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:19:18,727-Speed 128.86 samples/sec Loss 0.5267 Epoch: 19 Global Step: 190740 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:19:38,452-Speed 129.79 samples/sec Loss 0.6339 Epoch: 19 Global Step: 190745 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:19:58,347-Speed 128.68 samples/sec Loss 0.6904 Epoch: 19 Global Step: 190750 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:20:18,251-Speed 128.62 samples/sec Loss 0.7038 Epoch: 19 Global Step: 190755 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:20:37,999-Speed 129.64 samples/sec Loss 0.6937 Epoch: 19 Global Step: 190760 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:20:57,937-Speed 128.41 samples/sec Loss 0.6461 Epoch: 19 Global Step: 190765 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:21:17,626-Speed 130.03 samples/sec Loss 0.7161 Epoch: 19 Global Step: 190770 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:21:37,764-Speed 127.13 samples/sec Loss 0.5980 Epoch: 19 Global Step: 190775 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:21:57,422-Speed 130.23 samples/sec Loss 0.7086 Epoch: 19 Global Step: 190780 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:22:17,280-Speed 128.92 samples/sec Loss 0.6536 Epoch: 19 Global Step: 190785 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:22:37,099-Speed 129.17 samples/sec Loss 0.6604 Epoch: 19 Global Step: 190790 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:22:56,998-Speed 128.66 samples/sec Loss 0.6906 Epoch: 19 Global Step: 190795 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:23:16,892-Speed 128.69 samples/sec Loss 0.6004 Epoch: 19 Global Step: 190800 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:23:36,726-Speed 129.08 samples/sec Loss 0.6775 Epoch: 19 Global Step: 190805 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:23:56,628-Speed 128.64 samples/sec Loss 0.6623 Epoch: 19 Global Step: 190810 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:24:16,354-Speed 129.78 samples/sec Loss 0.6397 Epoch: 19 Global Step: 190815 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:24:36,342-Speed 128.08 samples/sec Loss 0.6986 Epoch: 19 Global Step: 190820 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:24:56,127-Speed 129.40 samples/sec Loss 0.6651 Epoch: 19 Global Step: 190825 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:25:15,774-Speed 130.31 samples/sec Loss 0.6101 Epoch: 19 Global Step: 190830 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:25:35,619-Speed 129.00 samples/sec Loss 0.6068 Epoch: 19 Global Step: 190835 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:25:55,497-Speed 128.79 samples/sec Loss 0.7170 Epoch: 19 Global Step: 190840 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:26:15,314-Speed 129.19 samples/sec Loss 0.5605 Epoch: 19 Global Step: 190845 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:26:35,233-Speed 128.53 samples/sec Loss 0.6814 Epoch: 19 Global Step: 190850 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:26:55,019-Speed 129.39 samples/sec Loss 0.7625 Epoch: 19 Global Step: 190855 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:27:14,799-Speed 129.44 samples/sec Loss 0.7286 Epoch: 19 Global Step: 190860 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:27:34,843-Speed 127.73 samples/sec Loss 0.6287 Epoch: 19 Global Step: 190865 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:27:54,732-Speed 128.72 samples/sec Loss 0.6574 Epoch: 19 Global Step: 190870 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:28:14,371-Speed 130.35 samples/sec Loss 0.7071 Epoch: 19 Global Step: 190875 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:28:34,208-Speed 129.06 samples/sec Loss 0.6793 Epoch: 19 Global Step: 190880 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:28:54,309-Speed 127.36 samples/sec Loss 0.6440 Epoch: 19 Global Step: 190885 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:29:14,192-Speed 128.76 samples/sec Loss 0.6819 Epoch: 19 Global Step: 190890 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:29:34,003-Speed 129.23 samples/sec Loss 0.7010 Epoch: 19 Global Step: 190895 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:29:53,759-Speed 129.59 samples/sec Loss 0.6636 Epoch: 19 Global Step: 190900 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:30:13,454-Speed 129.99 samples/sec Loss 0.5357 Epoch: 19 Global Step: 190905 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:30:33,408-Speed 128.31 samples/sec Loss 0.6648 Epoch: 19 Global Step: 190910 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:30:53,234-Speed 129.13 samples/sec Loss 0.6853 Epoch: 19 Global Step: 190915 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:31:12,942-Speed 129.90 samples/sec Loss 0.6571 Epoch: 19 Global Step: 190920 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:31:32,287-Speed 132.34 samples/sec Loss 0.6632 Epoch: 19 Global Step: 190925 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:31:52,196-Speed 128.60 samples/sec Loss 0.5564 Epoch: 19 Global Step: 190930 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:32:11,883-Speed 130.04 samples/sec Loss 0.6006 Epoch: 19 Global Step: 190935 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:32:31,742-Speed 128.92 samples/sec Loss 0.6886 Epoch: 19 Global Step: 190940 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:32:51,309-Speed 130.84 samples/sec Loss 0.6590 Epoch: 19 Global Step: 190945 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:33:11,227-Speed 128.53 samples/sec Loss 0.6294 Epoch: 19 Global Step: 190950 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:33:30,650-Speed 131.81 samples/sec Loss 0.6331 Epoch: 19 Global Step: 190955 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:33:50,407-Speed 129.59 samples/sec Loss 0.6986 Epoch: 19 Global Step: 190960 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:34:10,335-Speed 128.47 samples/sec Loss 0.8483 Epoch: 19 Global Step: 190965 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:34:30,252-Speed 128.54 samples/sec Loss 0.7157 Epoch: 19 Global Step: 190970 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:34:49,713-Speed 131.55 samples/sec Loss 0.6738 Epoch: 19 Global Step: 190975 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:35:09,451-Speed 129.71 samples/sec Loss 0.7113 Epoch: 19 Global Step: 190980 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:35:29,300-Speed 128.98 samples/sec Loss 0.6539 Epoch: 19 Global Step: 190985 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:35:48,832-Speed 131.07 samples/sec Loss 0.6551 Epoch: 19 Global Step: 190990 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:36:08,792-Speed 128.26 samples/sec Loss 0.6414 Epoch: 19 Global Step: 190995 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:36:28,212-Speed 131.83 samples/sec Loss 0.7686 Epoch: 19 Global Step: 191000 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:36:47,946-Speed 129.73 samples/sec Loss 0.6429 Epoch: 19 Global Step: 191005 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:37:07,695-Speed 129.63 samples/sec Loss 0.6633 Epoch: 19 Global Step: 191010 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:37:27,611-Speed 128.54 samples/sec Loss 0.5668 Epoch: 19 Global Step: 191015 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:37:47,021-Speed 131.89 samples/sec Loss 0.7410 Epoch: 19 Global Step: 191020 Fp16 Grad Scale: 4096 Required: 70 hours Training: 2021-06-16 08:38:07,000-Speed 128.14 samples/sec Loss 0.6871 Epoch: 19 Global Step: 191025 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:38:26,701-Speed 129.95 samples/sec Loss 0.7295 Epoch: 19 Global Step: 191030 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:38:46,254-Speed 130.94 samples/sec Loss 0.6120 Epoch: 19 Global Step: 191035 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:39:06,065-Speed 129.23 samples/sec Loss 0.6594 Epoch: 19 Global Step: 191040 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:39:25,673-Speed 130.56 samples/sec Loss 0.5460 Epoch: 19 Global Step: 191045 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:39:45,579-Speed 128.62 samples/sec Loss 0.6564 Epoch: 19 Global Step: 191050 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:40:05,380-Speed 129.29 samples/sec Loss 0.6881 Epoch: 19 Global Step: 191055 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:40:25,220-Speed 129.04 samples/sec Loss 0.5786 Epoch: 19 Global Step: 191060 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:40:44,675-Speed 131.59 samples/sec Loss 0.6126 Epoch: 19 Global Step: 191065 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:41:04,625-Speed 128.33 samples/sec Loss 0.6542 Epoch: 19 Global Step: 191070 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:41:24,480-Speed 128.94 samples/sec Loss 0.6356 Epoch: 19 Global Step: 191075 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:41:44,091-Speed 130.55 samples/sec Loss 0.7123 Epoch: 19 Global Step: 191080 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:42:03,827-Speed 129.72 samples/sec Loss 0.7525 Epoch: 19 Global Step: 191085 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:42:23,448-Speed 130.48 samples/sec Loss 0.6401 Epoch: 19 Global Step: 191090 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:42:43,295-Speed 128.99 samples/sec Loss 0.5901 Epoch: 19 Global Step: 191095 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:43:03,183-Speed 128.73 samples/sec Loss 0.6379 Epoch: 19 Global Step: 191100 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:43:22,918-Speed 129.72 samples/sec Loss 0.6168 Epoch: 19 Global Step: 191105 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:43:42,505-Speed 130.71 samples/sec Loss 0.6495 Epoch: 19 Global Step: 191110 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:44:02,429-Speed 128.49 samples/sec Loss 0.6231 Epoch: 19 Global Step: 191115 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:44:22,152-Speed 129.80 samples/sec Loss 0.6702 Epoch: 19 Global Step: 191120 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:44:41,772-Speed 130.49 samples/sec Loss 0.7117 Epoch: 19 Global Step: 191125 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:45:01,520-Speed 129.64 samples/sec Loss 0.6586 Epoch: 19 Global Step: 191130 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:45:21,121-Speed 130.61 samples/sec Loss 0.6796 Epoch: 19 Global Step: 191135 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:45:40,854-Speed 129.74 samples/sec Loss 0.7045 Epoch: 19 Global Step: 191140 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:46:00,829-Speed 128.16 samples/sec Loss 0.6363 Epoch: 19 Global Step: 191145 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:46:20,541-Speed 129.88 samples/sec Loss 0.6598 Epoch: 19 Global Step: 191150 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:46:40,061-Speed 131.16 samples/sec Loss 0.5620 Epoch: 19 Global Step: 191155 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:46:59,994-Speed 128.44 samples/sec Loss 0.7090 Epoch: 19 Global Step: 191160 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:47:19,698-Speed 129.93 samples/sec Loss 0.5618 Epoch: 19 Global Step: 191165 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:47:39,307-Speed 130.56 samples/sec Loss 0.6955 Epoch: 19 Global Step: 191170 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:47:59,033-Speed 129.79 samples/sec Loss 0.7094 Epoch: 19 Global Step: 191175 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:48:18,643-Speed 130.55 samples/sec Loss 0.7680 Epoch: 19 Global Step: 191180 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:48:38,354-Speed 129.89 samples/sec Loss 0.6636 Epoch: 19 Global Step: 191185 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:48:58,232-Speed 128.80 samples/sec Loss 0.6427 Epoch: 19 Global Step: 191190 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:49:17,956-Speed 129.79 samples/sec Loss 0.6284 Epoch: 19 Global Step: 191195 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:49:37,651-Speed 129.99 samples/sec Loss 0.5920 Epoch: 19 Global Step: 191200 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:49:57,548-Speed 128.67 samples/sec Loss 0.7065 Epoch: 19 Global Step: 191205 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:50:17,295-Speed 129.65 samples/sec Loss 0.5651 Epoch: 19 Global Step: 191210 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:50:36,876-Speed 130.75 samples/sec Loss 0.6483 Epoch: 19 Global Step: 191215 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:50:56,597-Speed 129.82 samples/sec Loss 0.6631 Epoch: 19 Global Step: 191220 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:51:16,187-Speed 130.68 samples/sec Loss 0.7428 Epoch: 19 Global Step: 191225 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:51:35,878-Speed 130.02 samples/sec Loss 0.6809 Epoch: 19 Global Step: 191230 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:51:55,656-Speed 129.44 samples/sec Loss 0.7675 Epoch: 19 Global Step: 191235 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:52:15,402-Speed 129.65 samples/sec Loss 0.6894 Epoch: 19 Global Step: 191240 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:52:35,135-Speed 129.74 samples/sec Loss 0.6395 Epoch: 19 Global Step: 191245 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:52:54,903-Speed 129.51 samples/sec Loss 0.7307 Epoch: 19 Global Step: 191250 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:53:14,654-Speed 129.62 samples/sec Loss 0.5512 Epoch: 19 Global Step: 191255 Fp16 Grad Scale: 8192 Required: 70 hours Training: 2021-06-16 08:53:34,211-Speed 130.91 samples/sec Loss 0.7043 Epoch: 19 Global Step: 191260 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 08:53:53,917-Speed 129.92 samples/sec Loss 0.7584 Epoch: 19 Global Step: 191265 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 08:54:13,083-Speed 133.58 samples/sec Loss 0.6444 Epoch: 19 Global Step: 191270 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:54:32,649-Speed 130.84 samples/sec Loss 0.6198 Epoch: 19 Global Step: 191275 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:54:52,763-Speed 127.29 samples/sec Loss 0.7405 Epoch: 19 Global Step: 191280 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:55:12,532-Speed 129.50 samples/sec Loss 0.7025 Epoch: 19 Global Step: 191285 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:55:32,380-Speed 128.98 samples/sec Loss 0.7054 Epoch: 19 Global Step: 191290 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:55:52,787-Speed 125.45 samples/sec Loss 0.7226 Epoch: 19 Global Step: 191295 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:56:12,684-Speed 128.67 samples/sec Loss 0.6450 Epoch: 19 Global Step: 191300 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:56:32,629-Speed 128.36 samples/sec Loss 0.6387 Epoch: 19 Global Step: 191305 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:56:52,730-Speed 127.37 samples/sec Loss 0.6821 Epoch: 19 Global Step: 191310 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:57:12,514-Speed 129.40 samples/sec Loss 0.7052 Epoch: 19 Global Step: 191315 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:57:32,272-Speed 129.57 samples/sec Loss 0.6617 Epoch: 19 Global Step: 191320 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:57:52,289-Speed 127.90 samples/sec Loss 0.6653 Epoch: 19 Global Step: 191325 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:58:22,113-Speed 85.84 samples/sec Loss 0.7181 Epoch: 19 Global Step: 191330 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:58:59,688-Speed 68.14 samples/sec Loss 0.5917 Epoch: 19 Global Step: 191335 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 08:59:38,258-Speed 66.38 samples/sec Loss 0.7307 Epoch: 19 Global Step: 191340 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:00:20,207-Speed 61.03 samples/sec Loss 0.6557 Epoch: 19 Global Step: 191345 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:01:01,436-Speed 62.09 samples/sec Loss 0.6644 Epoch: 19 Global Step: 191350 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:01:44,169-Speed 59.91 samples/sec Loss 0.6338 Epoch: 19 Global Step: 191355 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:02:28,341-Speed 57.96 samples/sec Loss 0.6469 Epoch: 19 Global Step: 191360 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:03:12,688-Speed 57.73 samples/sec Loss 0.6417 Epoch: 19 Global Step: 191365 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:03:55,789-Speed 59.40 samples/sec Loss 0.6586 Epoch: 19 Global Step: 191370 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:04:32,120-Speed 70.47 samples/sec Loss 0.6206 Epoch: 19 Global Step: 191375 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:05:14,454-Speed 60.47 samples/sec Loss 0.6945 Epoch: 19 Global Step: 191380 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:05:58,569-Speed 58.03 samples/sec Loss 0.6903 Epoch: 19 Global Step: 191385 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:06:42,349-Speed 58.48 samples/sec Loss 0.6111 Epoch: 19 Global Step: 191390 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:07:28,129-Speed 55.92 samples/sec Loss 0.6238 Epoch: 19 Global Step: 191395 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:08:10,543-Speed 60.37 samples/sec Loss 0.7602 Epoch: 19 Global Step: 191400 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:08:56,049-Speed 56.27 samples/sec Loss 0.6458 Epoch: 19 Global Step: 191405 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:09:29,568-Speed 76.38 samples/sec Loss 0.7257 Epoch: 19 Global Step: 191410 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:10:16,897-Speed 54.10 samples/sec Loss 0.7108 Epoch: 19 Global Step: 191415 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:10:59,905-Speed 59.54 samples/sec Loss 0.7250 Epoch: 19 Global Step: 191420 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:11:41,774-Speed 61.15 samples/sec Loss 0.8069 Epoch: 19 Global Step: 191425 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:12:24,456-Speed 59.98 samples/sec Loss 0.5969 Epoch: 19 Global Step: 191430 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:13:10,443-Speed 55.67 samples/sec Loss 0.6685 Epoch: 19 Global Step: 191435 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:13:51,374-Speed 62.56 samples/sec Loss 0.6641 Epoch: 19 Global Step: 191440 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:14:33,451-Speed 60.85 samples/sec Loss 0.5524 Epoch: 19 Global Step: 191445 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:15:14,539-Speed 62.31 samples/sec Loss 0.7526 Epoch: 19 Global Step: 191450 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:15:57,910-Speed 59.04 samples/sec Loss 0.6082 Epoch: 19 Global Step: 191455 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:16:37,572-Speed 64.56 samples/sec Loss 0.5868 Epoch: 19 Global Step: 191460 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:17:18,350-Speed 62.80 samples/sec Loss 0.7857 Epoch: 19 Global Step: 191465 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:17:59,708-Speed 61.90 samples/sec Loss 0.6377 Epoch: 19 Global Step: 191470 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:18:40,151-Speed 63.31 samples/sec Loss 0.6439 Epoch: 19 Global Step: 191475 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:19:17,608-Speed 68.36 samples/sec Loss 0.7529 Epoch: 19 Global Step: 191480 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:19:48,522-Speed 82.82 samples/sec Loss 0.6609 Epoch: 19 Global Step: 191485 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:20:08,559-Speed 127.77 samples/sec Loss 0.7514 Epoch: 19 Global Step: 191490 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:20:28,521-Speed 128.25 samples/sec Loss 0.7483 Epoch: 19 Global Step: 191495 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:20:48,242-Speed 129.82 samples/sec Loss 0.7994 Epoch: 19 Global Step: 191500 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:21:08,632-Speed 125.56 samples/sec Loss 0.6665 Epoch: 19 Global Step: 191505 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:21:28,418-Speed 129.39 samples/sec Loss 0.6272 Epoch: 19 Global Step: 191510 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:21:48,465-Speed 127.71 samples/sec Loss 0.6915 Epoch: 19 Global Step: 191515 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:22:08,224-Speed 129.57 samples/sec Loss 0.6405 Epoch: 19 Global Step: 191520 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:22:27,938-Speed 129.86 samples/sec Loss 0.5382 Epoch: 19 Global Step: 191525 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:22:48,131-Speed 126.79 samples/sec Loss 0.6655 Epoch: 19 Global Step: 191530 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:23:07,851-Speed 129.82 samples/sec Loss 0.5498 Epoch: 19 Global Step: 191535 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:23:27,761-Speed 128.58 samples/sec Loss 0.6067 Epoch: 19 Global Step: 191540 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:23:47,422-Speed 130.22 samples/sec Loss 0.6777 Epoch: 19 Global Step: 191545 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:24:07,445-Speed 127.86 samples/sec Loss 0.6716 Epoch: 19 Global Step: 191550 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:24:27,255-Speed 129.23 samples/sec Loss 0.6062 Epoch: 19 Global Step: 191555 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:24:47,098-Speed 129.02 samples/sec Loss 0.6216 Epoch: 19 Global Step: 191560 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:25:06,836-Speed 129.71 samples/sec Loss 0.6370 Epoch: 19 Global Step: 191565 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:25:26,720-Speed 128.75 samples/sec Loss 0.5933 Epoch: 19 Global Step: 191570 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:25:46,727-Speed 127.96 samples/sec Loss 0.6567 Epoch: 19 Global Step: 191575 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:26:06,454-Speed 129.78 samples/sec Loss 0.6230 Epoch: 19 Global Step: 191580 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:26:26,338-Speed 128.75 samples/sec Loss 0.6630 Epoch: 19 Global Step: 191585 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:26:46,041-Speed 129.94 samples/sec Loss 0.6919 Epoch: 19 Global Step: 191590 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:27:06,064-Speed 127.86 samples/sec Loss 0.6847 Epoch: 19 Global Step: 191595 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:27:25,875-Speed 129.23 samples/sec Loss 0.6542 Epoch: 19 Global Step: 191600 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:27:45,737-Speed 128.90 samples/sec Loss 0.7745 Epoch: 19 Global Step: 191605 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:28:05,459-Speed 129.81 samples/sec Loss 0.6549 Epoch: 19 Global Step: 191610 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:28:25,288-Speed 129.11 samples/sec Loss 0.7517 Epoch: 19 Global Step: 191615 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:28:45,163-Speed 128.81 samples/sec Loss 0.6990 Epoch: 19 Global Step: 191620 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:29:04,952-Speed 129.37 samples/sec Loss 0.7149 Epoch: 19 Global Step: 191625 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:29:24,817-Speed 128.88 samples/sec Loss 0.6834 Epoch: 19 Global Step: 191630 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:29:44,542-Speed 129.79 samples/sec Loss 0.7145 Epoch: 19 Global Step: 191635 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:30:04,597-Speed 127.66 samples/sec Loss 0.8124 Epoch: 19 Global Step: 191640 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:30:23,903-Speed 132.61 samples/sec Loss 0.7575 Epoch: 19 Global Step: 191645 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:30:43,788-Speed 128.75 samples/sec Loss 0.6264 Epoch: 19 Global Step: 191650 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:31:03,554-Speed 129.52 samples/sec Loss 0.6869 Epoch: 19 Global Step: 191655 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:31:23,458-Speed 128.62 samples/sec Loss 0.6582 Epoch: 19 Global Step: 191660 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:31:43,427-Speed 128.20 samples/sec Loss 0.6845 Epoch: 19 Global Step: 191665 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:32:03,253-Speed 129.13 samples/sec Loss 0.7109 Epoch: 19 Global Step: 191670 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:32:22,958-Speed 129.93 samples/sec Loss 0.6518 Epoch: 19 Global Step: 191675 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:32:42,670-Speed 129.88 samples/sec Loss 0.7003 Epoch: 19 Global Step: 191680 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:33:02,765-Speed 127.40 samples/sec Loss 0.6295 Epoch: 19 Global Step: 191685 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:33:22,654-Speed 128.72 samples/sec Loss 0.6708 Epoch: 19 Global Step: 191690 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:33:42,379-Speed 129.79 samples/sec Loss 0.7364 Epoch: 19 Global Step: 191695 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:34:02,042-Speed 130.20 samples/sec Loss 0.6811 Epoch: 19 Global Step: 191700 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:34:21,920-Speed 128.80 samples/sec Loss 0.6400 Epoch: 19 Global Step: 191705 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:34:42,014-Speed 127.40 samples/sec Loss 0.7280 Epoch: 19 Global Step: 191710 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:35:01,919-Speed 128.62 samples/sec Loss 0.6405 Epoch: 19 Global Step: 191715 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:35:21,680-Speed 129.55 samples/sec Loss 0.6636 Epoch: 19 Global Step: 191720 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:35:41,471-Speed 129.36 samples/sec Loss 0.6299 Epoch: 19 Global Step: 191725 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:36:01,536-Speed 127.59 samples/sec Loss 0.7069 Epoch: 19 Global Step: 191730 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:36:21,500-Speed 128.23 samples/sec Loss 0.6531 Epoch: 19 Global Step: 191735 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:36:41,216-Speed 129.85 samples/sec Loss 0.7012 Epoch: 19 Global Step: 191740 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:37:01,011-Speed 129.33 samples/sec Loss 0.6279 Epoch: 19 Global Step: 191745 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:37:20,726-Speed 129.86 samples/sec Loss 0.6553 Epoch: 19 Global Step: 191750 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:37:40,775-Speed 127.69 samples/sec Loss 0.6531 Epoch: 19 Global Step: 191755 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:38:00,691-Speed 128.55 samples/sec Loss 0.6074 Epoch: 19 Global Step: 191760 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:38:20,339-Speed 130.30 samples/sec Loss 0.6105 Epoch: 19 Global Step: 191765 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:38:40,231-Speed 128.71 samples/sec Loss 0.5724 Epoch: 19 Global Step: 191770 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:39:00,116-Speed 128.74 samples/sec Loss 0.7157 Epoch: 19 Global Step: 191775 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:39:20,049-Speed 128.43 samples/sec Loss 0.6967 Epoch: 19 Global Step: 191780 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:39:39,756-Speed 129.91 samples/sec Loss 0.7386 Epoch: 19 Global Step: 191785 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:39:59,656-Speed 128.65 samples/sec Loss 0.6543 Epoch: 19 Global Step: 191790 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:40:19,365-Speed 129.89 samples/sec Loss 0.7010 Epoch: 19 Global Step: 191795 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:40:39,752-Speed 125.58 samples/sec Loss 0.6373 Epoch: 19 Global Step: 191800 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:40:59,454-Speed 129.94 samples/sec Loss 0.6791 Epoch: 19 Global Step: 191805 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:41:19,116-Speed 130.21 samples/sec Loss 0.7019 Epoch: 19 Global Step: 191810 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:41:38,976-Speed 128.91 samples/sec Loss 0.7345 Epoch: 19 Global Step: 191815 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:41:59,124-Speed 127.07 samples/sec Loss 0.7012 Epoch: 19 Global Step: 191820 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:42:19,057-Speed 128.44 samples/sec Loss 0.7272 Epoch: 19 Global Step: 191825 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:42:38,794-Speed 129.71 samples/sec Loss 0.7023 Epoch: 19 Global Step: 191830 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:42:58,566-Speed 129.48 samples/sec Loss 0.6652 Epoch: 19 Global Step: 191835 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:43:18,239-Speed 130.13 samples/sec Loss 0.7929 Epoch: 19 Global Step: 191840 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:43:38,517-Speed 126.25 samples/sec Loss 0.6581 Epoch: 19 Global Step: 191845 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:43:58,224-Speed 129.91 samples/sec Loss 0.6558 Epoch: 19 Global Step: 191850 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:44:17,971-Speed 129.65 samples/sec Loss 0.7165 Epoch: 19 Global Step: 191855 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:44:37,805-Speed 129.08 samples/sec Loss 0.6401 Epoch: 19 Global Step: 191860 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:44:57,808-Speed 127.99 samples/sec Loss 0.6627 Epoch: 19 Global Step: 191865 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:45:16,973-Speed 133.58 samples/sec Loss 0.6931 Epoch: 19 Global Step: 191870 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:45:36,943-Speed 128.19 samples/sec Loss 0.6773 Epoch: 19 Global Step: 191875 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:45:57,004-Speed 127.62 samples/sec Loss 0.6847 Epoch: 19 Global Step: 191880 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:46:16,446-Speed 131.68 samples/sec Loss 0.7073 Epoch: 19 Global Step: 191885 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:46:36,527-Speed 127.49 samples/sec Loss 0.7319 Epoch: 19 Global Step: 191890 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:46:56,247-Speed 129.82 samples/sec Loss 0.6609 Epoch: 19 Global Step: 191895 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:47:15,855-Speed 130.57 samples/sec Loss 0.7702 Epoch: 19 Global Step: 191900 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:47:35,578-Speed 129.80 samples/sec Loss 0.6699 Epoch: 19 Global Step: 191905 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:47:55,122-Speed 130.99 samples/sec Loss 0.7162 Epoch: 19 Global Step: 191910 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:48:15,012-Speed 128.72 samples/sec Loss 0.6358 Epoch: 19 Global Step: 191915 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:48:34,902-Speed 128.72 samples/sec Loss 0.6473 Epoch: 19 Global Step: 191920 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:48:54,531-Speed 130.42 samples/sec Loss 0.7502 Epoch: 19 Global Step: 191925 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:49:14,125-Speed 130.66 samples/sec Loss 0.7137 Epoch: 19 Global Step: 191930 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:49:34,001-Speed 128.80 samples/sec Loss 0.5629 Epoch: 19 Global Step: 191935 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:49:53,715-Speed 129.86 samples/sec Loss 0.6289 Epoch: 19 Global Step: 191940 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:50:13,318-Speed 130.61 samples/sec Loss 0.6942 Epoch: 19 Global Step: 191945 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:50:33,123-Speed 129.26 samples/sec Loss 0.5646 Epoch: 19 Global Step: 191950 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:50:52,592-Speed 131.50 samples/sec Loss 0.6685 Epoch: 19 Global Step: 191955 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:51:12,458-Speed 128.87 samples/sec Loss 0.6231 Epoch: 19 Global Step: 191960 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:51:32,371-Speed 128.57 samples/sec Loss 0.7154 Epoch: 19 Global Step: 191965 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:51:52,104-Speed 129.74 samples/sec Loss 0.7190 Epoch: 19 Global Step: 191970 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:52:11,680-Speed 130.78 samples/sec Loss 0.6872 Epoch: 19 Global Step: 191975 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:52:31,582-Speed 128.64 samples/sec Loss 0.7197 Epoch: 19 Global Step: 191980 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:52:51,358-Speed 129.46 samples/sec Loss 0.6637 Epoch: 19 Global Step: 191985 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:53:10,979-Speed 130.48 samples/sec Loss 0.5421 Epoch: 19 Global Step: 191990 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:53:30,669-Speed 130.02 samples/sec Loss 0.5305 Epoch: 19 Global Step: 191995 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 09:53:50,038-Speed 132.18 samples/sec Loss 0.6600 Epoch: 19 Global Step: 192000 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:54:09,753-Speed 129.86 samples/sec Loss 0.6343 Epoch: 19 Global Step: 192005 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:54:29,846-Speed 127.41 samples/sec Loss 0.7250 Epoch: 19 Global Step: 192010 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:54:49,509-Speed 130.19 samples/sec Loss 0.7365 Epoch: 19 Global Step: 192015 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:55:09,448-Speed 128.40 samples/sec Loss 0.6226 Epoch: 19 Global Step: 192020 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:55:29,790-Speed 125.86 samples/sec Loss 0.7442 Epoch: 19 Global Step: 192025 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:55:49,641-Speed 128.97 samples/sec Loss 0.6017 Epoch: 19 Global Step: 192030 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:56:09,383-Speed 129.68 samples/sec Loss 0.7338 Epoch: 19 Global Step: 192035 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:56:29,284-Speed 128.65 samples/sec Loss 0.6222 Epoch: 19 Global Step: 192040 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:56:49,143-Speed 128.91 samples/sec Loss 0.6986 Epoch: 19 Global Step: 192045 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:57:09,176-Speed 127.79 samples/sec Loss 0.6536 Epoch: 19 Global Step: 192050 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:57:29,293-Speed 127.26 samples/sec Loss 0.6605 Epoch: 19 Global Step: 192055 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:57:49,021-Speed 129.77 samples/sec Loss 0.6075 Epoch: 19 Global Step: 192060 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:58:08,880-Speed 128.91 samples/sec Loss 0.5946 Epoch: 19 Global Step: 192065 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:58:28,501-Speed 130.48 samples/sec Loss 0.6446 Epoch: 19 Global Step: 192070 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:58:48,466-Speed 128.23 samples/sec Loss 0.6145 Epoch: 19 Global Step: 192075 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:59:08,188-Speed 129.81 samples/sec Loss 0.6039 Epoch: 19 Global Step: 192080 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:59:27,844-Speed 130.25 samples/sec Loss 0.6991 Epoch: 19 Global Step: 192085 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 09:59:47,700-Speed 128.93 samples/sec Loss 0.6963 Epoch: 19 Global Step: 192090 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 10:00:07,721-Speed 127.87 samples/sec Loss 0.6970 Epoch: 19 Global Step: 192095 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 10:00:27,880-Speed 127.00 samples/sec Loss 0.6518 Epoch: 19 Global Step: 192100 Fp16 Grad Scale: 8192 Required: 69 hours Training: 2021-06-16 10:00:47,222-Speed 132.36 samples/sec Loss 0.7594 Epoch: 19 Global Step: 192105 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 10:01:06,950-Speed 129.77 samples/sec Loss 0.6597 Epoch: 19 Global Step: 192110 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 10:01:26,662-Speed 129.87 samples/sec Loss 0.6878 Epoch: 19 Global Step: 192115 Fp16 Grad Scale: 4096 Required: 69 hours Training: 2021-06-16 10:01:46,626-Speed 128.24 samples/sec Loss 0.6613 Epoch: 19 Global Step: 192120 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:02:06,364-Speed 129.71 samples/sec Loss 0.7311 Epoch: 19 Global Step: 192125 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:02:26,160-Speed 129.33 samples/sec Loss 0.6558 Epoch: 19 Global Step: 192130 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:02:45,875-Speed 129.86 samples/sec Loss 0.6690 Epoch: 19 Global Step: 192135 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:03:05,998-Speed 127.22 samples/sec Loss 0.6377 Epoch: 19 Global Step: 192140 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:03:26,062-Speed 127.60 samples/sec Loss 0.7424 Epoch: 19 Global Step: 192145 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:03:45,864-Speed 129.29 samples/sec Loss 0.6016 Epoch: 19 Global Step: 192150 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:04:05,582-Speed 129.84 samples/sec Loss 0.7253 Epoch: 19 Global Step: 192155 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:04:25,240-Speed 130.23 samples/sec Loss 0.6803 Epoch: 19 Global Step: 192160 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:04:45,168-Speed 128.47 samples/sec Loss 0.6990 Epoch: 19 Global Step: 192165 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:05:04,888-Speed 129.83 samples/sec Loss 0.7356 Epoch: 19 Global Step: 192170 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:05:24,707-Speed 129.18 samples/sec Loss 0.6591 Epoch: 19 Global Step: 192175 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:05:44,413-Speed 129.91 samples/sec Loss 0.6802 Epoch: 19 Global Step: 192180 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:06:04,361-Speed 128.34 samples/sec Loss 0.7424 Epoch: 19 Global Step: 192185 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:06:24,410-Speed 127.70 samples/sec Loss 0.5930 Epoch: 19 Global Step: 192190 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:06:44,373-Speed 128.24 samples/sec Loss 0.6157 Epoch: 19 Global Step: 192195 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:07:04,086-Speed 129.87 samples/sec Loss 0.7501 Epoch: 19 Global Step: 192200 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:07:23,889-Speed 129.28 samples/sec Loss 0.7735 Epoch: 19 Global Step: 192205 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:07:43,595-Speed 129.92 samples/sec Loss 0.6065 Epoch: 19 Global Step: 192210 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:08:03,351-Speed 129.59 samples/sec Loss 0.6618 Epoch: 19 Global Step: 192215 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:08:23,217-Speed 128.87 samples/sec Loss 0.6222 Epoch: 19 Global Step: 192220 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:08:42,847-Speed 130.42 samples/sec Loss 0.6240 Epoch: 19 Global Step: 192225 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:09:02,950-Speed 127.35 samples/sec Loss 0.7392 Epoch: 19 Global Step: 192230 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:09:22,952-Speed 127.99 samples/sec Loss 0.7374 Epoch: 19 Global Step: 192235 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:09:42,811-Speed 128.92 samples/sec Loss 0.7391 Epoch: 19 Global Step: 192240 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:10:02,535-Speed 129.79 samples/sec Loss 0.7948 Epoch: 19 Global Step: 192245 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:10:22,348-Speed 129.21 samples/sec Loss 0.6902 Epoch: 19 Global Step: 192250 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:10:42,078-Speed 129.76 samples/sec Loss 0.7201 Epoch: 19 Global Step: 192255 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:11:01,960-Speed 128.77 samples/sec Loss 0.6777 Epoch: 19 Global Step: 192260 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:11:21,749-Speed 129.38 samples/sec Loss 0.7356 Epoch: 19 Global Step: 192265 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:11:41,458-Speed 129.90 samples/sec Loss 0.6994 Epoch: 19 Global Step: 192270 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:12:01,604-Speed 127.08 samples/sec Loss 0.6888 Epoch: 19 Global Step: 192275 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:12:21,544-Speed 128.39 samples/sec Loss 0.6224 Epoch: 19 Global Step: 192280 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:12:41,502-Speed 128.27 samples/sec Loss 0.7027 Epoch: 19 Global Step: 192285 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:13:01,234-Speed 129.75 samples/sec Loss 0.7434 Epoch: 19 Global Step: 192290 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:13:21,037-Speed 129.28 samples/sec Loss 0.6960 Epoch: 19 Global Step: 192295 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:13:40,854-Speed 129.19 samples/sec Loss 0.6713 Epoch: 19 Global Step: 192300 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:14:00,665-Speed 129.23 samples/sec Loss 0.6490 Epoch: 19 Global Step: 192305 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:14:20,376-Speed 129.88 samples/sec Loss 0.6890 Epoch: 19 Global Step: 192310 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:14:40,136-Speed 129.56 samples/sec Loss 0.6119 Epoch: 19 Global Step: 192315 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:14:59,836-Speed 129.96 samples/sec Loss 0.6733 Epoch: 19 Global Step: 192320 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:15:19,545-Speed 129.90 samples/sec Loss 0.7426 Epoch: 19 Global Step: 192325 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:15:39,485-Speed 128.39 samples/sec Loss 0.6793 Epoch: 19 Global Step: 192330 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:15:59,169-Speed 130.06 samples/sec Loss 0.7100 Epoch: 19 Global Step: 192335 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:16:19,148-Speed 128.14 samples/sec Loss 0.6578 Epoch: 19 Global Step: 192340 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:16:38,868-Speed 129.82 samples/sec Loss 0.7062 Epoch: 19 Global Step: 192345 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:16:58,840-Speed 128.19 samples/sec Loss 0.6256 Epoch: 19 Global Step: 192350 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:17:18,619-Speed 129.43 samples/sec Loss 0.6586 Epoch: 19 Global Step: 192355 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:17:38,204-Speed 130.73 samples/sec Loss 0.6404 Epoch: 19 Global Step: 192360 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:17:58,020-Speed 129.19 samples/sec Loss 0.6230 Epoch: 19 Global Step: 192365 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:18:18,012-Speed 128.06 samples/sec Loss 0.7097 Epoch: 19 Global Step: 192370 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:18:37,463-Speed 131.62 samples/sec Loss 0.7218 Epoch: 19 Global Step: 192375 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:18:57,219-Speed 129.59 samples/sec Loss 0.8219 Epoch: 19 Global Step: 192380 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:19:16,793-Speed 130.80 samples/sec Loss 0.6454 Epoch: 19 Global Step: 192385 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:19:36,487-Speed 130.00 samples/sec Loss 0.6199 Epoch: 19 Global Step: 192390 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:19:56,388-Speed 128.64 samples/sec Loss 0.5778 Epoch: 19 Global Step: 192395 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:20:16,214-Speed 129.13 samples/sec Loss 0.8349 Epoch: 19 Global Step: 192400 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:20:35,707-Speed 131.34 samples/sec Loss 0.6388 Epoch: 19 Global Step: 192405 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:20:55,536-Speed 129.11 samples/sec Loss 0.6582 Epoch: 19 Global Step: 192410 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:21:15,507-Speed 128.20 samples/sec Loss 0.6188 Epoch: 19 Global Step: 192415 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:21:35,022-Speed 131.19 samples/sec Loss 0.6418 Epoch: 19 Global Step: 192420 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:21:54,781-Speed 129.57 samples/sec Loss 0.6467 Epoch: 19 Global Step: 192425 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:22:14,285-Speed 131.27 samples/sec Loss 0.6071 Epoch: 19 Global Step: 192430 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:22:34,048-Speed 129.54 samples/sec Loss 0.7011 Epoch: 19 Global Step: 192435 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:22:53,824-Speed 129.45 samples/sec Loss 0.6248 Epoch: 19 Global Step: 192440 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:23:13,854-Speed 127.82 samples/sec Loss 0.6483 Epoch: 19 Global Step: 192445 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:23:33,259-Speed 131.93 samples/sec Loss 0.7257 Epoch: 19 Global Step: 192450 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:23:52,987-Speed 129.77 samples/sec Loss 0.6708 Epoch: 19 Global Step: 192455 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:24:12,866-Speed 128.79 samples/sec Loss 0.6736 Epoch: 19 Global Step: 192460 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:24:32,562-Speed 129.99 samples/sec Loss 0.8319 Epoch: 19 Global Step: 192465 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:24:52,441-Speed 128.78 samples/sec Loss 0.7062 Epoch: 19 Global Step: 192470 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:25:11,907-Speed 131.52 samples/sec Loss 0.6926 Epoch: 19 Global Step: 192475 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:25:31,706-Speed 129.31 samples/sec Loss 0.6304 Epoch: 19 Global Step: 192480 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:25:51,509-Speed 129.28 samples/sec Loss 0.6687 Epoch: 19 Global Step: 192485 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:26:11,483-Speed 128.17 samples/sec Loss 0.7096 Epoch: 19 Global Step: 192490 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:26:30,935-Speed 131.61 samples/sec Loss 0.7293 Epoch: 19 Global Step: 192495 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:26:50,722-Speed 129.38 samples/sec Loss 0.6291 Epoch: 19 Global Step: 192500 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:27:10,563-Speed 129.03 samples/sec Loss 0.6015 Epoch: 19 Global Step: 192505 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:27:30,159-Speed 130.65 samples/sec Loss 0.7749 Epoch: 19 Global Step: 192510 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:27:49,961-Speed 129.29 samples/sec Loss 0.6883 Epoch: 19 Global Step: 192515 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:28:09,679-Speed 129.84 samples/sec Loss 0.7122 Epoch: 19 Global Step: 192520 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:28:29,419-Speed 129.69 samples/sec Loss 0.7180 Epoch: 19 Global Step: 192525 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:28:49,175-Speed 129.59 samples/sec Loss 0.8094 Epoch: 19 Global Step: 192530 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:29:09,220-Speed 127.72 samples/sec Loss 0.6486 Epoch: 19 Global Step: 192535 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:29:28,296-Speed 134.20 samples/sec Loss 0.7298 Epoch: 19 Global Step: 192540 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:29:48,193-Speed 128.67 samples/sec Loss 0.6996 Epoch: 19 Global Step: 192545 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:30:08,620-Speed 125.33 samples/sec Loss 0.6826 Epoch: 19 Global Step: 192550 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:30:28,383-Speed 129.54 samples/sec Loss 0.7105 Epoch: 19 Global Step: 192555 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:30:48,499-Speed 127.27 samples/sec Loss 0.6745 Epoch: 19 Global Step: 192560 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:31:08,222-Speed 129.80 samples/sec Loss 0.6364 Epoch: 19 Global Step: 192565 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:31:27,971-Speed 129.64 samples/sec Loss 0.6891 Epoch: 19 Global Step: 192570 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:31:47,981-Speed 127.94 samples/sec Loss 0.6432 Epoch: 19 Global Step: 192575 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:32:07,953-Speed 128.19 samples/sec Loss 0.6285 Epoch: 19 Global Step: 192580 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:32:27,578-Speed 130.45 samples/sec Loss 0.7320 Epoch: 19 Global Step: 192585 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:32:47,520-Speed 128.38 samples/sec Loss 0.6774 Epoch: 19 Global Step: 192590 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:33:07,180-Speed 130.22 samples/sec Loss 0.6622 Epoch: 19 Global Step: 192595 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:33:26,938-Speed 129.58 samples/sec Loss 0.6932 Epoch: 19 Global Step: 192600 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:33:46,905-Speed 128.22 samples/sec Loss 0.6640 Epoch: 19 Global Step: 192605 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:34:06,548-Speed 130.33 samples/sec Loss 0.6341 Epoch: 19 Global Step: 192610 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:34:26,638-Speed 127.43 samples/sec Loss 0.6736 Epoch: 19 Global Step: 192615 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:34:46,585-Speed 128.35 samples/sec Loss 0.7279 Epoch: 19 Global Step: 192620 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:35:06,501-Speed 128.54 samples/sec Loss 0.7438 Epoch: 19 Global Step: 192625 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:35:26,205-Speed 129.93 samples/sec Loss 0.7646 Epoch: 19 Global Step: 192630 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:35:45,714-Speed 131.23 samples/sec Loss 0.6946 Epoch: 19 Global Step: 192635 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:36:05,436-Speed 129.81 samples/sec Loss 0.7830 Epoch: 19 Global Step: 192640 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:36:25,342-Speed 128.61 samples/sec Loss 0.7322 Epoch: 19 Global Step: 192645 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:36:45,128-Speed 129.39 samples/sec Loss 0.6996 Epoch: 19 Global Step: 192650 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:37:04,851-Speed 129.80 samples/sec Loss 0.6858 Epoch: 19 Global Step: 192655 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:37:25,023-Speed 126.92 samples/sec Loss 0.6821 Epoch: 19 Global Step: 192660 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:37:45,095-Speed 127.55 samples/sec Loss 0.6075 Epoch: 19 Global Step: 192665 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:38:04,812-Speed 129.84 samples/sec Loss 0.7260 Epoch: 19 Global Step: 192670 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:38:24,504-Speed 130.01 samples/sec Loss 0.7061 Epoch: 19 Global Step: 192675 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:38:44,380-Speed 128.80 samples/sec Loss 0.7073 Epoch: 19 Global Step: 192680 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:39:04,076-Speed 129.99 samples/sec Loss 0.6962 Epoch: 19 Global Step: 192685 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:39:23,990-Speed 128.56 samples/sec Loss 0.6752 Epoch: 19 Global Step: 192690 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:39:43,825-Speed 129.07 samples/sec Loss 0.6864 Epoch: 19 Global Step: 192695 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:40:03,552-Speed 129.78 samples/sec Loss 0.8159 Epoch: 19 Global Step: 192700 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:40:23,597-Speed 127.72 samples/sec Loss 0.6078 Epoch: 19 Global Step: 192705 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:40:43,817-Speed 126.62 samples/sec Loss 0.6317 Epoch: 19 Global Step: 192710 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:41:03,585-Speed 129.51 samples/sec Loss 0.6583 Epoch: 19 Global Step: 192715 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:41:23,293-Speed 129.90 samples/sec Loss 0.7139 Epoch: 19 Global Step: 192720 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:41:43,198-Speed 128.62 samples/sec Loss 0.6540 Epoch: 19 Global Step: 192725 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:42:02,911-Speed 129.86 samples/sec Loss 0.6051 Epoch: 19 Global Step: 192730 Fp16 Grad Scale: 2048 Required: 68 hours Training: 2021-06-16 10:42:22,992-Speed 127.49 samples/sec Loss 0.7656 Epoch: 19 Global Step: 192735 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:42:42,839-Speed 129.00 samples/sec Loss 0.6334 Epoch: 19 Global Step: 192740 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:43:02,553-Speed 129.86 samples/sec Loss 0.6548 Epoch: 19 Global Step: 192745 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:43:22,757-Speed 126.71 samples/sec Loss 0.6212 Epoch: 19 Global Step: 192750 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:43:42,985-Speed 126.57 samples/sec Loss 0.5487 Epoch: 19 Global Step: 192755 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:44:02,895-Speed 128.58 samples/sec Loss 0.5887 Epoch: 19 Global Step: 192760 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:44:22,646-Speed 129.62 samples/sec Loss 0.6174 Epoch: 19 Global Step: 192765 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:44:42,475-Speed 129.11 samples/sec Loss 0.7379 Epoch: 19 Global Step: 192770 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:45:02,073-Speed 130.64 samples/sec Loss 0.6952 Epoch: 19 Global Step: 192775 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:45:21,958-Speed 128.74 samples/sec Loss 0.6981 Epoch: 19 Global Step: 192780 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:45:41,680-Speed 129.82 samples/sec Loss 0.7200 Epoch: 19 Global Step: 192785 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:46:01,549-Speed 128.85 samples/sec Loss 0.7517 Epoch: 19 Global Step: 192790 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:46:21,442-Speed 128.69 samples/sec Loss 0.6418 Epoch: 19 Global Step: 192795 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:46:41,571-Speed 127.19 samples/sec Loss 0.6399 Epoch: 19 Global Step: 192800 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:47:01,277-Speed 129.92 samples/sec Loss 0.6867 Epoch: 19 Global Step: 192805 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:47:21,017-Speed 129.69 samples/sec Loss 0.5883 Epoch: 19 Global Step: 192810 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:47:40,875-Speed 128.93 samples/sec Loss 0.8454 Epoch: 19 Global Step: 192815 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:48:00,813-Speed 128.40 samples/sec Loss 0.6867 Epoch: 19 Global Step: 192820 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:48:20,706-Speed 128.70 samples/sec Loss 0.6231 Epoch: 19 Global Step: 192825 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:48:40,502-Speed 129.33 samples/sec Loss 0.6452 Epoch: 19 Global Step: 192830 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:49:00,352-Speed 128.97 samples/sec Loss 0.6875 Epoch: 19 Global Step: 192835 Fp16 Grad Scale: 8192 Required: 68 hours Training: 2021-06-16 10:49:19,797-Speed 131.66 samples/sec Loss 0.6429 Epoch: 19 Global Step: 192840 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:49:39,562-Speed 129.52 samples/sec Loss 0.7496 Epoch: 19 Global Step: 192845 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:49:59,488-Speed 128.49 samples/sec Loss 0.6023 Epoch: 19 Global Step: 192850 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:50:19,205-Speed 129.85 samples/sec Loss 0.6368 Epoch: 19 Global Step: 192855 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:50:39,190-Speed 128.10 samples/sec Loss 0.6422 Epoch: 19 Global Step: 192860 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:50:58,690-Speed 131.29 samples/sec Loss 0.7793 Epoch: 19 Global Step: 192865 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:51:18,724-Speed 127.79 samples/sec Loss 0.6161 Epoch: 19 Global Step: 192870 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:51:38,530-Speed 129.26 samples/sec Loss 0.7282 Epoch: 19 Global Step: 192875 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:51:58,164-Speed 130.39 samples/sec Loss 0.6445 Epoch: 19 Global Step: 192880 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:52:17,953-Speed 129.37 samples/sec Loss 0.6765 Epoch: 19 Global Step: 192885 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:52:37,754-Speed 129.29 samples/sec Loss 0.7214 Epoch: 19 Global Step: 192890 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:52:57,426-Speed 130.14 samples/sec Loss 0.6954 Epoch: 19 Global Step: 192895 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:53:17,136-Speed 129.89 samples/sec Loss 0.8667 Epoch: 19 Global Step: 192900 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:53:36,752-Speed 130.52 samples/sec Loss 0.7027 Epoch: 19 Global Step: 192905 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:53:56,475-Speed 129.80 samples/sec Loss 0.6912 Epoch: 19 Global Step: 192910 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:54:16,317-Speed 129.02 samples/sec Loss 0.7085 Epoch: 19 Global Step: 192915 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:54:36,113-Speed 129.33 samples/sec Loss 0.6798 Epoch: 19 Global Step: 192920 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:54:55,742-Speed 130.43 samples/sec Loss 0.6781 Epoch: 19 Global Step: 192925 Fp16 Grad Scale: 4096 Required: 68 hours Training: 2021-06-16 10:55:15,446-Speed 129.93 samples/sec Loss 0.7368 Epoch: 19 Global Step: 192930 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 10:55:35,427-Speed 128.13 samples/sec Loss 0.6601 Epoch: 19 Global Step: 192935 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 10:55:54,923-Speed 131.32 samples/sec Loss 0.6309 Epoch: 19 Global Step: 192940 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:56:14,672-Speed 129.64 samples/sec Loss 0.7166 Epoch: 19 Global Step: 192945 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:56:34,231-Speed 130.89 samples/sec Loss 0.7351 Epoch: 19 Global Step: 192950 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:56:53,866-Speed 130.38 samples/sec Loss 0.8145 Epoch: 19 Global Step: 192955 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:57:13,660-Speed 129.34 samples/sec Loss 0.6025 Epoch: 19 Global Step: 192960 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:57:33,832-Speed 126.91 samples/sec Loss 0.7490 Epoch: 19 Global Step: 192965 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:57:53,259-Speed 131.78 samples/sec Loss 0.6859 Epoch: 19 Global Step: 192970 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:58:13,031-Speed 129.48 samples/sec Loss 0.7351 Epoch: 19 Global Step: 192975 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:58:33,038-Speed 127.96 samples/sec Loss 0.7758 Epoch: 19 Global Step: 192980 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:58:52,603-Speed 130.85 samples/sec Loss 0.6835 Epoch: 19 Global Step: 192985 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:59:12,396-Speed 129.35 samples/sec Loss 0.8070 Epoch: 19 Global Step: 192990 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:59:31,931-Speed 131.05 samples/sec Loss 0.6905 Epoch: 19 Global Step: 192995 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 10:59:51,590-Speed 130.24 samples/sec Loss 0.7089 Epoch: 19 Global Step: 193000 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:00:11,367-Speed 129.45 samples/sec Loss 0.7404 Epoch: 19 Global Step: 193005 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:00:31,301-Speed 128.42 samples/sec Loss 0.5977 Epoch: 19 Global Step: 193010 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:00:50,715-Speed 131.87 samples/sec Loss 0.6777 Epoch: 19 Global Step: 193015 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:01:10,035-Speed 132.51 samples/sec Loss 0.6860 Epoch: 19 Global Step: 193020 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:01:29,840-Speed 129.27 samples/sec Loss 0.6813 Epoch: 19 Global Step: 193025 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:01:49,988-Speed 127.06 samples/sec Loss 0.7364 Epoch: 19 Global Step: 193030 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:02:09,885-Speed 128.67 samples/sec Loss 0.5878 Epoch: 19 Global Step: 193035 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:02:29,578-Speed 130.00 samples/sec Loss 0.6807 Epoch: 19 Global Step: 193040 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:02:49,221-Speed 130.34 samples/sec Loss 0.7785 Epoch: 19 Global Step: 193045 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:03:09,285-Speed 127.59 samples/sec Loss 0.6482 Epoch: 19 Global Step: 193050 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:03:29,189-Speed 128.62 samples/sec Loss 0.6054 Epoch: 19 Global Step: 193055 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:03:48,897-Speed 129.90 samples/sec Loss 0.7401 Epoch: 19 Global Step: 193060 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:04:08,682-Speed 129.40 samples/sec Loss 0.6983 Epoch: 19 Global Step: 193065 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:04:28,502-Speed 129.17 samples/sec Loss 0.7026 Epoch: 19 Global Step: 193070 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:04:48,252-Speed 129.63 samples/sec Loss 0.7124 Epoch: 19 Global Step: 193075 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:05:08,119-Speed 128.86 samples/sec Loss 0.6201 Epoch: 19 Global Step: 193080 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:05:27,830-Speed 129.88 samples/sec Loss 0.7058 Epoch: 19 Global Step: 193085 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:05:47,995-Speed 126.95 samples/sec Loss 0.6817 Epoch: 19 Global Step: 193090 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:06:07,924-Speed 128.47 samples/sec Loss 0.7648 Epoch: 19 Global Step: 193095 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:06:27,790-Speed 128.87 samples/sec Loss 0.6573 Epoch: 19 Global Step: 193100 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:06:47,453-Speed 130.19 samples/sec Loss 0.6744 Epoch: 19 Global Step: 193105 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:07:07,294-Speed 129.04 samples/sec Loss 0.7551 Epoch: 19 Global Step: 193110 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:07:27,079-Speed 129.40 samples/sec Loss 0.6879 Epoch: 19 Global Step: 193115 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:07:46,797-Speed 129.84 samples/sec Loss 0.6909 Epoch: 19 Global Step: 193120 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:08:06,681-Speed 128.76 samples/sec Loss 0.6836 Epoch: 19 Global Step: 193125 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:08:26,349-Speed 130.17 samples/sec Loss 0.6947 Epoch: 19 Global Step: 193130 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:08:46,548-Speed 126.74 samples/sec Loss 0.6733 Epoch: 19 Global Step: 193135 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:09:06,892-Speed 125.84 samples/sec Loss 0.6319 Epoch: 19 Global Step: 193140 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:09:26,884-Speed 128.06 samples/sec Loss 0.6730 Epoch: 19 Global Step: 193145 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:09:46,590-Speed 129.92 samples/sec Loss 0.6621 Epoch: 19 Global Step: 193150 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:10:06,398-Speed 129.25 samples/sec Loss 0.6652 Epoch: 19 Global Step: 193155 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:10:25,766-Speed 132.18 samples/sec Loss 0.7095 Epoch: 19 Global Step: 193160 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:10:45,534-Speed 129.51 samples/sec Loss 0.6917 Epoch: 19 Global Step: 193165 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:11:05,373-Speed 129.04 samples/sec Loss 0.7300 Epoch: 19 Global Step: 193170 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:11:25,075-Speed 129.95 samples/sec Loss 0.7482 Epoch: 19 Global Step: 193175 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:11:45,310-Speed 126.52 samples/sec Loss 0.6529 Epoch: 19 Global Step: 193180 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:12:05,213-Speed 128.63 samples/sec Loss 0.6075 Epoch: 19 Global Step: 193185 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:12:25,089-Speed 128.80 samples/sec Loss 0.6040 Epoch: 19 Global Step: 193190 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:12:44,802-Speed 129.87 samples/sec Loss 0.7686 Epoch: 19 Global Step: 193195 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:13:04,730-Speed 128.47 samples/sec Loss 0.8081 Epoch: 19 Global Step: 193200 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:13:24,641-Speed 128.58 samples/sec Loss 0.7001 Epoch: 19 Global Step: 193205 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:13:44,511-Speed 128.85 samples/sec Loss 0.6676 Epoch: 19 Global Step: 193210 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:14:04,352-Speed 129.03 samples/sec Loss 0.5835 Epoch: 19 Global Step: 193215 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:14:24,089-Speed 129.71 samples/sec Loss 0.6166 Epoch: 19 Global Step: 193220 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:14:44,309-Speed 126.62 samples/sec Loss 0.7290 Epoch: 19 Global Step: 193225 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:15:04,399-Speed 127.43 samples/sec Loss 0.6236 Epoch: 19 Global Step: 193230 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:15:24,130-Speed 129.75 samples/sec Loss 0.6315 Epoch: 19 Global Step: 193235 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:15:43,873-Speed 129.67 samples/sec Loss 0.7275 Epoch: 19 Global Step: 193240 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:16:03,972-Speed 127.38 samples/sec Loss 0.6250 Epoch: 19 Global Step: 193245 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:16:23,741-Speed 129.50 samples/sec Loss 0.7174 Epoch: 19 Global Step: 193250 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:16:43,548-Speed 129.26 samples/sec Loss 0.6920 Epoch: 19 Global Step: 193255 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:17:03,283-Speed 129.72 samples/sec Loss 0.6425 Epoch: 19 Global Step: 193260 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:17:23,035-Speed 129.61 samples/sec Loss 0.6094 Epoch: 19 Global Step: 193265 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:17:43,257-Speed 126.60 samples/sec Loss 0.7251 Epoch: 19 Global Step: 193270 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:18:03,340-Speed 127.48 samples/sec Loss 0.7107 Epoch: 19 Global Step: 193275 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:18:23,020-Speed 130.09 samples/sec Loss 0.5900 Epoch: 19 Global Step: 193280 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:18:42,847-Speed 129.12 samples/sec Loss 0.6612 Epoch: 19 Global Step: 193285 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:19:02,517-Speed 130.16 samples/sec Loss 0.8089 Epoch: 19 Global Step: 193290 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:19:22,304-Speed 129.38 samples/sec Loss 0.7273 Epoch: 19 Global Step: 193295 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:19:42,133-Speed 129.12 samples/sec Loss 0.6758 Epoch: 19 Global Step: 193300 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:20:01,854-Speed 129.81 samples/sec Loss 0.7173 Epoch: 19 Global Step: 193305 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:20:21,743-Speed 128.72 samples/sec Loss 0.7437 Epoch: 19 Global Step: 193310 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:20:41,800-Speed 127.64 samples/sec Loss 0.6379 Epoch: 19 Global Step: 193315 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:21:01,780-Speed 128.13 samples/sec Loss 0.6066 Epoch: 19 Global Step: 193320 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:21:21,448-Speed 130.16 samples/sec Loss 0.7112 Epoch: 19 Global Step: 193325 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:21:41,355-Speed 128.60 samples/sec Loss 0.6705 Epoch: 19 Global Step: 193330 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:22:00,984-Speed 130.43 samples/sec Loss 0.6360 Epoch: 19 Global Step: 193335 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:22:20,799-Speed 129.20 samples/sec Loss 0.7768 Epoch: 19 Global Step: 193340 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:22:40,545-Speed 129.66 samples/sec Loss 0.5919 Epoch: 19 Global Step: 193345 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:23:00,535-Speed 128.07 samples/sec Loss 0.6723 Epoch: 19 Global Step: 193350 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:23:20,355-Speed 129.17 samples/sec Loss 0.6815 Epoch: 19 Global Step: 193355 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:23:40,412-Speed 127.64 samples/sec Loss 0.7595 Epoch: 19 Global Step: 193360 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:24:00,402-Speed 128.07 samples/sec Loss 0.7408 Epoch: 19 Global Step: 193365 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:24:20,101-Speed 129.96 samples/sec Loss 0.7021 Epoch: 19 Global Step: 193370 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:24:39,998-Speed 128.67 samples/sec Loss 0.6213 Epoch: 19 Global Step: 193375 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:24:59,825-Speed 129.12 samples/sec Loss 0.6769 Epoch: 19 Global Step: 193380 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:25:19,641-Speed 129.20 samples/sec Loss 0.6307 Epoch: 19 Global Step: 193385 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:25:39,390-Speed 129.64 samples/sec Loss 0.6954 Epoch: 19 Global Step: 193390 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:25:59,166-Speed 129.45 samples/sec Loss 0.6832 Epoch: 19 Global Step: 193395 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:26:19,014-Speed 128.99 samples/sec Loss 0.6774 Epoch: 19 Global Step: 193400 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:26:39,043-Speed 127.82 samples/sec Loss 0.6297 Epoch: 19 Global Step: 193405 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:26:59,021-Speed 128.15 samples/sec Loss 0.7448 Epoch: 19 Global Step: 193410 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:27:18,775-Speed 129.60 samples/sec Loss 0.7005 Epoch: 19 Global Step: 193415 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:27:38,573-Speed 129.31 samples/sec Loss 0.7081 Epoch: 19 Global Step: 193420 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:27:58,253-Speed 130.08 samples/sec Loss 0.7290 Epoch: 19 Global Step: 193425 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:28:18,228-Speed 128.17 samples/sec Loss 0.7060 Epoch: 19 Global Step: 193430 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:28:37,831-Speed 130.60 samples/sec Loss 0.5543 Epoch: 19 Global Step: 193435 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:28:57,584-Speed 129.60 samples/sec Loss 0.6468 Epoch: 19 Global Step: 193440 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:29:17,393-Speed 129.25 samples/sec Loss 0.5804 Epoch: 19 Global Step: 193445 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:29:37,452-Speed 127.62 samples/sec Loss 0.8289 Epoch: 19 Global Step: 193450 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:29:57,519-Speed 127.58 samples/sec Loss 0.6983 Epoch: 19 Global Step: 193455 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:30:17,241-Speed 129.81 samples/sec Loss 0.6934 Epoch: 19 Global Step: 193460 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:30:37,045-Speed 129.27 samples/sec Loss 0.8227 Epoch: 19 Global Step: 193465 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:30:56,726-Speed 130.08 samples/sec Loss 0.7427 Epoch: 19 Global Step: 193470 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:31:16,580-Speed 128.95 samples/sec Loss 0.8182 Epoch: 19 Global Step: 193475 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:31:36,428-Speed 128.99 samples/sec Loss 0.6766 Epoch: 19 Global Step: 193480 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:31:56,169-Speed 129.68 samples/sec Loss 0.6663 Epoch: 19 Global Step: 193485 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:32:15,969-Speed 129.30 samples/sec Loss 0.7309 Epoch: 19 Global Step: 193490 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:32:35,902-Speed 128.43 samples/sec Loss 0.7601 Epoch: 19 Global Step: 193495 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:32:56,001-Speed 127.38 samples/sec Loss 0.6710 Epoch: 19 Global Step: 193500 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:33:15,858-Speed 128.93 samples/sec Loss 0.6122 Epoch: 19 Global Step: 193505 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:33:35,630-Speed 129.48 samples/sec Loss 0.6968 Epoch: 19 Global Step: 193510 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:33:55,322-Speed 130.01 samples/sec Loss 0.6599 Epoch: 19 Global Step: 193515 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:34:15,283-Speed 128.25 samples/sec Loss 0.5736 Epoch: 19 Global Step: 193520 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:34:35,053-Speed 129.50 samples/sec Loss 0.8365 Epoch: 19 Global Step: 193525 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:34:54,833-Speed 129.43 samples/sec Loss 0.6766 Epoch: 19 Global Step: 193530 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:35:14,595-Speed 129.55 samples/sec Loss 0.6608 Epoch: 19 Global Step: 193535 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:35:34,678-Speed 127.48 samples/sec Loss 0.6303 Epoch: 19 Global Step: 193540 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:35:54,697-Speed 127.88 samples/sec Loss 0.7181 Epoch: 19 Global Step: 193545 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:36:14,460-Speed 129.54 samples/sec Loss 0.6608 Epoch: 19 Global Step: 193550 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:36:34,271-Speed 129.23 samples/sec Loss 0.7059 Epoch: 19 Global Step: 193555 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:36:53,998-Speed 129.77 samples/sec Loss 0.7502 Epoch: 19 Global Step: 193560 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:37:13,993-Speed 128.04 samples/sec Loss 0.6866 Epoch: 19 Global Step: 193565 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:37:33,689-Speed 129.98 samples/sec Loss 0.6594 Epoch: 19 Global Step: 193570 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:37:53,849-Speed 126.99 samples/sec Loss 0.7397 Epoch: 19 Global Step: 193575 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:38:13,487-Speed 130.36 samples/sec Loss 0.6943 Epoch: 19 Global Step: 193580 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:38:33,724-Speed 126.51 samples/sec Loss 0.6755 Epoch: 19 Global Step: 193585 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:38:53,767-Speed 127.73 samples/sec Loss 0.7304 Epoch: 19 Global Step: 193590 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:39:13,694-Speed 128.47 samples/sec Loss 0.7199 Epoch: 19 Global Step: 193595 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:39:33,411-Speed 129.84 samples/sec Loss 0.6712 Epoch: 19 Global Step: 193600 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:39:53,311-Speed 128.65 samples/sec Loss 0.7059 Epoch: 19 Global Step: 193605 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:40:13,172-Speed 128.90 samples/sec Loss 0.6646 Epoch: 19 Global Step: 193610 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:40:32,932-Speed 129.56 samples/sec Loss 0.6751 Epoch: 19 Global Step: 193615 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:40:52,822-Speed 128.72 samples/sec Loss 0.6217 Epoch: 19 Global Step: 193620 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:41:12,534-Speed 129.88 samples/sec Loss 0.6734 Epoch: 19 Global Step: 193625 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:41:32,774-Speed 126.48 samples/sec Loss 0.6296 Epoch: 19 Global Step: 193630 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:41:52,689-Speed 128.56 samples/sec Loss 0.5839 Epoch: 19 Global Step: 193635 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:42:12,565-Speed 128.80 samples/sec Loss 0.6848 Epoch: 19 Global Step: 193640 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:42:32,333-Speed 129.50 samples/sec Loss 0.7534 Epoch: 19 Global Step: 193645 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:42:52,233-Speed 128.65 samples/sec Loss 0.6168 Epoch: 19 Global Step: 193650 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:43:11,999-Speed 129.53 samples/sec Loss 0.6302 Epoch: 19 Global Step: 193655 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:43:31,706-Speed 129.91 samples/sec Loss 0.5984 Epoch: 19 Global Step: 193660 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:43:51,554-Speed 128.99 samples/sec Loss 0.6222 Epoch: 19 Global Step: 193665 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:44:11,252-Speed 129.97 samples/sec Loss 0.7682 Epoch: 19 Global Step: 193670 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:44:31,431-Speed 126.86 samples/sec Loss 0.7918 Epoch: 19 Global Step: 193675 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:44:51,276-Speed 129.01 samples/sec Loss 0.7435 Epoch: 19 Global Step: 193680 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:45:11,155-Speed 128.78 samples/sec Loss 0.6982 Epoch: 19 Global Step: 193685 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:45:30,874-Speed 129.83 samples/sec Loss 0.7447 Epoch: 19 Global Step: 193690 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:45:50,719-Speed 129.01 samples/sec Loss 0.6387 Epoch: 19 Global Step: 193695 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:46:10,495-Speed 129.46 samples/sec Loss 0.6321 Epoch: 19 Global Step: 193700 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:46:30,321-Speed 129.13 samples/sec Loss 0.6752 Epoch: 19 Global Step: 193705 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:46:50,051-Speed 129.76 samples/sec Loss 0.8041 Epoch: 19 Global Step: 193710 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:47:09,794-Speed 129.68 samples/sec Loss 0.6807 Epoch: 19 Global Step: 193715 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:47:29,954-Speed 126.99 samples/sec Loss 0.8243 Epoch: 19 Global Step: 193720 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:47:50,039-Speed 127.47 samples/sec Loss 0.7418 Epoch: 19 Global Step: 193725 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:48:09,743-Speed 129.93 samples/sec Loss 0.6251 Epoch: 19 Global Step: 193730 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:48:29,537-Speed 129.34 samples/sec Loss 0.6536 Epoch: 19 Global Step: 193735 Fp16 Grad Scale: 8192 Required: 67 hours Training: 2021-06-16 11:48:48,852-Speed 132.54 samples/sec Loss 0.7127 Epoch: 19 Global Step: 193740 Fp16 Grad Scale: 4096 Required: 67 hours Training: 2021-06-16 11:49:08,619-Speed 129.52 samples/sec Loss 0.7889 Epoch: 19 Global Step: 193745 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:49:28,475-Speed 128.94 samples/sec Loss 0.7359 Epoch: 19 Global Step: 193750 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:49:48,236-Speed 129.55 samples/sec Loss 0.6466 Epoch: 19 Global Step: 193755 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:50:08,077-Speed 129.04 samples/sec Loss 0.7014 Epoch: 19 Global Step: 193760 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:50:28,133-Speed 127.65 samples/sec Loss 0.7676 Epoch: 19 Global Step: 193765 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:50:47,868-Speed 129.73 samples/sec Loss 0.7575 Epoch: 19 Global Step: 193770 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:51:07,838-Speed 128.20 samples/sec Loss 0.7116 Epoch: 19 Global Step: 193775 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:51:27,898-Speed 127.62 samples/sec Loss 0.6600 Epoch: 19 Global Step: 193780 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:51:47,527-Speed 130.43 samples/sec Loss 0.5798 Epoch: 19 Global Step: 193785 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:52:07,355-Speed 129.12 samples/sec Loss 0.6571 Epoch: 19 Global Step: 193790 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:52:27,224-Speed 128.85 samples/sec Loss 0.6304 Epoch: 19 Global Step: 193795 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:52:46,971-Speed 129.64 samples/sec Loss 0.6467 Epoch: 19 Global Step: 193800 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:53:06,924-Speed 128.31 samples/sec Loss 0.6721 Epoch: 19 Global Step: 193805 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:53:26,880-Speed 128.29 samples/sec Loss 0.7325 Epoch: 19 Global Step: 193810 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:53:46,762-Speed 128.77 samples/sec Loss 0.6781 Epoch: 19 Global Step: 193815 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:54:06,635-Speed 128.82 samples/sec Loss 0.7219 Epoch: 19 Global Step: 193820 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:54:26,632-Speed 128.03 samples/sec Loss 0.6810 Epoch: 19 Global Step: 193825 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:54:46,424-Speed 129.36 samples/sec Loss 0.6535 Epoch: 19 Global Step: 193830 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:55:06,405-Speed 128.12 samples/sec Loss 0.6841 Epoch: 19 Global Step: 193835 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 11:55:26,124-Speed 129.83 samples/sec Loss 0.7232 Epoch: 19 Global Step: 193840 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:55:45,970-Speed 129.00 samples/sec Loss 0.6662 Epoch: 19 Global Step: 193845 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:56:05,632-Speed 130.20 samples/sec Loss 0.7502 Epoch: 19 Global Step: 193850 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:56:25,627-Speed 128.04 samples/sec Loss 0.8157 Epoch: 19 Global Step: 193855 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:56:45,472-Speed 129.01 samples/sec Loss 0.6525 Epoch: 19 Global Step: 193860 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:57:05,190-Speed 129.83 samples/sec Loss 0.7420 Epoch: 19 Global Step: 193865 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:57:25,214-Speed 127.86 samples/sec Loss 0.7013 Epoch: 19 Global Step: 193870 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:57:44,910-Speed 129.98 samples/sec Loss 0.6519 Epoch: 19 Global Step: 193875 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:58:04,787-Speed 128.80 samples/sec Loss 0.6965 Epoch: 19 Global Step: 193880 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:58:24,525-Speed 129.70 samples/sec Loss 0.6824 Epoch: 19 Global Step: 193885 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:58:44,431-Speed 128.61 samples/sec Loss 0.6502 Epoch: 19 Global Step: 193890 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:59:04,107-Speed 130.11 samples/sec Loss 0.7682 Epoch: 19 Global Step: 193895 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:59:24,233-Speed 127.20 samples/sec Loss 0.6594 Epoch: 19 Global Step: 193900 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 11:59:43,950-Speed 129.85 samples/sec Loss 0.7236 Epoch: 19 Global Step: 193905 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:00:03,676-Speed 129.79 samples/sec Loss 0.6599 Epoch: 19 Global Step: 193910 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:00:23,661-Speed 128.10 samples/sec Loss 0.6801 Epoch: 19 Global Step: 193915 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:00:43,454-Speed 129.34 samples/sec Loss 0.6993 Epoch: 19 Global Step: 193920 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:01:03,252-Speed 129.31 samples/sec Loss 0.7014 Epoch: 19 Global Step: 193925 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:01:22,940-Speed 130.04 samples/sec Loss 0.8063 Epoch: 19 Global Step: 193930 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:01:42,798-Speed 128.92 samples/sec Loss 0.8591 Epoch: 19 Global Step: 193935 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:02:02,421-Speed 130.47 samples/sec Loss 0.6637 Epoch: 19 Global Step: 193940 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:02:21,861-Speed 131.69 samples/sec Loss 0.6868 Epoch: 19 Global Step: 193945 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:02:41,573-Speed 129.87 samples/sec Loss 0.7366 Epoch: 19 Global Step: 193950 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:03:01,708-Speed 127.15 samples/sec Loss 0.5672 Epoch: 19 Global Step: 193955 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:03:21,684-Speed 128.16 samples/sec Loss 0.6214 Epoch: 19 Global Step: 193960 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:03:41,580-Speed 128.67 samples/sec Loss 0.6960 Epoch: 19 Global Step: 193965 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:04:01,305-Speed 129.79 samples/sec Loss 0.6592 Epoch: 19 Global Step: 193970 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:04:21,010-Speed 129.93 samples/sec Loss 0.6968 Epoch: 19 Global Step: 193975 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:04:40,586-Speed 130.78 samples/sec Loss 0.7051 Epoch: 19 Global Step: 193980 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:05:00,027-Speed 131.69 samples/sec Loss 0.6502 Epoch: 19 Global Step: 193985 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:05:19,639-Speed 130.54 samples/sec Loss 0.7223 Epoch: 19 Global Step: 193990 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:05:39,164-Speed 131.12 samples/sec Loss 0.7570 Epoch: 19 Global Step: 193995 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:05:58,637-Speed 131.48 samples/sec Loss 0.5899 Epoch: 19 Global Step: 194000 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:06:18,409-Speed 129.48 samples/sec Loss 0.6853 Epoch: 19 Global Step: 194005 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:06:38,065-Speed 130.25 samples/sec Loss 0.6418 Epoch: 19 Global Step: 194010 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:06:57,735-Speed 130.15 samples/sec Loss 0.7684 Epoch: 19 Global Step: 194015 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:07:17,364-Speed 130.43 samples/sec Loss 0.7101 Epoch: 19 Global Step: 194020 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:07:36,921-Speed 130.91 samples/sec Loss 0.6595 Epoch: 19 Global Step: 194025 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:07:56,393-Speed 131.48 samples/sec Loss 0.6656 Epoch: 19 Global Step: 194030 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:08:15,973-Speed 130.76 samples/sec Loss 0.7088 Epoch: 19 Global Step: 194035 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:08:35,557-Speed 130.73 samples/sec Loss 0.7492 Epoch: 19 Global Step: 194040 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:08:55,109-Speed 130.94 samples/sec Loss 0.8022 Epoch: 19 Global Step: 194045 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:09:14,837-Speed 129.77 samples/sec Loss 0.6951 Epoch: 19 Global Step: 194050 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:09:34,471-Speed 130.40 samples/sec Loss 0.7593 Epoch: 19 Global Step: 194055 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:09:54,002-Speed 131.08 samples/sec Loss 0.6726 Epoch: 19 Global Step: 194060 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:10:13,625-Speed 130.47 samples/sec Loss 0.6542 Epoch: 19 Global Step: 194065 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:10:33,125-Speed 131.29 samples/sec Loss 0.6812 Epoch: 19 Global Step: 194070 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:10:52,731-Speed 130.58 samples/sec Loss 0.6458 Epoch: 19 Global Step: 194075 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:11:12,245-Speed 131.20 samples/sec Loss 0.6996 Epoch: 19 Global Step: 194080 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:11:31,801-Speed 130.91 samples/sec Loss 0.6840 Epoch: 19 Global Step: 194085 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:11:51,552-Speed 129.63 samples/sec Loss 0.7090 Epoch: 19 Global Step: 194090 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:12:11,335-Speed 129.41 samples/sec Loss 0.7085 Epoch: 19 Global Step: 194095 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:12:30,956-Speed 130.48 samples/sec Loss 0.7151 Epoch: 19 Global Step: 194100 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:12:50,530-Speed 130.79 samples/sec Loss 0.7189 Epoch: 19 Global Step: 194105 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:13:09,703-Speed 133.53 samples/sec Loss 0.6676 Epoch: 19 Global Step: 194110 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:13:29,240-Speed 131.04 samples/sec Loss 0.7396 Epoch: 19 Global Step: 194115 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:13:48,793-Speed 130.94 samples/sec Loss 0.6629 Epoch: 19 Global Step: 194120 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:14:08,328-Speed 131.05 samples/sec Loss 0.6875 Epoch: 19 Global Step: 194125 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:14:27,944-Speed 130.51 samples/sec Loss 0.7326 Epoch: 19 Global Step: 194130 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:14:47,557-Speed 130.53 samples/sec Loss 0.7309 Epoch: 19 Global Step: 194135 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:15:07,433-Speed 128.81 samples/sec Loss 0.7675 Epoch: 19 Global Step: 194140 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:15:27,109-Speed 130.12 samples/sec Loss 0.6914 Epoch: 19 Global Step: 194145 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:15:46,440-Speed 132.43 samples/sec Loss 0.7339 Epoch: 19 Global Step: 194150 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:16:06,303-Speed 128.89 samples/sec Loss 0.6826 Epoch: 19 Global Step: 194155 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:16:26,033-Speed 129.76 samples/sec Loss 0.6242 Epoch: 19 Global Step: 194160 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:16:45,934-Speed 128.64 samples/sec Loss 0.6966 Epoch: 19 Global Step: 194165 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:17:05,518-Speed 130.72 samples/sec Loss 0.6905 Epoch: 19 Global Step: 194170 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:17:25,477-Speed 128.28 samples/sec Loss 0.6974 Epoch: 19 Global Step: 194175 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:17:45,163-Speed 130.04 samples/sec Loss 0.6608 Epoch: 19 Global Step: 194180 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:18:04,968-Speed 129.27 samples/sec Loss 0.7059 Epoch: 19 Global Step: 194185 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:18:24,861-Speed 128.69 samples/sec Loss 0.7449 Epoch: 19 Global Step: 194190 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:18:44,753-Speed 128.70 samples/sec Loss 0.6531 Epoch: 19 Global Step: 194195 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:19:04,683-Speed 128.46 samples/sec Loss 0.7173 Epoch: 19 Global Step: 194200 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:19:24,413-Speed 129.75 samples/sec Loss 0.7403 Epoch: 19 Global Step: 194205 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:19:44,500-Speed 127.45 samples/sec Loss 0.7548 Epoch: 19 Global Step: 194210 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:20:04,153-Speed 130.27 samples/sec Loss 0.6799 Epoch: 19 Global Step: 194215 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:20:23,989-Speed 129.06 samples/sec Loss 0.7801 Epoch: 19 Global Step: 194220 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:20:43,772-Speed 129.41 samples/sec Loss 0.6589 Epoch: 19 Global Step: 194225 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:21:03,881-Speed 127.31 samples/sec Loss 0.5940 Epoch: 19 Global Step: 194230 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:21:23,586-Speed 129.92 samples/sec Loss 0.7141 Epoch: 19 Global Step: 194235 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:21:43,628-Speed 127.74 samples/sec Loss 0.6834 Epoch: 19 Global Step: 194240 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:22:03,566-Speed 128.40 samples/sec Loss 0.7940 Epoch: 19 Global Step: 194245 Fp16 Grad Scale: 2048 Required: 66 hours Training: 2021-06-16 12:22:23,186-Speed 130.48 samples/sec Loss 0.7949 Epoch: 19 Global Step: 194250 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:22:43,277-Speed 127.43 samples/sec Loss 0.6615 Epoch: 19 Global Step: 194255 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:23:03,068-Speed 129.36 samples/sec Loss 0.6658 Epoch: 19 Global Step: 194260 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:23:22,790-Speed 129.81 samples/sec Loss 0.7315 Epoch: 19 Global Step: 194265 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:23:42,588-Speed 129.32 samples/sec Loss 0.7140 Epoch: 19 Global Step: 194270 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:24:02,413-Speed 129.14 samples/sec Loss 0.5860 Epoch: 19 Global Step: 194275 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:24:22,101-Speed 130.04 samples/sec Loss 0.6866 Epoch: 19 Global Step: 194280 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:24:42,268-Speed 126.94 samples/sec Loss 0.6638 Epoch: 19 Global Step: 194285 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:25:02,074-Speed 129.26 samples/sec Loss 0.7431 Epoch: 19 Global Step: 194290 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:25:21,750-Speed 130.12 samples/sec Loss 0.8044 Epoch: 19 Global Step: 194295 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:25:41,699-Speed 128.34 samples/sec Loss 0.6493 Epoch: 19 Global Step: 194300 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:26:01,596-Speed 128.66 samples/sec Loss 0.6568 Epoch: 19 Global Step: 194305 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:26:21,313-Speed 129.84 samples/sec Loss 0.6953 Epoch: 19 Global Step: 194310 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:26:41,146-Speed 129.09 samples/sec Loss 0.7254 Epoch: 19 Global Step: 194315 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:27:01,007-Speed 128.90 samples/sec Loss 0.7165 Epoch: 19 Global Step: 194320 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:27:20,607-Speed 130.62 samples/sec Loss 0.7416 Epoch: 19 Global Step: 194325 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:27:40,797-Speed 126.80 samples/sec Loss 0.6772 Epoch: 19 Global Step: 194330 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:28:00,588-Speed 129.36 samples/sec Loss 0.7107 Epoch: 19 Global Step: 194335 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:28:20,473-Speed 128.75 samples/sec Loss 0.7527 Epoch: 19 Global Step: 194340 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:28:40,309-Speed 129.06 samples/sec Loss 0.6407 Epoch: 19 Global Step: 194345 Fp16 Grad Scale: 4096 Required: 66 hours Training: 2021-06-16 12:29:00,161-Speed 128.96 samples/sec Loss 0.6725 Epoch: 19 Global Step: 194350 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:29:19,874-Speed 129.87 samples/sec Loss 0.7439 Epoch: 19 Global Step: 194355 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:29:39,829-Speed 128.29 samples/sec Loss 0.6868 Epoch: 19 Global Step: 194360 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:29:59,546-Speed 129.85 samples/sec Loss 0.6791 Epoch: 19 Global Step: 194365 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:30:19,337-Speed 129.36 samples/sec Loss 0.7530 Epoch: 19 Global Step: 194370 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:30:39,791-Speed 125.16 samples/sec Loss 0.6161 Epoch: 19 Global Step: 194375 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:30:59,610-Speed 129.18 samples/sec Loss 0.6888 Epoch: 19 Global Step: 194380 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:31:19,463-Speed 128.95 samples/sec Loss 0.7881 Epoch: 19 Global Step: 194385 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:31:39,542-Speed 127.51 samples/sec Loss 0.7097 Epoch: 19 Global Step: 194390 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:31:59,155-Speed 130.53 samples/sec Loss 0.6796 Epoch: 19 Global Step: 194395 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:32:19,065-Speed 128.59 samples/sec Loss 0.7348 Epoch: 19 Global Step: 194400 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:32:38,852-Speed 129.38 samples/sec Loss 0.6223 Epoch: 19 Global Step: 194405 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:32:58,614-Speed 129.55 samples/sec Loss 0.6914 Epoch: 19 Global Step: 194410 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:33:18,463-Speed 128.98 samples/sec Loss 0.6739 Epoch: 19 Global Step: 194415 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:33:38,389-Speed 128.48 samples/sec Loss 0.6663 Epoch: 19 Global Step: 194420 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:33:58,374-Speed 128.10 samples/sec Loss 0.6656 Epoch: 19 Global Step: 194425 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:34:18,084-Speed 129.89 samples/sec Loss 0.9002 Epoch: 19 Global Step: 194430 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:34:38,149-Speed 127.59 samples/sec Loss 0.7529 Epoch: 19 Global Step: 194435 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:34:57,807-Speed 130.23 samples/sec Loss 0.7021 Epoch: 19 Global Step: 194440 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:35:17,697-Speed 128.71 samples/sec Loss 0.6869 Epoch: 19 Global Step: 194445 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:35:37,628-Speed 128.45 samples/sec Loss 0.6837 Epoch: 19 Global Step: 194450 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:35:57,364-Speed 129.72 samples/sec Loss 0.7135 Epoch: 19 Global Step: 194455 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:36:17,161-Speed 129.32 samples/sec Loss 0.7051 Epoch: 19 Global Step: 194460 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:36:37,191-Speed 127.81 samples/sec Loss 0.7387 Epoch: 19 Global Step: 194465 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:36:57,103-Speed 128.57 samples/sec Loss 0.6474 Epoch: 19 Global Step: 194470 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:37:17,064-Speed 128.26 samples/sec Loss 0.7434 Epoch: 19 Global Step: 194475 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:37:37,042-Speed 128.15 samples/sec Loss 0.7018 Epoch: 19 Global Step: 194480 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:37:56,688-Speed 130.31 samples/sec Loss 0.6574 Epoch: 19 Global Step: 194485 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:38:16,645-Speed 128.28 samples/sec Loss 0.6521 Epoch: 19 Global Step: 194490 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:38:36,384-Speed 129.70 samples/sec Loss 0.6726 Epoch: 19 Global Step: 194495 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:38:56,195-Speed 129.23 samples/sec Loss 0.6768 Epoch: 19 Global Step: 194500 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:39:15,899-Speed 129.93 samples/sec Loss 0.7012 Epoch: 19 Global Step: 194505 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:39:35,915-Speed 127.90 samples/sec Loss 0.7423 Epoch: 19 Global Step: 194510 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:39:55,826-Speed 128.58 samples/sec Loss 0.7710 Epoch: 19 Global Step: 194515 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:40:15,504-Speed 130.11 samples/sec Loss 0.7689 Epoch: 19 Global Step: 194520 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:40:35,473-Speed 128.20 samples/sec Loss 0.6812 Epoch: 19 Global Step: 194525 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:40:55,302-Speed 129.10 samples/sec Loss 0.7475 Epoch: 19 Global Step: 194530 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:41:15,288-Speed 128.10 samples/sec Loss 0.7083 Epoch: 19 Global Step: 194535 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:41:34,971-Speed 130.06 samples/sec Loss 0.7597 Epoch: 19 Global Step: 194540 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:41:54,694-Speed 129.80 samples/sec Loss 0.7327 Epoch: 19 Global Step: 194545 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:42:14,392-Speed 129.97 samples/sec Loss 0.6712 Epoch: 19 Global Step: 194550 Fp16 Grad Scale: 8192 Required: 66 hours Training: 2021-06-16 12:42:34,398-Speed 127.97 samples/sec Loss 0.6629 Epoch: 19 Global Step: 194555 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:42:53,842-Speed 131.67 samples/sec Loss 0.6410 Epoch: 19 Global Step: 194560 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:43:13,555-Speed 129.87 samples/sec Loss 0.6810 Epoch: 19 Global Step: 194565 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:43:33,609-Speed 127.66 samples/sec Loss 0.7250 Epoch: 19 Global Step: 194570 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:43:53,390-Speed 129.42 samples/sec Loss 0.7346 Epoch: 19 Global Step: 194575 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:44:13,299-Speed 128.59 samples/sec Loss 0.7847 Epoch: 19 Global Step: 194580 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:44:33,030-Speed 129.75 samples/sec Loss 0.7194 Epoch: 19 Global Step: 194585 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:44:52,820-Speed 129.36 samples/sec Loss 0.5489 Epoch: 19 Global Step: 194590 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:45:12,562-Speed 129.68 samples/sec Loss 0.6370 Epoch: 19 Global Step: 194595 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:45:32,333-Speed 129.49 samples/sec Loss 0.6042 Epoch: 19 Global Step: 194600 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:45:52,528-Speed 126.77 samples/sec Loss 0.6799 Epoch: 19 Global Step: 194605 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:46:12,499-Speed 128.19 samples/sec Loss 0.6656 Epoch: 19 Global Step: 194610 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:46:32,484-Speed 128.10 samples/sec Loss 0.6354 Epoch: 19 Global Step: 194615 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:46:52,247-Speed 129.54 samples/sec Loss 0.6745 Epoch: 19 Global Step: 194620 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:47:12,032-Speed 129.39 samples/sec Loss 0.6432 Epoch: 19 Global Step: 194625 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:47:31,817-Speed 129.40 samples/sec Loss 0.6533 Epoch: 19 Global Step: 194630 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:47:51,630-Speed 129.22 samples/sec Loss 0.7012 Epoch: 19 Global Step: 194635 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:48:11,282-Speed 130.27 samples/sec Loss 0.7799 Epoch: 19 Global Step: 194640 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:48:31,132-Speed 128.98 samples/sec Loss 0.7646 Epoch: 19 Global Step: 194645 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:48:50,914-Speed 129.42 samples/sec Loss 0.6368 Epoch: 19 Global Step: 194650 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:49:10,894-Speed 128.13 samples/sec Loss 0.6748 Epoch: 19 Global Step: 194655 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:49:30,902-Speed 127.96 samples/sec Loss 0.8149 Epoch: 19 Global Step: 194660 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:49:50,686-Speed 129.41 samples/sec Loss 0.7288 Epoch: 19 Global Step: 194665 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:50:10,438-Speed 129.61 samples/sec Loss 0.6161 Epoch: 19 Global Step: 194670 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:50:30,262-Speed 129.15 samples/sec Loss 0.7073 Epoch: 19 Global Step: 194675 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:50:50,024-Speed 129.55 samples/sec Loss 0.7029 Epoch: 19 Global Step: 194680 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:51:09,658-Speed 130.39 samples/sec Loss 0.7371 Epoch: 19 Global Step: 194685 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:51:29,546-Speed 128.73 samples/sec Loss 0.6818 Epoch: 19 Global Step: 194690 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:51:49,258-Speed 129.88 samples/sec Loss 0.7616 Epoch: 19 Global Step: 194695 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:52:09,282-Speed 127.85 samples/sec Loss 0.6594 Epoch: 19 Global Step: 194700 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:52:29,380-Speed 127.39 samples/sec Loss 0.7564 Epoch: 19 Global Step: 194705 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:52:49,274-Speed 128.69 samples/sec Loss 0.7284 Epoch: 19 Global Step: 194710 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:53:08,990-Speed 129.85 samples/sec Loss 0.7597 Epoch: 19 Global Step: 194715 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:53:28,829-Speed 129.04 samples/sec Loss 0.8263 Epoch: 19 Global Step: 194720 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:53:48,601-Speed 129.48 samples/sec Loss 0.6141 Epoch: 19 Global Step: 194725 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:54:08,280-Speed 130.10 samples/sec Loss 0.7186 Epoch: 19 Global Step: 194730 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:54:28,052-Speed 129.48 samples/sec Loss 0.6975 Epoch: 19 Global Step: 194735 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:54:47,900-Speed 128.98 samples/sec Loss 0.7597 Epoch: 19 Global Step: 194740 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:55:08,082-Speed 126.86 samples/sec Loss 0.6804 Epoch: 19 Global Step: 194745 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:55:28,158-Speed 127.52 samples/sec Loss 0.6889 Epoch: 19 Global Step: 194750 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:55:48,312-Speed 127.03 samples/sec Loss 0.7688 Epoch: 19 Global Step: 194755 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:56:08,285-Speed 128.17 samples/sec Loss 0.6110 Epoch: 19 Global Step: 194760 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:56:28,034-Speed 129.63 samples/sec Loss 0.6933 Epoch: 19 Global Step: 194765 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:56:47,971-Speed 128.42 samples/sec Loss 0.7363 Epoch: 19 Global Step: 194770 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:57:07,733-Speed 129.55 samples/sec Loss 0.6834 Epoch: 19 Global Step: 194775 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:57:27,638-Speed 128.62 samples/sec Loss 0.6104 Epoch: 19 Global Step: 194780 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:57:47,259-Speed 130.48 samples/sec Loss 0.7089 Epoch: 19 Global Step: 194785 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:58:07,405-Speed 127.08 samples/sec Loss 0.6329 Epoch: 19 Global Step: 194790 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:58:27,371-Speed 128.23 samples/sec Loss 0.7036 Epoch: 19 Global Step: 194795 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:58:47,230-Speed 128.91 samples/sec Loss 0.6659 Epoch: 19 Global Step: 194800 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 12:59:06,453-Speed 133.18 samples/sec Loss 0.6149 Epoch: 19 Global Step: 194805 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:59:26,355-Speed 128.64 samples/sec Loss 0.5984 Epoch: 19 Global Step: 194810 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 12:59:46,138-Speed 129.41 samples/sec Loss 0.7955 Epoch: 19 Global Step: 194815 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:00:05,819-Speed 130.08 samples/sec Loss 0.6730 Epoch: 19 Global Step: 194820 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:00:25,700-Speed 128.78 samples/sec Loss 0.6890 Epoch: 19 Global Step: 194825 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:00:45,319-Speed 130.49 samples/sec Loss 0.6362 Epoch: 19 Global Step: 194830 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:01:05,943-Speed 124.13 samples/sec Loss 0.6591 Epoch: 19 Global Step: 194835 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:01:25,916-Speed 128.18 samples/sec Loss 0.8361 Epoch: 19 Global Step: 194840 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:01:45,820-Speed 128.62 samples/sec Loss 0.7327 Epoch: 19 Global Step: 194845 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:02:05,460-Speed 130.35 samples/sec Loss 0.6665 Epoch: 19 Global Step: 194850 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:02:25,329-Speed 128.86 samples/sec Loss 0.7450 Epoch: 19 Global Step: 194855 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:02:45,107-Speed 129.44 samples/sec Loss 0.6764 Epoch: 19 Global Step: 194860 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:03:04,941-Speed 129.08 samples/sec Loss 0.7458 Epoch: 19 Global Step: 194865 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:03:24,745-Speed 129.27 samples/sec Loss 0.6619 Epoch: 19 Global Step: 194870 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:03:44,566-Speed 129.16 samples/sec Loss 0.7456 Epoch: 19 Global Step: 194875 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:04:04,749-Speed 126.85 samples/sec Loss 0.7483 Epoch: 19 Global Step: 194880 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:04:24,797-Speed 127.70 samples/sec Loss 0.6993 Epoch: 19 Global Step: 194885 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:04:44,445-Speed 130.30 samples/sec Loss 0.7971 Epoch: 19 Global Step: 194890 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:05:04,175-Speed 129.75 samples/sec Loss 0.7130 Epoch: 19 Global Step: 194895 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:05:24,058-Speed 128.77 samples/sec Loss 0.7703 Epoch: 19 Global Step: 194900 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:05:43,834-Speed 129.45 samples/sec Loss 0.7103 Epoch: 19 Global Step: 194905 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:06:03,719-Speed 128.75 samples/sec Loss 0.7092 Epoch: 19 Global Step: 194910 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:06:23,453-Speed 129.73 samples/sec Loss 0.7275 Epoch: 19 Global Step: 194915 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:06:43,155-Speed 129.95 samples/sec Loss 0.7028 Epoch: 19 Global Step: 194920 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:07:03,370-Speed 126.64 samples/sec Loss 0.6203 Epoch: 19 Global Step: 194925 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:07:23,454-Speed 127.47 samples/sec Loss 0.7234 Epoch: 19 Global Step: 194930 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:07:43,183-Speed 129.76 samples/sec Loss 0.7869 Epoch: 19 Global Step: 194935 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:08:03,092-Speed 128.59 samples/sec Loss 0.7810 Epoch: 19 Global Step: 194940 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:08:22,919-Speed 129.12 samples/sec Loss 0.7479 Epoch: 19 Global Step: 194945 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:08:42,739-Speed 129.17 samples/sec Loss 0.8156 Epoch: 19 Global Step: 194950 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:09:02,643-Speed 128.62 samples/sec Loss 0.7330 Epoch: 19 Global Step: 194955 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:09:22,372-Speed 129.77 samples/sec Loss 0.6813 Epoch: 19 Global Step: 194960 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:09:42,213-Speed 129.03 samples/sec Loss 0.7964 Epoch: 19 Global Step: 194965 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:10:02,265-Speed 127.67 samples/sec Loss 0.7346 Epoch: 19 Global Step: 194970 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:10:22,366-Speed 127.36 samples/sec Loss 0.6981 Epoch: 19 Global Step: 194975 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:10:42,072-Speed 129.91 samples/sec Loss 0.7380 Epoch: 19 Global Step: 194980 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:11:01,914-Speed 129.03 samples/sec Loss 0.7614 Epoch: 19 Global Step: 194985 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:11:21,613-Speed 129.96 samples/sec Loss 0.6221 Epoch: 19 Global Step: 194990 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:11:41,437-Speed 129.15 samples/sec Loss 0.7838 Epoch: 19 Global Step: 194995 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:12:01,479-Speed 127.74 samples/sec Loss 0.7843 Epoch: 19 Global Step: 195000 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:12:21,228-Speed 129.63 samples/sec Loss 0.7359 Epoch: 19 Global Step: 195005 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:12:41,020-Speed 129.35 samples/sec Loss 0.7193 Epoch: 19 Global Step: 195010 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:13:01,096-Speed 127.52 samples/sec Loss 0.6325 Epoch: 19 Global Step: 195015 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:13:20,408-Speed 132.57 samples/sec Loss 0.7495 Epoch: 19 Global Step: 195020 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:13:40,260-Speed 128.96 samples/sec Loss 0.6954 Epoch: 19 Global Step: 195025 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:14:00,306-Speed 127.71 samples/sec Loss 0.6839 Epoch: 19 Global Step: 195030 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:14:20,046-Speed 129.69 samples/sec Loss 0.7279 Epoch: 19 Global Step: 195035 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:14:40,111-Speed 127.59 samples/sec Loss 0.6938 Epoch: 19 Global Step: 195040 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:14:59,827-Speed 129.85 samples/sec Loss 0.7362 Epoch: 19 Global Step: 195045 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:15:19,606-Speed 129.44 samples/sec Loss 0.7249 Epoch: 19 Global Step: 195050 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:15:39,358-Speed 129.61 samples/sec Loss 0.7061 Epoch: 19 Global Step: 195055 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:15:59,382-Speed 127.85 samples/sec Loss 0.7452 Epoch: 19 Global Step: 195060 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:16:19,373-Speed 128.06 samples/sec Loss 0.7473 Epoch: 19 Global Step: 195065 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:16:39,107-Speed 129.73 samples/sec Loss 0.7342 Epoch: 19 Global Step: 195070 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:16:59,177-Speed 127.56 samples/sec Loss 0.6389 Epoch: 19 Global Step: 195075 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:17:18,871-Speed 129.99 samples/sec Loss 0.7424 Epoch: 19 Global Step: 195080 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:17:38,844-Speed 128.18 samples/sec Loss 0.6522 Epoch: 19 Global Step: 195085 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:17:58,481-Speed 130.37 samples/sec Loss 0.7607 Epoch: 19 Global Step: 195090 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:18:18,359-Speed 128.80 samples/sec Loss 0.6868 Epoch: 19 Global Step: 195095 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:18:38,086-Speed 129.78 samples/sec Loss 0.6039 Epoch: 19 Global Step: 195100 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:18:58,103-Speed 127.89 samples/sec Loss 0.6467 Epoch: 19 Global Step: 195105 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:19:18,030-Speed 128.47 samples/sec Loss 0.7322 Epoch: 19 Global Step: 195110 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:19:37,679-Speed 130.29 samples/sec Loss 0.7075 Epoch: 19 Global Step: 195115 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:19:57,739-Speed 127.63 samples/sec Loss 0.7266 Epoch: 19 Global Step: 195120 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:20:17,583-Speed 129.01 samples/sec Loss 0.7162 Epoch: 19 Global Step: 195125 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:20:37,382-Speed 129.31 samples/sec Loss 0.6843 Epoch: 19 Global Step: 195130 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:20:57,108-Speed 129.79 samples/sec Loss 0.6931 Epoch: 19 Global Step: 195135 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:21:16,938-Speed 129.11 samples/sec Loss 0.7349 Epoch: 19 Global Step: 195140 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:21:36,609-Speed 130.14 samples/sec Loss 0.6073 Epoch: 19 Global Step: 195145 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:21:56,906-Speed 126.14 samples/sec Loss 0.6900 Epoch: 19 Global Step: 195150 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:22:16,693-Speed 129.38 samples/sec Loss 0.6162 Epoch: 19 Global Step: 195155 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:22:36,296-Speed 130.60 samples/sec Loss 0.7747 Epoch: 19 Global Step: 195160 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:22:56,379-Speed 127.48 samples/sec Loss 0.7271 Epoch: 19 Global Step: 195165 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:23:16,226-Speed 128.99 samples/sec Loss 0.6796 Epoch: 19 Global Step: 195170 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:23:36,046-Speed 129.16 samples/sec Loss 0.7523 Epoch: 19 Global Step: 195175 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:23:55,780-Speed 129.73 samples/sec Loss 0.7075 Epoch: 19 Global Step: 195180 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:24:15,566-Speed 129.39 samples/sec Loss 0.7377 Epoch: 19 Global Step: 195185 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:24:35,270-Speed 129.93 samples/sec Loss 0.6730 Epoch: 19 Global Step: 195190 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:24:55,340-Speed 127.56 samples/sec Loss 0.6395 Epoch: 19 Global Step: 195195 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:25:15,202-Speed 128.90 samples/sec Loss 0.6710 Epoch: 19 Global Step: 195200 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:25:34,863-Speed 130.21 samples/sec Loss 0.7683 Epoch: 19 Global Step: 195205 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:25:54,934-Speed 127.56 samples/sec Loss 0.6663 Epoch: 19 Global Step: 195210 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:26:15,543-Speed 124.22 samples/sec Loss 0.6446 Epoch: 19 Global Step: 195215 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:26:35,327-Speed 129.40 samples/sec Loss 0.6562 Epoch: 19 Global Step: 195220 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:26:55,089-Speed 129.55 samples/sec Loss 0.7779 Epoch: 19 Global Step: 195225 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:27:14,956-Speed 128.86 samples/sec Loss 0.6834 Epoch: 19 Global Step: 195230 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:27:34,622-Speed 130.18 samples/sec Loss 0.7887 Epoch: 19 Global Step: 195235 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:27:54,811-Speed 126.81 samples/sec Loss 0.7116 Epoch: 19 Global Step: 195240 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:28:14,517-Speed 129.91 samples/sec Loss 0.7172 Epoch: 19 Global Step: 195245 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:28:34,254-Speed 129.71 samples/sec Loss 0.6555 Epoch: 19 Global Step: 195250 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:28:54,301-Speed 127.71 samples/sec Loss 0.7127 Epoch: 19 Global Step: 195255 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:29:14,191-Speed 128.71 samples/sec Loss 0.7211 Epoch: 19 Global Step: 195260 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:29:33,905-Speed 129.86 samples/sec Loss 0.7049 Epoch: 19 Global Step: 195265 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:29:53,663-Speed 129.57 samples/sec Loss 0.6828 Epoch: 19 Global Step: 195270 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:30:13,559-Speed 128.68 samples/sec Loss 0.6638 Epoch: 19 Global Step: 195275 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:30:33,264-Speed 129.92 samples/sec Loss 0.7169 Epoch: 19 Global Step: 195280 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:30:53,425-Speed 126.99 samples/sec Loss 0.7732 Epoch: 19 Global Step: 195285 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:31:13,181-Speed 129.59 samples/sec Loss 0.7070 Epoch: 19 Global Step: 195290 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:31:32,886-Speed 129.92 samples/sec Loss 0.7350 Epoch: 19 Global Step: 195295 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:31:52,893-Speed 127.96 samples/sec Loss 0.7675 Epoch: 19 Global Step: 195300 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:32:12,792-Speed 128.65 samples/sec Loss 0.6775 Epoch: 19 Global Step: 195305 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:32:32,566-Speed 129.47 samples/sec Loss 0.7784 Epoch: 19 Global Step: 195310 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:32:52,493-Speed 128.47 samples/sec Loss 0.7934 Epoch: 19 Global Step: 195315 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:33:12,101-Speed 130.57 samples/sec Loss 0.7885 Epoch: 19 Global Step: 195320 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:33:31,857-Speed 129.59 samples/sec Loss 0.7695 Epoch: 19 Global Step: 195325 Fp16 Grad Scale: 8192 Required: 65 hours Training: 2021-06-16 13:33:51,267-Speed 131.90 samples/sec Loss 0.7042 Epoch: 19 Global Step: 195330 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:34:11,051-Speed 129.41 samples/sec Loss 0.6861 Epoch: 19 Global Step: 195335 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:34:31,147-Speed 127.39 samples/sec Loss 0.7840 Epoch: 19 Global Step: 195340 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:34:51,093-Speed 128.36 samples/sec Loss 0.7534 Epoch: 19 Global Step: 195345 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:35:10,950-Speed 128.93 samples/sec Loss 0.6926 Epoch: 19 Global Step: 195350 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:35:30,702-Speed 129.61 samples/sec Loss 0.5833 Epoch: 19 Global Step: 195355 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:35:50,585-Speed 128.76 samples/sec Loss 0.6726 Epoch: 19 Global Step: 195360 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:36:10,038-Speed 131.61 samples/sec Loss 0.7699 Epoch: 19 Global Step: 195365 Fp16 Grad Scale: 4096 Required: 65 hours Training: 2021-06-16 13:36:29,501-Speed 131.54 samples/sec Loss 0.7570 Epoch: 19 Global Step: 195370 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:36:49,700-Speed 126.74 samples/sec Loss 0.8146 Epoch: 19 Global Step: 195375 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:37:09,244-Speed 130.99 samples/sec Loss 0.7983 Epoch: 19 Global Step: 195380 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:37:28,853-Speed 130.56 samples/sec Loss 0.7578 Epoch: 19 Global Step: 195385 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:37:48,510-Speed 130.24 samples/sec Loss 0.7124 Epoch: 19 Global Step: 195390 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:38:08,086-Speed 130.78 samples/sec Loss 0.5367 Epoch: 19 Global Step: 195395 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:38:27,587-Speed 131.29 samples/sec Loss 0.6963 Epoch: 19 Global Step: 195400 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:38:47,212-Speed 130.45 samples/sec Loss 0.7285 Epoch: 19 Global Step: 195405 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:39:06,694-Speed 131.41 samples/sec Loss 0.7441 Epoch: 19 Global Step: 195410 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:39:26,285-Speed 130.68 samples/sec Loss 0.6912 Epoch: 19 Global Step: 195415 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:39:45,822-Speed 131.05 samples/sec Loss 0.7561 Epoch: 19 Global Step: 195420 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:40:05,366-Speed 131.00 samples/sec Loss 0.7805 Epoch: 19 Global Step: 195425 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:40:24,947-Speed 130.75 samples/sec Loss 0.7900 Epoch: 19 Global Step: 195430 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:40:44,675-Speed 129.77 samples/sec Loss 0.7102 Epoch: 19 Global Step: 195435 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:41:04,215-Speed 131.02 samples/sec Loss 0.6295 Epoch: 19 Global Step: 195440 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:41:23,753-Speed 131.03 samples/sec Loss 0.6961 Epoch: 19 Global Step: 195445 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:41:43,437-Speed 130.06 samples/sec Loss 0.8197 Epoch: 19 Global Step: 195450 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:42:02,873-Speed 131.72 samples/sec Loss 0.6467 Epoch: 19 Global Step: 195455 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:42:22,471-Speed 130.63 samples/sec Loss 0.7338 Epoch: 19 Global Step: 195460 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:42:41,917-Speed 131.66 samples/sec Loss 0.7177 Epoch: 19 Global Step: 195465 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:43:01,468-Speed 130.95 samples/sec Loss 0.7441 Epoch: 19 Global Step: 195470 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:43:21,082-Speed 130.53 samples/sec Loss 0.6784 Epoch: 19 Global Step: 195475 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:43:40,902-Speed 129.17 samples/sec Loss 0.6266 Epoch: 19 Global Step: 195480 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:44:00,345-Speed 131.68 samples/sec Loss 0.7275 Epoch: 19 Global Step: 195485 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:44:19,896-Speed 130.94 samples/sec Loss 0.7136 Epoch: 19 Global Step: 195490 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:44:39,791-Speed 128.69 samples/sec Loss 0.7557 Epoch: 19 Global Step: 195495 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:44:58,773-Speed 134.87 samples/sec Loss 0.6846 Epoch: 19 Global Step: 195500 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:45:18,384-Speed 130.55 samples/sec Loss 0.7872 Epoch: 19 Global Step: 195505 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:45:37,885-Speed 131.28 samples/sec Loss 0.7296 Epoch: 19 Global Step: 195510 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:45:57,426-Speed 131.01 samples/sec Loss 0.6316 Epoch: 19 Global Step: 195515 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:46:17,037-Speed 130.55 samples/sec Loss 0.7927 Epoch: 19 Global Step: 195520 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:46:36,786-Speed 129.64 samples/sec Loss 0.6574 Epoch: 19 Global Step: 195525 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:46:56,352-Speed 130.85 samples/sec Loss 0.7174 Epoch: 19 Global Step: 195530 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:47:15,763-Speed 131.89 samples/sec Loss 0.7567 Epoch: 19 Global Step: 195535 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:47:35,419-Speed 130.25 samples/sec Loss 0.7540 Epoch: 19 Global Step: 195540 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:47:54,937-Speed 131.17 samples/sec Loss 0.6867 Epoch: 19 Global Step: 195545 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:48:14,466-Speed 131.10 samples/sec Loss 0.7758 Epoch: 19 Global Step: 195550 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:48:34,028-Speed 130.87 samples/sec Loss 0.7199 Epoch: 19 Global Step: 195555 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:48:53,555-Speed 131.11 samples/sec Loss 0.7619 Epoch: 19 Global Step: 195560 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:49:13,092-Speed 131.04 samples/sec Loss 0.6730 Epoch: 19 Global Step: 195565 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:49:32,807-Speed 129.86 samples/sec Loss 0.7206 Epoch: 19 Global Step: 195570 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:49:52,520-Speed 129.87 samples/sec Loss 0.5781 Epoch: 19 Global Step: 195575 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:50:11,979-Speed 131.57 samples/sec Loss 0.6090 Epoch: 19 Global Step: 195580 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:50:31,746-Speed 129.51 samples/sec Loss 0.7425 Epoch: 19 Global Step: 195585 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:50:51,312-Speed 130.85 samples/sec Loss 0.5888 Epoch: 19 Global Step: 195590 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:51:10,956-Speed 130.33 samples/sec Loss 0.7637 Epoch: 19 Global Step: 195595 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 13:51:30,577-Speed 130.48 samples/sec Loss 0.7505 Epoch: 19 Global Step: 195600 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:51:49,998-Speed 131.82 samples/sec Loss 0.6845 Epoch: 19 Global Step: 195605 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:52:09,518-Speed 131.16 samples/sec Loss 0.7081 Epoch: 19 Global Step: 195610 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:52:29,300-Speed 129.42 samples/sec Loss 0.7744 Epoch: 19 Global Step: 195615 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:52:48,995-Speed 129.99 samples/sec Loss 0.6730 Epoch: 19 Global Step: 195620 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:53:08,448-Speed 131.60 samples/sec Loss 0.6486 Epoch: 19 Global Step: 195625 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:53:27,908-Speed 131.56 samples/sec Loss 0.7121 Epoch: 19 Global Step: 195630 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:53:47,496-Speed 130.70 samples/sec Loss 0.7805 Epoch: 19 Global Step: 195635 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:54:07,015-Speed 131.16 samples/sec Loss 0.7093 Epoch: 19 Global Step: 195640 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:54:26,625-Speed 130.55 samples/sec Loss 0.7300 Epoch: 19 Global Step: 195645 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:54:46,098-Speed 131.47 samples/sec Loss 0.7052 Epoch: 19 Global Step: 195650 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:55:05,544-Speed 131.65 samples/sec Loss 0.7715 Epoch: 19 Global Step: 195655 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:55:25,345-Speed 129.29 samples/sec Loss 0.7355 Epoch: 19 Global Step: 195660 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:55:45,058-Speed 129.87 samples/sec Loss 0.7432 Epoch: 19 Global Step: 195665 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:56:04,559-Speed 131.28 samples/sec Loss 0.7679 Epoch: 19 Global Step: 195670 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:56:24,040-Speed 131.42 samples/sec Loss 0.7700 Epoch: 19 Global Step: 195675 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:56:44,028-Speed 128.08 samples/sec Loss 0.7254 Epoch: 19 Global Step: 195680 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:57:03,553-Speed 131.12 samples/sec Loss 0.6865 Epoch: 19 Global Step: 195685 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:57:23,154-Speed 130.61 samples/sec Loss 0.7540 Epoch: 19 Global Step: 195690 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:57:42,681-Speed 131.11 samples/sec Loss 0.6724 Epoch: 19 Global Step: 195695 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:58:02,149-Speed 131.51 samples/sec Loss 0.7063 Epoch: 19 Global Step: 195700 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:58:21,916-Speed 129.52 samples/sec Loss 0.6405 Epoch: 19 Global Step: 195705 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:58:41,633-Speed 129.84 samples/sec Loss 0.6890 Epoch: 19 Global Step: 195710 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:59:01,209-Speed 130.78 samples/sec Loss 0.6933 Epoch: 19 Global Step: 195715 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:59:20,665-Speed 131.59 samples/sec Loss 0.8328 Epoch: 19 Global Step: 195720 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:59:40,231-Speed 130.85 samples/sec Loss 0.6866 Epoch: 19 Global Step: 195725 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 13:59:59,776-Speed 130.99 samples/sec Loss 0.7511 Epoch: 19 Global Step: 195730 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:00:19,408-Speed 130.41 samples/sec Loss 0.5726 Epoch: 19 Global Step: 195735 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:00:38,933-Speed 131.12 samples/sec Loss 0.7210 Epoch: 19 Global Step: 195740 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:00:58,375-Speed 131.69 samples/sec Loss 0.7036 Epoch: 19 Global Step: 195745 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:01:18,177-Speed 129.29 samples/sec Loss 0.8185 Epoch: 19 Global Step: 195750 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:01:37,848-Speed 130.14 samples/sec Loss 0.6311 Epoch: 19 Global Step: 195755 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:01:57,349-Speed 131.28 samples/sec Loss 0.8019 Epoch: 19 Global Step: 195760 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:02:16,773-Speed 131.80 samples/sec Loss 0.7137 Epoch: 19 Global Step: 195765 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:02:36,378-Speed 130.59 samples/sec Loss 0.7802 Epoch: 19 Global Step: 195770 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:02:55,931-Speed 130.93 samples/sec Loss 0.6892 Epoch: 19 Global Step: 195775 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:03:15,390-Speed 131.57 samples/sec Loss 0.7964 Epoch: 19 Global Step: 195780 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:03:34,534-Speed 133.73 samples/sec Loss 0.6941 Epoch: 19 Global Step: 195785 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:03:54,154-Speed 130.48 samples/sec Loss 0.7029 Epoch: 19 Global Step: 195790 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:04:14,191-Speed 127.77 samples/sec Loss 0.7410 Epoch: 19 Global Step: 195795 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:04:33,990-Speed 129.30 samples/sec Loss 0.6643 Epoch: 19 Global Step: 195800 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:04:54,089-Speed 127.37 samples/sec Loss 0.7291 Epoch: 19 Global Step: 195805 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:05:14,043-Speed 128.30 samples/sec Loss 0.7063 Epoch: 19 Global Step: 195810 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:05:33,514-Speed 131.48 samples/sec Loss 0.7826 Epoch: 19 Global Step: 195815 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:05:53,457-Speed 128.37 samples/sec Loss 0.6808 Epoch: 19 Global Step: 195820 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:06:13,326-Speed 128.85 samples/sec Loss 0.7545 Epoch: 19 Global Step: 195825 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:06:33,032-Speed 129.91 samples/sec Loss 0.7819 Epoch: 19 Global Step: 195830 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:06:52,828-Speed 129.32 samples/sec Loss 0.7692 Epoch: 19 Global Step: 195835 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:07:12,510-Speed 130.08 samples/sec Loss 0.6397 Epoch: 19 Global Step: 195840 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:07:32,261-Speed 129.62 samples/sec Loss 0.8877 Epoch: 19 Global Step: 195845 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:07:52,310-Speed 127.69 samples/sec Loss 0.7514 Epoch: 19 Global Step: 195850 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:08:11,910-Speed 130.62 samples/sec Loss 0.7377 Epoch: 19 Global Step: 195855 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:08:32,005-Speed 127.40 samples/sec Loss 0.6821 Epoch: 19 Global Step: 195860 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:08:51,933-Speed 128.47 samples/sec Loss 0.6884 Epoch: 19 Global Step: 195865 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:09:11,682-Speed 129.63 samples/sec Loss 0.7602 Epoch: 19 Global Step: 195870 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:09:31,400-Speed 129.84 samples/sec Loss 0.7763 Epoch: 19 Global Step: 195875 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:09:51,200-Speed 129.30 samples/sec Loss 0.8176 Epoch: 19 Global Step: 195880 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:10:10,988-Speed 129.38 samples/sec Loss 0.7096 Epoch: 19 Global Step: 195885 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:10:30,837-Speed 128.98 samples/sec Loss 0.8248 Epoch: 19 Global Step: 195890 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:10:50,739-Speed 128.64 samples/sec Loss 0.7094 Epoch: 19 Global Step: 195895 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:11:10,421-Speed 130.08 samples/sec Loss 0.6980 Epoch: 19 Global Step: 195900 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:11:30,464-Speed 127.73 samples/sec Loss 0.7026 Epoch: 19 Global Step: 195905 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:11:50,563-Speed 127.37 samples/sec Loss 0.7669 Epoch: 19 Global Step: 195910 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:12:10,181-Speed 130.50 samples/sec Loss 0.7291 Epoch: 19 Global Step: 195915 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:12:30,068-Speed 128.73 samples/sec Loss 0.7211 Epoch: 19 Global Step: 195920 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:12:49,714-Speed 130.31 samples/sec Loss 0.6823 Epoch: 19 Global Step: 195925 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:13:09,445-Speed 129.75 samples/sec Loss 0.6963 Epoch: 19 Global Step: 195930 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:13:29,380-Speed 128.43 samples/sec Loss 0.7718 Epoch: 19 Global Step: 195935 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:13:49,299-Speed 128.52 samples/sec Loss 0.7993 Epoch: 19 Global Step: 195940 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:14:09,155-Speed 128.94 samples/sec Loss 0.6806 Epoch: 19 Global Step: 195945 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:14:28,876-Speed 129.81 samples/sec Loss 0.6001 Epoch: 19 Global Step: 195950 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:14:49,039-Speed 126.97 samples/sec Loss 0.6710 Epoch: 19 Global Step: 195955 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:15:08,768-Speed 129.77 samples/sec Loss 0.7692 Epoch: 19 Global Step: 195960 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:15:28,661-Speed 128.69 samples/sec Loss 0.6860 Epoch: 19 Global Step: 195965 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:15:48,374-Speed 129.87 samples/sec Loss 0.7832 Epoch: 19 Global Step: 195970 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:16:08,116-Speed 129.68 samples/sec Loss 0.6936 Epoch: 19 Global Step: 195975 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:16:28,040-Speed 128.50 samples/sec Loss 0.7660 Epoch: 19 Global Step: 195980 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:16:47,884-Speed 129.02 samples/sec Loss 0.7199 Epoch: 19 Global Step: 195985 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:17:07,797-Speed 128.57 samples/sec Loss 0.7246 Epoch: 19 Global Step: 195990 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:17:27,489-Speed 130.01 samples/sec Loss 0.7102 Epoch: 19 Global Step: 195995 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:17:47,676-Speed 126.82 samples/sec Loss 0.7058 Epoch: 19 Global Step: 196000 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:18:07,678-Speed 127.99 samples/sec Loss 0.7442 Epoch: 19 Global Step: 196005 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:18:27,431-Speed 129.60 samples/sec Loss 0.6475 Epoch: 19 Global Step: 196010 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:18:47,158-Speed 129.78 samples/sec Loss 0.6355 Epoch: 19 Global Step: 196015 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:19:07,111-Speed 128.31 samples/sec Loss 0.7777 Epoch: 19 Global Step: 196020 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:19:26,898-Speed 129.38 samples/sec Loss 0.8096 Epoch: 19 Global Step: 196025 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:19:46,849-Speed 128.32 samples/sec Loss 0.8565 Epoch: 19 Global Step: 196030 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:20:06,694-Speed 129.01 samples/sec Loss 0.6926 Epoch: 19 Global Step: 196035 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:20:26,361-Speed 130.18 samples/sec Loss 0.7283 Epoch: 19 Global Step: 196040 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:20:46,508-Speed 127.07 samples/sec Loss 0.7188 Epoch: 19 Global Step: 196045 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:21:06,267-Speed 129.57 samples/sec Loss 0.7293 Epoch: 19 Global Step: 196050 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:21:26,020-Speed 129.61 samples/sec Loss 0.7206 Epoch: 19 Global Step: 196055 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:21:45,700-Speed 130.09 samples/sec Loss 0.7791 Epoch: 19 Global Step: 196060 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:22:05,603-Speed 128.63 samples/sec Loss 0.6727 Epoch: 19 Global Step: 196065 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:22:25,392-Speed 129.37 samples/sec Loss 0.7431 Epoch: 19 Global Step: 196070 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:22:45,478-Speed 127.46 samples/sec Loss 0.7092 Epoch: 19 Global Step: 196075 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:23:05,259-Speed 129.43 samples/sec Loss 0.7910 Epoch: 19 Global Step: 196080 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:23:25,021-Speed 129.54 samples/sec Loss 0.7246 Epoch: 19 Global Step: 196085 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:23:45,090-Speed 127.57 samples/sec Loss 0.7306 Epoch: 19 Global Step: 196090 Fp16 Grad Scale: 8192 Required: 64 hours Training: 2021-06-16 14:24:04,542-Speed 131.61 samples/sec Loss 0.7276 Epoch: 19 Global Step: 196095 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:24:24,244-Speed 129.94 samples/sec Loss 0.7095 Epoch: 19 Global Step: 196100 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:24:44,146-Speed 128.64 samples/sec Loss 0.7833 Epoch: 19 Global Step: 196105 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:25:04,055-Speed 128.59 samples/sec Loss 0.6803 Epoch: 19 Global Step: 196110 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:25:23,739-Speed 130.06 samples/sec Loss 0.7253 Epoch: 19 Global Step: 196115 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:25:43,562-Speed 129.15 samples/sec Loss 0.7277 Epoch: 19 Global Step: 196120 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:26:03,256-Speed 129.99 samples/sec Loss 0.7418 Epoch: 19 Global Step: 196125 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:26:22,775-Speed 131.16 samples/sec Loss 0.7579 Epoch: 19 Global Step: 196130 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:26:42,603-Speed 129.12 samples/sec Loss 0.7390 Epoch: 19 Global Step: 196135 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:27:02,269-Speed 130.18 samples/sec Loss 0.6358 Epoch: 19 Global Step: 196140 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:27:21,855-Speed 130.71 samples/sec Loss 0.6491 Epoch: 19 Global Step: 196145 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:27:41,455-Speed 130.61 samples/sec Loss 0.7761 Epoch: 19 Global Step: 196150 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:28:01,058-Speed 130.61 samples/sec Loss 0.7559 Epoch: 19 Global Step: 196155 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:28:20,600-Speed 131.00 samples/sec Loss 0.7116 Epoch: 19 Global Step: 196160 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:28:40,225-Speed 130.46 samples/sec Loss 0.7706 Epoch: 19 Global Step: 196165 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:28:59,713-Speed 131.37 samples/sec Loss 0.7626 Epoch: 19 Global Step: 196170 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:29:19,241-Speed 131.10 samples/sec Loss 0.7508 Epoch: 19 Global Step: 196175 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:29:39,047-Speed 129.26 samples/sec Loss 0.6606 Epoch: 19 Global Step: 196180 Fp16 Grad Scale: 4096 Required: 64 hours Training: 2021-06-16 14:29:58,759-Speed 129.87 samples/sec Loss 0.6951 Epoch: 19 Global Step: 196185 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:30:18,292-Speed 131.06 samples/sec Loss 0.6896 Epoch: 19 Global Step: 196190 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:30:37,900-Speed 130.57 samples/sec Loss 0.7309 Epoch: 19 Global Step: 196195 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:30:57,517-Speed 130.51 samples/sec Loss 0.7135 Epoch: 19 Global Step: 196200 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:31:17,063-Speed 130.98 samples/sec Loss 0.6706 Epoch: 19 Global Step: 196205 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:31:36,823-Speed 129.56 samples/sec Loss 0.8179 Epoch: 19 Global Step: 196210 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:31:56,406-Speed 130.73 samples/sec Loss 0.6745 Epoch: 19 Global Step: 196215 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:32:15,943-Speed 131.04 samples/sec Loss 0.7151 Epoch: 19 Global Step: 196220 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:32:35,696-Speed 129.61 samples/sec Loss 0.6791 Epoch: 19 Global Step: 196225 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:32:55,415-Speed 129.82 samples/sec Loss 0.7848 Epoch: 19 Global Step: 196230 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:33:14,948-Speed 131.07 samples/sec Loss 0.7336 Epoch: 19 Global Step: 196235 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:33:34,765-Speed 129.19 samples/sec Loss 0.7349 Epoch: 19 Global Step: 196240 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:33:54,291-Speed 131.12 samples/sec Loss 0.6448 Epoch: 19 Global Step: 196245 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:34:14,390-Speed 127.37 samples/sec Loss 0.7896 Epoch: 19 Global Step: 196250 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:34:34,064-Speed 130.13 samples/sec Loss 0.6783 Epoch: 19 Global Step: 196255 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:34:53,649-Speed 130.72 samples/sec Loss 0.6568 Epoch: 19 Global Step: 196260 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:35:13,154-Speed 131.25 samples/sec Loss 0.6959 Epoch: 19 Global Step: 196265 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:35:32,899-Speed 129.66 samples/sec Loss 0.7077 Epoch: 19 Global Step: 196270 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:35:52,634-Speed 129.73 samples/sec Loss 0.8405 Epoch: 19 Global Step: 196275 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:36:12,145-Speed 131.21 samples/sec Loss 0.7387 Epoch: 19 Global Step: 196280 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:36:31,757-Speed 130.54 samples/sec Loss 0.7515 Epoch: 19 Global Step: 196285 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:36:51,423-Speed 130.19 samples/sec Loss 0.8506 Epoch: 19 Global Step: 196290 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:37:11,042-Speed 130.49 samples/sec Loss 0.6918 Epoch: 19 Global Step: 196295 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:37:30,643-Speed 130.62 samples/sec Loss 0.6903 Epoch: 19 Global Step: 196300 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:37:50,161-Speed 131.16 samples/sec Loss 0.8629 Epoch: 19 Global Step: 196305 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:38:09,765-Speed 130.60 samples/sec Loss 0.7727 Epoch: 19 Global Step: 196310 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:38:29,700-Speed 128.42 samples/sec Loss 0.8071 Epoch: 19 Global Step: 196315 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:38:49,142-Speed 131.68 samples/sec Loss 0.7356 Epoch: 19 Global Step: 196320 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:39:08,672-Speed 131.09 samples/sec Loss 0.6305 Epoch: 19 Global Step: 196325 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:39:28,227-Speed 130.92 samples/sec Loss 0.7579 Epoch: 19 Global Step: 196330 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:39:47,740-Speed 131.21 samples/sec Loss 0.6857 Epoch: 19 Global Step: 196335 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:40:07,344-Speed 130.59 samples/sec Loss 0.7692 Epoch: 19 Global Step: 196340 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:40:26,777-Speed 131.74 samples/sec Loss 0.6036 Epoch: 19 Global Step: 196345 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:40:46,247-Speed 131.49 samples/sec Loss 0.7458 Epoch: 19 Global Step: 196350 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:41:05,813-Speed 130.84 samples/sec Loss 0.6413 Epoch: 19 Global Step: 196355 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:41:25,735-Speed 128.51 samples/sec Loss 0.6623 Epoch: 19 Global Step: 196360 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:41:45,154-Speed 131.84 samples/sec Loss 0.7889 Epoch: 19 Global Step: 196365 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:42:04,875-Speed 129.81 samples/sec Loss 0.6643 Epoch: 19 Global Step: 196370 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:42:24,482-Speed 130.58 samples/sec Loss 0.7522 Epoch: 19 Global Step: 196375 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:42:44,023-Speed 131.01 samples/sec Loss 0.6798 Epoch: 19 Global Step: 196380 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:43:03,629-Speed 130.58 samples/sec Loss 0.6911 Epoch: 19 Global Step: 196385 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:43:23,164-Speed 131.05 samples/sec Loss 0.7348 Epoch: 19 Global Step: 196390 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:43:42,314-Speed 133.69 samples/sec Loss 0.6958 Epoch: 19 Global Step: 196395 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:44:01,799-Speed 131.39 samples/sec Loss 0.7235 Epoch: 19 Global Step: 196400 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:44:21,716-Speed 128.54 samples/sec Loss 0.6594 Epoch: 19 Global Step: 196405 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:44:41,139-Speed 131.82 samples/sec Loss 0.6775 Epoch: 19 Global Step: 196410 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:45:00,768-Speed 130.42 samples/sec Loss 0.6848 Epoch: 19 Global Step: 196415 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:45:20,325-Speed 130.91 samples/sec Loss 0.6781 Epoch: 19 Global Step: 196420 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:45:39,878-Speed 130.93 samples/sec Loss 0.7121 Epoch: 19 Global Step: 196425 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:45:59,482-Speed 130.59 samples/sec Loss 0.6462 Epoch: 19 Global Step: 196430 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:46:19,051-Speed 130.82 samples/sec Loss 0.6421 Epoch: 19 Global Step: 196435 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:46:38,584-Speed 131.07 samples/sec Loss 0.7148 Epoch: 19 Global Step: 196440 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:46:58,117-Speed 131.07 samples/sec Loss 0.6568 Epoch: 19 Global Step: 196445 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:47:18,034-Speed 128.54 samples/sec Loss 0.6980 Epoch: 19 Global Step: 196450 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:47:37,499-Speed 131.52 samples/sec Loss 0.7617 Epoch: 19 Global Step: 196455 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:47:57,319-Speed 129.17 samples/sec Loss 0.7523 Epoch: 19 Global Step: 196460 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:48:16,838-Speed 131.18 samples/sec Loss 0.7424 Epoch: 19 Global Step: 196465 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:48:36,359-Speed 131.15 samples/sec Loss 0.7792 Epoch: 19 Global Step: 196470 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:48:55,974-Speed 130.52 samples/sec Loss 0.7290 Epoch: 19 Global Step: 196475 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:49:15,551-Speed 130.77 samples/sec Loss 0.7712 Epoch: 19 Global Step: 196480 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:49:35,164-Speed 130.53 samples/sec Loss 0.7193 Epoch: 19 Global Step: 196485 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:49:54,672-Speed 131.24 samples/sec Loss 0.7763 Epoch: 19 Global Step: 196490 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:50:14,566-Speed 128.69 samples/sec Loss 0.7817 Epoch: 19 Global Step: 196495 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:50:33,992-Speed 131.79 samples/sec Loss 0.8452 Epoch: 19 Global Step: 196500 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:50:53,615-Speed 130.47 samples/sec Loss 0.7070 Epoch: 19 Global Step: 196505 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:51:13,191-Speed 130.78 samples/sec Loss 0.6761 Epoch: 19 Global Step: 196510 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:51:32,737-Speed 130.99 samples/sec Loss 0.7717 Epoch: 19 Global Step: 196515 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:51:51,906-Speed 133.55 samples/sec Loss 0.7065 Epoch: 19 Global Step: 196520 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:52:11,689-Speed 129.41 samples/sec Loss 0.7768 Epoch: 19 Global Step: 196525 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:52:31,488-Speed 129.31 samples/sec Loss 0.7836 Epoch: 19 Global Step: 196530 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:52:51,184-Speed 129.98 samples/sec Loss 0.7271 Epoch: 19 Global Step: 196535 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:53:10,943-Speed 129.57 samples/sec Loss 0.7421 Epoch: 19 Global Step: 196540 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:53:30,799-Speed 128.94 samples/sec Loss 0.7534 Epoch: 19 Global Step: 196545 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:53:50,764-Speed 128.23 samples/sec Loss 0.8244 Epoch: 19 Global Step: 196550 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:54:10,640-Speed 128.81 samples/sec Loss 0.7434 Epoch: 19 Global Step: 196555 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:54:30,359-Speed 129.82 samples/sec Loss 0.6926 Epoch: 19 Global Step: 196560 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:54:50,429-Speed 127.56 samples/sec Loss 0.7657 Epoch: 19 Global Step: 196565 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:55:10,208-Speed 129.44 samples/sec Loss 0.7309 Epoch: 19 Global Step: 196570 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:55:30,044-Speed 129.06 samples/sec Loss 0.7301 Epoch: 19 Global Step: 196575 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:55:49,793-Speed 129.64 samples/sec Loss 0.6575 Epoch: 19 Global Step: 196580 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:56:09,730-Speed 128.41 samples/sec Loss 0.6591 Epoch: 19 Global Step: 196585 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:56:29,468-Speed 129.70 samples/sec Loss 0.7257 Epoch: 19 Global Step: 196590 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:56:49,499-Speed 127.81 samples/sec Loss 0.7237 Epoch: 19 Global Step: 196595 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:57:09,364-Speed 128.88 samples/sec Loss 0.7580 Epoch: 19 Global Step: 196600 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:57:29,098-Speed 129.73 samples/sec Loss 0.7057 Epoch: 19 Global Step: 196605 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:57:49,197-Speed 127.38 samples/sec Loss 0.7926 Epoch: 19 Global Step: 196610 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:58:08,977-Speed 129.43 samples/sec Loss 0.7482 Epoch: 19 Global Step: 196615 Fp16 Grad Scale: 4096 Required: 63 hours Training: 2021-06-16 14:58:28,862-Speed 128.75 samples/sec Loss 0.6615 Epoch: 19 Global Step: 196620 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:58:48,569-Speed 129.91 samples/sec Loss 0.7375 Epoch: 19 Global Step: 196625 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:59:08,520-Speed 128.32 samples/sec Loss 0.7275 Epoch: 19 Global Step: 196630 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:59:28,136-Speed 130.51 samples/sec Loss 0.7272 Epoch: 19 Global Step: 196635 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 14:59:48,182-Speed 127.71 samples/sec Loss 0.7238 Epoch: 19 Global Step: 196640 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:00:08,032-Speed 128.97 samples/sec Loss 0.6562 Epoch: 19 Global Step: 196645 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:00:27,782-Speed 129.63 samples/sec Loss 0.6827 Epoch: 19 Global Step: 196650 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:00:47,755-Speed 128.18 samples/sec Loss 0.7649 Epoch: 19 Global Step: 196655 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:01:07,425-Speed 130.16 samples/sec Loss 0.6901 Epoch: 19 Global Step: 196660 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:01:27,346-Speed 128.51 samples/sec Loss 0.8506 Epoch: 19 Global Step: 196665 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:01:47,072-Speed 129.78 samples/sec Loss 0.8042 Epoch: 19 Global Step: 196670 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:02:06,981-Speed 128.59 samples/sec Loss 0.7780 Epoch: 19 Global Step: 196675 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:02:26,580-Speed 130.62 samples/sec Loss 0.7854 Epoch: 19 Global Step: 196680 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:02:46,712-Speed 127.17 samples/sec Loss 0.7052 Epoch: 19 Global Step: 196685 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:03:06,564-Speed 128.96 samples/sec Loss 0.6760 Epoch: 19 Global Step: 196690 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:03:26,286-Speed 129.81 samples/sec Loss 0.6442 Epoch: 19 Global Step: 196695 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:03:46,349-Speed 127.61 samples/sec Loss 0.7124 Epoch: 19 Global Step: 196700 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:04:06,082-Speed 129.74 samples/sec Loss 0.7414 Epoch: 19 Global Step: 196705 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:04:25,942-Speed 128.91 samples/sec Loss 0.7185 Epoch: 19 Global Step: 196710 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:04:45,711-Speed 129.50 samples/sec Loss 0.7254 Epoch: 19 Global Step: 196715 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:05:05,594-Speed 128.76 samples/sec Loss 0.7332 Epoch: 19 Global Step: 196720 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:05:25,190-Speed 130.64 samples/sec Loss 0.7434 Epoch: 19 Global Step: 196725 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:05:45,345-Speed 127.02 samples/sec Loss 0.7787 Epoch: 19 Global Step: 196730 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:06:05,061-Speed 129.85 samples/sec Loss 0.7213 Epoch: 19 Global Step: 196735 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:06:24,811-Speed 129.63 samples/sec Loss 0.7190 Epoch: 19 Global Step: 196740 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:06:44,752-Speed 128.39 samples/sec Loss 0.7165 Epoch: 19 Global Step: 196745 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:07:04,531-Speed 129.43 samples/sec Loss 0.7168 Epoch: 19 Global Step: 196750 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:07:24,306-Speed 129.46 samples/sec Loss 0.6920 Epoch: 19 Global Step: 196755 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:07:44,099-Speed 129.34 samples/sec Loss 0.6388 Epoch: 19 Global Step: 196760 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:08:03,978-Speed 128.79 samples/sec Loss 0.7445 Epoch: 19 Global Step: 196765 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:08:23,595-Speed 130.50 samples/sec Loss 0.8055 Epoch: 19 Global Step: 196770 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:08:43,679-Speed 127.47 samples/sec Loss 0.6778 Epoch: 19 Global Step: 196775 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:09:03,465-Speed 129.39 samples/sec Loss 0.6527 Epoch: 19 Global Step: 196780 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:09:23,164-Speed 129.97 samples/sec Loss 0.7350 Epoch: 19 Global Step: 196785 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:09:43,114-Speed 128.33 samples/sec Loss 0.7832 Epoch: 19 Global Step: 196790 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:10:02,912-Speed 129.31 samples/sec Loss 0.7771 Epoch: 19 Global Step: 196795 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:10:22,646-Speed 129.74 samples/sec Loss 0.7053 Epoch: 19 Global Step: 196800 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:10:42,591-Speed 128.35 samples/sec Loss 0.7953 Epoch: 19 Global Step: 196805 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:11:02,338-Speed 129.65 samples/sec Loss 0.7479 Epoch: 19 Global Step: 196810 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:11:21,996-Speed 130.24 samples/sec Loss 0.7523 Epoch: 19 Global Step: 196815 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:11:42,068-Speed 127.54 samples/sec Loss 0.6819 Epoch: 19 Global Step: 196820 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:12:01,779-Speed 129.89 samples/sec Loss 0.7869 Epoch: 19 Global Step: 196825 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:12:21,662-Speed 128.76 samples/sec Loss 0.7191 Epoch: 19 Global Step: 196830 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:12:41,464-Speed 129.29 samples/sec Loss 0.7300 Epoch: 19 Global Step: 196835 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:13:01,306-Speed 129.02 samples/sec Loss 0.6981 Epoch: 19 Global Step: 196840 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:13:21,073-Speed 129.51 samples/sec Loss 0.7058 Epoch: 19 Global Step: 196845 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:13:40,986-Speed 128.57 samples/sec Loss 0.7219 Epoch: 19 Global Step: 196850 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:14:00,729-Speed 129.67 samples/sec Loss 0.7078 Epoch: 19 Global Step: 196855 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:14:20,531-Speed 129.29 samples/sec Loss 0.7173 Epoch: 19 Global Step: 196860 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:14:40,551-Speed 127.88 samples/sec Loss 0.8655 Epoch: 19 Global Step: 196865 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:15:00,336-Speed 129.40 samples/sec Loss 0.7471 Epoch: 19 Global Step: 196870 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:15:20,198-Speed 128.90 samples/sec Loss 0.7017 Epoch: 19 Global Step: 196875 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:15:40,300-Speed 127.36 samples/sec Loss 0.6759 Epoch: 19 Global Step: 196880 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:16:00,012-Speed 129.88 samples/sec Loss 0.7288 Epoch: 19 Global Step: 196885 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:16:19,774-Speed 129.55 samples/sec Loss 0.7046 Epoch: 19 Global Step: 196890 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:16:39,626-Speed 128.96 samples/sec Loss 0.6859 Epoch: 19 Global Step: 196895 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:16:59,376-Speed 129.63 samples/sec Loss 0.7262 Epoch: 19 Global Step: 196900 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:17:19,152-Speed 129.45 samples/sec Loss 0.6529 Epoch: 19 Global Step: 196905 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:17:39,217-Speed 127.59 samples/sec Loss 0.6602 Epoch: 19 Global Step: 196910 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:17:59,034-Speed 129.19 samples/sec Loss 0.6528 Epoch: 19 Global Step: 196915 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:18:18,857-Speed 129.15 samples/sec Loss 0.8165 Epoch: 19 Global Step: 196920 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:18:38,863-Speed 127.97 samples/sec Loss 0.7773 Epoch: 19 Global Step: 196925 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:18:58,482-Speed 130.49 samples/sec Loss 0.6560 Epoch: 19 Global Step: 196930 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:19:18,390-Speed 128.60 samples/sec Loss 0.8048 Epoch: 19 Global Step: 196935 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:19:38,100-Speed 129.89 samples/sec Loss 0.6238 Epoch: 19 Global Step: 196940 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:19:57,845-Speed 129.66 samples/sec Loss 0.7699 Epoch: 19 Global Step: 196945 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:20:17,725-Speed 128.77 samples/sec Loss 0.7556 Epoch: 19 Global Step: 196950 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:20:37,801-Speed 127.52 samples/sec Loss 0.8096 Epoch: 19 Global Step: 196955 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:20:57,698-Speed 128.67 samples/sec Loss 0.5702 Epoch: 19 Global Step: 196960 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:21:17,433-Speed 129.72 samples/sec Loss 0.6490 Epoch: 19 Global Step: 196965 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:21:37,539-Speed 127.33 samples/sec Loss 0.7175 Epoch: 19 Global Step: 196970 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:21:57,264-Speed 129.79 samples/sec Loss 0.6590 Epoch: 19 Global Step: 196975 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:22:17,224-Speed 128.26 samples/sec Loss 0.7938 Epoch: 19 Global Step: 196980 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:22:36,888-Speed 130.19 samples/sec Loss 0.8040 Epoch: 19 Global Step: 196985 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:22:56,654-Speed 129.53 samples/sec Loss 0.7176 Epoch: 19 Global Step: 196990 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:23:16,510-Speed 128.94 samples/sec Loss 0.6637 Epoch: 19 Global Step: 196995 Fp16 Grad Scale: 8192 Required: 63 hours Training: 2021-06-16 15:23:36,488-Speed 128.14 samples/sec Loss 0.7589 Epoch: 19 Global Step: 197000 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:23:56,343-Speed 128.94 samples/sec Loss 0.7548 Epoch: 19 Global Step: 197005 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:24:16,066-Speed 129.80 samples/sec Loss 0.6698 Epoch: 19 Global Step: 197010 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:24:36,142-Speed 127.52 samples/sec Loss 0.7535 Epoch: 19 Global Step: 197015 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:24:55,866-Speed 129.79 samples/sec Loss 0.7166 Epoch: 19 Global Step: 197020 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:25:15,719-Speed 128.96 samples/sec Loss 0.7589 Epoch: 19 Global Step: 197025 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:25:35,328-Speed 130.55 samples/sec Loss 0.7522 Epoch: 19 Global Step: 197030 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:25:55,112-Speed 129.41 samples/sec Loss 0.6804 Epoch: 19 Global Step: 197035 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:26:15,058-Speed 128.35 samples/sec Loss 0.7559 Epoch: 19 Global Step: 197040 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:26:35,114-Speed 127.65 samples/sec Loss 0.7008 Epoch: 19 Global Step: 197045 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:26:54,950-Speed 129.07 samples/sec Loss 0.7670 Epoch: 19 Global Step: 197050 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:27:14,678-Speed 129.77 samples/sec Loss 0.7612 Epoch: 19 Global Step: 197055 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:27:34,652-Speed 128.17 samples/sec Loss 0.8239 Epoch: 19 Global Step: 197060 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:27:54,465-Speed 129.21 samples/sec Loss 0.6262 Epoch: 19 Global Step: 197065 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:28:13,888-Speed 131.81 samples/sec Loss 0.7530 Epoch: 19 Global Step: 197070 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:28:33,494-Speed 130.58 samples/sec Loss 0.6655 Epoch: 19 Global Step: 197075 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:28:53,417-Speed 128.50 samples/sec Loss 0.6514 Epoch: 19 Global Step: 197080 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:29:13,032-Speed 130.52 samples/sec Loss 0.7437 Epoch: 19 Global Step: 197085 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:29:32,864-Speed 129.09 samples/sec Loss 0.6861 Epoch: 19 Global Step: 197090 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:29:52,721-Speed 128.93 samples/sec Loss 0.7709 Epoch: 19 Global Step: 197095 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:30:12,369-Speed 130.29 samples/sec Loss 0.7918 Epoch: 19 Global Step: 197100 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:30:32,436-Speed 127.58 samples/sec Loss 0.8529 Epoch: 19 Global Step: 197105 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:30:51,947-Speed 131.21 samples/sec Loss 0.7759 Epoch: 19 Global Step: 197110 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:31:11,689-Speed 129.68 samples/sec Loss 0.7248 Epoch: 19 Global Step: 197115 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:31:31,239-Speed 130.95 samples/sec Loss 0.6979 Epoch: 19 Global Step: 197120 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:31:50,849-Speed 130.55 samples/sec Loss 0.7082 Epoch: 19 Global Step: 197125 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:32:10,350-Speed 131.28 samples/sec Loss 0.6751 Epoch: 19 Global Step: 197130 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:32:29,906-Speed 130.91 samples/sec Loss 0.7429 Epoch: 19 Global Step: 197135 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:32:49,401-Speed 131.33 samples/sec Loss 0.7861 Epoch: 19 Global Step: 197140 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:33:08,939-Speed 131.03 samples/sec Loss 0.7041 Epoch: 19 Global Step: 197145 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:33:29,195-Speed 126.39 samples/sec Loss 0.6646 Epoch: 19 Global Step: 197150 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:33:48,847-Speed 130.28 samples/sec Loss 0.7869 Epoch: 19 Global Step: 197155 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:34:08,275-Speed 131.78 samples/sec Loss 0.7241 Epoch: 19 Global Step: 197160 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:34:27,795-Speed 131.15 samples/sec Loss 0.7151 Epoch: 19 Global Step: 197165 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:34:47,376-Speed 130.75 samples/sec Loss 0.7775 Epoch: 19 Global Step: 197170 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:35:06,822-Speed 131.66 samples/sec Loss 0.6883 Epoch: 19 Global Step: 197175 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:35:26,428-Speed 130.58 samples/sec Loss 0.6205 Epoch: 19 Global Step: 197180 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:35:46,047-Speed 130.49 samples/sec Loss 0.6986 Epoch: 19 Global Step: 197185 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:36:05,594-Speed 130.97 samples/sec Loss 0.7921 Epoch: 19 Global Step: 197190 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:36:25,403-Speed 129.24 samples/sec Loss 0.6830 Epoch: 19 Global Step: 197195 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:36:45,044-Speed 130.35 samples/sec Loss 0.7857 Epoch: 19 Global Step: 197200 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:37:04,118-Speed 134.22 samples/sec Loss 0.7548 Epoch: 19 Global Step: 197205 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:37:23,603-Speed 131.39 samples/sec Loss 0.7741 Epoch: 19 Global Step: 197210 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:37:43,193-Speed 130.69 samples/sec Loss 0.7643 Epoch: 19 Global Step: 197215 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:38:02,736-Speed 131.00 samples/sec Loss 0.7574 Epoch: 19 Global Step: 197220 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:38:22,341-Speed 130.59 samples/sec Loss 0.7797 Epoch: 19 Global Step: 197225 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:38:41,889-Speed 130.97 samples/sec Loss 0.7121 Epoch: 19 Global Step: 197230 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:39:01,506-Speed 130.51 samples/sec Loss 0.7095 Epoch: 19 Global Step: 197235 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:39:21,261-Speed 129.59 samples/sec Loss 0.7548 Epoch: 19 Global Step: 197240 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:39:40,897-Speed 130.38 samples/sec Loss 0.8612 Epoch: 19 Global Step: 197245 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:40:00,428-Speed 131.08 samples/sec Loss 0.7346 Epoch: 19 Global Step: 197250 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:40:19,948-Speed 131.15 samples/sec Loss 0.7460 Epoch: 19 Global Step: 197255 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:40:39,503-Speed 130.92 samples/sec Loss 0.7975 Epoch: 19 Global Step: 197260 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:40:58,996-Speed 131.34 samples/sec Loss 0.6460 Epoch: 19 Global Step: 197265 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:41:18,595-Speed 130.63 samples/sec Loss 0.7676 Epoch: 19 Global Step: 197270 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:41:38,141-Speed 130.98 samples/sec Loss 0.7107 Epoch: 19 Global Step: 197275 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:41:57,664-Speed 131.14 samples/sec Loss 0.6936 Epoch: 19 Global Step: 197280 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:42:17,426-Speed 129.55 samples/sec Loss 0.7303 Epoch: 19 Global Step: 197285 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:42:37,068-Speed 130.34 samples/sec Loss 0.7530 Epoch: 19 Global Step: 197290 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:42:56,649-Speed 130.75 samples/sec Loss 0.6623 Epoch: 19 Global Step: 197295 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:43:16,163-Speed 131.20 samples/sec Loss 0.8828 Epoch: 19 Global Step: 197300 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 15:43:35,724-Speed 130.89 samples/sec Loss 0.7699 Epoch: 19 Global Step: 197305 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:43:55,296-Speed 130.80 samples/sec Loss 0.7304 Epoch: 19 Global Step: 197310 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:44:14,919-Speed 130.47 samples/sec Loss 0.7653 Epoch: 19 Global Step: 197315 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:44:34,431-Speed 131.21 samples/sec Loss 0.8076 Epoch: 19 Global Step: 197320 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:44:53,917-Speed 131.38 samples/sec Loss 0.7289 Epoch: 19 Global Step: 197325 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:45:13,721-Speed 129.27 samples/sec Loss 0.7487 Epoch: 19 Global Step: 197330 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:45:33,269-Speed 130.97 samples/sec Loss 0.6079 Epoch: 19 Global Step: 197335 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:45:52,971-Speed 129.94 samples/sec Loss 0.6687 Epoch: 19 Global Step: 197340 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:46:12,593-Speed 130.47 samples/sec Loss 0.6949 Epoch: 19 Global Step: 197345 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:46:32,202-Speed 130.56 samples/sec Loss 0.7607 Epoch: 19 Global Step: 197350 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:46:51,653-Speed 131.62 samples/sec Loss 0.7272 Epoch: 19 Global Step: 197355 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:47:11,266-Speed 130.54 samples/sec Loss 0.7495 Epoch: 19 Global Step: 197360 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:47:30,820-Speed 130.93 samples/sec Loss 0.7759 Epoch: 19 Global Step: 197365 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:47:50,274-Speed 131.60 samples/sec Loss 0.7824 Epoch: 19 Global Step: 197370 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:48:10,068-Speed 129.34 samples/sec Loss 0.7850 Epoch: 19 Global Step: 197375 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:48:29,547-Speed 131.43 samples/sec Loss 0.7447 Epoch: 19 Global Step: 197380 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:48:49,257-Speed 129.89 samples/sec Loss 0.7697 Epoch: 19 Global Step: 197385 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:49:08,688-Speed 131.76 samples/sec Loss 0.7022 Epoch: 19 Global Step: 197390 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:49:28,907-Speed 126.62 samples/sec Loss 0.8458 Epoch: 19 Global Step: 197395 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:49:48,442-Speed 131.06 samples/sec Loss 0.6526 Epoch: 19 Global Step: 197400 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:50:08,064-Speed 130.47 samples/sec Loss 0.6963 Epoch: 19 Global Step: 197405 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:50:27,632-Speed 130.83 samples/sec Loss 0.7613 Epoch: 19 Global Step: 197410 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:50:47,073-Speed 131.69 samples/sec Loss 0.7218 Epoch: 19 Global Step: 197415 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:51:06,871-Speed 129.31 samples/sec Loss 0.7416 Epoch: 19 Global Step: 197420 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:51:26,536-Speed 130.19 samples/sec Loss 0.8099 Epoch: 19 Global Step: 197425 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:51:46,255-Speed 129.83 samples/sec Loss 0.7665 Epoch: 19 Global Step: 197430 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:52:05,781-Speed 131.11 samples/sec Loss 0.6552 Epoch: 19 Global Step: 197435 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:52:25,507-Speed 129.79 samples/sec Loss 0.6813 Epoch: 19 Global Step: 197440 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:52:45,030-Speed 131.14 samples/sec Loss 0.6839 Epoch: 19 Global Step: 197445 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:53:04,731-Speed 129.95 samples/sec Loss 0.7511 Epoch: 19 Global Step: 197450 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:53:24,313-Speed 130.74 samples/sec Loss 0.8071 Epoch: 19 Global Step: 197455 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:53:43,851-Speed 131.04 samples/sec Loss 0.7791 Epoch: 19 Global Step: 197460 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:54:03,638-Speed 129.38 samples/sec Loss 0.6800 Epoch: 19 Global Step: 197465 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:54:23,201-Speed 130.86 samples/sec Loss 0.6622 Epoch: 19 Global Step: 197470 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:54:42,908-Speed 129.91 samples/sec Loss 0.8025 Epoch: 19 Global Step: 197475 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:55:02,457-Speed 130.96 samples/sec Loss 0.7574 Epoch: 19 Global Step: 197480 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:55:22,066-Speed 130.56 samples/sec Loss 0.6984 Epoch: 19 Global Step: 197485 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:55:41,851-Speed 129.40 samples/sec Loss 0.7896 Epoch: 19 Global Step: 197490 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:56:01,703-Speed 128.96 samples/sec Loss 0.6996 Epoch: 19 Global Step: 197495 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:56:21,668-Speed 128.24 samples/sec Loss 0.7739 Epoch: 19 Global Step: 197500 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:56:41,229-Speed 130.87 samples/sec Loss 0.7371 Epoch: 19 Global Step: 197505 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:57:01,002-Speed 129.48 samples/sec Loss 0.7283 Epoch: 19 Global Step: 197510 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:57:20,796-Speed 129.34 samples/sec Loss 0.7651 Epoch: 19 Global Step: 197515 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:57:40,321-Speed 131.12 samples/sec Loss 0.9279 Epoch: 19 Global Step: 197520 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:57:59,859-Speed 131.03 samples/sec Loss 0.8277 Epoch: 19 Global Step: 197525 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:58:19,480-Speed 130.48 samples/sec Loss 0.7232 Epoch: 19 Global Step: 197530 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:58:39,028-Speed 130.97 samples/sec Loss 0.8289 Epoch: 19 Global Step: 197535 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:58:58,504-Speed 131.45 samples/sec Loss 0.7149 Epoch: 19 Global Step: 197540 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:59:18,071-Speed 130.84 samples/sec Loss 0.7281 Epoch: 19 Global Step: 197545 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:59:37,639-Speed 130.84 samples/sec Loss 0.7864 Epoch: 19 Global Step: 197550 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 15:59:57,407-Speed 129.51 samples/sec Loss 0.7242 Epoch: 19 Global Step: 197555 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:00:17,122-Speed 129.86 samples/sec Loss 0.7406 Epoch: 19 Global Step: 197560 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:00:36,630-Speed 131.23 samples/sec Loss 0.7620 Epoch: 19 Global Step: 197565 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:00:56,197-Speed 130.84 samples/sec Loss 0.7085 Epoch: 19 Global Step: 197570 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:01:15,427-Speed 133.13 samples/sec Loss 0.7213 Epoch: 19 Global Step: 197575 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:01:35,194-Speed 129.52 samples/sec Loss 0.7556 Epoch: 19 Global Step: 197580 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:01:54,991-Speed 129.32 samples/sec Loss 0.7797 Epoch: 19 Global Step: 197585 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:02:14,749-Speed 129.57 samples/sec Loss 0.7654 Epoch: 19 Global Step: 197590 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:02:34,694-Speed 128.36 samples/sec Loss 0.6826 Epoch: 19 Global Step: 197595 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:02:54,321-Speed 130.44 samples/sec Loss 0.7260 Epoch: 19 Global Step: 197600 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:03:14,174-Speed 128.96 samples/sec Loss 0.7045 Epoch: 19 Global Step: 197605 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:03:34,138-Speed 128.23 samples/sec Loss 0.7734 Epoch: 19 Global Step: 197610 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:03:54,036-Speed 128.66 samples/sec Loss 0.7002 Epoch: 19 Global Step: 197615 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:04:13,657-Speed 130.48 samples/sec Loss 0.6714 Epoch: 19 Global Step: 197620 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:04:33,754-Speed 127.39 samples/sec Loss 0.7076 Epoch: 19 Global Step: 197625 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:04:53,801-Speed 127.70 samples/sec Loss 0.7359 Epoch: 19 Global Step: 197630 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:05:13,774-Speed 128.18 samples/sec Loss 0.6932 Epoch: 19 Global Step: 197635 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:05:33,527-Speed 129.61 samples/sec Loss 0.8143 Epoch: 19 Global Step: 197640 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:05:53,394-Speed 128.86 samples/sec Loss 0.7721 Epoch: 19 Global Step: 197645 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:06:13,137-Speed 129.67 samples/sec Loss 0.7534 Epoch: 19 Global Step: 197650 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:06:33,174-Speed 127.77 samples/sec Loss 0.7190 Epoch: 19 Global Step: 197655 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:06:53,032-Speed 128.92 samples/sec Loss 0.7531 Epoch: 19 Global Step: 197660 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:07:12,719-Speed 130.05 samples/sec Loss 0.8841 Epoch: 19 Global Step: 197665 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:07:32,787-Speed 127.57 samples/sec Loss 0.8258 Epoch: 19 Global Step: 197670 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:07:52,524-Speed 129.71 samples/sec Loss 0.6649 Epoch: 19 Global Step: 197675 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:08:12,406-Speed 128.76 samples/sec Loss 0.7162 Epoch: 19 Global Step: 197680 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:08:32,331-Speed 128.49 samples/sec Loss 0.7445 Epoch: 19 Global Step: 197685 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:08:52,198-Speed 128.86 samples/sec Loss 0.6780 Epoch: 19 Global Step: 197690 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:09:11,877-Speed 130.10 samples/sec Loss 0.6506 Epoch: 19 Global Step: 197695 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:09:31,254-Speed 132.12 samples/sec Loss 0.7708 Epoch: 19 Global Step: 197700 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:09:51,041-Speed 129.38 samples/sec Loss 0.7580 Epoch: 19 Global Step: 197705 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:10:11,105-Speed 127.60 samples/sec Loss 0.7282 Epoch: 19 Global Step: 197710 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:10:31,098-Speed 128.05 samples/sec Loss 0.7470 Epoch: 19 Global Step: 197715 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:10:50,881-Speed 129.41 samples/sec Loss 0.7071 Epoch: 19 Global Step: 197720 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:11:10,660-Speed 129.44 samples/sec Loss 0.7035 Epoch: 19 Global Step: 197725 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:11:30,351-Speed 130.01 samples/sec Loss 0.8219 Epoch: 19 Global Step: 197730 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:11:50,017-Speed 130.18 samples/sec Loss 0.7575 Epoch: 19 Global Step: 197735 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:12:09,455-Speed 131.71 samples/sec Loss 0.7409 Epoch: 19 Global Step: 197740 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:12:29,058-Speed 130.60 samples/sec Loss 0.8120 Epoch: 19 Global Step: 197745 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:12:48,659-Speed 130.61 samples/sec Loss 0.7728 Epoch: 19 Global Step: 197750 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:13:08,217-Speed 130.90 samples/sec Loss 0.8492 Epoch: 19 Global Step: 197755 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:13:28,016-Speed 129.31 samples/sec Loss 0.6921 Epoch: 19 Global Step: 197760 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:13:47,639-Speed 130.47 samples/sec Loss 0.7465 Epoch: 19 Global Step: 197765 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:14:07,256-Speed 130.50 samples/sec Loss 0.7432 Epoch: 19 Global Step: 197770 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:14:26,805-Speed 130.96 samples/sec Loss 0.7399 Epoch: 19 Global Step: 197775 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:14:46,391-Speed 130.71 samples/sec Loss 0.7630 Epoch: 19 Global Step: 197780 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:15:05,885-Speed 131.33 samples/sec Loss 0.7614 Epoch: 19 Global Step: 197785 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:15:25,498-Speed 130.54 samples/sec Loss 0.7100 Epoch: 19 Global Step: 197790 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:15:45,098-Speed 130.62 samples/sec Loss 0.7527 Epoch: 19 Global Step: 197795 Fp16 Grad Scale: 4096 Required: 62 hours Training: 2021-06-16 16:16:04,661-Speed 130.87 samples/sec Loss 0.7553 Epoch: 19 Global Step: 197800 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:16:24,471-Speed 129.24 samples/sec Loss 0.7588 Epoch: 19 Global Step: 197805 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:16:44,072-Speed 130.61 samples/sec Loss 0.6669 Epoch: 19 Global Step: 197810 Fp16 Grad Scale: 8192 Required: 62 hours Training: 2021-06-16 16:17:03,671-Speed 130.62 samples/sec Loss 0.8081 Epoch: 19 Global Step: 197815 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:17:23,221-Speed 130.95 samples/sec Loss 0.7747 Epoch: 19 Global Step: 197820 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:17:42,833-Speed 130.54 samples/sec Loss 0.5990 Epoch: 19 Global Step: 197825 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:18:02,352-Speed 131.16 samples/sec Loss 0.7789 Epoch: 19 Global Step: 197830 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:18:21,880-Speed 131.10 samples/sec Loss 0.7108 Epoch: 19 Global Step: 197835 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:18:41,523-Speed 130.33 samples/sec Loss 0.6924 Epoch: 19 Global Step: 197840 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:19:01,087-Speed 130.86 samples/sec Loss 0.7654 Epoch: 19 Global Step: 197845 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:19:20,876-Speed 129.37 samples/sec Loss 0.7626 Epoch: 19 Global Step: 197850 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:19:40,534-Speed 130.23 samples/sec Loss 0.6913 Epoch: 19 Global Step: 197855 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:20:00,092-Speed 130.90 samples/sec Loss 0.6987 Epoch: 19 Global Step: 197860 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:20:19,649-Speed 130.91 samples/sec Loss 0.7286 Epoch: 19 Global Step: 197865 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:20:39,249-Speed 130.62 samples/sec Loss 0.6792 Epoch: 19 Global Step: 197870 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:20:58,388-Speed 133.76 samples/sec Loss 0.8007 Epoch: 19 Global Step: 197875 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:21:17,875-Speed 131.38 samples/sec Loss 0.8077 Epoch: 19 Global Step: 197880 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:21:37,448-Speed 130.80 samples/sec Loss 0.6643 Epoch: 19 Global Step: 197885 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:21:56,989-Speed 131.02 samples/sec Loss 0.7062 Epoch: 19 Global Step: 197890 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:22:16,810-Speed 129.17 samples/sec Loss 0.6598 Epoch: 19 Global Step: 197895 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:22:36,471-Speed 130.21 samples/sec Loss 0.6939 Epoch: 19 Global Step: 197900 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:22:56,003-Speed 131.07 samples/sec Loss 0.7259 Epoch: 19 Global Step: 197905 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:23:15,625-Speed 130.47 samples/sec Loss 0.7233 Epoch: 19 Global Step: 197910 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:23:35,226-Speed 130.62 samples/sec Loss 0.7713 Epoch: 19 Global Step: 197915 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:23:54,765-Speed 131.03 samples/sec Loss 0.7567 Epoch: 19 Global Step: 197920 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:24:14,377-Speed 130.54 samples/sec Loss 0.8082 Epoch: 19 Global Step: 197925 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:24:33,905-Speed 131.10 samples/sec Loss 0.8243 Epoch: 19 Global Step: 197930 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:24:53,497-Speed 130.67 samples/sec Loss 0.7579 Epoch: 19 Global Step: 197935 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:25:13,245-Speed 129.65 samples/sec Loss 0.7567 Epoch: 19 Global Step: 197940 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:25:32,879-Speed 130.39 samples/sec Loss 0.6783 Epoch: 19 Global Step: 197945 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:25:52,405-Speed 131.11 samples/sec Loss 0.7425 Epoch: 19 Global Step: 197950 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:26:12,032-Speed 130.44 samples/sec Loss 0.7620 Epoch: 19 Global Step: 197955 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:26:31,612-Speed 130.75 samples/sec Loss 0.7671 Epoch: 19 Global Step: 197960 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:26:51,165-Speed 130.94 samples/sec Loss 0.7589 Epoch: 19 Global Step: 197965 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:27:10,759-Speed 130.66 samples/sec Loss 0.7247 Epoch: 19 Global Step: 197970 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:27:30,275-Speed 131.18 samples/sec Loss 0.7655 Epoch: 19 Global Step: 197975 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:27:49,878-Speed 130.60 samples/sec Loss 0.7018 Epoch: 19 Global Step: 197980 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:28:09,681-Speed 129.28 samples/sec Loss 0.8485 Epoch: 19 Global Step: 197985 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:28:29,381-Speed 129.96 samples/sec Loss 0.7416 Epoch: 19 Global Step: 197990 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:28:48,879-Speed 131.30 samples/sec Loss 0.7033 Epoch: 19 Global Step: 197995 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:29:08,436-Speed 130.91 samples/sec Loss 0.7144 Epoch: 19 Global Step: 198000 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:29:28,009-Speed 130.80 samples/sec Loss 0.7507 Epoch: 19 Global Step: 198005 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:29:47,534-Speed 131.13 samples/sec Loss 0.7341 Epoch: 19 Global Step: 198010 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:30:07,152-Speed 130.49 samples/sec Loss 0.8696 Epoch: 19 Global Step: 198015 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:30:26,687-Speed 131.06 samples/sec Loss 0.7024 Epoch: 19 Global Step: 198020 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:30:46,241-Speed 130.92 samples/sec Loss 0.6524 Epoch: 19 Global Step: 198025 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:31:06,043-Speed 129.29 samples/sec Loss 0.7189 Epoch: 19 Global Step: 198030 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:31:25,499-Speed 131.59 samples/sec Loss 0.7391 Epoch: 19 Global Step: 198035 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:31:45,076-Speed 130.78 samples/sec Loss 0.7616 Epoch: 19 Global Step: 198040 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:32:04,648-Speed 130.80 samples/sec Loss 0.7042 Epoch: 19 Global Step: 198045 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:32:24,249-Speed 130.61 samples/sec Loss 0.8646 Epoch: 19 Global Step: 198050 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:32:43,682-Speed 131.74 samples/sec Loss 0.7317 Epoch: 19 Global Step: 198055 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:33:03,166-Speed 131.40 samples/sec Loss 0.6887 Epoch: 19 Global Step: 198060 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:33:22,858-Speed 130.01 samples/sec Loss 0.7722 Epoch: 19 Global Step: 198065 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:33:42,433-Speed 130.79 samples/sec Loss 0.7736 Epoch: 19 Global Step: 198070 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:34:02,264-Speed 129.09 samples/sec Loss 0.6375 Epoch: 19 Global Step: 198075 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:34:21,870-Speed 130.58 samples/sec Loss 0.7571 Epoch: 19 Global Step: 198080 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:34:41,402-Speed 131.07 samples/sec Loss 0.8296 Epoch: 19 Global Step: 198085 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:35:00,906-Speed 131.26 samples/sec Loss 0.7337 Epoch: 19 Global Step: 198090 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:35:20,071-Speed 133.58 samples/sec Loss 0.7479 Epoch: 19 Global Step: 198095 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:35:39,668-Speed 130.64 samples/sec Loss 0.6914 Epoch: 19 Global Step: 198100 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:35:59,387-Speed 129.84 samples/sec Loss 0.7316 Epoch: 19 Global Step: 198105 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:36:19,294-Speed 128.60 samples/sec Loss 0.7555 Epoch: 19 Global Step: 198110 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:36:38,951-Speed 130.25 samples/sec Loss 0.7370 Epoch: 19 Global Step: 198115 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:36:58,832-Speed 128.78 samples/sec Loss 0.7976 Epoch: 19 Global Step: 198120 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:37:18,391-Speed 130.89 samples/sec Loss 0.7283 Epoch: 19 Global Step: 198125 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:37:38,502-Speed 127.30 samples/sec Loss 0.7535 Epoch: 19 Global Step: 198130 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:37:58,236-Speed 129.73 samples/sec Loss 0.8014 Epoch: 19 Global Step: 198135 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:38:17,959-Speed 129.81 samples/sec Loss 0.7523 Epoch: 19 Global Step: 198140 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:38:37,933-Speed 128.17 samples/sec Loss 0.8163 Epoch: 19 Global Step: 198145 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:38:57,825-Speed 128.70 samples/sec Loss 0.7113 Epoch: 19 Global Step: 198150 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:39:17,634-Speed 129.23 samples/sec Loss 0.6920 Epoch: 19 Global Step: 198155 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:39:37,290-Speed 130.25 samples/sec Loss 0.7549 Epoch: 19 Global Step: 198160 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:39:57,173-Speed 128.76 samples/sec Loss 0.8301 Epoch: 19 Global Step: 198165 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:40:16,793-Speed 130.48 samples/sec Loss 0.7138 Epoch: 19 Global Step: 198170 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:40:37,021-Speed 126.56 samples/sec Loss 0.7888 Epoch: 19 Global Step: 198175 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:40:56,720-Speed 129.96 samples/sec Loss 0.6699 Epoch: 19 Global Step: 198180 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:41:16,494-Speed 129.47 samples/sec Loss 0.6671 Epoch: 19 Global Step: 198185 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:41:36,429-Speed 128.43 samples/sec Loss 0.7284 Epoch: 19 Global Step: 198190 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:41:56,394-Speed 128.23 samples/sec Loss 0.8037 Epoch: 19 Global Step: 198195 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:42:16,050-Speed 130.25 samples/sec Loss 0.7232 Epoch: 19 Global Step: 198200 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:42:35,974-Speed 128.49 samples/sec Loss 0.8083 Epoch: 19 Global Step: 198205 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:42:55,695-Speed 129.82 samples/sec Loss 0.7304 Epoch: 19 Global Step: 198210 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:43:15,334-Speed 130.36 samples/sec Loss 0.7265 Epoch: 19 Global Step: 198215 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:43:35,524-Speed 126.80 samples/sec Loss 0.8230 Epoch: 19 Global Step: 198220 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:43:55,254-Speed 129.76 samples/sec Loss 0.6970 Epoch: 19 Global Step: 198225 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:44:15,169-Speed 128.55 samples/sec Loss 0.6908 Epoch: 19 Global Step: 198230 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:44:35,013-Speed 129.01 samples/sec Loss 0.7145 Epoch: 19 Global Step: 198235 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:44:54,873-Speed 128.91 samples/sec Loss 0.7951 Epoch: 19 Global Step: 198240 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:45:14,595-Speed 129.81 samples/sec Loss 0.7695 Epoch: 19 Global Step: 198245 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:45:34,667-Speed 127.55 samples/sec Loss 0.6941 Epoch: 19 Global Step: 198250 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:45:53,935-Speed 132.88 samples/sec Loss 0.7189 Epoch: 19 Global Step: 198255 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:46:13,602-Speed 130.17 samples/sec Loss 0.7297 Epoch: 19 Global Step: 198260 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:46:33,526-Speed 128.50 samples/sec Loss 0.7118 Epoch: 19 Global Step: 198265 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:46:53,230-Speed 129.93 samples/sec Loss 0.8251 Epoch: 19 Global Step: 198270 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:47:13,131-Speed 128.64 samples/sec Loss 0.7053 Epoch: 19 Global Step: 198275 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:47:33,307-Speed 126.89 samples/sec Loss 0.6815 Epoch: 19 Global Step: 198280 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:47:53,189-Speed 128.76 samples/sec Loss 0.6037 Epoch: 19 Global Step: 198285 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:48:12,659-Speed 131.49 samples/sec Loss 0.8125 Epoch: 19 Global Step: 198290 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:48:32,256-Speed 130.64 samples/sec Loss 0.7636 Epoch: 19 Global Step: 198295 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:48:51,770-Speed 131.20 samples/sec Loss 0.7429 Epoch: 19 Global Step: 198300 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:49:11,378-Speed 130.56 samples/sec Loss 0.6784 Epoch: 19 Global Step: 198305 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:49:30,985-Speed 130.58 samples/sec Loss 0.7289 Epoch: 19 Global Step: 198310 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:49:50,458-Speed 131.47 samples/sec Loss 0.6748 Epoch: 19 Global Step: 198315 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:50:10,046-Speed 130.70 samples/sec Loss 0.7545 Epoch: 19 Global Step: 198320 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:50:29,916-Speed 128.85 samples/sec Loss 0.8331 Epoch: 19 Global Step: 198325 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:50:49,449-Speed 131.06 samples/sec Loss 0.6705 Epoch: 19 Global Step: 198330 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:51:09,011-Speed 130.87 samples/sec Loss 0.7811 Epoch: 19 Global Step: 198335 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:51:28,623-Speed 130.54 samples/sec Loss 0.6733 Epoch: 19 Global Step: 198340 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:51:48,204-Speed 130.75 samples/sec Loss 0.7501 Epoch: 19 Global Step: 198345 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:52:07,936-Speed 129.75 samples/sec Loss 0.7662 Epoch: 19 Global Step: 198350 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:52:27,533-Speed 130.64 samples/sec Loss 0.7460 Epoch: 19 Global Step: 198355 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:52:47,053-Speed 131.15 samples/sec Loss 0.8388 Epoch: 19 Global Step: 198360 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:53:06,666-Speed 130.53 samples/sec Loss 0.7348 Epoch: 19 Global Step: 198365 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 16:53:26,083-Speed 131.85 samples/sec Loss 0.6499 Epoch: 19 Global Step: 198370 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:53:45,659-Speed 130.78 samples/sec Loss 0.7261 Epoch: 19 Global Step: 198375 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:54:05,249-Speed 130.69 samples/sec Loss 0.7485 Epoch: 19 Global Step: 198380 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:54:24,863-Speed 130.53 samples/sec Loss 0.7695 Epoch: 19 Global Step: 198385 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:54:44,425-Speed 130.87 samples/sec Loss 0.7567 Epoch: 19 Global Step: 198390 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:55:03,970-Speed 130.98 samples/sec Loss 0.8376 Epoch: 19 Global Step: 198395 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:55:23,583-Speed 130.54 samples/sec Loss 0.7070 Epoch: 19 Global Step: 198400 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:55:43,047-Speed 131.53 samples/sec Loss 0.7475 Epoch: 19 Global Step: 198405 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:56:02,656-Speed 130.56 samples/sec Loss 0.7962 Epoch: 19 Global Step: 198410 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:56:22,446-Speed 129.37 samples/sec Loss 0.7877 Epoch: 19 Global Step: 198415 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:56:42,090-Speed 130.33 samples/sec Loss 0.8617 Epoch: 19 Global Step: 198420 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:57:01,584-Speed 131.33 samples/sec Loss 0.7516 Epoch: 19 Global Step: 198425 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:57:21,194-Speed 130.55 samples/sec Loss 0.7632 Epoch: 19 Global Step: 198430 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:57:40,722-Speed 131.10 samples/sec Loss 0.6535 Epoch: 19 Global Step: 198435 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:58:00,213-Speed 131.35 samples/sec Loss 0.7170 Epoch: 19 Global Step: 198440 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:58:19,832-Speed 130.49 samples/sec Loss 0.8317 Epoch: 19 Global Step: 198445 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:58:39,391-Speed 130.90 samples/sec Loss 0.7466 Epoch: 19 Global Step: 198450 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:58:58,977-Speed 130.71 samples/sec Loss 0.6797 Epoch: 19 Global Step: 198455 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:59:18,751-Speed 129.47 samples/sec Loss 0.7896 Epoch: 19 Global Step: 198460 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:59:38,404-Speed 130.27 samples/sec Loss 0.7119 Epoch: 19 Global Step: 198465 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 16:59:57,927-Speed 131.13 samples/sec Loss 0.7081 Epoch: 19 Global Step: 198470 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 17:00:17,539-Speed 130.54 samples/sec Loss 0.8478 Epoch: 19 Global Step: 198475 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 17:00:37,070-Speed 131.08 samples/sec Loss 0.7652 Epoch: 19 Global Step: 198480 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 17:00:56,819-Speed 129.63 samples/sec Loss 0.6403 Epoch: 19 Global Step: 198485 Fp16 Grad Scale: 8192 Required: 61 hours Training: 2021-06-16 17:01:16,219-Speed 131.97 samples/sec Loss 0.7262 Epoch: 19 Global Step: 198490 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:01:35,944-Speed 129.79 samples/sec Loss 0.8079 Epoch: 19 Global Step: 198495 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:01:55,869-Speed 128.49 samples/sec Loss 0.6614 Epoch: 19 Global Step: 198500 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:02:16,286-Speed 125.39 samples/sec Loss 0.7066 Epoch: 19 Global Step: 198505 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:02:36,051-Speed 129.53 samples/sec Loss 0.7359 Epoch: 19 Global Step: 198510 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:02:56,177-Speed 127.21 samples/sec Loss 0.7353 Epoch: 19 Global Step: 198515 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:03:16,299-Speed 127.23 samples/sec Loss 0.7284 Epoch: 19 Global Step: 198520 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:03:36,190-Speed 128.71 samples/sec Loss 0.7924 Epoch: 19 Global Step: 198525 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:03:56,264-Speed 127.54 samples/sec Loss 0.7676 Epoch: 19 Global Step: 198530 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:04:16,062-Speed 129.31 samples/sec Loss 0.7898 Epoch: 19 Global Step: 198535 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:04:35,775-Speed 129.87 samples/sec Loss 0.6951 Epoch: 19 Global Step: 198540 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:04:55,637-Speed 128.90 samples/sec Loss 0.7345 Epoch: 19 Global Step: 198545 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:05:15,494-Speed 128.93 samples/sec Loss 0.7965 Epoch: 19 Global Step: 198550 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:05:35,368-Speed 128.82 samples/sec Loss 0.7527 Epoch: 19 Global Step: 198555 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:05:55,436-Speed 127.57 samples/sec Loss 0.7819 Epoch: 19 Global Step: 198560 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:06:15,320-Speed 128.75 samples/sec Loss 0.6907 Epoch: 19 Global Step: 198565 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:06:35,042-Speed 129.81 samples/sec Loss 0.6934 Epoch: 19 Global Step: 198570 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:06:55,079-Speed 127.77 samples/sec Loss 0.6609 Epoch: 19 Global Step: 198575 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:07:14,868-Speed 129.37 samples/sec Loss 0.6882 Epoch: 19 Global Step: 198580 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:07:34,822-Speed 128.30 samples/sec Loss 0.7631 Epoch: 19 Global Step: 198585 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:07:54,122-Speed 132.65 samples/sec Loss 0.7401 Epoch: 19 Global Step: 198590 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:08:13,871-Speed 129.64 samples/sec Loss 0.7544 Epoch: 19 Global Step: 198595 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:08:33,717-Speed 129.00 samples/sec Loss 0.7289 Epoch: 19 Global Step: 198600 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:08:53,500-Speed 129.41 samples/sec Loss 0.6590 Epoch: 19 Global Step: 198605 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:09:13,316-Speed 129.19 samples/sec Loss 0.7171 Epoch: 19 Global Step: 198610 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:09:33,016-Speed 129.95 samples/sec Loss 0.7450 Epoch: 19 Global Step: 198615 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:09:53,072-Speed 127.65 samples/sec Loss 0.7183 Epoch: 19 Global Step: 198620 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:10:12,629-Speed 130.90 samples/sec Loss 0.8523 Epoch: 19 Global Step: 198625 Fp16 Grad Scale: 4096 Required: 61 hours Training: 2021-06-16 17:10:32,342-Speed 129.88 samples/sec Loss 0.7260 Epoch: 19 Global Step: 198630 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:10:51,883-Speed 131.01 samples/sec Loss 0.8163 Epoch: 19 Global Step: 198635 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:11:11,498-Speed 130.52 samples/sec Loss 0.7585 Epoch: 19 Global Step: 198640 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:11:31,050-Speed 130.94 samples/sec Loss 0.7461 Epoch: 19 Global Step: 198645 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:11:50,655-Speed 130.59 samples/sec Loss 0.8954 Epoch: 19 Global Step: 198650 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:12:10,160-Speed 131.25 samples/sec Loss 0.7828 Epoch: 19 Global Step: 198655 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:12:29,719-Speed 130.90 samples/sec Loss 0.7187 Epoch: 19 Global Step: 198660 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:12:49,517-Speed 129.31 samples/sec Loss 0.7361 Epoch: 19 Global Step: 198665 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:13:09,193-Speed 130.11 samples/sec Loss 0.7982 Epoch: 19 Global Step: 198670 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:13:28,621-Speed 131.78 samples/sec Loss 0.7863 Epoch: 19 Global Step: 198675 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:13:48,169-Speed 130.96 samples/sec Loss 0.6943 Epoch: 19 Global Step: 198680 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:14:07,740-Speed 130.81 samples/sec Loss 0.7471 Epoch: 19 Global Step: 198685 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:14:27,267-Speed 131.11 samples/sec Loss 0.8322 Epoch: 19 Global Step: 198690 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:14:46,876-Speed 130.56 samples/sec Loss 0.8717 Epoch: 19 Global Step: 198695 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:15:06,379-Speed 131.27 samples/sec Loss 0.7288 Epoch: 19 Global Step: 198700 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:15:25,914-Speed 131.06 samples/sec Loss 0.7138 Epoch: 19 Global Step: 198705 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:15:45,706-Speed 129.35 samples/sec Loss 0.7458 Epoch: 19 Global Step: 198710 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:16:05,387-Speed 130.09 samples/sec Loss 0.8205 Epoch: 19 Global Step: 198715 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:16:24,942-Speed 130.92 samples/sec Loss 0.7500 Epoch: 19 Global Step: 198720 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:16:44,478-Speed 131.05 samples/sec Loss 0.7462 Epoch: 19 Global Step: 198725 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:17:04,086-Speed 130.57 samples/sec Loss 0.8393 Epoch: 19 Global Step: 198730 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:17:23,638-Speed 130.94 samples/sec Loss 0.7364 Epoch: 19 Global Step: 198735 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:17:43,183-Speed 130.99 samples/sec Loss 0.7410 Epoch: 19 Global Step: 198740 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:18:02,744-Speed 130.88 samples/sec Loss 0.7655 Epoch: 19 Global Step: 198745 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:18:22,291-Speed 130.97 samples/sec Loss 0.9792 Epoch: 19 Global Step: 198750 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:18:42,081-Speed 129.37 samples/sec Loss 0.7479 Epoch: 19 Global Step: 198755 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:19:01,700-Speed 130.49 samples/sec Loss 0.7588 Epoch: 19 Global Step: 198760 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:19:21,322-Speed 130.48 samples/sec Loss 0.7138 Epoch: 19 Global Step: 198765 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:19:40,856-Speed 131.06 samples/sec Loss 0.7841 Epoch: 19 Global Step: 198770 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:20:00,453-Speed 130.64 samples/sec Loss 0.6809 Epoch: 19 Global Step: 198775 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:20:19,981-Speed 131.10 samples/sec Loss 0.8194 Epoch: 19 Global Step: 198780 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:20:39,480-Speed 131.30 samples/sec Loss 0.7165 Epoch: 19 Global Step: 198785 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:20:59,196-Speed 129.85 samples/sec Loss 0.7224 Epoch: 19 Global Step: 198790 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:21:18,629-Speed 131.74 samples/sec Loss 0.6803 Epoch: 19 Global Step: 198795 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:21:38,443-Speed 129.21 samples/sec Loss 0.7483 Epoch: 19 Global Step: 198800 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:21:57,990-Speed 130.97 samples/sec Loss 0.6421 Epoch: 19 Global Step: 198805 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:22:17,660-Speed 130.15 samples/sec Loss 0.6895 Epoch: 19 Global Step: 198810 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:22:37,142-Speed 131.41 samples/sec Loss 0.7176 Epoch: 19 Global Step: 198815 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:22:56,765-Speed 130.47 samples/sec Loss 0.6979 Epoch: 19 Global Step: 198820 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:23:16,283-Speed 131.17 samples/sec Loss 0.6871 Epoch: 19 Global Step: 198825 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:23:35,842-Speed 130.90 samples/sec Loss 0.7896 Epoch: 19 Global Step: 198830 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:23:55,459-Speed 130.51 samples/sec Loss 0.7838 Epoch: 19 Global Step: 198835 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:24:14,914-Speed 131.59 samples/sec Loss 0.7972 Epoch: 19 Global Step: 198840 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:24:34,773-Speed 128.92 samples/sec Loss 0.7392 Epoch: 19 Global Step: 198845 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:24:54,333-Speed 130.89 samples/sec Loss 0.6812 Epoch: 19 Global Step: 198850 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:25:14,030-Speed 129.97 samples/sec Loss 0.6344 Epoch: 19 Global Step: 198855 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:25:33,598-Speed 130.83 samples/sec Loss 0.7984 Epoch: 19 Global Step: 198860 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:25:53,217-Speed 130.50 samples/sec Loss 0.6876 Epoch: 19 Global Step: 198865 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:26:12,752-Speed 131.05 samples/sec Loss 0.7350 Epoch: 19 Global Step: 198870 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:26:32,302-Speed 130.95 samples/sec Loss 0.8541 Epoch: 19 Global Step: 198875 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:26:51,864-Speed 130.88 samples/sec Loss 0.7109 Epoch: 19 Global Step: 198880 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:27:11,411-Speed 130.98 samples/sec Loss 0.7080 Epoch: 19 Global Step: 198885 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:27:31,216-Speed 129.27 samples/sec Loss 0.7187 Epoch: 19 Global Step: 198890 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:27:50,914-Speed 129.97 samples/sec Loss 0.7680 Epoch: 19 Global Step: 198895 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:28:10,468-Speed 130.93 samples/sec Loss 0.8323 Epoch: 19 Global Step: 198900 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:28:30,026-Speed 130.90 samples/sec Loss 0.7625 Epoch: 19 Global Step: 198905 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:28:49,625-Speed 130.62 samples/sec Loss 0.6580 Epoch: 19 Global Step: 198910 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:29:09,135-Speed 131.23 samples/sec Loss 0.7790 Epoch: 19 Global Step: 198915 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:29:28,722-Speed 130.70 samples/sec Loss 0.7080 Epoch: 19 Global Step: 198920 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:29:48,251-Speed 131.10 samples/sec Loss 0.6576 Epoch: 19 Global Step: 198925 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:30:07,744-Speed 131.34 samples/sec Loss 0.7239 Epoch: 19 Global Step: 198930 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:30:27,540-Speed 129.33 samples/sec Loss 0.8028 Epoch: 19 Global Step: 198935 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:30:47,160-Speed 130.49 samples/sec Loss 0.7524 Epoch: 19 Global Step: 198940 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:31:06,722-Speed 130.87 samples/sec Loss 0.8094 Epoch: 19 Global Step: 198945 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:31:26,283-Speed 130.88 samples/sec Loss 0.7363 Epoch: 19 Global Step: 198950 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:31:45,854-Speed 130.81 samples/sec Loss 0.7366 Epoch: 19 Global Step: 198955 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:32:05,390-Speed 131.05 samples/sec Loss 0.6794 Epoch: 19 Global Step: 198960 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:32:25,016-Speed 130.45 samples/sec Loss 0.6599 Epoch: 19 Global Step: 198965 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:32:44,571-Speed 130.92 samples/sec Loss 0.6446 Epoch: 19 Global Step: 198970 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:33:04,129-Speed 130.90 samples/sec Loss 0.6441 Epoch: 19 Global Step: 198975 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:33:23,897-Speed 129.51 samples/sec Loss 0.8428 Epoch: 19 Global Step: 198980 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:33:43,642-Speed 129.66 samples/sec Loss 0.8172 Epoch: 19 Global Step: 198985 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:34:03,163-Speed 131.15 samples/sec Loss 0.7503 Epoch: 19 Global Step: 198990 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:34:22,815-Speed 130.28 samples/sec Loss 0.7107 Epoch: 19 Global Step: 198995 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:34:42,284-Speed 131.50 samples/sec Loss 0.6783 Epoch: 19 Global Step: 199000 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:35:01,832-Speed 130.97 samples/sec Loss 0.6892 Epoch: 19 Global Step: 199005 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:35:21,413-Speed 130.74 samples/sec Loss 0.7422 Epoch: 19 Global Step: 199010 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:35:40,951-Speed 131.04 samples/sec Loss 0.7934 Epoch: 19 Global Step: 199015 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:36:00,168-Speed 133.22 samples/sec Loss 0.7397 Epoch: 19 Global Step: 199020 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:36:19,908-Speed 129.69 samples/sec Loss 0.7499 Epoch: 19 Global Step: 199025 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:36:39,577-Speed 130.16 samples/sec Loss 0.7673 Epoch: 19 Global Step: 199030 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:36:59,040-Speed 131.54 samples/sec Loss 0.8190 Epoch: 19 Global Step: 199035 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:37:18,712-Speed 130.14 samples/sec Loss 0.7635 Epoch: 19 Global Step: 199040 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:37:38,289-Speed 130.78 samples/sec Loss 0.7573 Epoch: 19 Global Step: 199045 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:37:57,873-Speed 130.73 samples/sec Loss 0.8042 Epoch: 19 Global Step: 199050 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:38:17,462-Speed 130.69 samples/sec Loss 0.7533 Epoch: 19 Global Step: 199055 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:38:36,918-Speed 131.59 samples/sec Loss 0.7297 Epoch: 19 Global Step: 199060 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:38:56,527-Speed 130.56 samples/sec Loss 0.7552 Epoch: 19 Global Step: 199065 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:39:16,367-Speed 129.04 samples/sec Loss 0.6442 Epoch: 19 Global Step: 199070 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:39:36,612-Speed 126.46 samples/sec Loss 0.7066 Epoch: 19 Global Step: 199075 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:39:56,203-Speed 130.68 samples/sec Loss 0.7412 Epoch: 19 Global Step: 199080 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:40:15,931-Speed 129.77 samples/sec Loss 0.7333 Epoch: 19 Global Step: 199085 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:40:35,471-Speed 131.02 samples/sec Loss 0.7379 Epoch: 19 Global Step: 199090 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:40:55,059-Speed 130.70 samples/sec Loss 0.7489 Epoch: 19 Global Step: 199095 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:41:14,559-Speed 131.29 samples/sec Loss 0.7782 Epoch: 19 Global Step: 199100 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:41:34,070-Speed 131.21 samples/sec Loss 0.8166 Epoch: 19 Global Step: 199105 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:41:53,660-Speed 130.69 samples/sec Loss 0.7269 Epoch: 19 Global Step: 199110 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:42:13,568-Speed 128.60 samples/sec Loss 0.7252 Epoch: 19 Global Step: 199115 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:42:32,997-Speed 131.77 samples/sec Loss 0.8421 Epoch: 19 Global Step: 199120 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:42:52,661-Speed 130.19 samples/sec Loss 0.7631 Epoch: 19 Global Step: 199125 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:43:12,259-Speed 130.63 samples/sec Loss 0.7263 Epoch: 19 Global Step: 199130 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:43:31,791-Speed 131.07 samples/sec Loss 0.7258 Epoch: 19 Global Step: 199135 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:43:51,417-Speed 130.45 samples/sec Loss 0.6955 Epoch: 19 Global Step: 199140 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:44:10,986-Speed 130.83 samples/sec Loss 0.6943 Epoch: 19 Global Step: 199145 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:44:30,546-Speed 130.89 samples/sec Loss 0.6974 Epoch: 19 Global Step: 199150 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:44:50,142-Speed 130.64 samples/sec Loss 0.7793 Epoch: 19 Global Step: 199155 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:45:10,040-Speed 128.67 samples/sec Loss 0.7498 Epoch: 19 Global Step: 199160 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:45:29,479-Speed 131.70 samples/sec Loss 0.7211 Epoch: 19 Global Step: 199165 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:45:49,147-Speed 130.16 samples/sec Loss 0.7877 Epoch: 19 Global Step: 199170 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:46:08,732-Speed 130.72 samples/sec Loss 0.6652 Epoch: 19 Global Step: 199175 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:46:28,264-Speed 131.07 samples/sec Loss 0.7492 Epoch: 19 Global Step: 199180 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:46:47,881-Speed 130.51 samples/sec Loss 0.7361 Epoch: 19 Global Step: 199185 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:47:07,046-Speed 133.58 samples/sec Loss 0.7079 Epoch: 19 Global Step: 199190 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:47:26,976-Speed 128.46 samples/sec Loss 0.7457 Epoch: 19 Global Step: 199195 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:47:46,781-Speed 129.27 samples/sec Loss 0.7794 Epoch: 19 Global Step: 199200 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:48:06,566-Speed 129.40 samples/sec Loss 0.7329 Epoch: 19 Global Step: 199205 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:48:26,418-Speed 128.96 samples/sec Loss 0.7683 Epoch: 19 Global Step: 199210 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:48:46,306-Speed 128.73 samples/sec Loss 0.7180 Epoch: 19 Global Step: 199215 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:49:06,245-Speed 128.40 samples/sec Loss 0.6867 Epoch: 19 Global Step: 199220 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:49:25,884-Speed 130.36 samples/sec Loss 0.6769 Epoch: 19 Global Step: 199225 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:49:45,954-Speed 127.56 samples/sec Loss 0.8338 Epoch: 19 Global Step: 199230 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:50:05,651-Speed 129.97 samples/sec Loss 0.6774 Epoch: 19 Global Step: 199235 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:50:25,365-Speed 129.87 samples/sec Loss 0.7391 Epoch: 19 Global Step: 199240 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:50:45,290-Speed 128.48 samples/sec Loss 0.7975 Epoch: 19 Global Step: 199245 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:51:05,174-Speed 128.76 samples/sec Loss 0.6691 Epoch: 19 Global Step: 199250 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:51:24,885-Speed 129.88 samples/sec Loss 0.8375 Epoch: 19 Global Step: 199255 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:51:44,725-Speed 129.04 samples/sec Loss 0.7073 Epoch: 19 Global Step: 199260 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:52:04,665-Speed 128.39 samples/sec Loss 0.7692 Epoch: 19 Global Step: 199265 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:52:24,305-Speed 130.35 samples/sec Loss 0.6738 Epoch: 19 Global Step: 199270 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:52:44,245-Speed 128.39 samples/sec Loss 0.6735 Epoch: 19 Global Step: 199275 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:53:03,896-Speed 130.28 samples/sec Loss 0.6640 Epoch: 19 Global Step: 199280 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:53:39,918-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-16 17:54:07,576-Speed 40.20 samples/sec Loss 0.5098 Epoch: 20 Global Step: 199285 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 17:54:27,466-Speed 128.71 samples/sec Loss 0.4379 Epoch: 20 Global Step: 199290 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:54:47,139-Speed 130.14 samples/sec Loss 0.5486 Epoch: 20 Global Step: 199295 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:55:07,107-Speed 128.21 samples/sec Loss 0.4649 Epoch: 20 Global Step: 199300 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:55:27,018-Speed 128.57 samples/sec Loss 0.4374 Epoch: 20 Global Step: 199305 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:55:46,807-Speed 129.37 samples/sec Loss 0.5255 Epoch: 20 Global Step: 199310 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:56:06,409-Speed 130.61 samples/sec Loss 0.4234 Epoch: 20 Global Step: 199315 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:56:26,688-Speed 126.24 samples/sec Loss 0.5463 Epoch: 20 Global Step: 199320 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:56:46,436-Speed 129.64 samples/sec Loss 0.5389 Epoch: 20 Global Step: 199325 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:57:06,589-Speed 127.03 samples/sec Loss 0.5363 Epoch: 20 Global Step: 199330 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:57:26,194-Speed 130.59 samples/sec Loss 0.4794 Epoch: 20 Global Step: 199335 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:57:46,007-Speed 129.21 samples/sec Loss 0.5218 Epoch: 20 Global Step: 199340 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:58:05,815-Speed 129.26 samples/sec Loss 0.4835 Epoch: 20 Global Step: 199345 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:58:25,885-Speed 127.56 samples/sec Loss 0.5159 Epoch: 20 Global Step: 199350 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:58:45,597-Speed 129.88 samples/sec Loss 0.5735 Epoch: 20 Global Step: 199355 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:59:05,244-Speed 130.31 samples/sec Loss 0.4546 Epoch: 20 Global Step: 199360 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:59:25,215-Speed 128.19 samples/sec Loss 0.5217 Epoch: 20 Global Step: 199365 Fp16 Grad Scale: 8192 Required: 60 hours Training: 2021-06-16 17:59:44,411-Speed 133.37 samples/sec Loss 0.4759 Epoch: 20 Global Step: 199370 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:00:04,501-Speed 127.43 samples/sec Loss 0.4730 Epoch: 20 Global Step: 199375 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:00:24,166-Speed 130.19 samples/sec Loss 0.5013 Epoch: 20 Global Step: 199380 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:00:44,040-Speed 128.82 samples/sec Loss 0.4992 Epoch: 20 Global Step: 199385 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:01:03,838-Speed 129.31 samples/sec Loss 0.4920 Epoch: 20 Global Step: 199390 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:01:23,615-Speed 129.45 samples/sec Loss 0.4812 Epoch: 20 Global Step: 199395 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:01:43,304-Speed 130.03 samples/sec Loss 0.5127 Epoch: 20 Global Step: 199400 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:02:02,972-Speed 130.16 samples/sec Loss 0.4499 Epoch: 20 Global Step: 199405 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:02:22,914-Speed 128.38 samples/sec Loss 0.5103 Epoch: 20 Global Step: 199410 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:02:42,602-Speed 130.04 samples/sec Loss 0.6217 Epoch: 20 Global Step: 199415 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:03:02,835-Speed 126.53 samples/sec Loss 0.5560 Epoch: 20 Global Step: 199420 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:03:22,495-Speed 130.21 samples/sec Loss 0.5043 Epoch: 20 Global Step: 199425 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:03:42,458-Speed 128.25 samples/sec Loss 0.5736 Epoch: 20 Global Step: 199430 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:04:02,157-Speed 129.96 samples/sec Loss 0.4964 Epoch: 20 Global Step: 199435 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:04:21,946-Speed 129.37 samples/sec Loss 0.5165 Epoch: 20 Global Step: 199440 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:04:41,661-Speed 129.86 samples/sec Loss 0.4860 Epoch: 20 Global Step: 199445 Fp16 Grad Scale: 4096 Required: 60 hours Training: 2021-06-16 18:05:01,406-Speed 129.66 samples/sec Loss 0.5186 Epoch: 20 Global Step: 199450 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:05:21,339-Speed 128.44 samples/sec Loss 0.5538 Epoch: 20 Global Step: 199455 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:05:41,037-Speed 129.97 samples/sec Loss 0.5193 Epoch: 20 Global Step: 199460 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:06:01,296-Speed 126.37 samples/sec Loss 0.5104 Epoch: 20 Global Step: 199465 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:06:21,182-Speed 128.74 samples/sec Loss 0.5276 Epoch: 20 Global Step: 199470 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:06:40,930-Speed 129.64 samples/sec Loss 0.5152 Epoch: 20 Global Step: 199475 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:07:00,618-Speed 130.04 samples/sec Loss 0.5167 Epoch: 20 Global Step: 199480 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:07:20,364-Speed 129.65 samples/sec Loss 0.5278 Epoch: 20 Global Step: 199485 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:07:40,179-Speed 129.20 samples/sec Loss 0.4885 Epoch: 20 Global Step: 199490 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:08:00,072-Speed 128.69 samples/sec Loss 0.5664 Epoch: 20 Global Step: 199495 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:08:19,806-Speed 129.73 samples/sec Loss 0.5379 Epoch: 20 Global Step: 199500 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:08:39,463-Speed 130.24 samples/sec Loss 0.5928 Epoch: 20 Global Step: 199505 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:08:59,658-Speed 126.77 samples/sec Loss 0.5320 Epoch: 20 Global Step: 199510 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:09:19,514-Speed 128.93 samples/sec Loss 0.4537 Epoch: 20 Global Step: 199515 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:09:39,265-Speed 129.62 samples/sec Loss 0.4876 Epoch: 20 Global Step: 199520 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:09:59,072-Speed 129.26 samples/sec Loss 0.5811 Epoch: 20 Global Step: 199525 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:10:18,786-Speed 129.86 samples/sec Loss 0.5578 Epoch: 20 Global Step: 199530 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:10:38,554-Speed 129.51 samples/sec Loss 0.4650 Epoch: 20 Global Step: 199535 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:10:58,318-Speed 129.53 samples/sec Loss 0.4912 Epoch: 20 Global Step: 199540 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:11:18,124-Speed 129.26 samples/sec Loss 0.4830 Epoch: 20 Global Step: 199545 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:11:37,953-Speed 129.11 samples/sec Loss 0.4686 Epoch: 20 Global Step: 199550 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:11:57,957-Speed 127.98 samples/sec Loss 0.4665 Epoch: 20 Global Step: 199555 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:12:17,914-Speed 128.28 samples/sec Loss 0.4853 Epoch: 20 Global Step: 199560 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:12:37,616-Speed 129.95 samples/sec Loss 0.4655 Epoch: 20 Global Step: 199565 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:12:57,554-Speed 128.40 samples/sec Loss 0.5037 Epoch: 20 Global Step: 199570 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:13:17,197-Speed 130.34 samples/sec Loss 0.5188 Epoch: 20 Global Step: 199575 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:13:36,933-Speed 129.72 samples/sec Loss 0.5501 Epoch: 20 Global Step: 199580 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:13:56,813-Speed 128.78 samples/sec Loss 0.4813 Epoch: 20 Global Step: 199585 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:14:16,541-Speed 129.78 samples/sec Loss 0.4790 Epoch: 20 Global Step: 199590 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:14:36,343-Speed 129.28 samples/sec Loss 0.4973 Epoch: 20 Global Step: 199595 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:14:56,552-Speed 126.69 samples/sec Loss 0.5230 Epoch: 20 Global Step: 199600 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:15:16,486-Speed 128.43 samples/sec Loss 0.4749 Epoch: 20 Global Step: 199605 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:15:36,247-Speed 129.55 samples/sec Loss 0.5458 Epoch: 20 Global Step: 199610 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:15:56,075-Speed 129.12 samples/sec Loss 0.5110 Epoch: 20 Global Step: 199615 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:16:15,772-Speed 129.98 samples/sec Loss 0.5256 Epoch: 20 Global Step: 199620 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:16:35,655-Speed 128.75 samples/sec Loss 0.5203 Epoch: 20 Global Step: 199625 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:16:55,376-Speed 129.82 samples/sec Loss 0.4671 Epoch: 20 Global Step: 199630 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:17:15,142-Speed 129.53 samples/sec Loss 0.5838 Epoch: 20 Global Step: 199635 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:17:34,936-Speed 129.34 samples/sec Loss 0.4938 Epoch: 20 Global Step: 199640 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:17:55,338-Speed 125.48 samples/sec Loss 0.4518 Epoch: 20 Global Step: 199645 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:18:15,119-Speed 129.42 samples/sec Loss 0.5135 Epoch: 20 Global Step: 199650 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:18:34,918-Speed 129.31 samples/sec Loss 0.4932 Epoch: 20 Global Step: 199655 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:18:54,707-Speed 129.37 samples/sec Loss 0.4923 Epoch: 20 Global Step: 199660 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:19:14,338-Speed 130.41 samples/sec Loss 0.5904 Epoch: 20 Global Step: 199665 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:19:34,198-Speed 128.91 samples/sec Loss 0.5271 Epoch: 20 Global Step: 199670 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:19:53,907-Speed 129.89 samples/sec Loss 0.4971 Epoch: 20 Global Step: 199675 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:20:13,821-Speed 128.56 samples/sec Loss 0.4911 Epoch: 20 Global Step: 199680 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:20:33,432-Speed 130.55 samples/sec Loss 0.5941 Epoch: 20 Global Step: 199685 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:20:53,675-Speed 126.47 samples/sec Loss 0.5783 Epoch: 20 Global Step: 199690 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:21:13,453-Speed 129.44 samples/sec Loss 0.5697 Epoch: 20 Global Step: 199695 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:21:33,282-Speed 129.11 samples/sec Loss 0.4770 Epoch: 20 Global Step: 199700 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:21:53,046-Speed 129.54 samples/sec Loss 0.5219 Epoch: 20 Global Step: 199705 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:22:12,830-Speed 129.40 samples/sec Loss 0.4579 Epoch: 20 Global Step: 199710 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:22:32,559-Speed 129.76 samples/sec Loss 0.5987 Epoch: 20 Global Step: 199715 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:22:52,352-Speed 129.35 samples/sec Loss 0.5756 Epoch: 20 Global Step: 199720 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:23:12,301-Speed 128.33 samples/sec Loss 0.5036 Epoch: 20 Global Step: 199725 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:23:31,905-Speed 130.59 samples/sec Loss 0.4706 Epoch: 20 Global Step: 199730 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:23:52,236-Speed 125.92 samples/sec Loss 0.4571 Epoch: 20 Global Step: 199735 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:24:12,060-Speed 129.14 samples/sec Loss 0.4866 Epoch: 20 Global Step: 199740 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:24:31,922-Speed 128.89 samples/sec Loss 0.5354 Epoch: 20 Global Step: 199745 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:24:51,623-Speed 129.95 samples/sec Loss 0.5669 Epoch: 20 Global Step: 199750 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:25:11,521-Speed 128.66 samples/sec Loss 0.4432 Epoch: 20 Global Step: 199755 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:25:31,252-Speed 129.75 samples/sec Loss 0.5013 Epoch: 20 Global Step: 199760 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:25:51,063-Speed 129.23 samples/sec Loss 0.4921 Epoch: 20 Global Step: 199765 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:26:10,943-Speed 128.78 samples/sec Loss 0.5110 Epoch: 20 Global Step: 199770 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:26:30,628-Speed 130.05 samples/sec Loss 0.5373 Epoch: 20 Global Step: 199775 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:26:50,566-Speed 128.40 samples/sec Loss 0.5275 Epoch: 20 Global Step: 199780 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:27:10,542-Speed 128.16 samples/sec Loss 0.5516 Epoch: 20 Global Step: 199785 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:27:30,376-Speed 129.07 samples/sec Loss 0.5155 Epoch: 20 Global Step: 199790 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:27:50,113-Speed 129.72 samples/sec Loss 0.4852 Epoch: 20 Global Step: 199795 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:28:09,989-Speed 128.81 samples/sec Loss 0.5415 Epoch: 20 Global Step: 199800 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:28:29,776-Speed 129.38 samples/sec Loss 0.4789 Epoch: 20 Global Step: 199805 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:28:49,753-Speed 128.15 samples/sec Loss 0.4685 Epoch: 20 Global Step: 199810 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:29:09,482-Speed 129.77 samples/sec Loss 0.4664 Epoch: 20 Global Step: 199815 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:29:29,175-Speed 130.00 samples/sec Loss 0.5897 Epoch: 20 Global Step: 199820 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:29:49,534-Speed 125.75 samples/sec Loss 0.6633 Epoch: 20 Global Step: 199825 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:30:09,493-Speed 128.27 samples/sec Loss 0.4458 Epoch: 20 Global Step: 199830 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:30:29,227-Speed 129.73 samples/sec Loss 0.6328 Epoch: 20 Global Step: 199835 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:30:48,983-Speed 129.59 samples/sec Loss 0.4849 Epoch: 20 Global Step: 199840 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:31:08,786-Speed 129.28 samples/sec Loss 0.4925 Epoch: 20 Global Step: 199845 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:31:28,522-Speed 129.72 samples/sec Loss 0.5149 Epoch: 20 Global Step: 199850 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:31:48,477-Speed 128.30 samples/sec Loss 0.5658 Epoch: 20 Global Step: 199855 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:32:08,214-Speed 129.72 samples/sec Loss 0.4913 Epoch: 20 Global Step: 199860 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:32:28,077-Speed 128.88 samples/sec Loss 0.4930 Epoch: 20 Global Step: 199865 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:32:48,393-Speed 126.01 samples/sec Loss 0.5432 Epoch: 20 Global Step: 199870 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:33:08,328-Speed 128.42 samples/sec Loss 0.5844 Epoch: 20 Global Step: 199875 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 18:33:27,951-Speed 130.47 samples/sec Loss 0.5809 Epoch: 20 Global Step: 199880 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:33:47,859-Speed 128.60 samples/sec Loss 0.5021 Epoch: 20 Global Step: 199885 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:34:07,566-Speed 129.91 samples/sec Loss 0.5399 Epoch: 20 Global Step: 199890 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:34:27,454-Speed 128.73 samples/sec Loss 0.6404 Epoch: 20 Global Step: 199895 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:34:47,209-Speed 129.59 samples/sec Loss 0.5356 Epoch: 20 Global Step: 199900 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:35:06,927-Speed 129.84 samples/sec Loss 0.5602 Epoch: 20 Global Step: 199905 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:35:26,823-Speed 128.67 samples/sec Loss 0.5548 Epoch: 20 Global Step: 199910 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:35:47,141-Speed 126.01 samples/sec Loss 0.5086 Epoch: 20 Global Step: 199915 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:36:06,937-Speed 129.32 samples/sec Loss 0.5917 Epoch: 20 Global Step: 199920 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:36:26,647-Speed 129.89 samples/sec Loss 0.5172 Epoch: 20 Global Step: 199925 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:36:46,499-Speed 128.96 samples/sec Loss 0.5222 Epoch: 20 Global Step: 199930 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:37:06,229-Speed 129.75 samples/sec Loss 0.4276 Epoch: 20 Global Step: 199935 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:37:26,106-Speed 128.80 samples/sec Loss 0.6549 Epoch: 20 Global Step: 199940 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:37:45,830-Speed 129.80 samples/sec Loss 0.4680 Epoch: 20 Global Step: 199945 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:38:05,577-Speed 129.65 samples/sec Loss 0.5513 Epoch: 20 Global Step: 199950 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:38:25,525-Speed 128.34 samples/sec Loss 0.5376 Epoch: 20 Global Step: 199955 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:38:45,965-Speed 125.25 samples/sec Loss 0.4793 Epoch: 20 Global Step: 199960 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:39:05,699-Speed 129.73 samples/sec Loss 0.5276 Epoch: 20 Global Step: 199965 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:39:25,445-Speed 129.65 samples/sec Loss 0.6011 Epoch: 20 Global Step: 199970 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:39:45,218-Speed 129.48 samples/sec Loss 0.4847 Epoch: 20 Global Step: 199975 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:40:05,257-Speed 127.76 samples/sec Loss 0.5503 Epoch: 20 Global Step: 199980 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:40:25,236-Speed 128.14 samples/sec Loss 0.5033 Epoch: 20 Global Step: 199985 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:40:44,971-Speed 129.73 samples/sec Loss 0.5573 Epoch: 20 Global Step: 199990 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:41:04,899-Speed 128.47 samples/sec Loss 0.5164 Epoch: 20 Global Step: 199995 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:41:24,588-Speed 130.03 samples/sec Loss 0.4492 Epoch: 20 Global Step: 200000 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 18:56:11,165-[lfw][200000]XNorm: 22.284483 Training: 2021-06-16 18:56:11,165-[lfw][200000]Accuracy-Flip: 0.99817+-0.00203 Training: 2021-06-16 18:56:11,165-[lfw][200000]Accuracy-Highest: 0.99833 Training: 2021-06-16 19:13:24,749-[cfp_fp][200000]XNorm: 23.828663 Training: 2021-06-16 19:13:24,750-[cfp_fp][200000]Accuracy-Flip: 0.99200+-0.00241 Training: 2021-06-16 19:13:24,750-[cfp_fp][200000]Accuracy-Highest: 0.99229 Training: 2021-06-16 19:28:12,737-[agedb_30][200000]XNorm: 23.415129 Training: 2021-06-16 19:28:12,737-[agedb_30][200000]Accuracy-Flip: 0.98667+-0.00573 Training: 2021-06-16 19:28:12,737-[agedb_30][200000]Accuracy-Highest: 0.98667 Training: 2021-06-16 19:28:32,940-Speed 0.91 samples/sec Loss 0.5537 Epoch: 20 Global Step: 200005 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:28:52,729-Speed 129.37 samples/sec Loss 0.4975 Epoch: 20 Global Step: 200010 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:29:12,322-Speed 130.66 samples/sec Loss 0.5661 Epoch: 20 Global Step: 200015 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:29:31,956-Speed 130.40 samples/sec Loss 0.5342 Epoch: 20 Global Step: 200020 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:29:51,910-Speed 128.30 samples/sec Loss 0.5326 Epoch: 20 Global Step: 200025 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:30:11,566-Speed 130.25 samples/sec Loss 0.5885 Epoch: 20 Global Step: 200030 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:30:31,398-Speed 129.09 samples/sec Loss 0.5169 Epoch: 20 Global Step: 200035 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:30:51,134-Speed 129.72 samples/sec Loss 0.5500 Epoch: 20 Global Step: 200040 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:31:10,914-Speed 129.43 samples/sec Loss 0.4849 Epoch: 20 Global Step: 200045 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:31:30,914-Speed 128.01 samples/sec Loss 0.4862 Epoch: 20 Global Step: 200050 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:31:50,818-Speed 128.62 samples/sec Loss 0.4567 Epoch: 20 Global Step: 200055 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:32:10,501-Speed 130.07 samples/sec Loss 0.5611 Epoch: 20 Global Step: 200060 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:32:30,285-Speed 129.40 samples/sec Loss 0.5360 Epoch: 20 Global Step: 200065 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:32:49,961-Speed 130.11 samples/sec Loss 0.4305 Epoch: 20 Global Step: 200070 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:33:09,605-Speed 130.33 samples/sec Loss 0.6015 Epoch: 20 Global Step: 200075 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:33:29,409-Speed 129.27 samples/sec Loss 0.4653 Epoch: 20 Global Step: 200080 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:33:49,135-Speed 129.78 samples/sec Loss 0.4604 Epoch: 20 Global Step: 200085 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:34:08,957-Speed 129.16 samples/sec Loss 0.4992 Epoch: 20 Global Step: 200090 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:34:29,164-Speed 126.69 samples/sec Loss 0.5471 Epoch: 20 Global Step: 200095 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:34:49,038-Speed 128.82 samples/sec Loss 0.4639 Epoch: 20 Global Step: 200100 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:35:08,766-Speed 129.77 samples/sec Loss 0.5129 Epoch: 20 Global Step: 200105 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:35:28,675-Speed 128.59 samples/sec Loss 0.5264 Epoch: 20 Global Step: 200110 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:35:48,473-Speed 129.32 samples/sec Loss 0.5914 Epoch: 20 Global Step: 200115 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:36:08,261-Speed 129.38 samples/sec Loss 0.5004 Epoch: 20 Global Step: 200120 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:36:27,948-Speed 130.04 samples/sec Loss 0.5669 Epoch: 20 Global Step: 200125 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:36:47,727-Speed 129.44 samples/sec Loss 0.5792 Epoch: 20 Global Step: 200130 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:37:07,565-Speed 129.05 samples/sec Loss 0.4679 Epoch: 20 Global Step: 200135 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:37:27,885-Speed 125.99 samples/sec Loss 0.6174 Epoch: 20 Global Step: 200140 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:37:47,601-Speed 129.85 samples/sec Loss 0.5689 Epoch: 20 Global Step: 200145 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:38:07,304-Speed 129.94 samples/sec Loss 0.5705 Epoch: 20 Global Step: 200150 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:38:27,193-Speed 128.72 samples/sec Loss 0.4261 Epoch: 20 Global Step: 200155 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:38:46,938-Speed 129.66 samples/sec Loss 0.5290 Epoch: 20 Global Step: 200160 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:39:06,718-Speed 129.43 samples/sec Loss 0.5224 Epoch: 20 Global Step: 200165 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:39:26,436-Speed 129.84 samples/sec Loss 0.5484 Epoch: 20 Global Step: 200170 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:39:46,330-Speed 128.69 samples/sec Loss 0.5782 Epoch: 20 Global Step: 200175 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:40:06,174-Speed 129.01 samples/sec Loss 0.5548 Epoch: 20 Global Step: 200180 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:40:26,704-Speed 124.70 samples/sec Loss 0.5039 Epoch: 20 Global Step: 200185 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:40:46,571-Speed 128.86 samples/sec Loss 0.4823 Epoch: 20 Global Step: 200190 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:41:06,472-Speed 128.64 samples/sec Loss 0.4709 Epoch: 20 Global Step: 200195 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:41:26,195-Speed 129.81 samples/sec Loss 0.5400 Epoch: 20 Global Step: 200200 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:41:45,924-Speed 129.77 samples/sec Loss 0.4652 Epoch: 20 Global Step: 200205 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:42:05,817-Speed 128.69 samples/sec Loss 0.4829 Epoch: 20 Global Step: 200210 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:42:25,437-Speed 130.49 samples/sec Loss 0.4567 Epoch: 20 Global Step: 200215 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:42:45,360-Speed 128.50 samples/sec Loss 0.6065 Epoch: 20 Global Step: 200220 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:43:05,133-Speed 129.47 samples/sec Loss 0.5050 Epoch: 20 Global Step: 200225 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:43:25,410-Speed 126.26 samples/sec Loss 0.5287 Epoch: 20 Global Step: 200230 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:43:45,023-Speed 130.53 samples/sec Loss 0.4547 Epoch: 20 Global Step: 200235 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:44:04,937-Speed 128.56 samples/sec Loss 0.5381 Epoch: 20 Global Step: 200240 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:44:24,721-Speed 129.41 samples/sec Loss 0.5510 Epoch: 20 Global Step: 200245 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:44:44,375-Speed 130.26 samples/sec Loss 0.5534 Epoch: 20 Global Step: 200250 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:45:04,234-Speed 128.92 samples/sec Loss 0.5589 Epoch: 20 Global Step: 200255 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:45:23,856-Speed 130.48 samples/sec Loss 0.5489 Epoch: 20 Global Step: 200260 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:45:43,787-Speed 128.45 samples/sec Loss 0.4963 Epoch: 20 Global Step: 200265 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:46:03,595-Speed 129.24 samples/sec Loss 0.5547 Epoch: 20 Global Step: 200270 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:46:23,869-Speed 126.28 samples/sec Loss 0.5936 Epoch: 20 Global Step: 200275 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:46:43,483-Speed 130.52 samples/sec Loss 0.5636 Epoch: 20 Global Step: 200280 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:47:03,430-Speed 128.35 samples/sec Loss 0.4899 Epoch: 20 Global Step: 200285 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:47:23,204-Speed 129.47 samples/sec Loss 0.5171 Epoch: 20 Global Step: 200290 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:47:42,887-Speed 130.07 samples/sec Loss 0.4162 Epoch: 20 Global Step: 200295 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:48:02,290-Speed 131.95 samples/sec Loss 0.5139 Epoch: 20 Global Step: 200300 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:48:21,898-Speed 130.57 samples/sec Loss 0.5689 Epoch: 20 Global Step: 200305 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:48:41,826-Speed 128.47 samples/sec Loss 0.5442 Epoch: 20 Global Step: 200310 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:49:01,534-Speed 129.90 samples/sec Loss 0.5491 Epoch: 20 Global Step: 200315 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:49:21,870-Speed 125.89 samples/sec Loss 0.5326 Epoch: 20 Global Step: 200320 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:49:41,508-Speed 130.36 samples/sec Loss 0.4864 Epoch: 20 Global Step: 200325 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:50:01,422-Speed 128.56 samples/sec Loss 0.5199 Epoch: 20 Global Step: 200330 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:50:21,206-Speed 129.40 samples/sec Loss 0.4995 Epoch: 20 Global Step: 200335 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:50:40,945-Speed 129.70 samples/sec Loss 0.5820 Epoch: 20 Global Step: 200340 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:51:00,708-Speed 129.55 samples/sec Loss 0.6144 Epoch: 20 Global Step: 200345 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:51:20,398-Speed 130.02 samples/sec Loss 0.6168 Epoch: 20 Global Step: 200350 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:51:40,283-Speed 128.75 samples/sec Loss 0.4784 Epoch: 20 Global Step: 200355 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:51:59,981-Speed 129.97 samples/sec Loss 0.4163 Epoch: 20 Global Step: 200360 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:52:20,249-Speed 126.32 samples/sec Loss 0.3994 Epoch: 20 Global Step: 200365 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:52:39,980-Speed 129.75 samples/sec Loss 0.6367 Epoch: 20 Global Step: 200370 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:52:59,795-Speed 129.20 samples/sec Loss 0.5139 Epoch: 20 Global Step: 200375 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:53:19,621-Speed 129.13 samples/sec Loss 0.5019 Epoch: 20 Global Step: 200380 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:53:39,488-Speed 128.87 samples/sec Loss 0.5437 Epoch: 20 Global Step: 200385 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:53:59,096-Speed 130.57 samples/sec Loss 0.5938 Epoch: 20 Global Step: 200390 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:54:19,231-Speed 127.14 samples/sec Loss 0.5634 Epoch: 20 Global Step: 200395 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:54:38,974-Speed 129.68 samples/sec Loss 0.4961 Epoch: 20 Global Step: 200400 Fp16 Grad Scale: 8192 Required: 59 hours Training: 2021-06-16 19:54:58,356-Speed 132.09 samples/sec Loss 0.5718 Epoch: 20 Global Step: 200405 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:55:18,375-Speed 127.88 samples/sec Loss 0.5280 Epoch: 20 Global Step: 200410 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:55:37,992-Speed 130.51 samples/sec Loss 0.4751 Epoch: 20 Global Step: 200415 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:55:57,547-Speed 130.92 samples/sec Loss 0.5314 Epoch: 20 Global Step: 200420 Fp16 Grad Scale: 4096 Required: 59 hours Training: 2021-06-16 19:56:17,187-Speed 130.35 samples/sec Loss 0.5152 Epoch: 20 Global Step: 200425 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:56:36,629-Speed 131.68 samples/sec Loss 0.5845 Epoch: 20 Global Step: 200430 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:56:56,106-Speed 131.45 samples/sec Loss 0.4544 Epoch: 20 Global Step: 200435 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:57:15,912-Speed 129.26 samples/sec Loss 0.4981 Epoch: 20 Global Step: 200440 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:57:35,568-Speed 130.24 samples/sec Loss 0.4863 Epoch: 20 Global Step: 200445 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:57:55,169-Speed 130.61 samples/sec Loss 0.6019 Epoch: 20 Global Step: 200450 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:58:14,844-Speed 130.12 samples/sec Loss 0.5735 Epoch: 20 Global Step: 200455 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:58:34,413-Speed 130.83 samples/sec Loss 0.4567 Epoch: 20 Global Step: 200460 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:58:53,988-Speed 130.78 samples/sec Loss 0.4754 Epoch: 20 Global Step: 200465 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:59:13,601-Speed 130.53 samples/sec Loss 0.5073 Epoch: 20 Global Step: 200470 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:59:33,084-Speed 131.41 samples/sec Loss 0.5863 Epoch: 20 Global Step: 200475 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 19:59:52,558-Speed 131.47 samples/sec Loss 0.5778 Epoch: 20 Global Step: 200480 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:00:12,171-Speed 130.53 samples/sec Loss 0.5348 Epoch: 20 Global Step: 200485 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:00:31,780-Speed 130.56 samples/sec Loss 0.5593 Epoch: 20 Global Step: 200490 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:00:51,399-Speed 130.49 samples/sec Loss 0.4765 Epoch: 20 Global Step: 200495 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:01:11,193-Speed 129.34 samples/sec Loss 0.5272 Epoch: 20 Global Step: 200500 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:01:30,747-Speed 130.93 samples/sec Loss 0.4875 Epoch: 20 Global Step: 200505 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:01:50,352-Speed 130.59 samples/sec Loss 0.5462 Epoch: 20 Global Step: 200510 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:02:09,967-Speed 130.52 samples/sec Loss 0.5516 Epoch: 20 Global Step: 200515 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:02:29,472-Speed 131.26 samples/sec Loss 0.5452 Epoch: 20 Global Step: 200520 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:02:49,073-Speed 130.61 samples/sec Loss 0.4689 Epoch: 20 Global Step: 200525 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:03:08,526-Speed 131.61 samples/sec Loss 0.5419 Epoch: 20 Global Step: 200530 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:03:28,183-Speed 130.24 samples/sec Loss 0.5752 Epoch: 20 Global Step: 200535 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:03:47,767-Speed 130.73 samples/sec Loss 0.5391 Epoch: 20 Global Step: 200540 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:04:07,668-Speed 128.64 samples/sec Loss 0.5281 Epoch: 20 Global Step: 200545 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:04:27,091-Speed 131.81 samples/sec Loss 0.6040 Epoch: 20 Global Step: 200550 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:04:46,696-Speed 130.59 samples/sec Loss 0.5051 Epoch: 20 Global Step: 200555 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:05:06,286-Speed 130.69 samples/sec Loss 0.5283 Epoch: 20 Global Step: 200560 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:05:25,815-Speed 131.09 samples/sec Loss 0.6308 Epoch: 20 Global Step: 200565 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:05:45,421-Speed 130.58 samples/sec Loss 0.5630 Epoch: 20 Global Step: 200570 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:06:04,579-Speed 133.63 samples/sec Loss 0.5755 Epoch: 20 Global Step: 200575 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:06:24,385-Speed 129.26 samples/sec Loss 0.5590 Epoch: 20 Global Step: 200580 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:06:44,203-Speed 129.18 samples/sec Loss 0.4831 Epoch: 20 Global Step: 200585 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:07:03,934-Speed 129.75 samples/sec Loss 0.5685 Epoch: 20 Global Step: 200590 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:07:23,777-Speed 129.02 samples/sec Loss 0.5405 Epoch: 20 Global Step: 200595 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:07:43,731-Speed 128.31 samples/sec Loss 0.5870 Epoch: 20 Global Step: 200600 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:08:03,602-Speed 128.83 samples/sec Loss 0.5633 Epoch: 20 Global Step: 200605 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:08:23,322-Speed 129.83 samples/sec Loss 0.5676 Epoch: 20 Global Step: 200610 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:08:43,329-Speed 127.96 samples/sec Loss 0.5349 Epoch: 20 Global Step: 200615 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:09:02,993-Speed 130.19 samples/sec Loss 0.5202 Epoch: 20 Global Step: 200620 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:09:22,907-Speed 128.56 samples/sec Loss 0.5899 Epoch: 20 Global Step: 200625 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:09:42,628-Speed 129.82 samples/sec Loss 0.6709 Epoch: 20 Global Step: 200630 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:10:02,535-Speed 128.61 samples/sec Loss 0.5546 Epoch: 20 Global Step: 200635 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:10:22,135-Speed 130.62 samples/sec Loss 0.4671 Epoch: 20 Global Step: 200640 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:10:42,151-Speed 127.90 samples/sec Loss 0.4867 Epoch: 20 Global Step: 200645 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:11:02,020-Speed 128.85 samples/sec Loss 0.4859 Epoch: 20 Global Step: 200650 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:11:21,640-Speed 130.49 samples/sec Loss 0.5711 Epoch: 20 Global Step: 200655 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:11:41,615-Speed 128.16 samples/sec Loss 0.5483 Epoch: 20 Global Step: 200660 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:12:01,416-Speed 129.30 samples/sec Loss 0.4888 Epoch: 20 Global Step: 200665 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:12:21,191-Speed 129.46 samples/sec Loss 0.4924 Epoch: 20 Global Step: 200670 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:12:40,956-Speed 129.53 samples/sec Loss 0.6110 Epoch: 20 Global Step: 200675 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:13:00,792-Speed 129.06 samples/sec Loss 0.5819 Epoch: 20 Global Step: 200680 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:13:20,521-Speed 129.76 samples/sec Loss 0.5895 Epoch: 20 Global Step: 200685 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:13:40,665-Speed 127.09 samples/sec Loss 0.5693 Epoch: 20 Global Step: 200690 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:14:00,391-Speed 129.79 samples/sec Loss 0.4887 Epoch: 20 Global Step: 200695 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:14:20,100-Speed 129.90 samples/sec Loss 0.4947 Epoch: 20 Global Step: 200700 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:14:40,030-Speed 128.46 samples/sec Loss 0.5336 Epoch: 20 Global Step: 200705 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:14:59,805-Speed 129.46 samples/sec Loss 0.5260 Epoch: 20 Global Step: 200710 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:15:19,573-Speed 129.51 samples/sec Loss 0.5434 Epoch: 20 Global Step: 200715 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:15:39,442-Speed 128.85 samples/sec Loss 0.3882 Epoch: 20 Global Step: 200720 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:15:59,132-Speed 130.02 samples/sec Loss 0.5242 Epoch: 20 Global Step: 200725 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:16:18,830-Speed 129.97 samples/sec Loss 0.5638 Epoch: 20 Global Step: 200730 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:16:39,003-Speed 126.91 samples/sec Loss 0.5451 Epoch: 20 Global Step: 200735 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:16:58,717-Speed 129.86 samples/sec Loss 0.4577 Epoch: 20 Global Step: 200740 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:17:18,507-Speed 129.37 samples/sec Loss 0.4429 Epoch: 20 Global Step: 200745 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:17:38,312-Speed 129.27 samples/sec Loss 0.5524 Epoch: 20 Global Step: 200750 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:17:58,107-Speed 129.33 samples/sec Loss 0.4787 Epoch: 20 Global Step: 200755 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:18:17,992-Speed 128.74 samples/sec Loss 0.5968 Epoch: 20 Global Step: 200760 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:18:37,893-Speed 128.64 samples/sec Loss 0.5629 Epoch: 20 Global Step: 200765 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:18:57,504-Speed 130.55 samples/sec Loss 0.5611 Epoch: 20 Global Step: 200770 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:19:17,161-Speed 130.24 samples/sec Loss 0.5135 Epoch: 20 Global Step: 200775 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:19:37,332-Speed 126.92 samples/sec Loss 0.5769 Epoch: 20 Global Step: 200780 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:19:57,060-Speed 129.77 samples/sec Loss 0.5148 Epoch: 20 Global Step: 200785 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:20:16,893-Speed 129.09 samples/sec Loss 0.4187 Epoch: 20 Global Step: 200790 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:20:36,742-Speed 128.98 samples/sec Loss 0.5034 Epoch: 20 Global Step: 200795 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:20:56,571-Speed 129.11 samples/sec Loss 0.6565 Epoch: 20 Global Step: 200800 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:21:16,272-Speed 129.95 samples/sec Loss 0.5649 Epoch: 20 Global Step: 200805 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:21:36,176-Speed 128.63 samples/sec Loss 0.5799 Epoch: 20 Global Step: 200810 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:21:55,795-Speed 130.49 samples/sec Loss 0.5456 Epoch: 20 Global Step: 200815 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:22:15,498-Speed 129.94 samples/sec Loss 0.6050 Epoch: 20 Global Step: 200820 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:22:35,610-Speed 127.29 samples/sec Loss 0.4799 Epoch: 20 Global Step: 200825 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:22:55,342-Speed 129.75 samples/sec Loss 0.5649 Epoch: 20 Global Step: 200830 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:23:15,125-Speed 129.41 samples/sec Loss 0.5237 Epoch: 20 Global Step: 200835 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:23:34,959-Speed 129.08 samples/sec Loss 0.6321 Epoch: 20 Global Step: 200840 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:23:54,870-Speed 128.57 samples/sec Loss 0.5779 Epoch: 20 Global Step: 200845 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:24:14,551-Speed 130.08 samples/sec Loss 0.4643 Epoch: 20 Global Step: 200850 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:24:34,452-Speed 128.65 samples/sec Loss 0.5795 Epoch: 20 Global Step: 200855 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:24:54,079-Speed 130.44 samples/sec Loss 0.5992 Epoch: 20 Global Step: 200860 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:25:13,871-Speed 129.35 samples/sec Loss 0.5139 Epoch: 20 Global Step: 200865 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:25:33,949-Speed 127.51 samples/sec Loss 0.4958 Epoch: 20 Global Step: 200870 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:25:53,744-Speed 129.33 samples/sec Loss 0.6210 Epoch: 20 Global Step: 200875 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:26:13,681-Speed 128.42 samples/sec Loss 0.5270 Epoch: 20 Global Step: 200880 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:26:33,659-Speed 128.14 samples/sec Loss 0.4223 Epoch: 20 Global Step: 200885 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:26:53,376-Speed 129.85 samples/sec Loss 0.5287 Epoch: 20 Global Step: 200890 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:27:13,276-Speed 128.65 samples/sec Loss 0.5009 Epoch: 20 Global Step: 200895 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:27:33,145-Speed 128.85 samples/sec Loss 0.5081 Epoch: 20 Global Step: 200900 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:27:52,757-Speed 130.54 samples/sec Loss 0.5925 Epoch: 20 Global Step: 200905 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:28:12,545-Speed 129.38 samples/sec Loss 0.5159 Epoch: 20 Global Step: 200910 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:28:32,539-Speed 128.04 samples/sec Loss 0.5314 Epoch: 20 Global Step: 200915 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:28:52,438-Speed 128.66 samples/sec Loss 0.5327 Epoch: 20 Global Step: 200920 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:29:12,048-Speed 130.55 samples/sec Loss 0.5278 Epoch: 20 Global Step: 200925 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:29:32,057-Speed 127.95 samples/sec Loss 0.5568 Epoch: 20 Global Step: 200930 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:29:51,708-Speed 130.28 samples/sec Loss 0.6502 Epoch: 20 Global Step: 200935 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:30:11,701-Speed 128.05 samples/sec Loss 0.5005 Epoch: 20 Global Step: 200940 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:30:31,452-Speed 129.62 samples/sec Loss 0.5694 Epoch: 20 Global Step: 200945 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:30:51,111-Speed 130.23 samples/sec Loss 0.6756 Epoch: 20 Global Step: 200950 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:31:10,493-Speed 132.09 samples/sec Loss 0.5617 Epoch: 20 Global Step: 200955 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:31:30,485-Speed 128.06 samples/sec Loss 0.5373 Epoch: 20 Global Step: 200960 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:31:50,393-Speed 128.60 samples/sec Loss 0.5990 Epoch: 20 Global Step: 200965 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:32:10,006-Speed 130.53 samples/sec Loss 0.5853 Epoch: 20 Global Step: 200970 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:32:30,078-Speed 127.55 samples/sec Loss 0.5283 Epoch: 20 Global Step: 200975 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:32:49,741-Speed 130.20 samples/sec Loss 0.5389 Epoch: 20 Global Step: 200980 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:33:09,718-Speed 128.16 samples/sec Loss 0.6039 Epoch: 20 Global Step: 200985 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:33:29,434-Speed 129.85 samples/sec Loss 0.5285 Epoch: 20 Global Step: 200990 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:33:49,243-Speed 129.24 samples/sec Loss 0.5239 Epoch: 20 Global Step: 200995 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:34:08,948-Speed 129.93 samples/sec Loss 0.5406 Epoch: 20 Global Step: 201000 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:34:28,888-Speed 128.39 samples/sec Loss 0.4901 Epoch: 20 Global Step: 201005 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:34:48,756-Speed 128.86 samples/sec Loss 0.5461 Epoch: 20 Global Step: 201010 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:35:08,365-Speed 130.56 samples/sec Loss 0.5322 Epoch: 20 Global Step: 201015 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:35:28,496-Speed 127.17 samples/sec Loss 0.4830 Epoch: 20 Global Step: 201020 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:35:48,283-Speed 129.39 samples/sec Loss 0.6307 Epoch: 20 Global Step: 201025 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:36:08,071-Speed 129.38 samples/sec Loss 0.4959 Epoch: 20 Global Step: 201030 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:36:27,870-Speed 129.30 samples/sec Loss 0.4934 Epoch: 20 Global Step: 201035 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:36:47,735-Speed 128.88 samples/sec Loss 0.5759 Epoch: 20 Global Step: 201040 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:37:07,443-Speed 129.90 samples/sec Loss 0.4896 Epoch: 20 Global Step: 201045 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:37:27,564-Speed 127.23 samples/sec Loss 0.5881 Epoch: 20 Global Step: 201050 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:37:47,292-Speed 129.77 samples/sec Loss 0.5376 Epoch: 20 Global Step: 201055 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:38:06,897-Speed 130.59 samples/sec Loss 0.5803 Epoch: 20 Global Step: 201060 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:38:26,963-Speed 127.59 samples/sec Loss 0.6936 Epoch: 20 Global Step: 201065 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:38:46,756-Speed 129.34 samples/sec Loss 0.4809 Epoch: 20 Global Step: 201070 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:39:06,556-Speed 129.30 samples/sec Loss 0.5393 Epoch: 20 Global Step: 201075 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:39:26,331-Speed 129.46 samples/sec Loss 0.4777 Epoch: 20 Global Step: 201080 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:39:46,088-Speed 129.58 samples/sec Loss 0.4663 Epoch: 20 Global Step: 201085 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:40:05,788-Speed 129.95 samples/sec Loss 0.5385 Epoch: 20 Global Step: 201090 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:40:26,138-Speed 125.80 samples/sec Loss 0.6032 Epoch: 20 Global Step: 201095 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:40:45,431-Speed 132.70 samples/sec Loss 0.5762 Epoch: 20 Global Step: 201100 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:41:05,093-Speed 130.21 samples/sec Loss 0.5402 Epoch: 20 Global Step: 201105 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:41:25,132-Speed 127.76 samples/sec Loss 0.5604 Epoch: 20 Global Step: 201110 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:41:45,029-Speed 128.67 samples/sec Loss 0.5030 Epoch: 20 Global Step: 201115 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:42:04,757-Speed 129.77 samples/sec Loss 0.5173 Epoch: 20 Global Step: 201120 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:42:24,482-Speed 129.79 samples/sec Loss 0.6027 Epoch: 20 Global Step: 201125 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:42:44,247-Speed 129.53 samples/sec Loss 0.5184 Epoch: 20 Global Step: 201130 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:43:03,962-Speed 129.86 samples/sec Loss 0.5831 Epoch: 20 Global Step: 201135 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:43:23,879-Speed 128.54 samples/sec Loss 0.5779 Epoch: 20 Global Step: 201140 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:43:43,607-Speed 129.77 samples/sec Loss 0.5477 Epoch: 20 Global Step: 201145 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:44:03,550-Speed 128.37 samples/sec Loss 0.5271 Epoch: 20 Global Step: 201150 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:44:23,600-Speed 127.69 samples/sec Loss 0.5221 Epoch: 20 Global Step: 201155 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:44:43,509-Speed 128.59 samples/sec Loss 0.5196 Epoch: 20 Global Step: 201160 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:45:03,311-Speed 129.29 samples/sec Loss 0.4917 Epoch: 20 Global Step: 201165 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:45:23,209-Speed 128.66 samples/sec Loss 0.5646 Epoch: 20 Global Step: 201170 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:45:42,838-Speed 130.43 samples/sec Loss 0.5427 Epoch: 20 Global Step: 201175 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:46:02,560-Speed 129.81 samples/sec Loss 0.5224 Epoch: 20 Global Step: 201180 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:46:22,492-Speed 128.44 samples/sec Loss 0.5288 Epoch: 20 Global Step: 201185 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:46:42,204-Speed 129.87 samples/sec Loss 0.5423 Epoch: 20 Global Step: 201190 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:47:02,236-Speed 127.81 samples/sec Loss 0.6044 Epoch: 20 Global Step: 201195 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:47:22,273-Speed 127.77 samples/sec Loss 0.5681 Epoch: 20 Global Step: 201200 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:47:42,199-Speed 128.48 samples/sec Loss 0.5577 Epoch: 20 Global Step: 201205 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:48:01,915-Speed 129.85 samples/sec Loss 0.5478 Epoch: 20 Global Step: 201210 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:48:21,731-Speed 129.19 samples/sec Loss 0.6264 Epoch: 20 Global Step: 201215 Fp16 Grad Scale: 8192 Required: 58 hours Training: 2021-06-16 20:48:40,983-Speed 132.98 samples/sec Loss 0.6326 Epoch: 20 Global Step: 201220 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:49:00,790-Speed 129.25 samples/sec Loss 0.5816 Epoch: 20 Global Step: 201225 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:49:20,578-Speed 129.38 samples/sec Loss 0.5288 Epoch: 20 Global Step: 201230 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:49:40,345-Speed 129.51 samples/sec Loss 0.5758 Epoch: 20 Global Step: 201235 Fp16 Grad Scale: 4096 Required: 58 hours Training: 2021-06-16 20:50:00,355-Speed 127.95 samples/sec Loss 0.6402 Epoch: 20 Global Step: 201240 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:50:20,068-Speed 129.87 samples/sec Loss 0.5026 Epoch: 20 Global Step: 201245 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:50:39,895-Speed 129.12 samples/sec Loss 0.5095 Epoch: 20 Global Step: 201250 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:50:59,683-Speed 129.38 samples/sec Loss 0.4847 Epoch: 20 Global Step: 201255 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:51:19,266-Speed 130.73 samples/sec Loss 0.5523 Epoch: 20 Global Step: 201260 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:51:39,059-Speed 129.35 samples/sec Loss 0.5175 Epoch: 20 Global Step: 201265 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:51:58,982-Speed 128.50 samples/sec Loss 0.6203 Epoch: 20 Global Step: 201270 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:52:18,758-Speed 129.45 samples/sec Loss 0.6484 Epoch: 20 Global Step: 201275 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:52:38,377-Speed 130.50 samples/sec Loss 0.4955 Epoch: 20 Global Step: 201280 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:52:58,146-Speed 129.50 samples/sec Loss 0.5911 Epoch: 20 Global Step: 201285 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:53:17,963-Speed 129.18 samples/sec Loss 0.6013 Epoch: 20 Global Step: 201290 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:53:37,574-Speed 130.55 samples/sec Loss 0.4530 Epoch: 20 Global Step: 201295 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:53:57,374-Speed 129.29 samples/sec Loss 0.5987 Epoch: 20 Global Step: 201300 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:54:16,969-Speed 130.65 samples/sec Loss 0.6199 Epoch: 20 Global Step: 201305 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:54:36,680-Speed 129.88 samples/sec Loss 0.5267 Epoch: 20 Global Step: 201310 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:54:56,662-Speed 128.12 samples/sec Loss 0.5282 Epoch: 20 Global Step: 201315 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 20:55:16,479-Speed 129.19 samples/sec Loss 0.5412 Epoch: 20 Global Step: 201320 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:55:36,316-Speed 129.06 samples/sec Loss 0.5746 Epoch: 20 Global Step: 201325 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:55:56,146-Speed 129.10 samples/sec Loss 0.5364 Epoch: 20 Global Step: 201330 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:56:16,120-Speed 128.17 samples/sec Loss 0.5254 Epoch: 20 Global Step: 201335 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:56:35,797-Speed 130.11 samples/sec Loss 0.4927 Epoch: 20 Global Step: 201340 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:56:55,612-Speed 129.20 samples/sec Loss 0.5187 Epoch: 20 Global Step: 201345 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:57:15,218-Speed 130.58 samples/sec Loss 0.4966 Epoch: 20 Global Step: 201350 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:57:34,902-Speed 130.06 samples/sec Loss 0.5502 Epoch: 20 Global Step: 201355 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:57:54,769-Speed 128.87 samples/sec Loss 0.5453 Epoch: 20 Global Step: 201360 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:58:14,518-Speed 129.63 samples/sec Loss 0.4984 Epoch: 20 Global Step: 201365 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:58:34,186-Speed 130.17 samples/sec Loss 0.5281 Epoch: 20 Global Step: 201370 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:58:54,018-Speed 129.09 samples/sec Loss 0.5811 Epoch: 20 Global Step: 201375 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:59:13,778-Speed 129.56 samples/sec Loss 0.5826 Epoch: 20 Global Step: 201380 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:59:33,339-Speed 130.87 samples/sec Loss 0.5790 Epoch: 20 Global Step: 201385 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 20:59:53,029-Speed 130.03 samples/sec Loss 0.4370 Epoch: 20 Global Step: 201390 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:00:12,634-Speed 130.58 samples/sec Loss 0.6070 Epoch: 20 Global Step: 201395 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:00:32,355-Speed 129.82 samples/sec Loss 0.5947 Epoch: 20 Global Step: 201400 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:00:52,134-Speed 129.43 samples/sec Loss 0.5556 Epoch: 20 Global Step: 201405 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:01:11,800-Speed 130.18 samples/sec Loss 0.5113 Epoch: 20 Global Step: 201410 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:01:31,410-Speed 130.55 samples/sec Loss 0.4922 Epoch: 20 Global Step: 201415 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:01:51,319-Speed 128.59 samples/sec Loss 0.5103 Epoch: 20 Global Step: 201420 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:02:11,202-Speed 128.76 samples/sec Loss 0.4823 Epoch: 20 Global Step: 201425 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:02:30,625-Speed 131.81 samples/sec Loss 0.5494 Epoch: 20 Global Step: 201430 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:02:49,955-Speed 132.45 samples/sec Loss 0.5401 Epoch: 20 Global Step: 201435 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:03:09,522-Speed 130.84 samples/sec Loss 0.4694 Epoch: 20 Global Step: 201440 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:03:29,283-Speed 129.56 samples/sec Loss 0.4945 Epoch: 20 Global Step: 201445 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:03:49,100-Speed 129.19 samples/sec Loss 0.5377 Epoch: 20 Global Step: 201450 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:04:08,932-Speed 129.09 samples/sec Loss 0.5471 Epoch: 20 Global Step: 201455 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:04:28,499-Speed 130.84 samples/sec Loss 0.5710 Epoch: 20 Global Step: 201460 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:04:48,434-Speed 128.43 samples/sec Loss 0.5055 Epoch: 20 Global Step: 201465 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:05:08,347-Speed 128.56 samples/sec Loss 0.6325 Epoch: 20 Global Step: 201470 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:05:27,771-Speed 131.81 samples/sec Loss 0.5252 Epoch: 20 Global Step: 201475 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:05:47,563-Speed 129.35 samples/sec Loss 0.5553 Epoch: 20 Global Step: 201480 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:06:07,149-Speed 130.71 samples/sec Loss 0.5199 Epoch: 20 Global Step: 201485 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:06:26,863-Speed 129.86 samples/sec Loss 0.5336 Epoch: 20 Global Step: 201490 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:06:46,673-Speed 129.24 samples/sec Loss 0.5846 Epoch: 20 Global Step: 201495 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:07:06,284-Speed 130.54 samples/sec Loss 0.5379 Epoch: 20 Global Step: 201500 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:07:26,038-Speed 129.60 samples/sec Loss 0.5089 Epoch: 20 Global Step: 201505 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:07:45,835-Speed 129.32 samples/sec Loss 0.6214 Epoch: 20 Global Step: 201510 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:08:05,658-Speed 129.15 samples/sec Loss 0.6018 Epoch: 20 Global Step: 201515 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:08:25,194-Speed 131.05 samples/sec Loss 0.6022 Epoch: 20 Global Step: 201520 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:08:44,918-Speed 129.80 samples/sec Loss 0.5211 Epoch: 20 Global Step: 201525 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:09:04,517-Speed 130.63 samples/sec Loss 0.6008 Epoch: 20 Global Step: 201530 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:09:24,197-Speed 130.09 samples/sec Loss 0.6816 Epoch: 20 Global Step: 201535 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:09:44,002-Speed 129.27 samples/sec Loss 0.4937 Epoch: 20 Global Step: 201540 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:10:03,132-Speed 133.83 samples/sec Loss 0.5624 Epoch: 20 Global Step: 201545 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:10:23,053-Speed 128.51 samples/sec Loss 0.5685 Epoch: 20 Global Step: 201550 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:10:42,808-Speed 129.60 samples/sec Loss 0.5638 Epoch: 20 Global Step: 201555 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:11:02,615-Speed 129.25 samples/sec Loss 0.5850 Epoch: 20 Global Step: 201560 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:11:22,426-Speed 129.22 samples/sec Loss 0.5205 Epoch: 20 Global Step: 201565 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:11:42,267-Speed 129.04 samples/sec Loss 0.5892 Epoch: 20 Global Step: 201570 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:12:02,106-Speed 129.04 samples/sec Loss 0.6025 Epoch: 20 Global Step: 201575 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:12:21,751-Speed 130.32 samples/sec Loss 0.6096 Epoch: 20 Global Step: 201580 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:12:41,725-Speed 128.17 samples/sec Loss 0.5916 Epoch: 20 Global Step: 201585 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:13:01,532-Speed 129.25 samples/sec Loss 0.5862 Epoch: 20 Global Step: 201590 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:13:21,320-Speed 129.38 samples/sec Loss 0.4525 Epoch: 20 Global Step: 201595 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:13:41,105-Speed 129.40 samples/sec Loss 0.5150 Epoch: 20 Global Step: 201600 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:14:00,867-Speed 129.54 samples/sec Loss 0.5724 Epoch: 20 Global Step: 201605 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:14:20,712-Speed 129.01 samples/sec Loss 0.5360 Epoch: 20 Global Step: 201610 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:14:40,550-Speed 129.05 samples/sec Loss 0.6405 Epoch: 20 Global Step: 201615 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:15:00,397-Speed 129.00 samples/sec Loss 0.5426 Epoch: 20 Global Step: 201620 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:15:20,118-Speed 129.82 samples/sec Loss 0.6475 Epoch: 20 Global Step: 201625 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:15:40,091-Speed 128.18 samples/sec Loss 0.5398 Epoch: 20 Global Step: 201630 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:15:59,892-Speed 129.29 samples/sec Loss 0.5457 Epoch: 20 Global Step: 201635 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:16:19,731-Speed 129.04 samples/sec Loss 0.5552 Epoch: 20 Global Step: 201640 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:16:39,457-Speed 129.79 samples/sec Loss 0.5608 Epoch: 20 Global Step: 201645 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:16:59,347-Speed 128.71 samples/sec Loss 0.5141 Epoch: 20 Global Step: 201650 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:17:19,240-Speed 128.69 samples/sec Loss 0.5771 Epoch: 20 Global Step: 201655 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:17:39,044-Speed 129.27 samples/sec Loss 0.6214 Epoch: 20 Global Step: 201660 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:17:59,126-Speed 127.49 samples/sec Loss 0.5210 Epoch: 20 Global Step: 201665 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:18:18,420-Speed 132.69 samples/sec Loss 0.6371 Epoch: 20 Global Step: 201670 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:18:38,174-Speed 129.61 samples/sec Loss 0.5610 Epoch: 20 Global Step: 201675 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:18:57,961-Speed 129.38 samples/sec Loss 0.5732 Epoch: 20 Global Step: 201680 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:19:17,910-Speed 128.34 samples/sec Loss 0.5204 Epoch: 20 Global Step: 201685 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:19:37,756-Speed 129.00 samples/sec Loss 0.5482 Epoch: 20 Global Step: 201690 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:19:57,249-Speed 131.34 samples/sec Loss 0.5732 Epoch: 20 Global Step: 201695 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:20:16,994-Speed 129.65 samples/sec Loss 0.6076 Epoch: 20 Global Step: 201700 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:20:36,903-Speed 128.60 samples/sec Loss 0.5521 Epoch: 20 Global Step: 201705 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:20:57,130-Speed 126.57 samples/sec Loss 0.5582 Epoch: 20 Global Step: 201710 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:21:17,002-Speed 128.83 samples/sec Loss 0.5298 Epoch: 20 Global Step: 201715 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:21:36,427-Speed 131.80 samples/sec Loss 0.5620 Epoch: 20 Global Step: 201720 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:21:56,746-Speed 126.00 samples/sec Loss 0.6184 Epoch: 20 Global Step: 201725 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:22:16,658-Speed 128.57 samples/sec Loss 0.5118 Epoch: 20 Global Step: 201730 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:22:36,579-Speed 128.52 samples/sec Loss 0.4964 Epoch: 20 Global Step: 201735 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:22:56,004-Speed 131.79 samples/sec Loss 0.4941 Epoch: 20 Global Step: 201740 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:23:16,131-Speed 127.19 samples/sec Loss 0.4661 Epoch: 20 Global Step: 201745 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:23:36,068-Speed 128.41 samples/sec Loss 0.5369 Epoch: 20 Global Step: 201750 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:23:55,621-Speed 130.94 samples/sec Loss 0.5087 Epoch: 20 Global Step: 201755 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:24:15,442-Speed 129.17 samples/sec Loss 0.6056 Epoch: 20 Global Step: 201760 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:24:35,431-Speed 128.07 samples/sec Loss 0.5755 Epoch: 20 Global Step: 201765 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:24:55,197-Speed 129.53 samples/sec Loss 0.6561 Epoch: 20 Global Step: 201770 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:25:14,540-Speed 132.36 samples/sec Loss 0.6346 Epoch: 20 Global Step: 201775 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:25:34,472-Speed 128.44 samples/sec Loss 0.5428 Epoch: 20 Global Step: 201780 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:25:53,884-Speed 131.89 samples/sec Loss 0.6009 Epoch: 20 Global Step: 201785 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:26:13,744-Speed 128.91 samples/sec Loss 0.4556 Epoch: 20 Global Step: 201790 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:26:33,605-Speed 128.90 samples/sec Loss 0.5293 Epoch: 20 Global Step: 201795 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:26:53,154-Speed 130.96 samples/sec Loss 0.5055 Epoch: 20 Global Step: 201800 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:27:13,034-Speed 128.78 samples/sec Loss 0.5495 Epoch: 20 Global Step: 201805 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:27:32,557-Speed 131.13 samples/sec Loss 0.6104 Epoch: 20 Global Step: 201810 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:27:52,331-Speed 129.47 samples/sec Loss 0.6484 Epoch: 20 Global Step: 201815 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:28:12,126-Speed 129.33 samples/sec Loss 0.5000 Epoch: 20 Global Step: 201820 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:28:31,989-Speed 128.89 samples/sec Loss 0.5541 Epoch: 20 Global Step: 201825 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:28:51,414-Speed 131.79 samples/sec Loss 0.5036 Epoch: 20 Global Step: 201830 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:29:11,301-Speed 128.74 samples/sec Loss 0.4926 Epoch: 20 Global Step: 201835 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:29:31,075-Speed 129.47 samples/sec Loss 0.6294 Epoch: 20 Global Step: 201840 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:29:50,639-Speed 130.86 samples/sec Loss 0.5424 Epoch: 20 Global Step: 201845 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:30:10,447-Speed 129.25 samples/sec Loss 0.5507 Epoch: 20 Global Step: 201850 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:30:30,005-Speed 130.90 samples/sec Loss 0.5854 Epoch: 20 Global Step: 201855 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:30:49,791-Speed 129.40 samples/sec Loss 0.5687 Epoch: 20 Global Step: 201860 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:31:09,630-Speed 129.05 samples/sec Loss 0.5697 Epoch: 20 Global Step: 201865 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:31:29,497-Speed 128.86 samples/sec Loss 0.5862 Epoch: 20 Global Step: 201870 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:31:48,910-Speed 131.88 samples/sec Loss 0.5583 Epoch: 20 Global Step: 201875 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:32:08,747-Speed 129.06 samples/sec Loss 0.6268 Epoch: 20 Global Step: 201880 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:32:28,500-Speed 129.61 samples/sec Loss 0.5605 Epoch: 20 Global Step: 201885 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:32:48,148-Speed 130.31 samples/sec Loss 0.5708 Epoch: 20 Global Step: 201890 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:33:08,020-Speed 128.83 samples/sec Loss 0.5750 Epoch: 20 Global Step: 201895 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:33:27,462-Speed 131.68 samples/sec Loss 0.5830 Epoch: 20 Global Step: 201900 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:33:47,432-Speed 128.20 samples/sec Loss 0.5809 Epoch: 20 Global Step: 201905 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:34:07,206-Speed 129.47 samples/sec Loss 0.5087 Epoch: 20 Global Step: 201910 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:34:27,077-Speed 128.83 samples/sec Loss 0.5351 Epoch: 20 Global Step: 201915 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:34:46,495-Speed 131.84 samples/sec Loss 0.5367 Epoch: 20 Global Step: 201920 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:35:06,302-Speed 129.25 samples/sec Loss 0.5148 Epoch: 20 Global Step: 201925 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:35:25,989-Speed 130.04 samples/sec Loss 0.5417 Epoch: 20 Global Step: 201930 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:35:45,536-Speed 130.98 samples/sec Loss 0.5900 Epoch: 20 Global Step: 201935 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:36:05,384-Speed 128.99 samples/sec Loss 0.5043 Epoch: 20 Global Step: 201940 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:36:24,970-Speed 130.71 samples/sec Loss 0.5196 Epoch: 20 Global Step: 201945 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:36:44,898-Speed 128.47 samples/sec Loss 0.5243 Epoch: 20 Global Step: 201950 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:37:04,656-Speed 129.58 samples/sec Loss 0.5404 Epoch: 20 Global Step: 201955 Fp16 Grad Scale: 8192 Required: 57 hours Training: 2021-06-16 21:37:24,058-Speed 131.95 samples/sec Loss 0.5683 Epoch: 20 Global Step: 201960 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:37:43,811-Speed 129.61 samples/sec Loss 0.5258 Epoch: 20 Global Step: 201965 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:38:03,570-Speed 129.57 samples/sec Loss 0.5609 Epoch: 20 Global Step: 201970 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:38:24,281-Speed 123.61 samples/sec Loss 0.5332 Epoch: 20 Global Step: 201975 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:38:44,474-Speed 126.78 samples/sec Loss 0.5500 Epoch: 20 Global Step: 201980 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:39:04,212-Speed 129.71 samples/sec Loss 0.5237 Epoch: 20 Global Step: 201985 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:39:24,007-Speed 129.33 samples/sec Loss 0.5386 Epoch: 20 Global Step: 201990 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:39:43,796-Speed 129.37 samples/sec Loss 0.4897 Epoch: 20 Global Step: 201995 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:40:03,862-Speed 127.59 samples/sec Loss 0.5848 Epoch: 20 Global Step: 202000 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:40:23,801-Speed 128.40 samples/sec Loss 0.5824 Epoch: 20 Global Step: 202005 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:40:43,464-Speed 130.20 samples/sec Loss 0.4701 Epoch: 20 Global Step: 202010 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:41:03,302-Speed 129.06 samples/sec Loss 0.5755 Epoch: 20 Global Step: 202015 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:41:23,059-Speed 129.57 samples/sec Loss 0.6608 Epoch: 20 Global Step: 202020 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:41:43,240-Speed 126.86 samples/sec Loss 0.5082 Epoch: 20 Global Step: 202025 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:42:02,928-Speed 130.03 samples/sec Loss 0.4582 Epoch: 20 Global Step: 202030 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:42:22,820-Speed 128.70 samples/sec Loss 0.5454 Epoch: 20 Global Step: 202035 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:42:42,598-Speed 129.44 samples/sec Loss 0.5151 Epoch: 20 Global Step: 202040 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:43:02,608-Speed 127.94 samples/sec Loss 0.5816 Epoch: 20 Global Step: 202045 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:43:22,318-Speed 129.89 samples/sec Loss 0.5911 Epoch: 20 Global Step: 202050 Fp16 Grad Scale: 4096 Required: 57 hours Training: 2021-06-16 21:43:42,122-Speed 129.27 samples/sec Loss 0.5714 Epoch: 20 Global Step: 202055 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:44:01,863-Speed 129.69 samples/sec Loss 0.5468 Epoch: 20 Global Step: 202060 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:44:21,600-Speed 129.71 samples/sec Loss 0.5884 Epoch: 20 Global Step: 202065 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:44:41,758-Speed 127.01 samples/sec Loss 0.5691 Epoch: 20 Global Step: 202070 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:45:01,378-Speed 130.48 samples/sec Loss 0.5907 Epoch: 20 Global Step: 202075 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:45:21,271-Speed 128.70 samples/sec Loss 0.5133 Epoch: 20 Global Step: 202080 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:45:41,063-Speed 129.35 samples/sec Loss 0.5837 Epoch: 20 Global Step: 202085 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:46:01,059-Speed 128.03 samples/sec Loss 0.5273 Epoch: 20 Global Step: 202090 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:46:20,771-Speed 129.88 samples/sec Loss 0.5244 Epoch: 20 Global Step: 202095 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:46:41,067-Speed 126.14 samples/sec Loss 0.6057 Epoch: 20 Global Step: 202100 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:47:00,780-Speed 129.87 samples/sec Loss 0.4939 Epoch: 20 Global Step: 202105 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:47:20,749-Speed 128.21 samples/sec Loss 0.5362 Epoch: 20 Global Step: 202110 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:47:40,754-Speed 127.97 samples/sec Loss 0.6377 Epoch: 20 Global Step: 202115 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:48:00,366-Speed 130.54 samples/sec Loss 0.5730 Epoch: 20 Global Step: 202120 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:48:19,836-Speed 131.50 samples/sec Loss 0.5261 Epoch: 20 Global Step: 202125 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:48:39,601-Speed 129.53 samples/sec Loss 0.5500 Epoch: 20 Global Step: 202130 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:48:59,199-Speed 130.63 samples/sec Loss 0.5033 Epoch: 20 Global Step: 202135 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:49:18,921-Speed 129.81 samples/sec Loss 0.5554 Epoch: 20 Global Step: 202140 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:49:38,472-Speed 130.95 samples/sec Loss 0.5605 Epoch: 20 Global Step: 202145 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:49:58,193-Speed 129.82 samples/sec Loss 0.6218 Epoch: 20 Global Step: 202150 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:50:17,678-Speed 131.39 samples/sec Loss 0.6840 Epoch: 20 Global Step: 202155 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:50:37,742-Speed 127.60 samples/sec Loss 0.5551 Epoch: 20 Global Step: 202160 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:50:57,559-Speed 129.18 samples/sec Loss 0.5694 Epoch: 20 Global Step: 202165 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:51:17,278-Speed 129.83 samples/sec Loss 0.5119 Epoch: 20 Global Step: 202170 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:51:37,093-Speed 129.20 samples/sec Loss 0.5019 Epoch: 20 Global Step: 202175 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:51:56,957-Speed 128.88 samples/sec Loss 0.6372 Epoch: 20 Global Step: 202180 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:52:16,692-Speed 129.73 samples/sec Loss 0.5453 Epoch: 20 Global Step: 202185 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:52:36,587-Speed 128.68 samples/sec Loss 0.5247 Epoch: 20 Global Step: 202190 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:52:56,876-Speed 126.19 samples/sec Loss 0.5097 Epoch: 20 Global Step: 202195 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:53:16,666-Speed 129.37 samples/sec Loss 0.5465 Epoch: 20 Global Step: 202200 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:53:37,009-Speed 125.84 samples/sec Loss 0.6268 Epoch: 20 Global Step: 202205 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:53:56,887-Speed 128.79 samples/sec Loss 0.5540 Epoch: 20 Global Step: 202210 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:54:16,630-Speed 129.67 samples/sec Loss 0.5384 Epoch: 20 Global Step: 202215 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:54:36,543-Speed 128.56 samples/sec Loss 0.5822 Epoch: 20 Global Step: 202220 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:54:56,343-Speed 129.31 samples/sec Loss 0.5802 Epoch: 20 Global Step: 202225 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:55:16,097-Speed 129.60 samples/sec Loss 0.5175 Epoch: 20 Global Step: 202230 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:55:35,946-Speed 128.98 samples/sec Loss 0.6084 Epoch: 20 Global Step: 202235 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:55:55,769-Speed 129.16 samples/sec Loss 0.6284 Epoch: 20 Global Step: 202240 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:56:15,571-Speed 129.28 samples/sec Loss 0.4721 Epoch: 20 Global Step: 202245 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:56:35,781-Speed 126.68 samples/sec Loss 0.6056 Epoch: 20 Global Step: 202250 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:56:55,646-Speed 128.87 samples/sec Loss 0.6385 Epoch: 20 Global Step: 202255 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:57:15,428-Speed 129.42 samples/sec Loss 0.5951 Epoch: 20 Global Step: 202260 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:57:35,240-Speed 129.22 samples/sec Loss 0.5274 Epoch: 20 Global Step: 202265 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:57:54,944-Speed 129.93 samples/sec Loss 0.5456 Epoch: 20 Global Step: 202270 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:58:14,838-Speed 128.69 samples/sec Loss 0.5032 Epoch: 20 Global Step: 202275 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 21:58:34,127-Speed 132.72 samples/sec Loss 0.5762 Epoch: 20 Global Step: 202280 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:58:53,928-Speed 129.29 samples/sec Loss 0.4939 Epoch: 20 Global Step: 202285 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:59:13,734-Speed 129.26 samples/sec Loss 0.5132 Epoch: 20 Global Step: 202290 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:59:34,099-Speed 125.72 samples/sec Loss 0.5250 Epoch: 20 Global Step: 202295 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 21:59:53,831-Speed 129.74 samples/sec Loss 0.5213 Epoch: 20 Global Step: 202300 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:00:13,593-Speed 129.55 samples/sec Loss 0.6004 Epoch: 20 Global Step: 202305 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:00:33,482-Speed 128.72 samples/sec Loss 0.6136 Epoch: 20 Global Step: 202310 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:00:53,199-Speed 129.85 samples/sec Loss 0.5429 Epoch: 20 Global Step: 202315 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:01:13,101-Speed 128.63 samples/sec Loss 0.6235 Epoch: 20 Global Step: 202320 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:01:32,815-Speed 129.86 samples/sec Loss 0.6213 Epoch: 20 Global Step: 202325 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:01:52,641-Speed 129.14 samples/sec Loss 0.6376 Epoch: 20 Global Step: 202330 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:02:12,486-Speed 129.00 samples/sec Loss 0.5925 Epoch: 20 Global Step: 202335 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:02:32,883-Speed 125.51 samples/sec Loss 0.6174 Epoch: 20 Global Step: 202340 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:02:52,901-Speed 127.89 samples/sec Loss 0.5317 Epoch: 20 Global Step: 202345 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:03:12,540-Speed 130.36 samples/sec Loss 0.5817 Epoch: 20 Global Step: 202350 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:03:32,469-Speed 128.46 samples/sec Loss 0.5141 Epoch: 20 Global Step: 202355 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:03:52,205-Speed 129.72 samples/sec Loss 0.6298 Epoch: 20 Global Step: 202360 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:04:12,114-Speed 128.59 samples/sec Loss 0.6146 Epoch: 20 Global Step: 202365 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:04:31,826-Speed 129.88 samples/sec Loss 0.6500 Epoch: 20 Global Step: 202370 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:04:51,608-Speed 129.42 samples/sec Loss 0.6255 Epoch: 20 Global Step: 202375 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:05:11,412-Speed 129.27 samples/sec Loss 0.5817 Epoch: 20 Global Step: 202380 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:05:31,781-Speed 125.69 samples/sec Loss 0.5240 Epoch: 20 Global Step: 202385 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:05:51,595-Speed 129.21 samples/sec Loss 0.5653 Epoch: 20 Global Step: 202390 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:06:11,412-Speed 129.19 samples/sec Loss 0.5641 Epoch: 20 Global Step: 202395 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:06:31,166-Speed 129.60 samples/sec Loss 0.6225 Epoch: 20 Global Step: 202400 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:06:50,842-Speed 130.11 samples/sec Loss 0.5497 Epoch: 20 Global Step: 202405 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:07:10,699-Speed 128.93 samples/sec Loss 0.5101 Epoch: 20 Global Step: 202410 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:07:30,418-Speed 129.83 samples/sec Loss 0.5458 Epoch: 20 Global Step: 202415 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:07:50,268-Speed 128.97 samples/sec Loss 0.5413 Epoch: 20 Global Step: 202420 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:08:10,070-Speed 129.29 samples/sec Loss 0.4387 Epoch: 20 Global Step: 202425 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:08:30,367-Speed 126.13 samples/sec Loss 0.6692 Epoch: 20 Global Step: 202430 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:08:50,093-Speed 129.79 samples/sec Loss 0.4388 Epoch: 20 Global Step: 202435 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:09:09,980-Speed 128.73 samples/sec Loss 0.6112 Epoch: 20 Global Step: 202440 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:09:29,751-Speed 129.49 samples/sec Loss 0.5500 Epoch: 20 Global Step: 202445 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:09:49,534-Speed 129.41 samples/sec Loss 0.6311 Epoch: 20 Global Step: 202450 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:10:09,318-Speed 129.41 samples/sec Loss 0.5571 Epoch: 20 Global Step: 202455 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:10:29,088-Speed 129.50 samples/sec Loss 0.6245 Epoch: 20 Global Step: 202460 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:10:48,950-Speed 128.89 samples/sec Loss 0.5243 Epoch: 20 Global Step: 202465 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:11:08,738-Speed 129.38 samples/sec Loss 0.5519 Epoch: 20 Global Step: 202470 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:11:29,234-Speed 124.91 samples/sec Loss 0.6524 Epoch: 20 Global Step: 202475 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:11:49,099-Speed 128.88 samples/sec Loss 0.5163 Epoch: 20 Global Step: 202480 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:12:08,964-Speed 128.87 samples/sec Loss 0.6693 Epoch: 20 Global Step: 202485 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:12:28,733-Speed 129.50 samples/sec Loss 0.5007 Epoch: 20 Global Step: 202490 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:12:48,548-Speed 129.20 samples/sec Loss 0.5736 Epoch: 20 Global Step: 202495 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:13:08,270-Speed 129.81 samples/sec Loss 0.5900 Epoch: 20 Global Step: 202500 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:13:28,020-Speed 129.62 samples/sec Loss 0.6047 Epoch: 20 Global Step: 202505 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:13:47,762-Speed 129.68 samples/sec Loss 0.5755 Epoch: 20 Global Step: 202510 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:14:07,535-Speed 129.47 samples/sec Loss 0.5876 Epoch: 20 Global Step: 202515 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:14:27,892-Speed 125.76 samples/sec Loss 0.6243 Epoch: 20 Global Step: 202520 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:14:47,794-Speed 128.64 samples/sec Loss 0.5786 Epoch: 20 Global Step: 202525 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:15:07,407-Speed 130.53 samples/sec Loss 0.6143 Epoch: 20 Global Step: 202530 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:15:27,247-Speed 129.04 samples/sec Loss 0.5984 Epoch: 20 Global Step: 202535 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:15:46,995-Speed 129.64 samples/sec Loss 0.5705 Epoch: 20 Global Step: 202540 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:16:06,728-Speed 129.74 samples/sec Loss 0.5758 Epoch: 20 Global Step: 202545 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:16:26,170-Speed 131.68 samples/sec Loss 0.5419 Epoch: 20 Global Step: 202550 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:16:45,649-Speed 131.43 samples/sec Loss 0.5863 Epoch: 20 Global Step: 202555 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:17:05,220-Speed 130.81 samples/sec Loss 0.5423 Epoch: 20 Global Step: 202560 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:17:24,979-Speed 129.57 samples/sec Loss 0.6060 Epoch: 20 Global Step: 202565 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:17:44,902-Speed 128.50 samples/sec Loss 0.5551 Epoch: 20 Global Step: 202570 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:18:04,371-Speed 131.50 samples/sec Loss 0.6029 Epoch: 20 Global Step: 202575 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:18:23,996-Speed 130.47 samples/sec Loss 0.5164 Epoch: 20 Global Step: 202580 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:18:43,914-Speed 128.53 samples/sec Loss 0.5316 Epoch: 20 Global Step: 202585 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:19:03,956-Speed 127.74 samples/sec Loss 0.5443 Epoch: 20 Global Step: 202590 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:19:23,744-Speed 129.38 samples/sec Loss 0.5455 Epoch: 20 Global Step: 202595 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:19:43,364-Speed 130.49 samples/sec Loss 0.5627 Epoch: 20 Global Step: 202600 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:20:02,872-Speed 131.23 samples/sec Loss 0.5076 Epoch: 20 Global Step: 202605 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:20:22,399-Speed 131.11 samples/sec Loss 0.6099 Epoch: 20 Global Step: 202610 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:20:42,164-Speed 129.53 samples/sec Loss 0.5237 Epoch: 20 Global Step: 202615 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:21:01,742-Speed 130.77 samples/sec Loss 0.6115 Epoch: 20 Global Step: 202620 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:21:21,579-Speed 129.06 samples/sec Loss 0.5455 Epoch: 20 Global Step: 202625 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:21:40,825-Speed 133.02 samples/sec Loss 0.6023 Epoch: 20 Global Step: 202630 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:22:00,851-Speed 127.84 samples/sec Loss 0.5705 Epoch: 20 Global Step: 202635 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:22:20,706-Speed 128.94 samples/sec Loss 0.5282 Epoch: 20 Global Step: 202640 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:22:40,343-Speed 130.37 samples/sec Loss 0.5225 Epoch: 20 Global Step: 202645 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:23:00,313-Speed 128.20 samples/sec Loss 0.6648 Epoch: 20 Global Step: 202650 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:23:19,918-Speed 130.58 samples/sec Loss 0.5371 Epoch: 20 Global Step: 202655 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:23:39,753-Speed 129.07 samples/sec Loss 0.6390 Epoch: 20 Global Step: 202660 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:23:59,537-Speed 129.41 samples/sec Loss 0.5374 Epoch: 20 Global Step: 202665 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:24:19,263-Speed 129.78 samples/sec Loss 0.4967 Epoch: 20 Global Step: 202670 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:24:39,238-Speed 128.16 samples/sec Loss 0.5250 Epoch: 20 Global Step: 202675 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:24:59,404-Speed 126.95 samples/sec Loss 0.5771 Epoch: 20 Global Step: 202680 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:25:19,267-Speed 128.89 samples/sec Loss 0.5790 Epoch: 20 Global Step: 202685 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:25:38,947-Speed 130.09 samples/sec Loss 0.4960 Epoch: 20 Global Step: 202690 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:25:58,960-Speed 127.92 samples/sec Loss 0.5391 Epoch: 20 Global Step: 202695 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:26:18,711-Speed 129.62 samples/sec Loss 0.5495 Epoch: 20 Global Step: 202700 Fp16 Grad Scale: 2048 Required: 56 hours Training: 2021-06-16 22:26:38,486-Speed 129.46 samples/sec Loss 0.5752 Epoch: 20 Global Step: 202705 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:26:58,375-Speed 128.72 samples/sec Loss 0.5217 Epoch: 20 Global Step: 202710 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:27:18,161-Speed 129.39 samples/sec Loss 0.5824 Epoch: 20 Global Step: 202715 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:27:38,099-Speed 128.40 samples/sec Loss 0.5527 Epoch: 20 Global Step: 202720 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:27:59,084-Speed 122.00 samples/sec Loss 0.4733 Epoch: 20 Global Step: 202725 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:28:18,900-Speed 129.19 samples/sec Loss 0.5983 Epoch: 20 Global Step: 202730 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:28:38,620-Speed 129.82 samples/sec Loss 0.6465 Epoch: 20 Global Step: 202735 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:28:58,464-Speed 129.02 samples/sec Loss 0.5751 Epoch: 20 Global Step: 202740 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:29:18,190-Speed 129.78 samples/sec Loss 0.4512 Epoch: 20 Global Step: 202745 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:29:38,098-Speed 128.60 samples/sec Loss 0.4668 Epoch: 20 Global Step: 202750 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:29:57,822-Speed 129.80 samples/sec Loss 0.5903 Epoch: 20 Global Step: 202755 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:30:17,817-Speed 128.04 samples/sec Loss 0.6495 Epoch: 20 Global Step: 202760 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:30:37,453-Speed 130.38 samples/sec Loss 0.5429 Epoch: 20 Global Step: 202765 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:30:57,869-Speed 125.40 samples/sec Loss 0.6476 Epoch: 20 Global Step: 202770 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:31:17,605-Speed 129.71 samples/sec Loss 0.5559 Epoch: 20 Global Step: 202775 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:31:37,496-Speed 128.71 samples/sec Loss 0.5741 Epoch: 20 Global Step: 202780 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:31:57,315-Speed 129.18 samples/sec Loss 0.6242 Epoch: 20 Global Step: 202785 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:32:17,006-Speed 130.02 samples/sec Loss 0.5239 Epoch: 20 Global Step: 202790 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:32:36,887-Speed 128.77 samples/sec Loss 0.6040 Epoch: 20 Global Step: 202795 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:32:56,615-Speed 129.77 samples/sec Loss 0.5706 Epoch: 20 Global Step: 202800 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:33:16,577-Speed 128.25 samples/sec Loss 0.5485 Epoch: 20 Global Step: 202805 Fp16 Grad Scale: 8192 Required: 56 hours Training: 2021-06-16 22:33:35,848-Speed 132.85 samples/sec Loss 0.5670 Epoch: 20 Global Step: 202810 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:33:56,156-Speed 126.06 samples/sec Loss 0.5345 Epoch: 20 Global Step: 202815 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:34:15,887-Speed 129.75 samples/sec Loss 0.5513 Epoch: 20 Global Step: 202820 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:34:35,874-Speed 128.09 samples/sec Loss 0.5040 Epoch: 20 Global Step: 202825 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:34:55,676-Speed 129.29 samples/sec Loss 0.6406 Epoch: 20 Global Step: 202830 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:35:15,491-Speed 129.21 samples/sec Loss 0.7099 Epoch: 20 Global Step: 202835 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:35:35,243-Speed 129.61 samples/sec Loss 0.5386 Epoch: 20 Global Step: 202840 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:35:54,998-Speed 129.60 samples/sec Loss 0.4807 Epoch: 20 Global Step: 202845 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:36:14,922-Speed 128.50 samples/sec Loss 0.6099 Epoch: 20 Global Step: 202850 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:36:34,789-Speed 128.86 samples/sec Loss 0.5665 Epoch: 20 Global Step: 202855 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:36:55,156-Speed 125.71 samples/sec Loss 0.5074 Epoch: 20 Global Step: 202860 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:37:15,168-Speed 127.92 samples/sec Loss 0.5313 Epoch: 20 Global Step: 202865 Fp16 Grad Scale: 4096 Required: 56 hours Training: 2021-06-16 22:37:35,002-Speed 129.08 samples/sec Loss 0.5833 Epoch: 20 Global Step: 202870 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:37:54,667-Speed 130.19 samples/sec Loss 0.6185 Epoch: 20 Global Step: 202875 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:38:14,532-Speed 128.88 samples/sec Loss 0.6557 Epoch: 20 Global Step: 202880 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:38:34,694-Speed 126.98 samples/sec Loss 0.5561 Epoch: 20 Global Step: 202885 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:38:54,522-Speed 129.11 samples/sec Loss 0.5530 Epoch: 20 Global Step: 202890 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:39:14,277-Speed 129.60 samples/sec Loss 0.5042 Epoch: 20 Global Step: 202895 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:39:33,947-Speed 130.15 samples/sec Loss 0.5808 Epoch: 20 Global Step: 202900 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:39:54,304-Speed 125.76 samples/sec Loss 0.5917 Epoch: 20 Global Step: 202905 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:40:14,264-Speed 128.26 samples/sec Loss 0.5645 Epoch: 20 Global Step: 202910 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:40:33,976-Speed 129.88 samples/sec Loss 0.6009 Epoch: 20 Global Step: 202915 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:40:53,626-Speed 130.28 samples/sec Loss 0.6010 Epoch: 20 Global Step: 202920 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:41:13,492-Speed 128.87 samples/sec Loss 0.6099 Epoch: 20 Global Step: 202925 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:41:33,211-Speed 129.83 samples/sec Loss 0.4834 Epoch: 20 Global Step: 202930 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:41:53,494-Speed 126.22 samples/sec Loss 0.4912 Epoch: 20 Global Step: 202935 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:42:13,221-Speed 129.78 samples/sec Loss 0.4822 Epoch: 20 Global Step: 202940 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:42:32,884-Speed 130.20 samples/sec Loss 0.6132 Epoch: 20 Global Step: 202945 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:42:53,300-Speed 125.40 samples/sec Loss 0.5637 Epoch: 20 Global Step: 202950 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:43:13,300-Speed 128.01 samples/sec Loss 0.5452 Epoch: 20 Global Step: 202955 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:43:32,981-Speed 130.07 samples/sec Loss 0.6190 Epoch: 20 Global Step: 202960 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:43:52,422-Speed 131.69 samples/sec Loss 0.6002 Epoch: 20 Global Step: 202965 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:44:11,864-Speed 131.68 samples/sec Loss 0.4790 Epoch: 20 Global Step: 202970 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:44:31,424-Speed 130.88 samples/sec Loss 0.6177 Epoch: 20 Global Step: 202975 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:44:51,029-Speed 130.59 samples/sec Loss 0.6141 Epoch: 20 Global Step: 202980 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:45:10,590-Speed 130.88 samples/sec Loss 0.5083 Epoch: 20 Global Step: 202985 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:45:30,682-Speed 127.42 samples/sec Loss 0.4948 Epoch: 20 Global Step: 202990 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:45:50,358-Speed 130.11 samples/sec Loss 0.5772 Epoch: 20 Global Step: 202995 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:46:10,214-Speed 128.94 samples/sec Loss 0.5797 Epoch: 20 Global Step: 203000 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:46:29,927-Speed 129.87 samples/sec Loss 0.6165 Epoch: 20 Global Step: 203005 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:46:49,748-Speed 129.16 samples/sec Loss 0.6219 Epoch: 20 Global Step: 203010 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:47:09,376-Speed 130.44 samples/sec Loss 0.5110 Epoch: 20 Global Step: 203015 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:47:29,554-Speed 126.88 samples/sec Loss 0.5492 Epoch: 20 Global Step: 203020 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:47:49,343-Speed 129.37 samples/sec Loss 0.5969 Epoch: 20 Global Step: 203025 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:48:08,984-Speed 130.34 samples/sec Loss 0.4922 Epoch: 20 Global Step: 203030 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:48:28,953-Speed 128.21 samples/sec Loss 0.6478 Epoch: 20 Global Step: 203035 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:48:48,874-Speed 128.52 samples/sec Loss 0.5821 Epoch: 20 Global Step: 203040 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:49:08,808-Speed 128.42 samples/sec Loss 0.5796 Epoch: 20 Global Step: 203045 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:49:28,527-Speed 129.83 samples/sec Loss 0.5565 Epoch: 20 Global Step: 203050 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:49:48,382-Speed 128.95 samples/sec Loss 0.4872 Epoch: 20 Global Step: 203055 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:50:08,091-Speed 129.90 samples/sec Loss 0.5552 Epoch: 20 Global Step: 203060 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:50:28,467-Speed 125.64 samples/sec Loss 0.5871 Epoch: 20 Global Step: 203065 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:50:48,253-Speed 129.39 samples/sec Loss 0.5435 Epoch: 20 Global Step: 203070 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:51:07,904-Speed 130.29 samples/sec Loss 0.5740 Epoch: 20 Global Step: 203075 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:51:27,771-Speed 128.86 samples/sec Loss 0.6024 Epoch: 20 Global Step: 203080 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:51:47,840-Speed 127.57 samples/sec Loss 0.5695 Epoch: 20 Global Step: 203085 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:52:07,613-Speed 129.48 samples/sec Loss 0.5521 Epoch: 20 Global Step: 203090 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:52:27,373-Speed 129.56 samples/sec Loss 0.5027 Epoch: 20 Global Step: 203095 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:52:47,149-Speed 129.45 samples/sec Loss 0.6240 Epoch: 20 Global Step: 203100 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:53:06,864-Speed 129.86 samples/sec Loss 0.6464 Epoch: 20 Global Step: 203105 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:53:27,082-Speed 126.63 samples/sec Loss 0.6171 Epoch: 20 Global Step: 203110 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:53:46,800-Speed 129.84 samples/sec Loss 0.5421 Epoch: 20 Global Step: 203115 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:54:06,498-Speed 129.97 samples/sec Loss 0.5193 Epoch: 20 Global Step: 203120 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:54:26,335-Speed 129.06 samples/sec Loss 0.6230 Epoch: 20 Global Step: 203125 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:54:46,439-Speed 127.35 samples/sec Loss 0.5022 Epoch: 20 Global Step: 203130 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:55:06,215-Speed 129.46 samples/sec Loss 0.6154 Epoch: 20 Global Step: 203135 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:55:25,962-Speed 129.64 samples/sec Loss 0.6081 Epoch: 20 Global Step: 203140 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:55:45,821-Speed 128.92 samples/sec Loss 0.6867 Epoch: 20 Global Step: 203145 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:56:05,507-Speed 130.04 samples/sec Loss 0.5267 Epoch: 20 Global Step: 203150 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:56:25,756-Speed 126.43 samples/sec Loss 0.6472 Epoch: 20 Global Step: 203155 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:56:45,471-Speed 129.85 samples/sec Loss 0.6029 Epoch: 20 Global Step: 203160 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:57:05,266-Speed 129.34 samples/sec Loss 0.4779 Epoch: 20 Global Step: 203165 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:57:24,956-Speed 130.02 samples/sec Loss 0.5827 Epoch: 20 Global Step: 203170 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:57:45,008-Speed 127.67 samples/sec Loss 0.6433 Epoch: 20 Global Step: 203175 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 22:58:04,163-Speed 133.65 samples/sec Loss 0.6262 Epoch: 20 Global Step: 203180 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:58:24,064-Speed 128.64 samples/sec Loss 0.5495 Epoch: 20 Global Step: 203185 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:58:43,771-Speed 129.91 samples/sec Loss 0.6384 Epoch: 20 Global Step: 203190 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:59:03,292-Speed 131.15 samples/sec Loss 0.5472 Epoch: 20 Global Step: 203195 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:59:22,814-Speed 131.14 samples/sec Loss 0.6426 Epoch: 20 Global Step: 203200 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 22:59:42,352-Speed 131.03 samples/sec Loss 0.5769 Epoch: 20 Global Step: 203205 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:00:02,182-Speed 129.10 samples/sec Loss 0.5443 Epoch: 20 Global Step: 203210 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:00:21,802-Speed 130.49 samples/sec Loss 0.5506 Epoch: 20 Global Step: 203215 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:00:41,668-Speed 128.87 samples/sec Loss 0.5871 Epoch: 20 Global Step: 203220 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:01:01,434-Speed 129.53 samples/sec Loss 0.4947 Epoch: 20 Global Step: 203225 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:01:21,200-Speed 129.52 samples/sec Loss 0.4953 Epoch: 20 Global Step: 203230 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:01:40,973-Speed 129.48 samples/sec Loss 0.5566 Epoch: 20 Global Step: 203235 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:02:00,611-Speed 130.36 samples/sec Loss 0.5479 Epoch: 20 Global Step: 203240 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:02:20,943-Speed 125.92 samples/sec Loss 0.6359 Epoch: 20 Global Step: 203245 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:02:40,852-Speed 128.59 samples/sec Loss 0.5738 Epoch: 20 Global Step: 203250 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:03:00,537-Speed 130.06 samples/sec Loss 0.5612 Epoch: 20 Global Step: 203255 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:03:20,204-Speed 130.17 samples/sec Loss 0.6839 Epoch: 20 Global Step: 203260 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:03:40,053-Speed 128.98 samples/sec Loss 0.6388 Epoch: 20 Global Step: 203265 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:03:59,837-Speed 129.41 samples/sec Loss 0.5145 Epoch: 20 Global Step: 203270 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:04:19,781-Speed 128.37 samples/sec Loss 0.6124 Epoch: 20 Global Step: 203275 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:04:39,601-Speed 129.17 samples/sec Loss 0.6221 Epoch: 20 Global Step: 203280 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 23:04:59,266-Speed 130.18 samples/sec Loss 0.5810 Epoch: 20 Global Step: 203285 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 23:05:19,695-Speed 125.32 samples/sec Loss 0.6006 Epoch: 20 Global Step: 203290 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 23:05:39,666-Speed 128.19 samples/sec Loss 0.6096 Epoch: 20 Global Step: 203295 Fp16 Grad Scale: 8192 Required: 55 hours Training: 2021-06-16 23:05:58,975-Speed 132.59 samples/sec Loss 0.5606 Epoch: 20 Global Step: 203300 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:06:18,624-Speed 130.29 samples/sec Loss 0.5799 Epoch: 20 Global Step: 203305 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:06:38,610-Speed 128.10 samples/sec Loss 0.5285 Epoch: 20 Global Step: 203310 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:06:58,369-Speed 129.57 samples/sec Loss 0.6239 Epoch: 20 Global Step: 203315 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:07:18,271-Speed 128.64 samples/sec Loss 0.5537 Epoch: 20 Global Step: 203320 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:07:37,883-Speed 130.54 samples/sec Loss 0.5329 Epoch: 20 Global Step: 203325 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:07:57,585-Speed 129.94 samples/sec Loss 0.5844 Epoch: 20 Global Step: 203330 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:08:17,963-Speed 125.63 samples/sec Loss 0.5130 Epoch: 20 Global Step: 203335 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:08:37,890-Speed 128.47 samples/sec Loss 0.5438 Epoch: 20 Global Step: 203340 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:08:57,489-Speed 130.62 samples/sec Loss 0.5351 Epoch: 20 Global Step: 203345 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:09:17,277-Speed 129.38 samples/sec Loss 0.5616 Epoch: 20 Global Step: 203350 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:09:36,967-Speed 130.03 samples/sec Loss 0.5610 Epoch: 20 Global Step: 203355 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:09:56,762-Speed 129.33 samples/sec Loss 0.5138 Epoch: 20 Global Step: 203360 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:10:16,614-Speed 128.96 samples/sec Loss 0.6818 Epoch: 20 Global Step: 203365 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:10:36,215-Speed 130.62 samples/sec Loss 0.5992 Epoch: 20 Global Step: 203370 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:10:55,981-Speed 129.53 samples/sec Loss 0.5862 Epoch: 20 Global Step: 203375 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:11:16,404-Speed 125.35 samples/sec Loss 0.6467 Epoch: 20 Global Step: 203380 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:11:35,829-Speed 131.79 samples/sec Loss 0.6022 Epoch: 20 Global Step: 203385 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:11:55,257-Speed 131.78 samples/sec Loss 0.6164 Epoch: 20 Global Step: 203390 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:12:14,877-Speed 130.49 samples/sec Loss 0.5613 Epoch: 20 Global Step: 203395 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:12:34,421-Speed 130.99 samples/sec Loss 0.5898 Epoch: 20 Global Step: 203400 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:12:53,999-Speed 130.76 samples/sec Loss 0.4848 Epoch: 20 Global Step: 203405 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:13:13,872-Speed 128.82 samples/sec Loss 0.5880 Epoch: 20 Global Step: 203410 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:13:33,313-Speed 131.69 samples/sec Loss 0.5617 Epoch: 20 Global Step: 203415 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:13:53,080-Speed 129.52 samples/sec Loss 0.6588 Epoch: 20 Global Step: 203420 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:14:13,095-Speed 127.91 samples/sec Loss 0.5627 Epoch: 20 Global Step: 203425 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:14:33,091-Speed 128.03 samples/sec Loss 0.6149 Epoch: 20 Global Step: 203430 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:14:52,806-Speed 129.85 samples/sec Loss 0.6735 Epoch: 20 Global Step: 203435 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:15:12,645-Speed 129.05 samples/sec Loss 0.5676 Epoch: 20 Global Step: 203440 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:15:32,282-Speed 130.37 samples/sec Loss 0.5886 Epoch: 20 Global Step: 203445 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:15:52,402-Speed 127.24 samples/sec Loss 0.6285 Epoch: 20 Global Step: 203450 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:16:12,274-Speed 128.83 samples/sec Loss 0.5758 Epoch: 20 Global Step: 203455 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:16:31,889-Speed 130.52 samples/sec Loss 0.6846 Epoch: 20 Global Step: 203460 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:16:51,783-Speed 128.69 samples/sec Loss 0.5219 Epoch: 20 Global Step: 203465 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:17:11,718-Speed 128.43 samples/sec Loss 0.5792 Epoch: 20 Global Step: 203470 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:17:31,484-Speed 129.52 samples/sec Loss 0.5868 Epoch: 20 Global Step: 203475 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:17:51,191-Speed 129.91 samples/sec Loss 0.5679 Epoch: 20 Global Step: 203480 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:18:11,007-Speed 129.20 samples/sec Loss 0.6007 Epoch: 20 Global Step: 203485 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:18:30,201-Speed 133.38 samples/sec Loss 0.5307 Epoch: 20 Global Step: 203490 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:18:50,137-Speed 128.42 samples/sec Loss 0.5946 Epoch: 20 Global Step: 203495 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:19:09,929-Speed 129.35 samples/sec Loss 0.5429 Epoch: 20 Global Step: 203500 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:19:29,395-Speed 131.52 samples/sec Loss 0.4744 Epoch: 20 Global Step: 203505 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:19:48,972-Speed 130.77 samples/sec Loss 0.6178 Epoch: 20 Global Step: 203510 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:20:08,711-Speed 129.70 samples/sec Loss 0.5863 Epoch: 20 Global Step: 203515 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:20:28,295-Speed 130.73 samples/sec Loss 0.6172 Epoch: 20 Global Step: 203520 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:20:48,004-Speed 129.90 samples/sec Loss 0.6092 Epoch: 20 Global Step: 203525 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:21:07,859-Speed 128.94 samples/sec Loss 0.6298 Epoch: 20 Global Step: 203530 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:21:27,465-Speed 130.58 samples/sec Loss 0.5839 Epoch: 20 Global Step: 203535 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:21:48,237-Speed 123.24 samples/sec Loss 0.5315 Epoch: 20 Global Step: 203540 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:22:07,930-Speed 130.00 samples/sec Loss 0.5598 Epoch: 20 Global Step: 203545 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:22:27,757-Speed 129.12 samples/sec Loss 0.6262 Epoch: 20 Global Step: 203550 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:22:47,452-Speed 129.99 samples/sec Loss 0.6037 Epoch: 20 Global Step: 203555 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:23:07,268-Speed 129.19 samples/sec Loss 0.4890 Epoch: 20 Global Step: 203560 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:23:28,431-Speed 120.97 samples/sec Loss 0.6477 Epoch: 20 Global Step: 203565 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:23:48,168-Speed 129.71 samples/sec Loss 0.7083 Epoch: 20 Global Step: 203570 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:24:08,060-Speed 128.71 samples/sec Loss 0.6747 Epoch: 20 Global Step: 203575 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:24:27,691-Speed 130.41 samples/sec Loss 0.5701 Epoch: 20 Global Step: 203580 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:24:48,124-Speed 125.29 samples/sec Loss 0.5642 Epoch: 20 Global Step: 203585 Fp16 Grad Scale: 2048 Required: 55 hours Training: 2021-06-16 23:25:07,919-Speed 129.33 samples/sec Loss 0.5564 Epoch: 20 Global Step: 203590 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:25:27,785-Speed 128.87 samples/sec Loss 0.6575 Epoch: 20 Global Step: 203595 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:25:47,499-Speed 129.86 samples/sec Loss 0.5778 Epoch: 20 Global Step: 203600 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:26:07,456-Speed 128.28 samples/sec Loss 0.6158 Epoch: 20 Global Step: 203605 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:26:27,237-Speed 129.43 samples/sec Loss 0.5865 Epoch: 20 Global Step: 203610 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:26:46,908-Speed 130.15 samples/sec Loss 0.6297 Epoch: 20 Global Step: 203615 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:27:06,785-Speed 128.80 samples/sec Loss 0.6162 Epoch: 20 Global Step: 203620 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:27:26,499-Speed 129.86 samples/sec Loss 0.5826 Epoch: 20 Global Step: 203625 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:27:46,933-Speed 125.29 samples/sec Loss 0.5561 Epoch: 20 Global Step: 203630 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:28:06,618-Speed 130.05 samples/sec Loss 0.5260 Epoch: 20 Global Step: 203635 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:28:26,377-Speed 129.56 samples/sec Loss 0.6030 Epoch: 20 Global Step: 203640 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:28:46,000-Speed 130.46 samples/sec Loss 0.4937 Epoch: 20 Global Step: 203645 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:29:05,799-Speed 129.31 samples/sec Loss 0.5501 Epoch: 20 Global Step: 203650 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:29:25,624-Speed 129.14 samples/sec Loss 0.6142 Epoch: 20 Global Step: 203655 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:29:45,252-Speed 130.43 samples/sec Loss 0.5794 Epoch: 20 Global Step: 203660 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:30:05,090-Speed 129.06 samples/sec Loss 0.5502 Epoch: 20 Global Step: 203665 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:30:24,715-Speed 130.45 samples/sec Loss 0.5893 Epoch: 20 Global Step: 203670 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:30:45,554-Speed 122.85 samples/sec Loss 0.5875 Epoch: 20 Global Step: 203675 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:31:05,312-Speed 129.57 samples/sec Loss 0.6573 Epoch: 20 Global Step: 203680 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:31:25,093-Speed 129.43 samples/sec Loss 0.6822 Epoch: 20 Global Step: 203685 Fp16 Grad Scale: 4096 Required: 55 hours Training: 2021-06-16 23:31:44,822-Speed 129.76 samples/sec Loss 0.5493 Epoch: 20 Global Step: 203690 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:32:04,794-Speed 128.19 samples/sec Loss 0.6575 Epoch: 20 Global Step: 203695 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:32:24,508-Speed 129.86 samples/sec Loss 0.6833 Epoch: 20 Global Step: 203700 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:32:44,250-Speed 129.68 samples/sec Loss 0.5553 Epoch: 20 Global Step: 203705 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:33:04,057-Speed 129.25 samples/sec Loss 0.6250 Epoch: 20 Global Step: 203710 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:33:23,786-Speed 129.76 samples/sec Loss 0.5805 Epoch: 20 Global Step: 203715 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:33:44,214-Speed 125.32 samples/sec Loss 0.5592 Epoch: 20 Global Step: 203720 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:34:04,088-Speed 128.82 samples/sec Loss 0.5561 Epoch: 20 Global Step: 203725 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:34:23,799-Speed 129.88 samples/sec Loss 0.5823 Epoch: 20 Global Step: 203730 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:34:43,568-Speed 129.50 samples/sec Loss 0.6172 Epoch: 20 Global Step: 203735 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:35:03,489-Speed 128.52 samples/sec Loss 0.5561 Epoch: 20 Global Step: 203740 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:35:23,224-Speed 129.72 samples/sec Loss 0.5722 Epoch: 20 Global Step: 203745 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:35:43,128-Speed 128.62 samples/sec Loss 0.6177 Epoch: 20 Global Step: 203750 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:36:02,827-Speed 129.97 samples/sec Loss 0.6211 Epoch: 20 Global Step: 203755 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:36:22,563-Speed 129.72 samples/sec Loss 0.5322 Epoch: 20 Global Step: 203760 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:36:42,968-Speed 125.47 samples/sec Loss 0.6018 Epoch: 20 Global Step: 203765 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:37:02,430-Speed 131.55 samples/sec Loss 0.6342 Epoch: 20 Global Step: 203770 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:37:22,131-Speed 129.94 samples/sec Loss 0.4793 Epoch: 20 Global Step: 203775 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:37:41,952-Speed 129.17 samples/sec Loss 0.6414 Epoch: 20 Global Step: 203780 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:38:01,874-Speed 128.51 samples/sec Loss 0.6122 Epoch: 20 Global Step: 203785 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:38:21,585-Speed 129.88 samples/sec Loss 0.6266 Epoch: 20 Global Step: 203790 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:38:41,585-Speed 128.01 samples/sec Loss 0.6228 Epoch: 20 Global Step: 203795 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:39:01,307-Speed 129.81 samples/sec Loss 0.6603 Epoch: 20 Global Step: 203800 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:39:21,034-Speed 129.78 samples/sec Loss 0.6832 Epoch: 20 Global Step: 203805 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:39:41,441-Speed 125.46 samples/sec Loss 0.6056 Epoch: 20 Global Step: 203810 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:40:01,309-Speed 128.86 samples/sec Loss 0.5829 Epoch: 20 Global Step: 203815 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:40:20,996-Speed 130.04 samples/sec Loss 0.6263 Epoch: 20 Global Step: 203820 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:40:40,870-Speed 128.82 samples/sec Loss 0.6083 Epoch: 20 Global Step: 203825 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:41:00,701-Speed 129.10 samples/sec Loss 0.6403 Epoch: 20 Global Step: 203830 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:41:20,440-Speed 129.70 samples/sec Loss 0.5902 Epoch: 20 Global Step: 203835 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:41:40,264-Speed 129.14 samples/sec Loss 0.5484 Epoch: 20 Global Step: 203840 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:41:59,873-Speed 130.56 samples/sec Loss 0.6106 Epoch: 20 Global Step: 203845 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:42:19,699-Speed 129.13 samples/sec Loss 0.5702 Epoch: 20 Global Step: 203850 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:42:39,959-Speed 126.36 samples/sec Loss 0.5686 Epoch: 20 Global Step: 203855 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:42:59,819-Speed 128.91 samples/sec Loss 0.5895 Epoch: 20 Global Step: 203860 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:43:19,455-Speed 130.38 samples/sec Loss 0.6982 Epoch: 20 Global Step: 203865 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:43:39,348-Speed 128.69 samples/sec Loss 0.6045 Epoch: 20 Global Step: 203870 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:43:59,057-Speed 129.90 samples/sec Loss 0.5768 Epoch: 20 Global Step: 203875 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:44:18,773-Speed 129.85 samples/sec Loss 0.6314 Epoch: 20 Global Step: 203880 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:44:38,651-Speed 128.79 samples/sec Loss 0.5822 Epoch: 20 Global Step: 203885 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:44:57,845-Speed 133.38 samples/sec Loss 0.6201 Epoch: 20 Global Step: 203890 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:45:17,421-Speed 130.78 samples/sec Loss 0.6694 Epoch: 20 Global Step: 203895 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:45:37,247-Speed 129.13 samples/sec Loss 0.6270 Epoch: 20 Global Step: 203900 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:45:56,970-Speed 129.80 samples/sec Loss 0.5089 Epoch: 20 Global Step: 203905 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:46:16,378-Speed 131.91 samples/sec Loss 0.5538 Epoch: 20 Global Step: 203910 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:46:36,261-Speed 128.75 samples/sec Loss 0.5716 Epoch: 20 Global Step: 203915 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:46:56,032-Speed 129.49 samples/sec Loss 0.6030 Epoch: 20 Global Step: 203920 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:47:15,730-Speed 129.97 samples/sec Loss 0.6270 Epoch: 20 Global Step: 203925 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:47:35,618-Speed 128.72 samples/sec Loss 0.4853 Epoch: 20 Global Step: 203930 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:47:55,335-Speed 129.85 samples/sec Loss 0.6640 Epoch: 20 Global Step: 203935 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:48:15,246-Speed 128.58 samples/sec Loss 0.6239 Epoch: 20 Global Step: 203940 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:48:35,244-Speed 128.02 samples/sec Loss 0.6351 Epoch: 20 Global Step: 203945 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:48:55,076-Speed 129.09 samples/sec Loss 0.5186 Epoch: 20 Global Step: 203950 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:49:14,777-Speed 129.95 samples/sec Loss 0.5576 Epoch: 20 Global Step: 203955 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:49:34,704-Speed 128.48 samples/sec Loss 0.6211 Epoch: 20 Global Step: 203960 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:49:54,471-Speed 129.51 samples/sec Loss 0.6440 Epoch: 20 Global Step: 203965 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:50:14,400-Speed 128.46 samples/sec Loss 0.6021 Epoch: 20 Global Step: 203970 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:50:34,259-Speed 128.92 samples/sec Loss 0.5234 Epoch: 20 Global Step: 203975 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:50:53,984-Speed 129.79 samples/sec Loss 0.5654 Epoch: 20 Global Step: 203980 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:51:13,949-Speed 128.23 samples/sec Loss 0.6096 Epoch: 20 Global Step: 203985 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:51:33,894-Speed 128.36 samples/sec Loss 0.6425 Epoch: 20 Global Step: 203990 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:51:53,691-Speed 129.32 samples/sec Loss 0.6962 Epoch: 20 Global Step: 203995 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:52:13,402-Speed 129.89 samples/sec Loss 0.6486 Epoch: 20 Global Step: 204000 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:52:33,320-Speed 128.53 samples/sec Loss 0.6049 Epoch: 20 Global Step: 204005 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:52:53,097-Speed 129.45 samples/sec Loss 0.5939 Epoch: 20 Global Step: 204010 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:53:13,105-Speed 127.95 samples/sec Loss 0.6403 Epoch: 20 Global Step: 204015 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:53:32,941-Speed 129.07 samples/sec Loss 0.5427 Epoch: 20 Global Step: 204020 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:53:52,656-Speed 129.86 samples/sec Loss 0.5007 Epoch: 20 Global Step: 204025 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:54:12,622-Speed 128.22 samples/sec Loss 0.5336 Epoch: 20 Global Step: 204030 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-16 23:54:31,896-Speed 132.83 samples/sec Loss 0.5508 Epoch: 20 Global Step: 204035 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:54:51,786-Speed 128.71 samples/sec Loss 0.5810 Epoch: 20 Global Step: 204040 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:55:11,539-Speed 129.61 samples/sec Loss 0.6889 Epoch: 20 Global Step: 204045 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:55:31,403-Speed 128.89 samples/sec Loss 0.5829 Epoch: 20 Global Step: 204050 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:55:50,861-Speed 131.57 samples/sec Loss 0.5611 Epoch: 20 Global Step: 204055 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:56:10,493-Speed 130.40 samples/sec Loss 0.5517 Epoch: 20 Global Step: 204060 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:56:29,957-Speed 131.53 samples/sec Loss 0.5758 Epoch: 20 Global Step: 204065 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:56:49,853-Speed 128.68 samples/sec Loss 0.5284 Epoch: 20 Global Step: 204070 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:57:09,630-Speed 129.45 samples/sec Loss 0.6277 Epoch: 20 Global Step: 204075 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:57:29,376-Speed 129.66 samples/sec Loss 0.5565 Epoch: 20 Global Step: 204080 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:57:49,207-Speed 129.10 samples/sec Loss 0.5156 Epoch: 20 Global Step: 204085 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:58:08,918-Speed 129.88 samples/sec Loss 0.6616 Epoch: 20 Global Step: 204090 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:58:28,739-Speed 129.16 samples/sec Loss 0.5544 Epoch: 20 Global Step: 204095 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:58:48,428-Speed 130.03 samples/sec Loss 0.5640 Epoch: 20 Global Step: 204100 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:59:08,844-Speed 125.39 samples/sec Loss 0.5715 Epoch: 20 Global Step: 204105 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:59:28,617-Speed 129.47 samples/sec Loss 0.4848 Epoch: 20 Global Step: 204110 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-16 23:59:48,509-Speed 128.71 samples/sec Loss 0.6104 Epoch: 20 Global Step: 204115 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:00:08,288-Speed 129.44 samples/sec Loss 0.5553 Epoch: 20 Global Step: 204120 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:00:28,187-Speed 128.65 samples/sec Loss 0.7043 Epoch: 20 Global Step: 204125 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:00:47,899-Speed 129.88 samples/sec Loss 0.6140 Epoch: 20 Global Step: 204130 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:01:07,669-Speed 129.50 samples/sec Loss 0.6333 Epoch: 20 Global Step: 204135 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:01:27,558-Speed 128.72 samples/sec Loss 0.6532 Epoch: 20 Global Step: 204140 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:01:47,293-Speed 129.72 samples/sec Loss 0.5543 Epoch: 20 Global Step: 204145 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:02:07,636-Speed 125.85 samples/sec Loss 0.6305 Epoch: 20 Global Step: 204150 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:02:27,551-Speed 128.55 samples/sec Loss 0.5551 Epoch: 20 Global Step: 204155 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:02:47,288-Speed 129.71 samples/sec Loss 0.6620 Epoch: 20 Global Step: 204160 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:03:07,020-Speed 129.74 samples/sec Loss 0.5924 Epoch: 20 Global Step: 204165 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:03:26,887-Speed 128.86 samples/sec Loss 0.5946 Epoch: 20 Global Step: 204170 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:03:46,493-Speed 130.59 samples/sec Loss 0.6792 Epoch: 20 Global Step: 204175 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:04:06,399-Speed 128.61 samples/sec Loss 0.5695 Epoch: 20 Global Step: 204180 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:04:26,122-Speed 129.81 samples/sec Loss 0.6498 Epoch: 20 Global Step: 204185 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:04:45,863-Speed 129.68 samples/sec Loss 0.5997 Epoch: 20 Global Step: 204190 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:05:06,061-Speed 126.75 samples/sec Loss 0.5583 Epoch: 20 Global Step: 204195 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:05:25,984-Speed 128.50 samples/sec Loss 0.5817 Epoch: 20 Global Step: 204200 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:05:45,262-Speed 132.80 samples/sec Loss 0.5728 Epoch: 20 Global Step: 204205 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:06:05,084-Speed 129.16 samples/sec Loss 0.6353 Epoch: 20 Global Step: 204210 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:06:24,872-Speed 129.38 samples/sec Loss 0.6742 Epoch: 20 Global Step: 204215 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:06:44,494-Speed 130.48 samples/sec Loss 0.5565 Epoch: 20 Global Step: 204220 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:07:04,394-Speed 128.65 samples/sec Loss 0.5851 Epoch: 20 Global Step: 204225 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:07:24,128-Speed 129.74 samples/sec Loss 0.6139 Epoch: 20 Global Step: 204230 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:07:44,018-Speed 128.71 samples/sec Loss 0.5940 Epoch: 20 Global Step: 204235 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:08:04,228-Speed 126.68 samples/sec Loss 0.5914 Epoch: 20 Global Step: 204240 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:08:24,110-Speed 128.77 samples/sec Loss 0.6036 Epoch: 20 Global Step: 204245 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:08:43,705-Speed 130.65 samples/sec Loss 0.5673 Epoch: 20 Global Step: 204250 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:09:03,623-Speed 128.53 samples/sec Loss 0.5991 Epoch: 20 Global Step: 204255 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:09:23,385-Speed 129.55 samples/sec Loss 0.5415 Epoch: 20 Global Step: 204260 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:09:43,083-Speed 129.97 samples/sec Loss 0.6054 Epoch: 20 Global Step: 204265 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:10:02,906-Speed 129.15 samples/sec Loss 0.5519 Epoch: 20 Global Step: 204270 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:10:22,517-Speed 130.54 samples/sec Loss 0.6187 Epoch: 20 Global Step: 204275 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:10:42,423-Speed 128.61 samples/sec Loss 0.6378 Epoch: 20 Global Step: 204280 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:11:02,604-Speed 126.86 samples/sec Loss 0.6033 Epoch: 20 Global Step: 204285 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:11:22,389-Speed 129.39 samples/sec Loss 0.6234 Epoch: 20 Global Step: 204290 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:11:41,997-Speed 130.56 samples/sec Loss 0.5605 Epoch: 20 Global Step: 204295 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:12:01,979-Speed 128.12 samples/sec Loss 0.6214 Epoch: 20 Global Step: 204300 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:12:21,597-Speed 130.50 samples/sec Loss 0.5961 Epoch: 20 Global Step: 204305 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:12:41,326-Speed 129.78 samples/sec Loss 0.4957 Epoch: 20 Global Step: 204310 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:13:01,136-Speed 129.24 samples/sec Loss 0.5289 Epoch: 20 Global Step: 204315 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:13:20,844-Speed 129.91 samples/sec Loss 0.5664 Epoch: 20 Global Step: 204320 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:13:40,762-Speed 128.54 samples/sec Loss 0.5820 Epoch: 20 Global Step: 204325 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:14:01,078-Speed 126.01 samples/sec Loss 0.6587 Epoch: 20 Global Step: 204330 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:14:20,862-Speed 129.41 samples/sec Loss 0.6006 Epoch: 20 Global Step: 204335 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:14:40,570-Speed 129.90 samples/sec Loss 0.5523 Epoch: 20 Global Step: 204340 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:15:01,043-Speed 125.05 samples/sec Loss 0.5717 Epoch: 20 Global Step: 204345 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:15:20,744-Speed 129.95 samples/sec Loss 0.6112 Epoch: 20 Global Step: 204350 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:15:40,590-Speed 129.00 samples/sec Loss 0.6313 Epoch: 20 Global Step: 204355 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:16:00,323-Speed 129.74 samples/sec Loss 0.5746 Epoch: 20 Global Step: 204360 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:16:19,588-Speed 132.90 samples/sec Loss 0.5531 Epoch: 20 Global Step: 204365 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:16:39,210-Speed 130.47 samples/sec Loss 0.6597 Epoch: 20 Global Step: 204370 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:16:59,024-Speed 129.21 samples/sec Loss 0.6789 Epoch: 20 Global Step: 204375 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:17:18,468-Speed 131.67 samples/sec Loss 0.6067 Epoch: 20 Global Step: 204380 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:17:38,076-Speed 130.57 samples/sec Loss 0.5464 Epoch: 20 Global Step: 204385 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:17:57,891-Speed 129.20 samples/sec Loss 0.6560 Epoch: 20 Global Step: 204390 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:18:17,451-Speed 130.88 samples/sec Loss 0.6123 Epoch: 20 Global Step: 204395 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:18:37,364-Speed 128.57 samples/sec Loss 0.6192 Epoch: 20 Global Step: 204400 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:18:57,087-Speed 129.81 samples/sec Loss 0.6430 Epoch: 20 Global Step: 204405 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:19:16,909-Speed 129.16 samples/sec Loss 0.6639 Epoch: 20 Global Step: 204410 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:19:36,671-Speed 129.55 samples/sec Loss 0.6071 Epoch: 20 Global Step: 204415 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:19:56,648-Speed 128.15 samples/sec Loss 0.6556 Epoch: 20 Global Step: 204420 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:20:16,370-Speed 129.81 samples/sec Loss 0.5789 Epoch: 20 Global Step: 204425 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:20:36,229-Speed 128.91 samples/sec Loss 0.5961 Epoch: 20 Global Step: 204430 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:20:56,006-Speed 129.45 samples/sec Loss 0.6629 Epoch: 20 Global Step: 204435 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:21:15,686-Speed 130.09 samples/sec Loss 0.5807 Epoch: 20 Global Step: 204440 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:21:35,737-Speed 127.68 samples/sec Loss 0.6299 Epoch: 20 Global Step: 204445 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:21:55,457-Speed 129.82 samples/sec Loss 0.5862 Epoch: 20 Global Step: 204450 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:22:15,419-Speed 128.25 samples/sec Loss 0.6200 Epoch: 20 Global Step: 204455 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:22:35,058-Speed 130.36 samples/sec Loss 0.6395 Epoch: 20 Global Step: 204460 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:22:55,017-Speed 128.28 samples/sec Loss 0.6103 Epoch: 20 Global Step: 204465 Fp16 Grad Scale: 8192 Required: 54 hours Training: 2021-06-17 00:23:14,112-Speed 134.06 samples/sec Loss 0.5953 Epoch: 20 Global Step: 204470 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:23:34,175-Speed 127.61 samples/sec Loss 0.4968 Epoch: 20 Global Step: 204475 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:23:53,972-Speed 129.32 samples/sec Loss 0.4554 Epoch: 20 Global Step: 204480 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:24:13,616-Speed 130.33 samples/sec Loss 0.6139 Epoch: 20 Global Step: 204485 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:24:33,350-Speed 129.73 samples/sec Loss 0.6112 Epoch: 20 Global Step: 204490 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:24:53,127-Speed 129.45 samples/sec Loss 0.5936 Epoch: 20 Global Step: 204495 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:25:12,702-Speed 130.79 samples/sec Loss 0.5712 Epoch: 20 Global Step: 204500 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:25:32,548-Speed 129.00 samples/sec Loss 0.6302 Epoch: 20 Global Step: 204505 Fp16 Grad Scale: 4096 Required: 54 hours Training: 2021-06-17 00:25:52,130-Speed 130.74 samples/sec Loss 0.5957 Epoch: 20 Global Step: 204510 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:26:12,492-Speed 125.73 samples/sec Loss 0.6152 Epoch: 20 Global Step: 204515 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:26:32,315-Speed 129.15 samples/sec Loss 0.6392 Epoch: 20 Global Step: 204520 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:26:52,099-Speed 129.41 samples/sec Loss 0.6156 Epoch: 20 Global Step: 204525 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:27:11,808-Speed 129.90 samples/sec Loss 0.5894 Epoch: 20 Global Step: 204530 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:27:31,619-Speed 129.23 samples/sec Loss 0.5611 Epoch: 20 Global Step: 204535 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:27:51,543-Speed 128.50 samples/sec Loss 0.5621 Epoch: 20 Global Step: 204540 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:28:11,059-Speed 131.18 samples/sec Loss 0.5352 Epoch: 20 Global Step: 204545 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:28:30,875-Speed 129.20 samples/sec Loss 0.5463 Epoch: 20 Global Step: 204550 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:28:50,449-Speed 130.80 samples/sec Loss 0.5463 Epoch: 20 Global Step: 204555 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:29:10,169-Speed 129.82 samples/sec Loss 0.5510 Epoch: 20 Global Step: 204560 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:29:29,953-Speed 129.40 samples/sec Loss 0.6263 Epoch: 20 Global Step: 204565 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:29:49,861-Speed 128.60 samples/sec Loss 0.6383 Epoch: 20 Global Step: 204570 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:30:09,288-Speed 131.78 samples/sec Loss 0.6275 Epoch: 20 Global Step: 204575 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:30:28,652-Speed 132.21 samples/sec Loss 0.6343 Epoch: 20 Global Step: 204580 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:30:48,572-Speed 128.52 samples/sec Loss 0.6553 Epoch: 20 Global Step: 204585 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:31:08,188-Speed 130.51 samples/sec Loss 0.5486 Epoch: 20 Global Step: 204590 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:31:27,854-Speed 130.18 samples/sec Loss 0.5857 Epoch: 20 Global Step: 204595 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:31:47,431-Speed 130.78 samples/sec Loss 0.6705 Epoch: 20 Global Step: 204600 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:32:07,170-Speed 129.69 samples/sec Loss 0.5513 Epoch: 20 Global Step: 204605 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:32:26,949-Speed 129.44 samples/sec Loss 0.5906 Epoch: 20 Global Step: 204610 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:32:46,845-Speed 128.67 samples/sec Loss 0.5871 Epoch: 20 Global Step: 204615 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:33:06,486-Speed 130.35 samples/sec Loss 0.5760 Epoch: 20 Global Step: 204620 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:33:26,304-Speed 129.18 samples/sec Loss 0.5901 Epoch: 20 Global Step: 204625 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:33:46,234-Speed 128.46 samples/sec Loss 0.5546 Epoch: 20 Global Step: 204630 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:34:05,755-Speed 131.15 samples/sec Loss 0.6202 Epoch: 20 Global Step: 204635 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:34:25,575-Speed 129.17 samples/sec Loss 0.5535 Epoch: 20 Global Step: 204640 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:34:45,159-Speed 130.73 samples/sec Loss 0.5668 Epoch: 20 Global Step: 204645 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:35:04,885-Speed 129.79 samples/sec Loss 0.6809 Epoch: 20 Global Step: 204650 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:35:24,686-Speed 129.29 samples/sec Loss 0.5909 Epoch: 20 Global Step: 204655 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:35:44,604-Speed 128.53 samples/sec Loss 0.6046 Epoch: 20 Global Step: 204660 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:36:04,277-Speed 130.14 samples/sec Loss 0.5201 Epoch: 20 Global Step: 204665 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:36:24,004-Speed 129.78 samples/sec Loss 0.6336 Epoch: 20 Global Step: 204670 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:36:43,921-Speed 128.54 samples/sec Loss 0.6540 Epoch: 20 Global Step: 204675 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:37:03,463-Speed 131.01 samples/sec Loss 0.6587 Epoch: 20 Global Step: 204680 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:37:23,435-Speed 128.19 samples/sec Loss 0.6309 Epoch: 20 Global Step: 204685 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:37:42,997-Speed 130.88 samples/sec Loss 0.5567 Epoch: 20 Global Step: 204690 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:38:02,620-Speed 130.47 samples/sec Loss 0.6171 Epoch: 20 Global Step: 204695 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:38:22,406-Speed 129.39 samples/sec Loss 0.5842 Epoch: 20 Global Step: 204700 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:38:42,177-Speed 129.49 samples/sec Loss 0.7288 Epoch: 20 Global Step: 204705 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:39:01,867-Speed 130.02 samples/sec Loss 0.6018 Epoch: 20 Global Step: 204710 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:39:21,660-Speed 129.34 samples/sec Loss 0.6508 Epoch: 20 Global Step: 204715 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:39:41,493-Speed 129.09 samples/sec Loss 0.5809 Epoch: 20 Global Step: 204720 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:40:01,743-Speed 126.43 samples/sec Loss 0.6151 Epoch: 20 Global Step: 204725 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:40:21,586-Speed 129.02 samples/sec Loss 0.6090 Epoch: 20 Global Step: 204730 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:40:41,154-Speed 130.83 samples/sec Loss 0.6296 Epoch: 20 Global Step: 204735 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:41:00,356-Speed 133.33 samples/sec Loss 0.6169 Epoch: 20 Global Step: 204740 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:41:20,459-Speed 127.35 samples/sec Loss 0.6216 Epoch: 20 Global Step: 204745 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:41:40,241-Speed 129.41 samples/sec Loss 0.5863 Epoch: 20 Global Step: 204750 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:42:00,312-Speed 127.56 samples/sec Loss 0.5630 Epoch: 20 Global Step: 204755 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:42:20,040-Speed 129.77 samples/sec Loss 0.5490 Epoch: 20 Global Step: 204760 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:42:39,854-Speed 129.20 samples/sec Loss 0.5699 Epoch: 20 Global Step: 204765 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:42:59,803-Speed 128.34 samples/sec Loss 0.6527 Epoch: 20 Global Step: 204770 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:43:19,640-Speed 129.05 samples/sec Loss 0.6450 Epoch: 20 Global Step: 204775 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:43:39,512-Speed 128.84 samples/sec Loss 0.5445 Epoch: 20 Global Step: 204780 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:43:59,130-Speed 130.50 samples/sec Loss 0.6981 Epoch: 20 Global Step: 204785 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:44:19,214-Speed 127.47 samples/sec Loss 0.6027 Epoch: 20 Global Step: 204790 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:44:39,024-Speed 129.24 samples/sec Loss 0.5426 Epoch: 20 Global Step: 204795 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:44:58,899-Speed 128.81 samples/sec Loss 0.5690 Epoch: 20 Global Step: 204800 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:45:18,641-Speed 129.67 samples/sec Loss 0.6278 Epoch: 20 Global Step: 204805 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:45:38,449-Speed 129.25 samples/sec Loss 0.5440 Epoch: 20 Global Step: 204810 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:45:58,413-Speed 128.24 samples/sec Loss 0.5852 Epoch: 20 Global Step: 204815 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:46:18,524-Speed 127.30 samples/sec Loss 0.6021 Epoch: 20 Global Step: 204820 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:46:38,290-Speed 129.53 samples/sec Loss 0.6542 Epoch: 20 Global Step: 204825 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:46:57,891-Speed 130.61 samples/sec Loss 0.6229 Epoch: 20 Global Step: 204830 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:47:17,956-Speed 127.59 samples/sec Loss 0.6653 Epoch: 20 Global Step: 204835 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:47:38,537-Speed 124.39 samples/sec Loss 0.6064 Epoch: 20 Global Step: 204840 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:47:58,318-Speed 129.43 samples/sec Loss 0.5217 Epoch: 20 Global Step: 204845 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:48:17,578-Speed 132.93 samples/sec Loss 0.5634 Epoch: 20 Global Step: 204850 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:48:37,352-Speed 129.47 samples/sec Loss 0.6417 Epoch: 20 Global Step: 204855 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:48:57,236-Speed 128.75 samples/sec Loss 0.6077 Epoch: 20 Global Step: 204860 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:49:17,188-Speed 128.31 samples/sec Loss 0.7305 Epoch: 20 Global Step: 204865 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:49:36,613-Speed 131.80 samples/sec Loss 0.6613 Epoch: 20 Global Step: 204870 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:49:56,271-Speed 130.24 samples/sec Loss 0.5541 Epoch: 20 Global Step: 204875 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:50:15,832-Speed 130.88 samples/sec Loss 0.6328 Epoch: 20 Global Step: 204880 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:50:35,770-Speed 128.40 samples/sec Loss 0.5866 Epoch: 20 Global Step: 204885 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:50:55,555-Speed 129.40 samples/sec Loss 0.5591 Epoch: 20 Global Step: 204890 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:51:15,372-Speed 129.19 samples/sec Loss 0.5827 Epoch: 20 Global Step: 204895 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:51:34,952-Speed 130.75 samples/sec Loss 0.6323 Epoch: 20 Global Step: 204900 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:51:54,804-Speed 128.96 samples/sec Loss 0.6050 Epoch: 20 Global Step: 204905 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:52:14,710-Speed 128.61 samples/sec Loss 0.4590 Epoch: 20 Global Step: 204910 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:52:34,219-Speed 131.23 samples/sec Loss 0.5675 Epoch: 20 Global Step: 204915 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:52:53,896-Speed 130.11 samples/sec Loss 0.6224 Epoch: 20 Global Step: 204920 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:53:13,455-Speed 130.90 samples/sec Loss 0.6338 Epoch: 20 Global Step: 204925 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:53:33,082-Speed 130.44 samples/sec Loss 0.5514 Epoch: 20 Global Step: 204930 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:53:52,865-Speed 129.41 samples/sec Loss 0.6385 Epoch: 20 Global Step: 204935 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:54:12,672-Speed 129.25 samples/sec Loss 0.5686 Epoch: 20 Global Step: 204940 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:54:32,282-Speed 130.55 samples/sec Loss 0.6074 Epoch: 20 Global Step: 204945 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:54:51,985-Speed 129.93 samples/sec Loss 0.5533 Epoch: 20 Global Step: 204950 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:55:11,890-Speed 128.62 samples/sec Loss 0.5860 Epoch: 20 Global Step: 204955 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:55:31,623-Speed 129.74 samples/sec Loss 0.5582 Epoch: 20 Global Step: 204960 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:55:51,367-Speed 129.66 samples/sec Loss 0.5893 Epoch: 20 Global Step: 204965 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:56:10,936-Speed 130.83 samples/sec Loss 0.5161 Epoch: 20 Global Step: 204970 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:56:30,546-Speed 130.55 samples/sec Loss 0.6083 Epoch: 20 Global Step: 204975 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:56:50,341-Speed 129.33 samples/sec Loss 0.6046 Epoch: 20 Global Step: 204980 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:57:10,340-Speed 128.01 samples/sec Loss 0.6560 Epoch: 20 Global Step: 204985 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:57:29,776-Speed 131.73 samples/sec Loss 0.5739 Epoch: 20 Global Step: 204990 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 00:57:49,102-Speed 132.47 samples/sec Loss 0.6271 Epoch: 20 Global Step: 204995 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:58:08,977-Speed 128.81 samples/sec Loss 0.5702 Epoch: 20 Global Step: 205000 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:58:28,516-Speed 131.03 samples/sec Loss 0.5408 Epoch: 20 Global Step: 205005 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:58:48,397-Speed 128.78 samples/sec Loss 0.5239 Epoch: 20 Global Step: 205010 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:59:07,935-Speed 131.03 samples/sec Loss 0.6240 Epoch: 20 Global Step: 205015 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:59:27,556-Speed 130.49 samples/sec Loss 0.5769 Epoch: 20 Global Step: 205020 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 00:59:47,325-Speed 129.50 samples/sec Loss 0.5430 Epoch: 20 Global Step: 205025 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:00:07,186-Speed 128.90 samples/sec Loss 0.5746 Epoch: 20 Global Step: 205030 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:00:26,606-Speed 131.83 samples/sec Loss 0.6127 Epoch: 20 Global Step: 205035 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:00:46,261-Speed 130.25 samples/sec Loss 0.6565 Epoch: 20 Global Step: 205040 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:01:06,917-Speed 123.95 samples/sec Loss 0.5946 Epoch: 20 Global Step: 205045 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:01:26,463-Speed 130.98 samples/sec Loss 0.6578 Epoch: 20 Global Step: 205050 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:01:46,366-Speed 128.63 samples/sec Loss 0.6597 Epoch: 20 Global Step: 205055 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:02:05,822-Speed 131.59 samples/sec Loss 0.5168 Epoch: 20 Global Step: 205060 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:02:25,431-Speed 130.56 samples/sec Loss 0.6247 Epoch: 20 Global Step: 205065 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:02:45,226-Speed 129.33 samples/sec Loss 0.6144 Epoch: 20 Global Step: 205070 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:03:05,183-Speed 128.28 samples/sec Loss 0.6601 Epoch: 20 Global Step: 205075 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:03:24,616-Speed 131.74 samples/sec Loss 0.6708 Epoch: 20 Global Step: 205080 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:03:44,319-Speed 129.93 samples/sec Loss 0.6374 Epoch: 20 Global Step: 205085 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:04:04,196-Speed 128.80 samples/sec Loss 0.4962 Epoch: 20 Global Step: 205090 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:04:23,745-Speed 130.96 samples/sec Loss 0.6054 Epoch: 20 Global Step: 205095 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 01:04:43,724-Speed 128.14 samples/sec Loss 0.5916 Epoch: 20 Global Step: 205100 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 01:05:03,146-Speed 131.82 samples/sec Loss 0.6055 Epoch: 20 Global Step: 205105 Fp16 Grad Scale: 8192 Required: 53 hours Training: 2021-06-17 01:05:22,437-Speed 132.71 samples/sec Loss 0.5682 Epoch: 20 Global Step: 205110 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:05:42,490-Speed 127.67 samples/sec Loss 0.5763 Epoch: 20 Global Step: 205115 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:06:02,415-Speed 128.49 samples/sec Loss 0.5938 Epoch: 20 Global Step: 205120 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:06:22,123-Speed 129.90 samples/sec Loss 0.5706 Epoch: 20 Global Step: 205125 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:06:41,862-Speed 129.70 samples/sec Loss 0.5276 Epoch: 20 Global Step: 205130 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:07:01,688-Speed 129.13 samples/sec Loss 0.6831 Epoch: 20 Global Step: 205135 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:07:21,388-Speed 129.95 samples/sec Loss 0.6027 Epoch: 20 Global Step: 205140 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:07:41,316-Speed 128.47 samples/sec Loss 0.5520 Epoch: 20 Global Step: 205145 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:08:01,018-Speed 129.94 samples/sec Loss 0.6005 Epoch: 20 Global Step: 205150 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:08:20,914-Speed 128.67 samples/sec Loss 0.5285 Epoch: 20 Global Step: 205155 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:08:40,961-Speed 127.71 samples/sec Loss 0.5476 Epoch: 20 Global Step: 205160 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:09:00,925-Speed 128.23 samples/sec Loss 0.5608 Epoch: 20 Global Step: 205165 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:09:20,619-Speed 130.00 samples/sec Loss 0.5226 Epoch: 20 Global Step: 205170 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:09:40,415-Speed 129.33 samples/sec Loss 0.6491 Epoch: 20 Global Step: 205175 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:10:00,153-Speed 129.71 samples/sec Loss 0.6000 Epoch: 20 Global Step: 205180 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:10:19,855-Speed 129.94 samples/sec Loss 0.5194 Epoch: 20 Global Step: 205185 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:10:39,805-Speed 128.33 samples/sec Loss 0.5898 Epoch: 20 Global Step: 205190 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:10:59,025-Speed 133.20 samples/sec Loss 0.5068 Epoch: 20 Global Step: 205195 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:11:19,107-Speed 127.48 samples/sec Loss 0.5947 Epoch: 20 Global Step: 205200 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:11:38,792-Speed 130.06 samples/sec Loss 0.6734 Epoch: 20 Global Step: 205205 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:11:58,605-Speed 129.22 samples/sec Loss 0.5542 Epoch: 20 Global Step: 205210 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:12:18,636-Speed 127.81 samples/sec Loss 0.7212 Epoch: 20 Global Step: 205215 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:12:38,429-Speed 129.34 samples/sec Loss 0.5271 Epoch: 20 Global Step: 205220 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:12:58,153-Speed 129.80 samples/sec Loss 0.5784 Epoch: 20 Global Step: 205225 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:13:18,020-Speed 128.86 samples/sec Loss 0.6045 Epoch: 20 Global Step: 205230 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:13:37,823-Speed 129.28 samples/sec Loss 0.5492 Epoch: 20 Global Step: 205235 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:13:57,558-Speed 129.72 samples/sec Loss 0.5432 Epoch: 20 Global Step: 205240 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:14:17,727-Speed 126.94 samples/sec Loss 0.6759 Epoch: 20 Global Step: 205245 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:14:37,446-Speed 129.83 samples/sec Loss 0.6106 Epoch: 20 Global Step: 205250 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:14:57,337-Speed 128.71 samples/sec Loss 0.5546 Epoch: 20 Global Step: 205255 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:15:17,264-Speed 128.48 samples/sec Loss 0.6273 Epoch: 20 Global Step: 205260 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:15:37,124-Speed 128.91 samples/sec Loss 0.5329 Epoch: 20 Global Step: 205265 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:15:56,825-Speed 129.95 samples/sec Loss 0.5150 Epoch: 20 Global Step: 205270 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:16:16,744-Speed 128.53 samples/sec Loss 0.5980 Epoch: 20 Global Step: 205275 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:16:36,547-Speed 129.28 samples/sec Loss 0.6519 Epoch: 20 Global Step: 205280 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:16:56,324-Speed 129.45 samples/sec Loss 0.5625 Epoch: 20 Global Step: 205285 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:17:16,375-Speed 127.68 samples/sec Loss 0.5773 Epoch: 20 Global Step: 205290 Fp16 Grad Scale: 2048 Required: 53 hours Training: 2021-06-17 01:17:36,094-Speed 129.83 samples/sec Loss 0.6436 Epoch: 20 Global Step: 205295 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:17:55,939-Speed 129.01 samples/sec Loss 0.6052 Epoch: 20 Global Step: 205300 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:18:15,871-Speed 128.45 samples/sec Loss 0.6893 Epoch: 20 Global Step: 205305 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:18:35,696-Speed 129.13 samples/sec Loss 0.5147 Epoch: 20 Global Step: 205310 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:18:55,409-Speed 129.87 samples/sec Loss 0.5608 Epoch: 20 Global Step: 205315 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:19:15,309-Speed 128.65 samples/sec Loss 0.4933 Epoch: 20 Global Step: 205320 Fp16 Grad Scale: 4096 Required: 53 hours Training: 2021-06-17 01:19:35,066-Speed 129.58 samples/sec Loss 0.7124 Epoch: 20 Global Step: 205325 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:19:54,887-Speed 129.17 samples/sec Loss 0.5592 Epoch: 20 Global Step: 205330 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:20:14,879-Speed 128.06 samples/sec Loss 0.7321 Epoch: 20 Global Step: 205335 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:20:34,656-Speed 129.45 samples/sec Loss 0.5709 Epoch: 20 Global Step: 205340 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:20:54,577-Speed 128.51 samples/sec Loss 0.6317 Epoch: 20 Global Step: 205345 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:21:14,649-Speed 127.55 samples/sec Loss 0.5910 Epoch: 20 Global Step: 205350 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:21:34,357-Speed 129.91 samples/sec Loss 0.5877 Epoch: 20 Global Step: 205355 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:21:54,853-Speed 124.91 samples/sec Loss 0.6108 Epoch: 20 Global Step: 205360 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:22:14,705-Speed 128.96 samples/sec Loss 0.5952 Epoch: 20 Global Step: 205365 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:22:34,589-Speed 128.75 samples/sec Loss 0.6207 Epoch: 20 Global Step: 205370 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:22:54,476-Speed 128.73 samples/sec Loss 0.5977 Epoch: 20 Global Step: 205375 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:23:14,483-Speed 127.96 samples/sec Loss 0.6238 Epoch: 20 Global Step: 205380 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:23:34,247-Speed 129.53 samples/sec Loss 0.5885 Epoch: 20 Global Step: 205385 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:23:54,149-Speed 128.64 samples/sec Loss 0.7206 Epoch: 20 Global Step: 205390 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:24:14,240-Speed 127.43 samples/sec Loss 0.5131 Epoch: 20 Global Step: 205395 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:24:33,954-Speed 129.87 samples/sec Loss 0.6581 Epoch: 20 Global Step: 205400 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:24:53,791-Speed 129.05 samples/sec Loss 0.5755 Epoch: 20 Global Step: 205405 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:25:13,628-Speed 129.06 samples/sec Loss 0.5363 Epoch: 20 Global Step: 205410 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:25:33,374-Speed 129.66 samples/sec Loss 0.5723 Epoch: 20 Global Step: 205415 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:25:53,245-Speed 128.84 samples/sec Loss 0.6161 Epoch: 20 Global Step: 205420 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:26:13,244-Speed 128.01 samples/sec Loss 0.6297 Epoch: 20 Global Step: 205425 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:26:33,069-Speed 129.14 samples/sec Loss 0.5700 Epoch: 20 Global Step: 205430 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:26:52,856-Speed 129.39 samples/sec Loss 0.5933 Epoch: 20 Global Step: 205435 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:27:12,873-Speed 127.89 samples/sec Loss 0.5984 Epoch: 20 Global Step: 205440 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:27:32,687-Speed 129.21 samples/sec Loss 0.6229 Epoch: 20 Global Step: 205445 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:27:52,450-Speed 129.54 samples/sec Loss 0.5544 Epoch: 20 Global Step: 205450 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:28:11,844-Speed 132.01 samples/sec Loss 0.6507 Epoch: 20 Global Step: 205455 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:28:31,580-Speed 129.72 samples/sec Loss 0.5433 Epoch: 20 Global Step: 205460 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:28:51,448-Speed 128.86 samples/sec Loss 0.5551 Epoch: 20 Global Step: 205465 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:29:11,492-Speed 127.72 samples/sec Loss 0.5856 Epoch: 20 Global Step: 205470 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:29:31,376-Speed 128.75 samples/sec Loss 0.6028 Epoch: 20 Global Step: 205475 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:29:51,203-Speed 129.13 samples/sec Loss 0.6128 Epoch: 20 Global Step: 205480 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:30:11,317-Speed 127.28 samples/sec Loss 0.7085 Epoch: 20 Global Step: 205485 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:30:31,066-Speed 129.64 samples/sec Loss 0.5653 Epoch: 20 Global Step: 205490 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:30:50,971-Speed 128.62 samples/sec Loss 0.5902 Epoch: 20 Global Step: 205495 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:31:10,758-Speed 129.38 samples/sec Loss 0.6317 Epoch: 20 Global Step: 205500 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:31:30,463-Speed 129.92 samples/sec Loss 0.6171 Epoch: 20 Global Step: 205505 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:31:50,312-Speed 128.98 samples/sec Loss 0.5978 Epoch: 20 Global Step: 205510 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:32:10,340-Speed 127.82 samples/sec Loss 0.5879 Epoch: 20 Global Step: 205515 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:32:30,309-Speed 128.21 samples/sec Loss 0.5373 Epoch: 20 Global Step: 205520 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:32:49,919-Speed 130.55 samples/sec Loss 0.6583 Epoch: 20 Global Step: 205525 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:33:10,021-Speed 127.36 samples/sec Loss 0.5517 Epoch: 20 Global Step: 205530 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:33:29,751-Speed 129.75 samples/sec Loss 0.7753 Epoch: 20 Global Step: 205535 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:33:49,854-Speed 127.35 samples/sec Loss 0.6059 Epoch: 20 Global Step: 205540 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:34:09,751-Speed 128.67 samples/sec Loss 0.6247 Epoch: 20 Global Step: 205545 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:34:29,453-Speed 129.95 samples/sec Loss 0.4595 Epoch: 20 Global Step: 205550 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:34:49,338-Speed 128.75 samples/sec Loss 0.6000 Epoch: 20 Global Step: 205555 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:35:09,250-Speed 128.57 samples/sec Loss 0.6243 Epoch: 20 Global Step: 205560 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:35:29,264-Speed 127.92 samples/sec Loss 0.6690 Epoch: 20 Global Step: 205565 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:35:48,938-Speed 130.13 samples/sec Loss 0.6549 Epoch: 20 Global Step: 205570 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:36:09,017-Speed 127.50 samples/sec Loss 0.6275 Epoch: 20 Global Step: 205575 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:36:29,337-Speed 125.99 samples/sec Loss 0.5832 Epoch: 20 Global Step: 205580 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:36:49,324-Speed 128.09 samples/sec Loss 0.6062 Epoch: 20 Global Step: 205585 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:37:08,943-Speed 130.49 samples/sec Loss 0.6148 Epoch: 20 Global Step: 205590 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:37:28,728-Speed 129.40 samples/sec Loss 0.5962 Epoch: 20 Global Step: 205595 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:37:48,434-Speed 129.92 samples/sec Loss 0.6244 Epoch: 20 Global Step: 205600 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:38:08,178-Speed 129.67 samples/sec Loss 0.5883 Epoch: 20 Global Step: 205605 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:38:28,115-Speed 128.41 samples/sec Loss 0.6477 Epoch: 20 Global Step: 205610 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:38:47,560-Speed 131.66 samples/sec Loss 0.6627 Epoch: 20 Global Step: 205615 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:39:07,365-Speed 129.27 samples/sec Loss 0.5410 Epoch: 20 Global Step: 205620 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:39:26,936-Speed 130.81 samples/sec Loss 0.5300 Epoch: 20 Global Step: 205625 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:39:46,727-Speed 129.36 samples/sec Loss 0.6837 Epoch: 20 Global Step: 205630 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:40:06,272-Speed 130.99 samples/sec Loss 0.6143 Epoch: 20 Global Step: 205635 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:40:25,901-Speed 130.43 samples/sec Loss 0.6404 Epoch: 20 Global Step: 205640 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:40:45,420-Speed 131.16 samples/sec Loss 0.5619 Epoch: 20 Global Step: 205645 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:41:04,992-Speed 130.80 samples/sec Loss 0.6798 Epoch: 20 Global Step: 205650 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:41:24,595-Speed 130.60 samples/sec Loss 0.6126 Epoch: 20 Global Step: 205655 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:41:44,050-Speed 131.59 samples/sec Loss 0.6005 Epoch: 20 Global Step: 205660 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:42:03,843-Speed 129.35 samples/sec Loss 0.6154 Epoch: 20 Global Step: 205665 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:42:23,400-Speed 130.91 samples/sec Loss 0.5808 Epoch: 20 Global Step: 205670 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:42:43,111-Speed 129.88 samples/sec Loss 0.5871 Epoch: 20 Global Step: 205675 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:43:02,564-Speed 131.61 samples/sec Loss 0.6124 Epoch: 20 Global Step: 205680 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:43:22,170-Speed 130.58 samples/sec Loss 0.7145 Epoch: 20 Global Step: 205685 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:43:41,755-Speed 130.72 samples/sec Loss 0.5739 Epoch: 20 Global Step: 205690 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:44:01,290-Speed 131.06 samples/sec Loss 0.6868 Epoch: 20 Global Step: 205695 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:44:20,904-Speed 130.53 samples/sec Loss 0.6859 Epoch: 20 Global Step: 205700 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:44:40,340-Speed 131.72 samples/sec Loss 0.6168 Epoch: 20 Global Step: 205705 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:45:00,135-Speed 129.33 samples/sec Loss 0.6070 Epoch: 20 Global Step: 205710 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:45:19,712-Speed 130.77 samples/sec Loss 0.6978 Epoch: 20 Global Step: 205715 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:45:39,318-Speed 130.58 samples/sec Loss 0.5811 Epoch: 20 Global Step: 205720 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:45:58,305-Speed 134.84 samples/sec Loss 0.5670 Epoch: 20 Global Step: 205725 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:46:18,080-Speed 129.46 samples/sec Loss 0.5442 Epoch: 20 Global Step: 205730 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:46:37,829-Speed 129.64 samples/sec Loss 0.7102 Epoch: 20 Global Step: 205735 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:46:57,560-Speed 129.76 samples/sec Loss 0.6249 Epoch: 20 Global Step: 205740 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:47:17,517-Speed 128.28 samples/sec Loss 0.6057 Epoch: 20 Global Step: 205745 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:47:37,139-Speed 130.47 samples/sec Loss 0.6311 Epoch: 20 Global Step: 205750 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:47:57,034-Speed 128.68 samples/sec Loss 0.5868 Epoch: 20 Global Step: 205755 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:48:16,744-Speed 129.89 samples/sec Loss 0.6145 Epoch: 20 Global Step: 205760 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:48:37,181-Speed 125.27 samples/sec Loss 0.6427 Epoch: 20 Global Step: 205765 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:48:56,787-Speed 130.58 samples/sec Loss 0.6627 Epoch: 20 Global Step: 205770 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:49:16,731-Speed 128.36 samples/sec Loss 0.6404 Epoch: 20 Global Step: 205775 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:49:36,442-Speed 129.88 samples/sec Loss 0.6908 Epoch: 20 Global Step: 205780 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:49:56,166-Speed 129.80 samples/sec Loss 0.6587 Epoch: 20 Global Step: 205785 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:50:16,121-Speed 128.29 samples/sec Loss 0.6680 Epoch: 20 Global Step: 205790 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:50:35,755-Speed 130.39 samples/sec Loss 0.5896 Epoch: 20 Global Step: 205795 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:50:55,705-Speed 128.33 samples/sec Loss 0.6604 Epoch: 20 Global Step: 205800 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:51:15,423-Speed 129.84 samples/sec Loss 0.5389 Epoch: 20 Global Step: 205805 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:51:35,792-Speed 125.69 samples/sec Loss 0.6911 Epoch: 20 Global Step: 205810 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:51:55,661-Speed 128.85 samples/sec Loss 0.6252 Epoch: 20 Global Step: 205815 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:52:15,564-Speed 128.63 samples/sec Loss 0.6452 Epoch: 20 Global Step: 205820 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:52:35,280-Speed 129.85 samples/sec Loss 0.5889 Epoch: 20 Global Step: 205825 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:52:55,017-Speed 129.71 samples/sec Loss 0.6402 Epoch: 20 Global Step: 205830 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:53:14,941-Speed 128.49 samples/sec Loss 0.5795 Epoch: 20 Global Step: 205835 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:53:34,648-Speed 129.91 samples/sec Loss 0.5225 Epoch: 20 Global Step: 205840 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:53:54,520-Speed 128.84 samples/sec Loss 0.6588 Epoch: 20 Global Step: 205845 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:54:14,228-Speed 129.90 samples/sec Loss 0.5933 Epoch: 20 Global Step: 205850 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:54:34,652-Speed 125.35 samples/sec Loss 0.6441 Epoch: 20 Global Step: 205855 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:54:54,310-Speed 130.24 samples/sec Loss 0.5521 Epoch: 20 Global Step: 205860 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:55:14,073-Speed 129.54 samples/sec Loss 0.6103 Epoch: 20 Global Step: 205865 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:55:33,802-Speed 129.76 samples/sec Loss 0.6063 Epoch: 20 Global Step: 205870 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:55:53,593-Speed 129.36 samples/sec Loss 0.6264 Epoch: 20 Global Step: 205875 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:56:13,383-Speed 129.37 samples/sec Loss 0.6515 Epoch: 20 Global Step: 205880 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 01:56:32,600-Speed 133.22 samples/sec Loss 0.6222 Epoch: 20 Global Step: 205885 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:56:52,452-Speed 128.97 samples/sec Loss 0.6302 Epoch: 20 Global Step: 205890 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:57:12,252-Speed 129.30 samples/sec Loss 0.6182 Epoch: 20 Global Step: 205895 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:57:32,706-Speed 125.17 samples/sec Loss 0.6008 Epoch: 20 Global Step: 205900 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:57:52,448-Speed 129.68 samples/sec Loss 0.6570 Epoch: 20 Global Step: 205905 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:58:12,302-Speed 128.94 samples/sec Loss 0.5451 Epoch: 20 Global Step: 205910 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:58:32,039-Speed 129.72 samples/sec Loss 0.5788 Epoch: 20 Global Step: 205915 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:58:51,874-Speed 129.07 samples/sec Loss 0.5539 Epoch: 20 Global Step: 205920 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:59:11,617-Speed 129.67 samples/sec Loss 0.6791 Epoch: 20 Global Step: 205925 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:59:31,384-Speed 129.52 samples/sec Loss 0.5831 Epoch: 20 Global Step: 205930 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 01:59:51,230-Speed 129.00 samples/sec Loss 0.6641 Epoch: 20 Global Step: 205935 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:00:10,922-Speed 130.01 samples/sec Loss 0.6015 Epoch: 20 Global Step: 205940 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:00:31,340-Speed 125.38 samples/sec Loss 0.5582 Epoch: 20 Global Step: 205945 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:00:51,093-Speed 129.61 samples/sec Loss 0.6520 Epoch: 20 Global Step: 205950 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:01:10,878-Speed 129.40 samples/sec Loss 0.5895 Epoch: 20 Global Step: 205955 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:01:30,562-Speed 130.06 samples/sec Loss 0.6247 Epoch: 20 Global Step: 205960 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:01:50,452-Speed 128.71 samples/sec Loss 0.7008 Epoch: 20 Global Step: 205965 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:02:10,234-Speed 129.42 samples/sec Loss 0.5022 Epoch: 20 Global Step: 205970 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:02:29,960-Speed 129.79 samples/sec Loss 0.6453 Epoch: 20 Global Step: 205975 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:02:49,818-Speed 128.92 samples/sec Loss 0.5828 Epoch: 20 Global Step: 205980 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:03:09,441-Speed 130.47 samples/sec Loss 0.6132 Epoch: 20 Global Step: 205985 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:03:29,812-Speed 125.67 samples/sec Loss 0.6476 Epoch: 20 Global Step: 205990 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:03:49,614-Speed 129.29 samples/sec Loss 0.5933 Epoch: 20 Global Step: 205995 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:04:09,406-Speed 129.35 samples/sec Loss 0.5351 Epoch: 20 Global Step: 206000 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:04:29,112-Speed 129.92 samples/sec Loss 0.6399 Epoch: 20 Global Step: 206005 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:04:48,977-Speed 128.88 samples/sec Loss 0.6048 Epoch: 20 Global Step: 206010 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:05:08,736-Speed 129.57 samples/sec Loss 0.6420 Epoch: 20 Global Step: 206015 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:05:28,432-Speed 129.98 samples/sec Loss 0.5671 Epoch: 20 Global Step: 206020 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:05:48,285-Speed 128.96 samples/sec Loss 0.6486 Epoch: 20 Global Step: 206025 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:06:07,915-Speed 130.41 samples/sec Loss 0.5519 Epoch: 20 Global Step: 206030 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:06:28,344-Speed 125.32 samples/sec Loss 0.6188 Epoch: 20 Global Step: 206035 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:06:48,177-Speed 129.09 samples/sec Loss 0.6928 Epoch: 20 Global Step: 206040 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:07:07,905-Speed 129.77 samples/sec Loss 0.6207 Epoch: 20 Global Step: 206045 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:07:27,558-Speed 130.27 samples/sec Loss 0.5281 Epoch: 20 Global Step: 206050 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:07:47,346-Speed 129.38 samples/sec Loss 0.6245 Epoch: 20 Global Step: 206055 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:08:07,160-Speed 129.21 samples/sec Loss 0.6102 Epoch: 20 Global Step: 206060 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:08:27,050-Speed 128.72 samples/sec Loss 0.6458 Epoch: 20 Global Step: 206065 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:08:46,786-Speed 129.72 samples/sec Loss 0.5628 Epoch: 20 Global Step: 206070 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:09:06,449-Speed 130.20 samples/sec Loss 0.7342 Epoch: 20 Global Step: 206075 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:09:26,851-Speed 125.48 samples/sec Loss 0.5922 Epoch: 20 Global Step: 206080 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:09:46,654-Speed 129.28 samples/sec Loss 0.6395 Epoch: 20 Global Step: 206085 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:10:06,316-Speed 130.20 samples/sec Loss 0.5870 Epoch: 20 Global Step: 206090 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:10:26,021-Speed 129.93 samples/sec Loss 0.5515 Epoch: 20 Global Step: 206095 Fp16 Grad Scale: 8192 Required: 52 hours Training: 2021-06-17 02:10:45,123-Speed 134.03 samples/sec Loss 0.6557 Epoch: 20 Global Step: 206100 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:11:04,751-Speed 130.43 samples/sec Loss 0.5564 Epoch: 20 Global Step: 206105 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:11:24,359-Speed 130.56 samples/sec Loss 0.6189 Epoch: 20 Global Step: 206110 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:11:43,891-Speed 131.08 samples/sec Loss 0.5049 Epoch: 20 Global Step: 206115 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:12:03,566-Speed 130.13 samples/sec Loss 0.6356 Epoch: 20 Global Step: 206120 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:12:23,322-Speed 129.59 samples/sec Loss 0.6218 Epoch: 20 Global Step: 206125 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:12:43,269-Speed 128.34 samples/sec Loss 0.7232 Epoch: 20 Global Step: 206130 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:13:02,971-Speed 129.95 samples/sec Loss 0.6520 Epoch: 20 Global Step: 206135 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:13:22,751-Speed 129.43 samples/sec Loss 0.6993 Epoch: 20 Global Step: 206140 Fp16 Grad Scale: 4096 Required: 52 hours Training: 2021-06-17 02:13:42,562-Speed 129.23 samples/sec Loss 0.6036 Epoch: 20 Global Step: 206145 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:14:02,640-Speed 127.51 samples/sec Loss 0.5505 Epoch: 20 Global Step: 206150 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:14:22,534-Speed 128.69 samples/sec Loss 0.6200 Epoch: 20 Global Step: 206155 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:14:42,152-Speed 130.49 samples/sec Loss 0.7300 Epoch: 20 Global Step: 206160 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:15:01,908-Speed 129.59 samples/sec Loss 0.6048 Epoch: 20 Global Step: 206165 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:15:22,150-Speed 126.48 samples/sec Loss 0.6203 Epoch: 20 Global Step: 206170 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:15:42,113-Speed 128.24 samples/sec Loss 0.6948 Epoch: 20 Global Step: 206175 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:16:01,788-Speed 130.12 samples/sec Loss 0.6520 Epoch: 20 Global Step: 206180 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:16:21,596-Speed 129.24 samples/sec Loss 0.4757 Epoch: 20 Global Step: 206185 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:16:41,435-Speed 129.05 samples/sec Loss 0.6092 Epoch: 20 Global Step: 206190 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:17:01,422-Speed 128.09 samples/sec Loss 0.5652 Epoch: 20 Global Step: 206195 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:17:21,284-Speed 128.90 samples/sec Loss 0.6131 Epoch: 20 Global Step: 206200 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:17:40,894-Speed 130.55 samples/sec Loss 0.5808 Epoch: 20 Global Step: 206205 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:18:00,706-Speed 129.22 samples/sec Loss 0.6960 Epoch: 20 Global Step: 206210 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:18:20,152-Speed 131.66 samples/sec Loss 0.5409 Epoch: 20 Global Step: 206215 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:18:39,939-Speed 129.39 samples/sec Loss 0.5316 Epoch: 20 Global Step: 206220 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:18:59,682-Speed 129.67 samples/sec Loss 0.6815 Epoch: 20 Global Step: 206225 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:19:19,492-Speed 129.24 samples/sec Loss 0.6076 Epoch: 20 Global Step: 206230 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:19:39,067-Speed 130.78 samples/sec Loss 0.6355 Epoch: 20 Global Step: 206235 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:19:58,803-Speed 129.72 samples/sec Loss 0.6184 Epoch: 20 Global Step: 206240 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:20:18,295-Speed 131.34 samples/sec Loss 0.5670 Epoch: 20 Global Step: 206245 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:20:37,945-Speed 130.29 samples/sec Loss 0.5721 Epoch: 20 Global Step: 206250 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:20:57,766-Speed 129.17 samples/sec Loss 0.6101 Epoch: 20 Global Step: 206255 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:21:17,563-Speed 129.32 samples/sec Loss 0.6176 Epoch: 20 Global Step: 206260 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:21:37,598-Speed 127.78 samples/sec Loss 0.6895 Epoch: 20 Global Step: 206265 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:21:57,223-Speed 130.46 samples/sec Loss 0.5479 Epoch: 20 Global Step: 206270 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:22:16,992-Speed 129.50 samples/sec Loss 0.6556 Epoch: 20 Global Step: 206275 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:22:36,849-Speed 128.93 samples/sec Loss 0.5671 Epoch: 20 Global Step: 206280 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:22:57,319-Speed 125.06 samples/sec Loss 0.5261 Epoch: 20 Global Step: 206285 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:23:17,025-Speed 129.92 samples/sec Loss 0.5391 Epoch: 20 Global Step: 206290 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:23:36,700-Speed 130.12 samples/sec Loss 0.6766 Epoch: 20 Global Step: 206295 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:23:56,559-Speed 128.91 samples/sec Loss 0.6802 Epoch: 20 Global Step: 206300 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:24:16,924-Speed 125.71 samples/sec Loss 0.6145 Epoch: 20 Global Step: 206305 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:24:36,828-Speed 128.63 samples/sec Loss 0.5671 Epoch: 20 Global Step: 206310 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:24:56,452-Speed 130.46 samples/sec Loss 0.5711 Epoch: 20 Global Step: 206315 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:25:16,102-Speed 130.29 samples/sec Loss 0.5858 Epoch: 20 Global Step: 206320 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:25:35,892-Speed 129.36 samples/sec Loss 0.6357 Epoch: 20 Global Step: 206325 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:25:56,303-Speed 125.43 samples/sec Loss 0.6287 Epoch: 20 Global Step: 206330 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:26:15,934-Speed 130.43 samples/sec Loss 0.5724 Epoch: 20 Global Step: 206335 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:26:35,588-Speed 130.26 samples/sec Loss 0.6017 Epoch: 20 Global Step: 206340 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:26:55,442-Speed 128.95 samples/sec Loss 0.5925 Epoch: 20 Global Step: 206345 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:27:15,177-Speed 129.73 samples/sec Loss 0.6011 Epoch: 20 Global Step: 206350 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:27:35,107-Speed 128.46 samples/sec Loss 0.6142 Epoch: 20 Global Step: 206355 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:27:54,736-Speed 130.43 samples/sec Loss 0.5913 Epoch: 20 Global Step: 206360 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:28:14,535-Speed 129.31 samples/sec Loss 0.5758 Epoch: 20 Global Step: 206365 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:28:34,247-Speed 129.87 samples/sec Loss 0.5719 Epoch: 20 Global Step: 206370 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:28:54,692-Speed 125.22 samples/sec Loss 0.6159 Epoch: 20 Global Step: 206375 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:29:14,294-Speed 130.61 samples/sec Loss 0.5962 Epoch: 20 Global Step: 206380 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:29:34,083-Speed 129.37 samples/sec Loss 0.5843 Epoch: 20 Global Step: 206385 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:29:53,855-Speed 129.48 samples/sec Loss 0.5584 Epoch: 20 Global Step: 206390 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:30:13,615-Speed 129.56 samples/sec Loss 0.5761 Epoch: 20 Global Step: 206395 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:30:33,461-Speed 129.00 samples/sec Loss 0.6637 Epoch: 20 Global Step: 206400 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:30:53,087-Speed 130.45 samples/sec Loss 0.5551 Epoch: 20 Global Step: 206405 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:31:12,902-Speed 129.21 samples/sec Loss 0.6419 Epoch: 20 Global Step: 206410 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:31:32,587-Speed 130.05 samples/sec Loss 0.5683 Epoch: 20 Global Step: 206415 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:31:52,973-Speed 125.58 samples/sec Loss 0.6144 Epoch: 20 Global Step: 206420 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:32:12,589-Speed 130.51 samples/sec Loss 0.6642 Epoch: 20 Global Step: 206425 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:32:31,946-Speed 132.25 samples/sec Loss 0.6107 Epoch: 20 Global Step: 206430 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:32:51,636-Speed 130.02 samples/sec Loss 0.6448 Epoch: 20 Global Step: 206435 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:33:11,421-Speed 129.40 samples/sec Loss 0.5940 Epoch: 20 Global Step: 206440 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:33:31,277-Speed 128.94 samples/sec Loss 0.5929 Epoch: 20 Global Step: 206445 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:33:50,930-Speed 130.27 samples/sec Loss 0.5933 Epoch: 20 Global Step: 206450 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:34:10,801-Speed 128.84 samples/sec Loss 0.5935 Epoch: 20 Global Step: 206455 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:34:30,512-Speed 129.88 samples/sec Loss 0.7303 Epoch: 20 Global Step: 206460 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:34:50,910-Speed 125.51 samples/sec Loss 0.7457 Epoch: 20 Global Step: 206465 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:35:10,526-Speed 130.51 samples/sec Loss 0.6825 Epoch: 20 Global Step: 206470 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:35:30,321-Speed 129.33 samples/sec Loss 0.5463 Epoch: 20 Global Step: 206475 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:35:50,138-Speed 129.19 samples/sec Loss 0.5378 Epoch: 20 Global Step: 206480 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:36:09,827-Speed 130.03 samples/sec Loss 0.6596 Epoch: 20 Global Step: 206485 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:36:29,697-Speed 128.85 samples/sec Loss 0.6675 Epoch: 20 Global Step: 206490 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:36:49,305-Speed 130.57 samples/sec Loss 0.5613 Epoch: 20 Global Step: 206495 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:37:09,186-Speed 128.77 samples/sec Loss 0.6852 Epoch: 20 Global Step: 206500 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:37:28,976-Speed 129.37 samples/sec Loss 0.6649 Epoch: 20 Global Step: 206505 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:37:49,241-Speed 126.33 samples/sec Loss 0.6213 Epoch: 20 Global Step: 206510 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:38:08,846-Speed 130.59 samples/sec Loss 0.5785 Epoch: 20 Global Step: 206515 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:38:28,967-Speed 127.23 samples/sec Loss 0.5932 Epoch: 20 Global Step: 206520 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:38:48,741-Speed 129.47 samples/sec Loss 0.5752 Epoch: 20 Global Step: 206525 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:39:07,897-Speed 133.65 samples/sec Loss 0.6079 Epoch: 20 Global Step: 206530 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:39:27,440-Speed 131.00 samples/sec Loss 0.6147 Epoch: 20 Global Step: 206535 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:39:46,924-Speed 131.40 samples/sec Loss 0.6194 Epoch: 20 Global Step: 206540 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:40:06,567-Speed 130.33 samples/sec Loss 0.6898 Epoch: 20 Global Step: 206545 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:40:26,345-Speed 129.44 samples/sec Loss 0.5820 Epoch: 20 Global Step: 206550 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:40:46,143-Speed 129.31 samples/sec Loss 0.6054 Epoch: 20 Global Step: 206555 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:41:05,838-Speed 129.99 samples/sec Loss 0.6360 Epoch: 20 Global Step: 206560 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:41:25,740-Speed 128.64 samples/sec Loss 0.7293 Epoch: 20 Global Step: 206565 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:41:45,468-Speed 129.77 samples/sec Loss 0.5552 Epoch: 20 Global Step: 206570 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:42:05,199-Speed 129.75 samples/sec Loss 0.7031 Epoch: 20 Global Step: 206575 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:42:25,255-Speed 127.65 samples/sec Loss 0.5697 Epoch: 20 Global Step: 206580 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:42:44,983-Speed 129.77 samples/sec Loss 0.5981 Epoch: 20 Global Step: 206585 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:43:04,865-Speed 128.77 samples/sec Loss 0.6114 Epoch: 20 Global Step: 206590 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:43:24,668-Speed 129.28 samples/sec Loss 0.6912 Epoch: 20 Global Step: 206595 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:43:44,686-Speed 127.89 samples/sec Loss 0.6550 Epoch: 20 Global Step: 206600 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:44:04,341-Speed 130.25 samples/sec Loss 0.6747 Epoch: 20 Global Step: 206605 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:44:24,208-Speed 128.86 samples/sec Loss 0.5700 Epoch: 20 Global Step: 206610 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:44:43,923-Speed 129.86 samples/sec Loss 0.6780 Epoch: 20 Global Step: 206615 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:45:03,900-Speed 128.16 samples/sec Loss 0.5942 Epoch: 20 Global Step: 206620 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:45:23,784-Speed 128.75 samples/sec Loss 0.5661 Epoch: 20 Global Step: 206625 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:45:43,638-Speed 128.95 samples/sec Loss 0.6873 Epoch: 20 Global Step: 206630 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:46:03,449-Speed 129.23 samples/sec Loss 0.6714 Epoch: 20 Global Step: 206635 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:46:23,256-Speed 129.26 samples/sec Loss 0.6487 Epoch: 20 Global Step: 206640 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:46:43,351-Speed 127.40 samples/sec Loss 0.5726 Epoch: 20 Global Step: 206645 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:47:03,142-Speed 129.36 samples/sec Loss 0.6686 Epoch: 20 Global Step: 206650 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:47:22,844-Speed 129.94 samples/sec Loss 0.5265 Epoch: 20 Global Step: 206655 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:47:42,619-Speed 129.46 samples/sec Loss 0.6710 Epoch: 20 Global Step: 206660 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:48:02,523-Speed 128.62 samples/sec Loss 0.6986 Epoch: 20 Global Step: 206665 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:48:22,440-Speed 128.54 samples/sec Loss 0.5714 Epoch: 20 Global Step: 206670 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:48:42,360-Speed 128.52 samples/sec Loss 0.6271 Epoch: 20 Global Step: 206675 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:49:02,077-Speed 129.85 samples/sec Loss 0.7480 Epoch: 20 Global Step: 206680 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:49:21,482-Speed 131.93 samples/sec Loss 0.5870 Epoch: 20 Global Step: 206685 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:49:41,068-Speed 130.72 samples/sec Loss 0.6445 Epoch: 20 Global Step: 206690 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:50:00,730-Speed 130.20 samples/sec Loss 0.6520 Epoch: 20 Global Step: 206695 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:50:20,303-Speed 130.80 samples/sec Loss 0.6337 Epoch: 20 Global Step: 206700 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:50:40,201-Speed 128.66 samples/sec Loss 0.5699 Epoch: 20 Global Step: 206705 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:50:59,452-Speed 132.99 samples/sec Loss 0.6342 Epoch: 20 Global Step: 206710 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:51:19,417-Speed 128.23 samples/sec Loss 0.6770 Epoch: 20 Global Step: 206715 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:51:39,291-Speed 128.81 samples/sec Loss 0.5706 Epoch: 20 Global Step: 206720 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:51:58,997-Speed 129.91 samples/sec Loss 0.6074 Epoch: 20 Global Step: 206725 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:52:18,988-Speed 128.07 samples/sec Loss 0.5877 Epoch: 20 Global Step: 206730 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:52:38,694-Speed 129.92 samples/sec Loss 0.5942 Epoch: 20 Global Step: 206735 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:52:58,479-Speed 129.40 samples/sec Loss 0.6062 Epoch: 20 Global Step: 206740 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:53:18,266-Speed 129.39 samples/sec Loss 0.6064 Epoch: 20 Global Step: 206745 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:53:38,046-Speed 129.43 samples/sec Loss 0.6732 Epoch: 20 Global Step: 206750 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:53:57,965-Speed 128.53 samples/sec Loss 0.6305 Epoch: 20 Global Step: 206755 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:54:18,352-Speed 125.57 samples/sec Loss 0.6399 Epoch: 20 Global Step: 206760 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:54:37,990-Speed 130.37 samples/sec Loss 0.6109 Epoch: 20 Global Step: 206765 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:54:57,710-Speed 129.82 samples/sec Loss 0.5588 Epoch: 20 Global Step: 206770 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:55:17,761-Speed 127.68 samples/sec Loss 0.5463 Epoch: 20 Global Step: 206775 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:55:37,414-Speed 130.27 samples/sec Loss 0.6514 Epoch: 20 Global Step: 206780 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:55:57,306-Speed 128.70 samples/sec Loss 0.5964 Epoch: 20 Global Step: 206785 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:56:16,923-Speed 130.51 samples/sec Loss 0.5579 Epoch: 20 Global Step: 206790 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:56:36,805-Speed 128.77 samples/sec Loss 0.6289 Epoch: 20 Global Step: 206795 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:56:56,596-Speed 129.35 samples/sec Loss 0.6351 Epoch: 20 Global Step: 206800 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:57:16,939-Speed 125.85 samples/sec Loss 0.6971 Epoch: 20 Global Step: 206805 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:57:36,592-Speed 130.26 samples/sec Loss 0.5604 Epoch: 20 Global Step: 206810 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 02:57:55,930-Speed 132.38 samples/sec Loss 0.6487 Epoch: 20 Global Step: 206815 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:58:15,759-Speed 129.11 samples/sec Loss 0.5756 Epoch: 20 Global Step: 206820 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:58:35,401-Speed 130.34 samples/sec Loss 0.5422 Epoch: 20 Global Step: 206825 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:58:55,279-Speed 128.79 samples/sec Loss 0.6252 Epoch: 20 Global Step: 206830 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:59:14,916-Speed 130.37 samples/sec Loss 0.7048 Epoch: 20 Global Step: 206835 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:59:34,888-Speed 128.19 samples/sec Loss 0.5668 Epoch: 20 Global Step: 206840 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 02:59:54,619-Speed 129.75 samples/sec Loss 0.6401 Epoch: 20 Global Step: 206845 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:00:14,913-Speed 126.15 samples/sec Loss 0.6067 Epoch: 20 Global Step: 206850 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:00:34,623-Speed 129.89 samples/sec Loss 0.6348 Epoch: 20 Global Step: 206855 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:00:54,495-Speed 128.83 samples/sec Loss 0.6213 Epoch: 20 Global Step: 206860 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:01:14,229-Speed 129.73 samples/sec Loss 0.5974 Epoch: 20 Global Step: 206865 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:01:33,903-Speed 130.14 samples/sec Loss 0.5882 Epoch: 20 Global Step: 206870 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:01:53,772-Speed 128.85 samples/sec Loss 0.7212 Epoch: 20 Global Step: 206875 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:02:13,464-Speed 130.01 samples/sec Loss 0.6525 Epoch: 20 Global Step: 206880 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:02:33,375-Speed 128.58 samples/sec Loss 0.7777 Epoch: 20 Global Step: 206885 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:02:53,150-Speed 129.46 samples/sec Loss 0.5596 Epoch: 20 Global Step: 206890 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:03:13,423-Speed 126.28 samples/sec Loss 0.6811 Epoch: 20 Global Step: 206895 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:03:33,192-Speed 129.50 samples/sec Loss 0.6233 Epoch: 20 Global Step: 206900 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:03:53,174-Speed 128.12 samples/sec Loss 0.6052 Epoch: 20 Global Step: 206905 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:04:12,882-Speed 129.90 samples/sec Loss 0.6942 Epoch: 20 Global Step: 206910 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:04:32,631-Speed 129.63 samples/sec Loss 0.6085 Epoch: 20 Global Step: 206915 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 03:04:52,532-Speed 128.64 samples/sec Loss 0.7546 Epoch: 20 Global Step: 206920 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 03:05:12,172-Speed 130.36 samples/sec Loss 0.6027 Epoch: 20 Global Step: 206925 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 03:05:32,145-Speed 128.18 samples/sec Loss 0.5795 Epoch: 20 Global Step: 206930 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 03:05:51,884-Speed 129.70 samples/sec Loss 0.6410 Epoch: 20 Global Step: 206935 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 03:06:12,137-Speed 126.40 samples/sec Loss 0.5979 Epoch: 20 Global Step: 206940 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 03:06:31,892-Speed 129.59 samples/sec Loss 0.6244 Epoch: 20 Global Step: 206945 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 03:06:51,830-Speed 128.41 samples/sec Loss 0.6864 Epoch: 20 Global Step: 206950 Fp16 Grad Scale: 8192 Required: 51 hours Training: 2021-06-17 03:07:11,067-Speed 133.08 samples/sec Loss 0.6155 Epoch: 20 Global Step: 206955 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:07:30,679-Speed 130.54 samples/sec Loss 0.5997 Epoch: 20 Global Step: 206960 Fp16 Grad Scale: 4096 Required: 51 hours Training: 2021-06-17 03:07:50,109-Speed 131.76 samples/sec Loss 0.6811 Epoch: 20 Global Step: 206965 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:08:09,615-Speed 131.25 samples/sec Loss 0.5809 Epoch: 20 Global Step: 206970 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:08:29,184-Speed 130.83 samples/sec Loss 0.5764 Epoch: 20 Global Step: 206975 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:08:48,762-Speed 130.77 samples/sec Loss 0.6004 Epoch: 20 Global Step: 206980 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:09:08,569-Speed 129.26 samples/sec Loss 0.7247 Epoch: 20 Global Step: 206985 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:09:28,060-Speed 131.34 samples/sec Loss 0.6359 Epoch: 20 Global Step: 206990 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:09:47,717-Speed 130.24 samples/sec Loss 0.6233 Epoch: 20 Global Step: 206995 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:10:07,267-Speed 130.96 samples/sec Loss 0.5969 Epoch: 20 Global Step: 207000 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:10:27,135-Speed 128.85 samples/sec Loss 0.6571 Epoch: 20 Global Step: 207005 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:10:46,664-Speed 131.09 samples/sec Loss 0.6416 Epoch: 20 Global Step: 207010 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:11:06,213-Speed 130.96 samples/sec Loss 0.6208 Epoch: 20 Global Step: 207015 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:11:25,772-Speed 130.89 samples/sec Loss 0.6631 Epoch: 20 Global Step: 207020 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:11:45,318-Speed 130.98 samples/sec Loss 0.5848 Epoch: 20 Global Step: 207025 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:12:05,124-Speed 129.26 samples/sec Loss 0.6285 Epoch: 20 Global Step: 207030 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:12:24,576-Speed 131.61 samples/sec Loss 0.6215 Epoch: 20 Global Step: 207035 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:12:44,183-Speed 130.57 samples/sec Loss 0.6265 Epoch: 20 Global Step: 207040 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:13:03,743-Speed 130.89 samples/sec Loss 0.5952 Epoch: 20 Global Step: 207045 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:13:23,354-Speed 130.55 samples/sec Loss 0.6671 Epoch: 20 Global Step: 207050 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:13:42,876-Speed 131.14 samples/sec Loss 0.6333 Epoch: 20 Global Step: 207055 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:14:02,389-Speed 131.20 samples/sec Loss 0.6694 Epoch: 20 Global Step: 207060 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:14:22,263-Speed 128.82 samples/sec Loss 0.6215 Epoch: 20 Global Step: 207065 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:14:41,723-Speed 131.56 samples/sec Loss 0.6662 Epoch: 20 Global Step: 207070 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:15:01,535-Speed 129.22 samples/sec Loss 0.6482 Epoch: 20 Global Step: 207075 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:15:20,996-Speed 131.55 samples/sec Loss 0.6918 Epoch: 20 Global Step: 207080 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:15:40,573-Speed 130.77 samples/sec Loss 0.6099 Epoch: 20 Global Step: 207085 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:16:00,281-Speed 129.90 samples/sec Loss 0.5961 Epoch: 20 Global Step: 207090 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:16:20,180-Speed 128.65 samples/sec Loss 0.5837 Epoch: 20 Global Step: 207095 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:16:39,831-Speed 130.28 samples/sec Loss 0.6374 Epoch: 20 Global Step: 207100 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:16:59,844-Speed 127.92 samples/sec Loss 0.5582 Epoch: 20 Global Step: 207105 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:17:19,636-Speed 129.35 samples/sec Loss 0.5975 Epoch: 20 Global Step: 207110 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:17:39,361-Speed 129.79 samples/sec Loss 0.5985 Epoch: 20 Global Step: 207115 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:17:59,225-Speed 128.88 samples/sec Loss 0.6504 Epoch: 20 Global Step: 207120 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:18:18,877-Speed 130.28 samples/sec Loss 0.5849 Epoch: 20 Global Step: 207125 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:18:39,076-Speed 126.74 samples/sec Loss 0.6756 Epoch: 20 Global Step: 207130 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:18:58,786-Speed 129.89 samples/sec Loss 0.6046 Epoch: 20 Global Step: 207135 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:19:18,494-Speed 129.91 samples/sec Loss 0.7823 Epoch: 20 Global Step: 207140 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:19:38,504-Speed 127.94 samples/sec Loss 0.6266 Epoch: 20 Global Step: 207145 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:19:58,478-Speed 128.17 samples/sec Loss 0.6191 Epoch: 20 Global Step: 207150 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:20:18,102-Speed 130.46 samples/sec Loss 0.6980 Epoch: 20 Global Step: 207155 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:20:38,002-Speed 128.65 samples/sec Loss 0.6410 Epoch: 20 Global Step: 207160 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:20:57,683-Speed 130.08 samples/sec Loss 0.6821 Epoch: 20 Global Step: 207165 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:21:17,449-Speed 129.52 samples/sec Loss 0.7119 Epoch: 20 Global Step: 207170 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:21:37,571-Speed 127.23 samples/sec Loss 0.5488 Epoch: 20 Global Step: 207175 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:21:57,306-Speed 129.73 samples/sec Loss 0.5891 Epoch: 20 Global Step: 207180 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:22:16,978-Speed 130.14 samples/sec Loss 0.5733 Epoch: 20 Global Step: 207185 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:22:36,999-Speed 127.88 samples/sec Loss 0.5745 Epoch: 20 Global Step: 207190 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:22:56,575-Speed 130.78 samples/sec Loss 0.6735 Epoch: 20 Global Step: 207195 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:23:16,219-Speed 130.33 samples/sec Loss 0.5327 Epoch: 20 Global Step: 207200 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:23:36,116-Speed 128.67 samples/sec Loss 0.6439 Epoch: 20 Global Step: 207205 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:23:55,803-Speed 130.04 samples/sec Loss 0.7170 Epoch: 20 Global Step: 207210 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:24:15,574-Speed 129.49 samples/sec Loss 0.5992 Epoch: 20 Global Step: 207215 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:24:35,818-Speed 126.46 samples/sec Loss 0.6522 Epoch: 20 Global Step: 207220 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:24:55,529-Speed 129.89 samples/sec Loss 0.6248 Epoch: 20 Global Step: 207225 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:25:15,327-Speed 129.31 samples/sec Loss 0.6875 Epoch: 20 Global Step: 207230 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:25:35,238-Speed 128.58 samples/sec Loss 0.6056 Epoch: 20 Global Step: 207235 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:25:55,304-Speed 127.59 samples/sec Loss 0.5889 Epoch: 20 Global Step: 207240 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:26:15,046-Speed 129.67 samples/sec Loss 0.6452 Epoch: 20 Global Step: 207245 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:26:34,926-Speed 128.78 samples/sec Loss 0.5689 Epoch: 20 Global Step: 207250 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:26:54,582-Speed 130.24 samples/sec Loss 0.6671 Epoch: 20 Global Step: 207255 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:27:14,336-Speed 129.60 samples/sec Loss 0.6203 Epoch: 20 Global Step: 207260 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:27:34,478-Speed 127.11 samples/sec Loss 0.5546 Epoch: 20 Global Step: 207265 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:27:54,212-Speed 129.73 samples/sec Loss 0.6173 Epoch: 20 Global Step: 207270 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:28:14,060-Speed 128.99 samples/sec Loss 0.6440 Epoch: 20 Global Step: 207275 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:28:33,992-Speed 128.45 samples/sec Loss 0.6299 Epoch: 20 Global Step: 207280 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:28:54,057-Speed 127.59 samples/sec Loss 0.5624 Epoch: 20 Global Step: 207285 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:29:13,865-Speed 129.24 samples/sec Loss 0.7011 Epoch: 20 Global Step: 207290 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:29:33,775-Speed 128.59 samples/sec Loss 0.6548 Epoch: 20 Global Step: 207295 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:29:53,508-Speed 129.74 samples/sec Loss 0.6048 Epoch: 20 Global Step: 207300 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:30:13,273-Speed 129.53 samples/sec Loss 0.6386 Epoch: 20 Global Step: 207305 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:30:33,376-Speed 127.35 samples/sec Loss 0.5954 Epoch: 20 Global Step: 207310 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:30:53,084-Speed 129.90 samples/sec Loss 0.6692 Epoch: 20 Global Step: 207315 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:31:12,905-Speed 129.17 samples/sec Loss 0.6535 Epoch: 20 Global Step: 207320 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:31:32,931-Speed 127.84 samples/sec Loss 0.5699 Epoch: 20 Global Step: 207325 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:31:52,746-Speed 129.21 samples/sec Loss 0.6821 Epoch: 20 Global Step: 207330 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:32:12,475-Speed 129.76 samples/sec Loss 0.6654 Epoch: 20 Global Step: 207335 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:32:32,288-Speed 129.22 samples/sec Loss 0.6757 Epoch: 20 Global Step: 207340 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:32:51,931-Speed 130.33 samples/sec Loss 0.6706 Epoch: 20 Global Step: 207345 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:33:11,721-Speed 129.37 samples/sec Loss 0.6345 Epoch: 20 Global Step: 207350 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:33:31,878-Speed 127.00 samples/sec Loss 0.6356 Epoch: 20 Global Step: 207355 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:33:51,589-Speed 129.89 samples/sec Loss 0.6625 Epoch: 20 Global Step: 207360 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:34:11,386-Speed 129.32 samples/sec Loss 0.6832 Epoch: 20 Global Step: 207365 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:34:31,524-Speed 127.13 samples/sec Loss 0.6166 Epoch: 20 Global Step: 207370 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:34:51,269-Speed 129.66 samples/sec Loss 0.6951 Epoch: 20 Global Step: 207375 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:35:10,575-Speed 132.61 samples/sec Loss 0.6619 Epoch: 20 Global Step: 207380 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:35:30,354-Speed 129.43 samples/sec Loss 0.6414 Epoch: 20 Global Step: 207385 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:35:50,013-Speed 130.23 samples/sec Loss 0.6015 Epoch: 20 Global Step: 207390 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:36:09,971-Speed 128.28 samples/sec Loss 0.7013 Epoch: 20 Global Step: 207395 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:36:29,704-Speed 129.74 samples/sec Loss 0.5483 Epoch: 20 Global Step: 207400 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:36:49,593-Speed 128.72 samples/sec Loss 0.6487 Epoch: 20 Global Step: 207405 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:37:09,586-Speed 128.05 samples/sec Loss 0.6588 Epoch: 20 Global Step: 207410 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:37:29,577-Speed 128.06 samples/sec Loss 0.6525 Epoch: 20 Global Step: 207415 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:37:49,005-Speed 131.78 samples/sec Loss 0.5295 Epoch: 20 Global Step: 207420 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:38:08,739-Speed 129.73 samples/sec Loss 0.6131 Epoch: 20 Global Step: 207425 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:38:28,289-Speed 130.95 samples/sec Loss 0.5514 Epoch: 20 Global Step: 207430 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:38:47,789-Speed 131.29 samples/sec Loss 0.6742 Epoch: 20 Global Step: 207435 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:39:07,401-Speed 130.54 samples/sec Loss 0.5703 Epoch: 20 Global Step: 207440 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:39:26,956-Speed 130.92 samples/sec Loss 0.6184 Epoch: 20 Global Step: 207445 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:39:46,566-Speed 130.55 samples/sec Loss 0.6437 Epoch: 20 Global Step: 207450 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:40:06,099-Speed 131.07 samples/sec Loss 0.6698 Epoch: 20 Global Step: 207455 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:40:25,903-Speed 129.28 samples/sec Loss 0.5687 Epoch: 20 Global Step: 207460 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:40:45,551-Speed 130.30 samples/sec Loss 0.5735 Epoch: 20 Global Step: 207465 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:41:05,274-Speed 129.81 samples/sec Loss 0.6309 Epoch: 20 Global Step: 207470 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:41:24,814-Speed 131.02 samples/sec Loss 0.7373 Epoch: 20 Global Step: 207475 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:41:44,345-Speed 131.08 samples/sec Loss 0.6022 Epoch: 20 Global Step: 207480 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:42:03,957-Speed 130.54 samples/sec Loss 0.7021 Epoch: 20 Global Step: 207485 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:42:23,488-Speed 131.08 samples/sec Loss 0.6477 Epoch: 20 Global Step: 207490 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:42:43,095-Speed 130.58 samples/sec Loss 0.7139 Epoch: 20 Global Step: 207495 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:43:02,632-Speed 131.04 samples/sec Loss 0.5255 Epoch: 20 Global Step: 207500 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:43:22,521-Speed 128.72 samples/sec Loss 0.5926 Epoch: 20 Global Step: 207505 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:43:41,981-Speed 131.56 samples/sec Loss 0.5816 Epoch: 20 Global Step: 207510 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:44:01,697-Speed 129.85 samples/sec Loss 0.6230 Epoch: 20 Global Step: 207515 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:44:21,269-Speed 130.80 samples/sec Loss 0.6087 Epoch: 20 Global Step: 207520 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:44:40,920-Speed 130.28 samples/sec Loss 0.5509 Epoch: 20 Global Step: 207525 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:45:00,856-Speed 128.41 samples/sec Loss 0.6920 Epoch: 20 Global Step: 207530 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:45:20,583-Speed 129.78 samples/sec Loss 0.6066 Epoch: 20 Global Step: 207535 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:45:40,395-Speed 129.22 samples/sec Loss 0.6124 Epoch: 20 Global Step: 207540 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:46:00,104-Speed 129.90 samples/sec Loss 0.6173 Epoch: 20 Global Step: 207545 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:46:20,758-Speed 123.95 samples/sec Loss 0.6694 Epoch: 20 Global Step: 207550 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:46:40,604-Speed 129.01 samples/sec Loss 0.6769 Epoch: 20 Global Step: 207555 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:47:00,555-Speed 128.32 samples/sec Loss 0.6240 Epoch: 20 Global Step: 207560 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:47:20,391-Speed 129.07 samples/sec Loss 0.6303 Epoch: 20 Global Step: 207565 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:47:40,140-Speed 129.64 samples/sec Loss 0.6523 Epoch: 20 Global Step: 207570 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:48:00,272-Speed 127.17 samples/sec Loss 0.6950 Epoch: 20 Global Step: 207575 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:48:20,030-Speed 129.57 samples/sec Loss 0.5491 Epoch: 20 Global Step: 207580 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:48:39,900-Speed 128.84 samples/sec Loss 0.6958 Epoch: 20 Global Step: 207585 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:48:59,549-Speed 130.29 samples/sec Loss 0.6796 Epoch: 20 Global Step: 207590 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:49:19,531-Speed 128.12 samples/sec Loss 0.6914 Epoch: 20 Global Step: 207595 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:49:39,265-Speed 129.73 samples/sec Loss 0.6644 Epoch: 20 Global Step: 207600 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:49:59,342-Speed 127.52 samples/sec Loss 0.6279 Epoch: 20 Global Step: 207605 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:50:18,989-Speed 130.31 samples/sec Loss 0.6549 Epoch: 20 Global Step: 207610 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:50:38,768-Speed 129.43 samples/sec Loss 0.6644 Epoch: 20 Global Step: 207615 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:50:58,834-Speed 127.59 samples/sec Loss 0.6937 Epoch: 20 Global Step: 207620 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:51:18,823-Speed 128.08 samples/sec Loss 0.6605 Epoch: 20 Global Step: 207625 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:51:38,426-Speed 130.60 samples/sec Loss 0.6397 Epoch: 20 Global Step: 207630 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:51:58,160-Speed 129.73 samples/sec Loss 0.6099 Epoch: 20 Global Step: 207635 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:52:17,676-Speed 131.19 samples/sec Loss 0.5586 Epoch: 20 Global Step: 207640 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:52:37,328-Speed 130.27 samples/sec Loss 0.7081 Epoch: 20 Global Step: 207645 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:52:57,434-Speed 127.33 samples/sec Loss 0.6414 Epoch: 20 Global Step: 207650 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:53:17,855-Speed 125.37 samples/sec Loss 0.5931 Epoch: 20 Global Step: 207655 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:53:37,614-Speed 129.57 samples/sec Loss 0.6205 Epoch: 20 Global Step: 207660 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:53:57,665-Speed 127.69 samples/sec Loss 0.6221 Epoch: 20 Global Step: 207665 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:54:17,581-Speed 128.54 samples/sec Loss 0.6280 Epoch: 20 Global Step: 207670 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:54:37,195-Speed 130.52 samples/sec Loss 0.5892 Epoch: 20 Global Step: 207675 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:54:56,940-Speed 129.66 samples/sec Loss 0.6221 Epoch: 20 Global Step: 207680 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:55:16,791-Speed 128.97 samples/sec Loss 0.6305 Epoch: 20 Global Step: 207685 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:55:36,500-Speed 129.90 samples/sec Loss 0.6296 Epoch: 20 Global Step: 207690 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:55:56,585-Speed 127.46 samples/sec Loss 0.6015 Epoch: 20 Global Step: 207695 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:56:16,284-Speed 129.96 samples/sec Loss 0.6289 Epoch: 20 Global Step: 207700 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:56:36,173-Speed 128.72 samples/sec Loss 0.5821 Epoch: 20 Global Step: 207705 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:56:56,157-Speed 128.11 samples/sec Loss 0.7025 Epoch: 20 Global Step: 207710 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:57:16,045-Speed 128.72 samples/sec Loss 0.6912 Epoch: 20 Global Step: 207715 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:57:35,749-Speed 129.93 samples/sec Loss 0.6559 Epoch: 20 Global Step: 207720 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:57:55,633-Speed 128.75 samples/sec Loss 0.6238 Epoch: 20 Global Step: 207725 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:58:15,440-Speed 129.25 samples/sec Loss 0.6611 Epoch: 20 Global Step: 207730 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:58:35,232-Speed 129.35 samples/sec Loss 0.7331 Epoch: 20 Global Step: 207735 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 03:58:55,343-Speed 127.29 samples/sec Loss 0.6653 Epoch: 20 Global Step: 207740 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:59:15,043-Speed 129.96 samples/sec Loss 0.5746 Epoch: 20 Global Step: 207745 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:59:34,971-Speed 128.47 samples/sec Loss 0.6186 Epoch: 20 Global Step: 207750 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 03:59:54,837-Speed 128.87 samples/sec Loss 0.7326 Epoch: 20 Global Step: 207755 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 04:00:14,692-Speed 128.94 samples/sec Loss 0.6337 Epoch: 20 Global Step: 207760 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 04:00:34,301-Speed 130.56 samples/sec Loss 0.6004 Epoch: 20 Global Step: 207765 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 04:00:54,263-Speed 128.25 samples/sec Loss 0.7285 Epoch: 20 Global Step: 207770 Fp16 Grad Scale: 8192 Required: 50 hours Training: 2021-06-17 04:01:13,535-Speed 132.85 samples/sec Loss 0.6563 Epoch: 20 Global Step: 207775 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 04:01:33,204-Speed 130.16 samples/sec Loss 0.6106 Epoch: 20 Global Step: 207780 Fp16 Grad Scale: 4096 Required: 50 hours Training: 2021-06-17 04:01:53,291-Speed 127.45 samples/sec Loss 0.5604 Epoch: 20 Global Step: 207785 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:02:12,905-Speed 130.53 samples/sec Loss 0.6541 Epoch: 20 Global Step: 207790 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:02:32,882-Speed 128.15 samples/sec Loss 0.6445 Epoch: 20 Global Step: 207795 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:02:53,039-Speed 127.01 samples/sec Loss 0.6885 Epoch: 20 Global Step: 207800 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:03:12,837-Speed 129.31 samples/sec Loss 0.7284 Epoch: 20 Global Step: 207805 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:03:32,358-Speed 131.14 samples/sec Loss 0.6322 Epoch: 20 Global Step: 207810 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:03:51,983-Speed 130.45 samples/sec Loss 0.5566 Epoch: 20 Global Step: 207815 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:04:11,523-Speed 131.02 samples/sec Loss 0.6489 Epoch: 20 Global Step: 207820 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:04:31,145-Speed 130.47 samples/sec Loss 0.6198 Epoch: 20 Global Step: 207825 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:04:50,743-Speed 130.64 samples/sec Loss 0.5726 Epoch: 20 Global Step: 207830 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:05:10,277-Speed 131.06 samples/sec Loss 0.6222 Epoch: 20 Global Step: 207835 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:05:29,887-Speed 130.55 samples/sec Loss 0.6224 Epoch: 20 Global Step: 207840 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:05:49,868-Speed 128.13 samples/sec Loss 0.6562 Epoch: 20 Global Step: 207845 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:06:09,311-Speed 131.67 samples/sec Loss 0.5504 Epoch: 20 Global Step: 207850 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:06:28,893-Speed 130.74 samples/sec Loss 0.5736 Epoch: 20 Global Step: 207855 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:06:48,531-Speed 130.36 samples/sec Loss 0.6401 Epoch: 20 Global Step: 207860 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:07:08,053-Speed 131.14 samples/sec Loss 0.7280 Epoch: 20 Global Step: 207865 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:07:27,659-Speed 130.58 samples/sec Loss 0.6531 Epoch: 20 Global Step: 207870 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:07:47,113-Speed 131.60 samples/sec Loss 0.5861 Epoch: 20 Global Step: 207875 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:08:06,678-Speed 130.86 samples/sec Loss 0.5057 Epoch: 20 Global Step: 207880 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:08:26,253-Speed 130.79 samples/sec Loss 0.6023 Epoch: 20 Global Step: 207885 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:08:45,925-Speed 130.14 samples/sec Loss 0.6805 Epoch: 20 Global Step: 207890 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:09:05,534-Speed 130.56 samples/sec Loss 0.5969 Epoch: 20 Global Step: 207895 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:09:24,817-Speed 132.77 samples/sec Loss 0.6234 Epoch: 20 Global Step: 207900 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:09:44,784-Speed 128.22 samples/sec Loss 0.8112 Epoch: 20 Global Step: 207905 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:10:04,729-Speed 128.36 samples/sec Loss 0.6265 Epoch: 20 Global Step: 207910 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:10:24,591-Speed 128.89 samples/sec Loss 0.6000 Epoch: 20 Global Step: 207915 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:10:44,209-Speed 130.50 samples/sec Loss 0.6160 Epoch: 20 Global Step: 207920 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:11:04,048-Speed 129.04 samples/sec Loss 0.5933 Epoch: 20 Global Step: 207925 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:11:23,913-Speed 128.88 samples/sec Loss 0.6285 Epoch: 20 Global Step: 207930 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:11:43,545-Speed 130.41 samples/sec Loss 0.5638 Epoch: 20 Global Step: 207935 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:12:03,405-Speed 128.91 samples/sec Loss 0.6627 Epoch: 20 Global Step: 207940 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:12:23,404-Speed 128.01 samples/sec Loss 0.6436 Epoch: 20 Global Step: 207945 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:12:43,371-Speed 128.22 samples/sec Loss 0.6559 Epoch: 20 Global Step: 207950 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:13:03,052-Speed 130.08 samples/sec Loss 0.7140 Epoch: 20 Global Step: 207955 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:13:23,091-Speed 127.76 samples/sec Loss 0.6565 Epoch: 20 Global Step: 207960 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:13:42,801-Speed 129.89 samples/sec Loss 0.6909 Epoch: 20 Global Step: 207965 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:14:02,763-Speed 128.25 samples/sec Loss 0.6986 Epoch: 20 Global Step: 207970 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:14:22,496-Speed 129.74 samples/sec Loss 0.6238 Epoch: 20 Global Step: 207975 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:14:42,251-Speed 129.60 samples/sec Loss 0.6907 Epoch: 20 Global Step: 207980 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:15:02,014-Speed 129.54 samples/sec Loss 0.6185 Epoch: 20 Global Step: 207985 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:15:22,033-Speed 127.88 samples/sec Loss 0.6201 Epoch: 20 Global Step: 207990 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:15:41,994-Speed 128.26 samples/sec Loss 0.6315 Epoch: 20 Global Step: 207995 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:16:01,707-Speed 129.87 samples/sec Loss 0.6191 Epoch: 20 Global Step: 208000 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:16:21,687-Speed 128.14 samples/sec Loss 0.5583 Epoch: 20 Global Step: 208005 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:16:41,421-Speed 129.73 samples/sec Loss 0.5637 Epoch: 20 Global Step: 208010 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:17:01,395-Speed 128.17 samples/sec Loss 0.5879 Epoch: 20 Global Step: 208015 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:17:20,998-Speed 130.60 samples/sec Loss 0.6618 Epoch: 20 Global Step: 208020 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:17:40,690-Speed 130.01 samples/sec Loss 0.6689 Epoch: 20 Global Step: 208025 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:18:00,455-Speed 129.53 samples/sec Loss 0.5742 Epoch: 20 Global Step: 208030 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:18:20,475-Speed 127.88 samples/sec Loss 0.7002 Epoch: 20 Global Step: 208035 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:18:40,011-Speed 131.05 samples/sec Loss 0.6638 Epoch: 20 Global Step: 208040 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:18:59,627-Speed 130.51 samples/sec Loss 0.6508 Epoch: 20 Global Step: 208045 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:19:19,721-Speed 127.41 samples/sec Loss 0.6956 Epoch: 20 Global Step: 208050 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:19:39,447-Speed 129.78 samples/sec Loss 0.6648 Epoch: 20 Global Step: 208055 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:19:59,324-Speed 128.80 samples/sec Loss 0.6684 Epoch: 20 Global Step: 208060 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:20:19,098-Speed 129.47 samples/sec Loss 0.6162 Epoch: 20 Global Step: 208065 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:20:38,800-Speed 129.95 samples/sec Loss 0.6486 Epoch: 20 Global Step: 208070 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:20:58,656-Speed 128.93 samples/sec Loss 0.5955 Epoch: 20 Global Step: 208075 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:21:18,681-Speed 127.85 samples/sec Loss 0.7004 Epoch: 20 Global Step: 208080 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:21:38,578-Speed 128.67 samples/sec Loss 0.6684 Epoch: 20 Global Step: 208085 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:21:58,253-Speed 130.12 samples/sec Loss 0.6453 Epoch: 20 Global Step: 208090 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:22:18,452-Speed 126.75 samples/sec Loss 0.6312 Epoch: 20 Global Step: 208095 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:22:38,186-Speed 129.73 samples/sec Loss 0.7429 Epoch: 20 Global Step: 208100 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:22:58,079-Speed 128.69 samples/sec Loss 0.6414 Epoch: 20 Global Step: 208105 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:23:17,724-Speed 130.32 samples/sec Loss 0.5887 Epoch: 20 Global Step: 208110 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:23:37,516-Speed 129.36 samples/sec Loss 0.5792 Epoch: 20 Global Step: 208115 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:23:57,377-Speed 128.90 samples/sec Loss 0.7115 Epoch: 20 Global Step: 208120 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:24:17,324-Speed 128.34 samples/sec Loss 0.5643 Epoch: 20 Global Step: 208125 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:24:37,204-Speed 128.78 samples/sec Loss 0.6120 Epoch: 20 Global Step: 208130 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:24:56,909-Speed 129.92 samples/sec Loss 0.6057 Epoch: 20 Global Step: 208135 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:25:17,017-Speed 127.32 samples/sec Loss 0.6753 Epoch: 20 Global Step: 208140 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:25:36,757-Speed 129.69 samples/sec Loss 0.6223 Epoch: 20 Global Step: 208145 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:25:56,632-Speed 128.81 samples/sec Loss 0.7343 Epoch: 20 Global Step: 208150 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:26:16,412-Speed 129.43 samples/sec Loss 0.7243 Epoch: 20 Global Step: 208155 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:26:36,469-Speed 127.64 samples/sec Loss 0.7531 Epoch: 20 Global Step: 208160 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:26:55,692-Speed 133.18 samples/sec Loss 0.6457 Epoch: 20 Global Step: 208165 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:27:15,510-Speed 129.18 samples/sec Loss 0.6486 Epoch: 20 Global Step: 208170 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:27:35,440-Speed 128.46 samples/sec Loss 0.6309 Epoch: 20 Global Step: 208175 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:27:55,442-Speed 127.99 samples/sec Loss 0.7367 Epoch: 20 Global Step: 208180 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:28:15,605-Speed 126.97 samples/sec Loss 0.7358 Epoch: 20 Global Step: 208185 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:28:35,266-Speed 130.21 samples/sec Loss 0.7309 Epoch: 20 Global Step: 208190 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:28:55,158-Speed 128.70 samples/sec Loss 0.6070 Epoch: 20 Global Step: 208195 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:29:15,027-Speed 128.85 samples/sec Loss 0.7096 Epoch: 20 Global Step: 208200 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:29:34,933-Speed 128.61 samples/sec Loss 0.6468 Epoch: 20 Global Step: 208205 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:29:54,620-Speed 130.04 samples/sec Loss 0.5898 Epoch: 20 Global Step: 208210 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:30:14,305-Speed 130.06 samples/sec Loss 0.6657 Epoch: 20 Global Step: 208215 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:30:34,168-Speed 128.89 samples/sec Loss 0.5381 Epoch: 20 Global Step: 208220 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:30:54,253-Speed 127.47 samples/sec Loss 0.7274 Epoch: 20 Global Step: 208225 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:31:14,334-Speed 127.49 samples/sec Loss 0.6618 Epoch: 20 Global Step: 208230 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:31:34,055-Speed 129.82 samples/sec Loss 0.5660 Epoch: 20 Global Step: 208235 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:31:53,929-Speed 128.82 samples/sec Loss 0.7140 Epoch: 20 Global Step: 208240 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:32:13,541-Speed 130.54 samples/sec Loss 0.5987 Epoch: 20 Global Step: 208245 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:32:33,460-Speed 128.52 samples/sec Loss 0.6465 Epoch: 20 Global Step: 208250 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:32:53,140-Speed 130.09 samples/sec Loss 0.5764 Epoch: 20 Global Step: 208255 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:33:12,802-Speed 130.21 samples/sec Loss 0.5804 Epoch: 20 Global Step: 208260 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:33:32,678-Speed 128.81 samples/sec Loss 0.6416 Epoch: 20 Global Step: 208265 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:33:52,742-Speed 127.60 samples/sec Loss 0.6649 Epoch: 20 Global Step: 208270 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:34:12,813-Speed 127.55 samples/sec Loss 0.5713 Epoch: 20 Global Step: 208275 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:34:32,480-Speed 130.17 samples/sec Loss 0.6708 Epoch: 20 Global Step: 208280 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:34:52,367-Speed 128.74 samples/sec Loss 0.5682 Epoch: 20 Global Step: 208285 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:35:12,125-Speed 129.57 samples/sec Loss 0.6403 Epoch: 20 Global Step: 208290 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:35:32,044-Speed 128.53 samples/sec Loss 0.6227 Epoch: 20 Global Step: 208295 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:35:51,753-Speed 129.90 samples/sec Loss 0.6001 Epoch: 20 Global Step: 208300 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:36:11,427-Speed 130.12 samples/sec Loss 0.6000 Epoch: 20 Global Step: 208305 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:36:31,306-Speed 128.79 samples/sec Loss 0.6527 Epoch: 20 Global Step: 208310 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:36:51,575-Speed 126.31 samples/sec Loss 0.5869 Epoch: 20 Global Step: 208315 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:37:11,637-Speed 127.61 samples/sec Loss 0.6179 Epoch: 20 Global Step: 208320 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:37:31,469-Speed 129.09 samples/sec Loss 0.6524 Epoch: 20 Global Step: 208325 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:37:51,206-Speed 129.71 samples/sec Loss 0.7281 Epoch: 20 Global Step: 208330 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:38:10,926-Speed 129.82 samples/sec Loss 0.6366 Epoch: 20 Global Step: 208335 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:38:30,840-Speed 128.56 samples/sec Loss 0.5836 Epoch: 20 Global Step: 208340 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:38:50,550-Speed 129.89 samples/sec Loss 0.6352 Epoch: 20 Global Step: 208345 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:39:10,383-Speed 129.08 samples/sec Loss 0.6816 Epoch: 20 Global Step: 208350 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:39:30,104-Speed 129.82 samples/sec Loss 0.5629 Epoch: 20 Global Step: 208355 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:39:50,221-Speed 127.26 samples/sec Loss 0.5846 Epoch: 20 Global Step: 208360 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:40:10,289-Speed 127.57 samples/sec Loss 0.6048 Epoch: 20 Global Step: 208365 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:40:30,092-Speed 129.28 samples/sec Loss 0.6209 Epoch: 20 Global Step: 208370 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:40:49,886-Speed 129.34 samples/sec Loss 0.6648 Epoch: 20 Global Step: 208375 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:41:09,758-Speed 128.83 samples/sec Loss 0.6661 Epoch: 20 Global Step: 208380 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:41:29,564-Speed 129.26 samples/sec Loss 0.6890 Epoch: 20 Global Step: 208385 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:41:49,291-Speed 129.78 samples/sec Loss 0.7283 Epoch: 20 Global Step: 208390 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:42:09,201-Speed 128.58 samples/sec Loss 0.6339 Epoch: 20 Global Step: 208395 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:42:29,006-Speed 129.27 samples/sec Loss 0.6363 Epoch: 20 Global Step: 208400 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:42:49,129-Speed 127.22 samples/sec Loss 0.5824 Epoch: 20 Global Step: 208405 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:43:09,138-Speed 127.95 samples/sec Loss 0.6271 Epoch: 20 Global Step: 208410 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:43:28,904-Speed 129.52 samples/sec Loss 0.6062 Epoch: 20 Global Step: 208415 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:43:48,680-Speed 129.46 samples/sec Loss 0.6120 Epoch: 20 Global Step: 208420 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:44:08,602-Speed 128.51 samples/sec Loss 0.5770 Epoch: 20 Global Step: 208425 Fp16 Grad Scale: 8192 Required: 49 hours Training: 2021-06-17 04:44:27,882-Speed 132.79 samples/sec Loss 0.7213 Epoch: 20 Global Step: 208430 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:44:47,291-Speed 131.90 samples/sec Loss 0.5775 Epoch: 20 Global Step: 208435 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:45:07,039-Speed 129.64 samples/sec Loss 0.6347 Epoch: 20 Global Step: 208440 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:45:26,787-Speed 129.64 samples/sec Loss 0.5644 Epoch: 20 Global Step: 208445 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:45:47,002-Speed 126.64 samples/sec Loss 0.6144 Epoch: 20 Global Step: 208450 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:46:06,658-Speed 130.25 samples/sec Loss 0.6608 Epoch: 20 Global Step: 208455 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:46:26,540-Speed 128.77 samples/sec Loss 0.6936 Epoch: 20 Global Step: 208460 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:46:46,507-Speed 128.21 samples/sec Loss 0.5829 Epoch: 20 Global Step: 208465 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:47:06,319-Speed 129.22 samples/sec Loss 0.7048 Epoch: 20 Global Step: 208470 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:47:26,034-Speed 129.86 samples/sec Loss 0.6644 Epoch: 20 Global Step: 208475 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:47:45,927-Speed 128.69 samples/sec Loss 0.6941 Epoch: 20 Global Step: 208480 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:48:05,723-Speed 129.34 samples/sec Loss 0.6656 Epoch: 20 Global Step: 208485 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:48:25,494-Speed 129.49 samples/sec Loss 0.5816 Epoch: 20 Global Step: 208490 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:48:45,664-Speed 126.93 samples/sec Loss 0.5938 Epoch: 20 Global Step: 208495 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:49:05,396-Speed 129.75 samples/sec Loss 0.6679 Epoch: 20 Global Step: 208500 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:49:25,357-Speed 128.26 samples/sec Loss 0.5972 Epoch: 20 Global Step: 208505 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:49:45,318-Speed 128.25 samples/sec Loss 0.6017 Epoch: 20 Global Step: 208510 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:50:05,215-Speed 128.67 samples/sec Loss 0.5972 Epoch: 20 Global Step: 208515 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:50:25,046-Speed 129.10 samples/sec Loss 0.7399 Epoch: 20 Global Step: 208520 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:50:44,943-Speed 128.68 samples/sec Loss 0.5722 Epoch: 20 Global Step: 208525 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:51:04,741-Speed 129.31 samples/sec Loss 0.6586 Epoch: 20 Global Step: 208530 Fp16 Grad Scale: 2048 Required: 49 hours Training: 2021-06-17 04:51:24,765-Speed 127.85 samples/sec Loss 0.5903 Epoch: 20 Global Step: 208535 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:51:44,800-Speed 127.79 samples/sec Loss 0.6651 Epoch: 20 Global Step: 208540 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:52:04,567-Speed 129.52 samples/sec Loss 0.7045 Epoch: 20 Global Step: 208545 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:52:24,443-Speed 128.80 samples/sec Loss 0.5741 Epoch: 20 Global Step: 208550 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:52:44,349-Speed 128.61 samples/sec Loss 0.5959 Epoch: 20 Global Step: 208555 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:53:04,160-Speed 129.23 samples/sec Loss 0.6605 Epoch: 20 Global Step: 208560 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:53:24,060-Speed 128.65 samples/sec Loss 0.6156 Epoch: 20 Global Step: 208565 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:53:43,929-Speed 128.85 samples/sec Loss 0.6336 Epoch: 20 Global Step: 208570 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:54:03,722-Speed 129.35 samples/sec Loss 0.6213 Epoch: 20 Global Step: 208575 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:54:23,508-Speed 129.39 samples/sec Loss 0.6833 Epoch: 20 Global Step: 208580 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:54:43,545-Speed 127.77 samples/sec Loss 0.6586 Epoch: 20 Global Step: 208585 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:55:03,300-Speed 129.60 samples/sec Loss 0.6132 Epoch: 20 Global Step: 208590 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:55:23,136-Speed 129.07 samples/sec Loss 0.6615 Epoch: 20 Global Step: 208595 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:55:43,100-Speed 128.24 samples/sec Loss 0.6227 Epoch: 20 Global Step: 208600 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:56:02,834-Speed 129.73 samples/sec Loss 0.6582 Epoch: 20 Global Step: 208605 Fp16 Grad Scale: 4096 Required: 49 hours Training: 2021-06-17 04:56:22,606-Speed 129.48 samples/sec Loss 0.6755 Epoch: 20 Global Step: 208610 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 04:56:42,522-Speed 128.55 samples/sec Loss 0.6036 Epoch: 20 Global Step: 208615 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 04:57:02,162-Speed 130.36 samples/sec Loss 0.6565 Epoch: 20 Global Step: 208620 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 04:57:21,944-Speed 129.42 samples/sec Loss 0.6100 Epoch: 20 Global Step: 208625 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 04:57:41,982-Speed 127.78 samples/sec Loss 0.5847 Epoch: 20 Global Step: 208630 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 04:58:01,866-Speed 128.76 samples/sec Loss 0.6382 Epoch: 20 Global Step: 208635 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 04:58:21,555-Speed 130.03 samples/sec Loss 0.7006 Epoch: 20 Global Step: 208640 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 04:58:41,541-Speed 128.10 samples/sec Loss 0.6493 Epoch: 20 Global Step: 208645 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 04:59:01,278-Speed 129.71 samples/sec Loss 0.6702 Epoch: 20 Global Step: 208650 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 04:59:21,086-Speed 129.25 samples/sec Loss 0.6005 Epoch: 20 Global Step: 208655 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 04:59:41,034-Speed 128.34 samples/sec Loss 0.6689 Epoch: 20 Global Step: 208660 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:00:00,647-Speed 130.53 samples/sec Loss 0.7568 Epoch: 20 Global Step: 208665 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:00:20,743-Speed 127.40 samples/sec Loss 0.6979 Epoch: 20 Global Step: 208670 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:00:40,774-Speed 127.81 samples/sec Loss 0.6454 Epoch: 20 Global Step: 208675 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:01:00,729-Speed 128.30 samples/sec Loss 0.6773 Epoch: 20 Global Step: 208680 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:01:20,324-Speed 130.65 samples/sec Loss 0.6385 Epoch: 20 Global Step: 208685 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:01:40,407-Speed 127.48 samples/sec Loss 0.6073 Epoch: 20 Global Step: 208690 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:02:00,244-Speed 129.06 samples/sec Loss 0.7206 Epoch: 20 Global Step: 208695 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:02:19,637-Speed 132.01 samples/sec Loss 0.6654 Epoch: 20 Global Step: 208700 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:02:39,315-Speed 130.10 samples/sec Loss 0.6892 Epoch: 20 Global Step: 208705 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:02:59,099-Speed 129.41 samples/sec Loss 0.5823 Epoch: 20 Global Step: 208710 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:03:18,898-Speed 129.31 samples/sec Loss 0.6035 Epoch: 20 Global Step: 208715 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:03:38,676-Speed 129.44 samples/sec Loss 0.6314 Epoch: 20 Global Step: 208720 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:03:58,584-Speed 128.60 samples/sec Loss 0.7371 Epoch: 20 Global Step: 208725 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:04:18,393-Speed 129.24 samples/sec Loss 0.5929 Epoch: 20 Global Step: 208730 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:04:38,774-Speed 125.62 samples/sec Loss 0.5949 Epoch: 20 Global Step: 208735 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:04:58,682-Speed 128.60 samples/sec Loss 0.6321 Epoch: 20 Global Step: 208740 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:05:18,510-Speed 129.11 samples/sec Loss 0.6296 Epoch: 20 Global Step: 208745 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:05:38,227-Speed 129.84 samples/sec Loss 0.7265 Epoch: 20 Global Step: 208750 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:05:58,006-Speed 129.44 samples/sec Loss 0.6022 Epoch: 20 Global Step: 208755 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:06:17,723-Speed 129.85 samples/sec Loss 0.6047 Epoch: 20 Global Step: 208760 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:06:37,514-Speed 129.36 samples/sec Loss 0.6307 Epoch: 20 Global Step: 208765 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:06:57,354-Speed 129.04 samples/sec Loss 0.6399 Epoch: 20 Global Step: 208770 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:07:17,363-Speed 127.95 samples/sec Loss 0.6854 Epoch: 20 Global Step: 208775 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:07:37,414-Speed 127.68 samples/sec Loss 0.5955 Epoch: 20 Global Step: 208780 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:07:57,305-Speed 128.71 samples/sec Loss 0.6787 Epoch: 20 Global Step: 208785 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:08:17,085-Speed 129.43 samples/sec Loss 0.6654 Epoch: 20 Global Step: 208790 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:08:36,849-Speed 129.54 samples/sec Loss 0.5442 Epoch: 20 Global Step: 208795 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:08:56,680-Speed 129.10 samples/sec Loss 0.5554 Epoch: 20 Global Step: 208800 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:09:16,296-Speed 130.51 samples/sec Loss 0.7530 Epoch: 20 Global Step: 208805 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:09:36,215-Speed 128.53 samples/sec Loss 0.6921 Epoch: 20 Global Step: 208810 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:09:55,537-Speed 132.49 samples/sec Loss 0.7106 Epoch: 20 Global Step: 208815 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:10:15,638-Speed 127.37 samples/sec Loss 0.7126 Epoch: 20 Global Step: 208820 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:10:35,609-Speed 128.19 samples/sec Loss 0.6053 Epoch: 20 Global Step: 208825 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:10:55,522-Speed 128.57 samples/sec Loss 0.5552 Epoch: 20 Global Step: 208830 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:11:15,319-Speed 129.31 samples/sec Loss 0.6970 Epoch: 20 Global Step: 208835 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:11:35,233-Speed 128.56 samples/sec Loss 0.6842 Epoch: 20 Global Step: 208840 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:11:55,100-Speed 128.86 samples/sec Loss 0.7010 Epoch: 20 Global Step: 208845 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:12:14,755-Speed 130.25 samples/sec Loss 0.6227 Epoch: 20 Global Step: 208850 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:12:34,684-Speed 128.46 samples/sec Loss 0.6166 Epoch: 20 Global Step: 208855 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:12:54,471-Speed 129.38 samples/sec Loss 0.6086 Epoch: 20 Global Step: 208860 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:13:14,431-Speed 128.27 samples/sec Loss 0.6821 Epoch: 20 Global Step: 208865 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:13:33,839-Speed 131.91 samples/sec Loss 0.6187 Epoch: 20 Global Step: 208870 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:13:53,572-Speed 129.74 samples/sec Loss 0.6164 Epoch: 20 Global Step: 208875 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:14:13,729-Speed 127.01 samples/sec Loss 0.6222 Epoch: 20 Global Step: 208880 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:14:33,522-Speed 129.35 samples/sec Loss 0.7528 Epoch: 20 Global Step: 208885 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:14:53,308-Speed 129.39 samples/sec Loss 0.6990 Epoch: 20 Global Step: 208890 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:15:13,059-Speed 129.62 samples/sec Loss 0.6850 Epoch: 20 Global Step: 208895 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:15:32,932-Speed 128.82 samples/sec Loss 0.6559 Epoch: 20 Global Step: 208900 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:15:52,716-Speed 129.41 samples/sec Loss 0.6737 Epoch: 20 Global Step: 208905 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:16:12,497-Speed 129.42 samples/sec Loss 0.7296 Epoch: 20 Global Step: 208910 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:16:32,210-Speed 129.87 samples/sec Loss 0.6721 Epoch: 20 Global Step: 208915 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:16:51,830-Speed 130.49 samples/sec Loss 0.5988 Epoch: 20 Global Step: 208920 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:17:11,552-Speed 129.82 samples/sec Loss 0.6468 Epoch: 20 Global Step: 208925 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:17:31,081-Speed 131.10 samples/sec Loss 0.6998 Epoch: 20 Global Step: 208930 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:17:50,949-Speed 128.86 samples/sec Loss 0.6786 Epoch: 20 Global Step: 208935 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:18:10,859-Speed 128.58 samples/sec Loss 0.6674 Epoch: 20 Global Step: 208940 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:18:30,655-Speed 129.32 samples/sec Loss 0.7024 Epoch: 20 Global Step: 208945 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:18:50,225-Speed 130.82 samples/sec Loss 0.6222 Epoch: 20 Global Step: 208950 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:19:10,091-Speed 128.87 samples/sec Loss 0.6231 Epoch: 20 Global Step: 208955 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:19:29,810-Speed 129.83 samples/sec Loss 0.6424 Epoch: 20 Global Step: 208960 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:19:49,407-Speed 130.64 samples/sec Loss 0.7033 Epoch: 20 Global Step: 208965 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:20:09,107-Speed 129.95 samples/sec Loss 0.6790 Epoch: 20 Global Step: 208970 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:20:28,574-Speed 131.52 samples/sec Loss 0.6338 Epoch: 20 Global Step: 208975 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:20:48,442-Speed 128.86 samples/sec Loss 0.6890 Epoch: 20 Global Step: 208980 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:21:08,256-Speed 129.20 samples/sec Loss 0.5929 Epoch: 20 Global Step: 208985 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:21:28,056-Speed 129.30 samples/sec Loss 0.7591 Epoch: 20 Global Step: 208990 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:21:47,614-Speed 130.90 samples/sec Loss 0.6532 Epoch: 20 Global Step: 208995 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:22:07,380-Speed 129.52 samples/sec Loss 0.7640 Epoch: 20 Global Step: 209000 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:22:27,196-Speed 129.19 samples/sec Loss 0.6148 Epoch: 20 Global Step: 209005 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:22:46,934-Speed 129.71 samples/sec Loss 0.6421 Epoch: 20 Global Step: 209010 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:23:06,657-Speed 129.80 samples/sec Loss 0.6628 Epoch: 20 Global Step: 209015 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:23:26,183-Speed 131.12 samples/sec Loss 0.6236 Epoch: 20 Global Step: 209020 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:23:46,000-Speed 129.19 samples/sec Loss 0.6714 Epoch: 20 Global Step: 209025 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:24:05,836-Speed 129.07 samples/sec Loss 0.6530 Epoch: 20 Global Step: 209030 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:24:25,633-Speed 129.32 samples/sec Loss 0.6443 Epoch: 20 Global Step: 209035 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:24:45,208-Speed 130.79 samples/sec Loss 0.6836 Epoch: 20 Global Step: 209040 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:25:04,966-Speed 129.58 samples/sec Loss 0.5935 Epoch: 20 Global Step: 209045 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:25:24,576-Speed 130.55 samples/sec Loss 0.6609 Epoch: 20 Global Step: 209050 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:25:44,183-Speed 130.58 samples/sec Loss 0.6307 Epoch: 20 Global Step: 209055 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:26:03,892-Speed 129.89 samples/sec Loss 0.6598 Epoch: 20 Global Step: 209060 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:26:23,192-Speed 132.65 samples/sec Loss 0.6809 Epoch: 20 Global Step: 209065 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:26:43,169-Speed 128.15 samples/sec Loss 0.6655 Epoch: 20 Global Step: 209070 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:27:03,150-Speed 128.13 samples/sec Loss 0.6116 Epoch: 20 Global Step: 209075 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:27:22,935-Speed 129.40 samples/sec Loss 0.6680 Epoch: 20 Global Step: 209080 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:27:42,690-Speed 129.59 samples/sec Loss 0.6212 Epoch: 20 Global Step: 209085 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:28:02,538-Speed 128.99 samples/sec Loss 0.6011 Epoch: 20 Global Step: 209090 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:28:22,195-Speed 130.24 samples/sec Loss 0.6506 Epoch: 20 Global Step: 209095 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:28:42,090-Speed 128.68 samples/sec Loss 0.7333 Epoch: 20 Global Step: 209100 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:29:01,874-Speed 129.40 samples/sec Loss 0.6008 Epoch: 20 Global Step: 209105 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:29:21,583-Speed 129.90 samples/sec Loss 0.6472 Epoch: 20 Global Step: 209110 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:29:41,594-Speed 127.93 samples/sec Loss 0.6995 Epoch: 20 Global Step: 209115 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:30:01,906-Speed 126.04 samples/sec Loss 0.6040 Epoch: 20 Global Step: 209120 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:30:21,657-Speed 129.62 samples/sec Loss 0.7446 Epoch: 20 Global Step: 209125 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:30:41,486-Speed 129.11 samples/sec Loss 0.6276 Epoch: 20 Global Step: 209130 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:31:01,274-Speed 129.38 samples/sec Loss 0.5923 Epoch: 20 Global Step: 209135 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:31:20,896-Speed 130.48 samples/sec Loss 0.5513 Epoch: 20 Global Step: 209140 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:31:40,792-Speed 128.67 samples/sec Loss 0.6381 Epoch: 20 Global Step: 209145 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:32:00,512-Speed 129.83 samples/sec Loss 0.7234 Epoch: 20 Global Step: 209150 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:32:20,371-Speed 128.91 samples/sec Loss 0.6320 Epoch: 20 Global Step: 209155 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:32:40,251-Speed 128.78 samples/sec Loss 0.6919 Epoch: 20 Global Step: 209160 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:33:00,341-Speed 127.43 samples/sec Loss 0.6353 Epoch: 20 Global Step: 209165 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:33:20,074-Speed 129.74 samples/sec Loss 0.7127 Epoch: 20 Global Step: 209170 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:33:39,825-Speed 129.62 samples/sec Loss 0.7033 Epoch: 20 Global Step: 209175 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:33:59,634-Speed 129.24 samples/sec Loss 0.5371 Epoch: 20 Global Step: 209180 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:34:19,257-Speed 130.47 samples/sec Loss 0.6244 Epoch: 20 Global Step: 209185 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:34:39,212-Speed 128.29 samples/sec Loss 0.5742 Epoch: 20 Global Step: 209190 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:34:58,971-Speed 129.57 samples/sec Loss 0.7223 Epoch: 20 Global Step: 209195 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:35:18,933-Speed 128.25 samples/sec Loss 0.6893 Epoch: 20 Global Step: 209200 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:35:38,902-Speed 128.20 samples/sec Loss 0.7256 Epoch: 20 Global Step: 209205 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:35:59,072-Speed 126.93 samples/sec Loss 0.5893 Epoch: 20 Global Step: 209210 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:36:18,716-Speed 130.33 samples/sec Loss 0.7388 Epoch: 20 Global Step: 209215 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:36:38,614-Speed 128.66 samples/sec Loss 0.5728 Epoch: 20 Global Step: 209220 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:36:58,319-Speed 129.92 samples/sec Loss 0.6406 Epoch: 20 Global Step: 209225 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:37:17,928-Speed 130.56 samples/sec Loss 0.5655 Epoch: 20 Global Step: 209230 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:37:37,924-Speed 128.03 samples/sec Loss 0.6809 Epoch: 20 Global Step: 209235 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:37:57,587-Speed 130.20 samples/sec Loss 0.6524 Epoch: 20 Global Step: 209240 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:38:48,483-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-17 05:38:59,553-Speed 41.31 samples/sec Loss 0.6152 Epoch: 21 Global Step: 209245 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:39:19,889-Speed 125.89 samples/sec Loss 0.4520 Epoch: 21 Global Step: 209250 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:39:39,645-Speed 129.59 samples/sec Loss 0.3648 Epoch: 21 Global Step: 209255 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:39:59,410-Speed 129.53 samples/sec Loss 0.3854 Epoch: 21 Global Step: 209260 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:40:19,606-Speed 126.76 samples/sec Loss 0.4273 Epoch: 21 Global Step: 209265 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:40:39,376-Speed 129.50 samples/sec Loss 0.4185 Epoch: 21 Global Step: 209270 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:40:59,216-Speed 129.04 samples/sec Loss 0.5015 Epoch: 21 Global Step: 209275 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:41:18,827-Speed 130.55 samples/sec Loss 0.4128 Epoch: 21 Global Step: 209280 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:41:38,771-Speed 128.36 samples/sec Loss 0.5371 Epoch: 21 Global Step: 209285 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:41:58,541-Speed 129.50 samples/sec Loss 0.4343 Epoch: 21 Global Step: 209290 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:42:18,721-Speed 126.86 samples/sec Loss 0.4389 Epoch: 21 Global Step: 209295 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:42:38,492-Speed 129.49 samples/sec Loss 0.3584 Epoch: 21 Global Step: 209300 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:42:58,209-Speed 129.84 samples/sec Loss 0.4000 Epoch: 21 Global Step: 209305 Fp16 Grad Scale: 8192 Required: 48 hours Training: 2021-06-17 05:43:17,967-Speed 129.58 samples/sec Loss 0.4075 Epoch: 21 Global Step: 209310 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:43:37,914-Speed 128.35 samples/sec Loss 0.4628 Epoch: 21 Global Step: 209315 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:43:57,667-Speed 129.61 samples/sec Loss 0.3730 Epoch: 21 Global Step: 209320 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:44:17,285-Speed 130.50 samples/sec Loss 0.4251 Epoch: 21 Global Step: 209325 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:44:37,187-Speed 128.63 samples/sec Loss 0.4311 Epoch: 21 Global Step: 209330 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:44:56,900-Speed 129.87 samples/sec Loss 0.4733 Epoch: 21 Global Step: 209335 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:45:16,864-Speed 128.23 samples/sec Loss 0.3995 Epoch: 21 Global Step: 209340 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:45:36,619-Speed 129.60 samples/sec Loss 0.3498 Epoch: 21 Global Step: 209345 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:45:56,290-Speed 130.15 samples/sec Loss 0.4135 Epoch: 21 Global Step: 209350 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:46:16,456-Speed 126.95 samples/sec Loss 0.4069 Epoch: 21 Global Step: 209355 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:46:36,350-Speed 128.69 samples/sec Loss 0.4266 Epoch: 21 Global Step: 209360 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:46:56,026-Speed 130.12 samples/sec Loss 0.4283 Epoch: 21 Global Step: 209365 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:47:15,751-Speed 129.79 samples/sec Loss 0.3882 Epoch: 21 Global Step: 209370 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:47:35,574-Speed 129.15 samples/sec Loss 0.3351 Epoch: 21 Global Step: 209375 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:47:55,360-Speed 129.39 samples/sec Loss 0.4004 Epoch: 21 Global Step: 209380 Fp16 Grad Scale: 4096 Required: 48 hours Training: 2021-06-17 05:48:14,931-Speed 130.81 samples/sec Loss 0.4399 Epoch: 21 Global Step: 209385 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:48:34,695-Speed 129.53 samples/sec Loss 0.4201 Epoch: 21 Global Step: 209390 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:48:54,461-Speed 129.52 samples/sec Loss 0.4122 Epoch: 21 Global Step: 209395 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:49:14,574-Speed 127.29 samples/sec Loss 0.3820 Epoch: 21 Global Step: 209400 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:49:34,500-Speed 128.48 samples/sec Loss 0.3732 Epoch: 21 Global Step: 209405 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:49:54,103-Speed 130.60 samples/sec Loss 0.4110 Epoch: 21 Global Step: 209410 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:50:13,816-Speed 129.87 samples/sec Loss 0.4003 Epoch: 21 Global Step: 209415 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:50:33,623-Speed 129.25 samples/sec Loss 0.4197 Epoch: 21 Global Step: 209420 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:50:53,418-Speed 129.33 samples/sec Loss 0.4562 Epoch: 21 Global Step: 209425 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:51:13,558-Speed 127.12 samples/sec Loss 0.3709 Epoch: 21 Global Step: 209430 Fp16 Grad Scale: 2048 Required: 48 hours Training: 2021-06-17 05:51:33,336-Speed 129.44 samples/sec Loss 0.4205 Epoch: 21 Global Step: 209435 Fp16 Grad Scale: 2048 Required: 47 hours Training: 2021-06-17 05:51:53,083-Speed 129.65 samples/sec Loss 0.3912 Epoch: 21 Global Step: 209440 Fp16 Grad Scale: 2048 Required: 47 hours Training: 2021-06-17 05:52:13,172-Speed 127.44 samples/sec Loss 0.3856 Epoch: 21 Global Step: 209445 Fp16 Grad Scale: 2048 Required: 47 hours Training: 2021-06-17 05:52:33,087-Speed 128.55 samples/sec Loss 0.4629 Epoch: 21 Global Step: 209450 Fp16 Grad Scale: 2048 Required: 47 hours Training: 2021-06-17 05:52:52,700-Speed 130.53 samples/sec Loss 0.4525 Epoch: 21 Global Step: 209455 Fp16 Grad Scale: 2048 Required: 47 hours Training: 2021-06-17 05:53:12,594-Speed 128.69 samples/sec Loss 0.4583 Epoch: 21 Global Step: 209460 Fp16 Grad Scale: 2048 Required: 47 hours Training: 2021-06-17 05:53:32,315-Speed 129.82 samples/sec Loss 0.4065 Epoch: 21 Global Step: 209465 Fp16 Grad Scale: 2048 Required: 47 hours Training: 2021-06-17 05:53:52,028-Speed 129.87 samples/sec Loss 0.3835 Epoch: 21 Global Step: 209470 Fp16 Grad Scale: 2048 Required: 47 hours Training: 2021-06-17 05:54:12,378-Speed 125.80 samples/sec Loss 0.4563 Epoch: 21 Global Step: 209475 Fp16 Grad Scale: 2048 Required: 47 hours Training: 2021-06-17 05:54:32,077-Speed 129.97 samples/sec Loss 0.4094 Epoch: 21 Global Step: 209480 Fp16 Grad Scale: 2048 Required: 47 hours Training: 2021-06-17 05:54:52,058-Speed 128.13 samples/sec Loss 0.3199 Epoch: 21 Global Step: 209485 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:55:12,116-Speed 127.64 samples/sec Loss 0.3479 Epoch: 21 Global Step: 209490 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:55:32,011-Speed 128.68 samples/sec Loss 0.3475 Epoch: 21 Global Step: 209495 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:55:51,688-Speed 130.10 samples/sec Loss 0.3738 Epoch: 21 Global Step: 209500 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:56:11,404-Speed 129.85 samples/sec Loss 0.4184 Epoch: 21 Global Step: 209505 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:56:31,350-Speed 128.35 samples/sec Loss 0.3998 Epoch: 21 Global Step: 209510 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:56:51,055-Speed 129.92 samples/sec Loss 0.4513 Epoch: 21 Global Step: 209515 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:57:11,250-Speed 126.77 samples/sec Loss 0.3842 Epoch: 21 Global Step: 209520 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:57:30,856-Speed 130.58 samples/sec Loss 0.3643 Epoch: 21 Global Step: 209525 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:57:50,700-Speed 129.02 samples/sec Loss 0.4094 Epoch: 21 Global Step: 209530 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:58:10,750-Speed 127.68 samples/sec Loss 0.3802 Epoch: 21 Global Step: 209535 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:58:30,676-Speed 128.49 samples/sec Loss 0.4083 Epoch: 21 Global Step: 209540 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:58:50,378-Speed 129.94 samples/sec Loss 0.3999 Epoch: 21 Global Step: 209545 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:59:10,160-Speed 129.42 samples/sec Loss 0.3759 Epoch: 21 Global Step: 209550 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:59:29,962-Speed 129.29 samples/sec Loss 0.3240 Epoch: 21 Global Step: 209555 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 05:59:49,721-Speed 129.57 samples/sec Loss 0.4041 Epoch: 21 Global Step: 209560 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:00:09,903-Speed 126.85 samples/sec Loss 0.3997 Epoch: 21 Global Step: 209565 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:00:29,499-Speed 130.65 samples/sec Loss 0.3635 Epoch: 21 Global Step: 209570 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:00:49,480-Speed 128.13 samples/sec Loss 0.3381 Epoch: 21 Global Step: 209575 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:01:09,378-Speed 128.66 samples/sec Loss 0.3901 Epoch: 21 Global Step: 209580 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:01:29,338-Speed 128.26 samples/sec Loss 0.4110 Epoch: 21 Global Step: 209585 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:01:49,096-Speed 129.58 samples/sec Loss 0.4214 Epoch: 21 Global Step: 209590 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:02:08,940-Speed 129.01 samples/sec Loss 0.3769 Epoch: 21 Global Step: 209595 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:02:28,746-Speed 129.26 samples/sec Loss 0.4196 Epoch: 21 Global Step: 209600 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:02:48,591-Speed 129.01 samples/sec Loss 0.3672 Epoch: 21 Global Step: 209605 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:03:08,827-Speed 126.51 samples/sec Loss 0.4199 Epoch: 21 Global Step: 209610 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:03:28,911-Speed 127.47 samples/sec Loss 0.3814 Epoch: 21 Global Step: 209615 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:03:48,894-Speed 128.11 samples/sec Loss 0.3891 Epoch: 21 Global Step: 209620 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:04:08,899-Speed 127.97 samples/sec Loss 0.3514 Epoch: 21 Global Step: 209625 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:04:28,739-Speed 129.04 samples/sec Loss 0.3972 Epoch: 21 Global Step: 209630 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:04:48,479-Speed 129.70 samples/sec Loss 0.3738 Epoch: 21 Global Step: 209635 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:05:08,233-Speed 129.60 samples/sec Loss 0.3325 Epoch: 21 Global Step: 209640 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:05:28,017-Speed 129.41 samples/sec Loss 0.3844 Epoch: 21 Global Step: 209645 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:05:47,939-Speed 128.50 samples/sec Loss 0.4686 Epoch: 21 Global Step: 209650 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:06:07,837-Speed 128.66 samples/sec Loss 0.3674 Epoch: 21 Global Step: 209655 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:06:27,478-Speed 130.35 samples/sec Loss 0.3628 Epoch: 21 Global Step: 209660 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:06:47,406-Speed 128.47 samples/sec Loss 0.3505 Epoch: 21 Global Step: 209665 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:07:07,488-Speed 127.49 samples/sec Loss 0.4172 Epoch: 21 Global Step: 209670 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:07:27,241-Speed 129.61 samples/sec Loss 0.4141 Epoch: 21 Global Step: 209675 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:07:46,889-Speed 130.30 samples/sec Loss 0.4508 Epoch: 21 Global Step: 209680 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:08:06,642-Speed 129.61 samples/sec Loss 0.3338 Epoch: 21 Global Step: 209685 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:08:26,424-Speed 129.41 samples/sec Loss 0.3387 Epoch: 21 Global Step: 209690 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:08:46,337-Speed 128.57 samples/sec Loss 0.3937 Epoch: 21 Global Step: 209695 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:09:06,259-Speed 128.51 samples/sec Loss 0.4088 Epoch: 21 Global Step: 209700 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:09:25,911-Speed 130.27 samples/sec Loss 0.4426 Epoch: 21 Global Step: 209705 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:09:45,850-Speed 128.40 samples/sec Loss 0.3463 Epoch: 21 Global Step: 209710 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:10:05,944-Speed 127.41 samples/sec Loss 0.3950 Epoch: 21 Global Step: 209715 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:10:25,698-Speed 129.60 samples/sec Loss 0.4479 Epoch: 21 Global Step: 209720 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:10:45,341-Speed 130.33 samples/sec Loss 0.4232 Epoch: 21 Global Step: 209725 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:11:05,139-Speed 129.32 samples/sec Loss 0.3399 Epoch: 21 Global Step: 209730 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:11:24,924-Speed 129.40 samples/sec Loss 0.4749 Epoch: 21 Global Step: 209735 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:11:44,871-Speed 128.35 samples/sec Loss 0.3677 Epoch: 21 Global Step: 209740 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:12:04,845-Speed 128.17 samples/sec Loss 0.3774 Epoch: 21 Global Step: 209745 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:12:24,726-Speed 128.78 samples/sec Loss 0.3785 Epoch: 21 Global Step: 209750 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:12:44,446-Speed 129.83 samples/sec Loss 0.3610 Epoch: 21 Global Step: 209755 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:13:04,556-Speed 127.31 samples/sec Loss 0.3184 Epoch: 21 Global Step: 209760 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:13:24,281-Speed 129.79 samples/sec Loss 0.3971 Epoch: 21 Global Step: 209765 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:13:44,147-Speed 128.87 samples/sec Loss 0.3547 Epoch: 21 Global Step: 209770 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:14:03,786-Speed 130.36 samples/sec Loss 0.3542 Epoch: 21 Global Step: 209775 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:14:23,564-Speed 129.44 samples/sec Loss 0.4151 Epoch: 21 Global Step: 209780 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:14:43,413-Speed 128.98 samples/sec Loss 0.4498 Epoch: 21 Global Step: 209785 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:15:03,325-Speed 128.57 samples/sec Loss 0.4025 Epoch: 21 Global Step: 209790 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:15:23,274-Speed 128.33 samples/sec Loss 0.3137 Epoch: 21 Global Step: 209795 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:15:43,018-Speed 129.66 samples/sec Loss 0.3581 Epoch: 21 Global Step: 209800 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:16:03,104-Speed 127.46 samples/sec Loss 0.3722 Epoch: 21 Global Step: 209805 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:16:22,870-Speed 129.52 samples/sec Loss 0.3767 Epoch: 21 Global Step: 209810 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:16:42,697-Speed 129.12 samples/sec Loss 0.4205 Epoch: 21 Global Step: 209815 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:17:02,479-Speed 129.42 samples/sec Loss 0.4175 Epoch: 21 Global Step: 209820 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:17:22,396-Speed 128.54 samples/sec Loss 0.4533 Epoch: 21 Global Step: 209825 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:17:42,063-Speed 130.18 samples/sec Loss 0.3348 Epoch: 21 Global Step: 209830 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:18:02,034-Speed 128.19 samples/sec Loss 0.4204 Epoch: 21 Global Step: 209835 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:18:21,446-Speed 131.88 samples/sec Loss 0.2975 Epoch: 21 Global Step: 209840 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:18:41,137-Speed 130.02 samples/sec Loss 0.3814 Epoch: 21 Global Step: 209845 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:19:00,950-Speed 129.21 samples/sec Loss 0.4056 Epoch: 21 Global Step: 209850 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:19:20,698-Speed 129.64 samples/sec Loss 0.4102 Epoch: 21 Global Step: 209855 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:19:40,717-Speed 127.89 samples/sec Loss 0.4164 Epoch: 21 Global Step: 209860 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:20:00,557-Speed 129.04 samples/sec Loss 0.3886 Epoch: 21 Global Step: 209865 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:20:20,435-Speed 128.79 samples/sec Loss 0.4042 Epoch: 21 Global Step: 209870 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:20:40,093-Speed 130.23 samples/sec Loss 0.3628 Epoch: 21 Global Step: 209875 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:21:00,150-Speed 127.64 samples/sec Loss 0.3873 Epoch: 21 Global Step: 209880 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:21:19,839-Speed 130.03 samples/sec Loss 0.3338 Epoch: 21 Global Step: 209885 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:21:39,964-Speed 127.21 samples/sec Loss 0.3593 Epoch: 21 Global Step: 209890 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:21:59,800-Speed 129.06 samples/sec Loss 0.3966 Epoch: 21 Global Step: 209895 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:22:19,485-Speed 130.06 samples/sec Loss 0.4748 Epoch: 21 Global Step: 209900 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:22:39,554-Speed 127.57 samples/sec Loss 0.3838 Epoch: 21 Global Step: 209905 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:22:59,517-Speed 128.24 samples/sec Loss 0.4327 Epoch: 21 Global Step: 209910 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:23:19,338-Speed 129.16 samples/sec Loss 0.3630 Epoch: 21 Global Step: 209915 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:23:38,964-Speed 130.45 samples/sec Loss 0.3210 Epoch: 21 Global Step: 209920 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:23:58,877-Speed 128.56 samples/sec Loss 0.3719 Epoch: 21 Global Step: 209925 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:24:18,634-Speed 129.58 samples/sec Loss 0.3801 Epoch: 21 Global Step: 209930 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:24:38,604-Speed 128.20 samples/sec Loss 0.3778 Epoch: 21 Global Step: 209935 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:24:58,387-Speed 129.41 samples/sec Loss 0.4221 Epoch: 21 Global Step: 209940 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:25:18,524-Speed 127.14 samples/sec Loss 0.4671 Epoch: 21 Global Step: 209945 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 06:25:38,122-Speed 130.64 samples/sec Loss 0.4114 Epoch: 21 Global Step: 209950 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:25:57,929-Speed 129.25 samples/sec Loss 0.4141 Epoch: 21 Global Step: 209955 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:26:17,701-Speed 129.48 samples/sec Loss 0.4602 Epoch: 21 Global Step: 209960 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:26:37,314-Speed 130.53 samples/sec Loss 0.4439 Epoch: 21 Global Step: 209965 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:26:57,282-Speed 128.21 samples/sec Loss 0.3666 Epoch: 21 Global Step: 209970 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:27:17,088-Speed 129.26 samples/sec Loss 0.3628 Epoch: 21 Global Step: 209975 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:27:37,014-Speed 128.48 samples/sec Loss 0.4074 Epoch: 21 Global Step: 209980 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:27:56,753-Speed 129.70 samples/sec Loss 0.3435 Epoch: 21 Global Step: 209985 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:28:16,475-Speed 129.81 samples/sec Loss 0.3361 Epoch: 21 Global Step: 209990 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:28:36,599-Speed 127.22 samples/sec Loss 0.3601 Epoch: 21 Global Step: 209995 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:28:56,491-Speed 128.70 samples/sec Loss 0.5012 Epoch: 21 Global Step: 210000 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 06:43:37,841-[lfw][210000]XNorm: 21.787501 Training: 2021-06-17 06:43:37,841-[lfw][210000]Accuracy-Flip: 0.99867+-0.00180 Training: 2021-06-17 06:43:37,841-[lfw][210000]Accuracy-Highest: 0.99867 Training: 2021-06-17 07:00:48,314-[cfp_fp][210000]XNorm: 23.439985 Training: 2021-06-17 07:00:48,314-[cfp_fp][210000]Accuracy-Flip: 0.99186+-0.00279 Training: 2021-06-17 07:00:48,314-[cfp_fp][210000]Accuracy-Highest: 0.99229 Training: 2021-06-17 07:15:35,381-[agedb_30][210000]XNorm: 22.757045 Training: 2021-06-17 07:15:35,381-[agedb_30][210000]Accuracy-Flip: 0.98450+-0.00633 Training: 2021-06-17 07:15:35,383-[agedb_30][210000]Accuracy-Highest: 0.98667 Training: 2021-06-17 07:15:55,483-Speed 0.91 samples/sec Loss 0.3861 Epoch: 21 Global Step: 210005 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:16:15,229-Speed 129.66 samples/sec Loss 0.3878 Epoch: 21 Global Step: 210010 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:16:35,405-Speed 126.89 samples/sec Loss 0.3474 Epoch: 21 Global Step: 210015 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:16:55,049-Speed 130.32 samples/sec Loss 0.3663 Epoch: 21 Global Step: 210020 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:17:15,028-Speed 128.13 samples/sec Loss 0.3677 Epoch: 21 Global Step: 210025 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:17:34,666-Speed 130.37 samples/sec Loss 0.3297 Epoch: 21 Global Step: 210030 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:17:54,416-Speed 129.63 samples/sec Loss 0.4578 Epoch: 21 Global Step: 210035 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:18:14,445-Speed 127.82 samples/sec Loss 0.3576 Epoch: 21 Global Step: 210040 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:18:34,252-Speed 129.25 samples/sec Loss 0.4669 Epoch: 21 Global Step: 210045 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:18:53,917-Speed 130.19 samples/sec Loss 0.3352 Epoch: 21 Global Step: 210050 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:19:14,079-Speed 126.98 samples/sec Loss 0.4319 Epoch: 21 Global Step: 210055 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:19:33,827-Speed 129.64 samples/sec Loss 0.3868 Epoch: 21 Global Step: 210060 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:19:53,598-Speed 129.49 samples/sec Loss 0.5175 Epoch: 21 Global Step: 210065 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:20:13,659-Speed 127.62 samples/sec Loss 0.3087 Epoch: 21 Global Step: 210070 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:20:33,353-Speed 129.99 samples/sec Loss 0.3636 Epoch: 21 Global Step: 210075 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:20:53,401-Speed 127.70 samples/sec Loss 0.3819 Epoch: 21 Global Step: 210080 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:21:13,412-Speed 127.94 samples/sec Loss 0.3548 Epoch: 21 Global Step: 210085 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:21:33,241-Speed 129.11 samples/sec Loss 0.3339 Epoch: 21 Global Step: 210090 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:21:52,937-Speed 129.98 samples/sec Loss 0.3493 Epoch: 21 Global Step: 210095 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:22:12,806-Speed 128.86 samples/sec Loss 0.4461 Epoch: 21 Global Step: 210100 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:22:32,715-Speed 128.59 samples/sec Loss 0.2960 Epoch: 21 Global Step: 210105 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:22:52,611-Speed 128.67 samples/sec Loss 0.3632 Epoch: 21 Global Step: 210110 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:23:12,541-Speed 128.46 samples/sec Loss 0.3499 Epoch: 21 Global Step: 210115 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:23:32,344-Speed 129.28 samples/sec Loss 0.3877 Epoch: 21 Global Step: 210120 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:23:52,309-Speed 128.23 samples/sec Loss 0.3961 Epoch: 21 Global Step: 210125 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:24:12,380-Speed 127.56 samples/sec Loss 0.3955 Epoch: 21 Global Step: 210130 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:24:32,130-Speed 129.62 samples/sec Loss 0.3278 Epoch: 21 Global Step: 210135 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:24:51,440-Speed 132.58 samples/sec Loss 0.4433 Epoch: 21 Global Step: 210140 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:25:11,349-Speed 128.59 samples/sec Loss 0.3352 Epoch: 21 Global Step: 210145 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:25:31,148-Speed 129.31 samples/sec Loss 0.3509 Epoch: 21 Global Step: 210150 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:25:51,055-Speed 128.61 samples/sec Loss 0.3860 Epoch: 21 Global Step: 210155 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:26:10,913-Speed 128.92 samples/sec Loss 0.4444 Epoch: 21 Global Step: 210160 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:26:30,813-Speed 128.65 samples/sec Loss 0.3369 Epoch: 21 Global Step: 210165 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:26:50,397-Speed 130.73 samples/sec Loss 0.4328 Epoch: 21 Global Step: 210170 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:27:10,297-Speed 128.65 samples/sec Loss 0.4038 Epoch: 21 Global Step: 210175 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:27:30,114-Speed 129.19 samples/sec Loss 0.3657 Epoch: 21 Global Step: 210180 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:27:49,973-Speed 128.92 samples/sec Loss 0.3936 Epoch: 21 Global Step: 210185 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:28:09,946-Speed 128.18 samples/sec Loss 0.3519 Epoch: 21 Global Step: 210190 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:28:29,977-Speed 127.81 samples/sec Loss 0.3778 Epoch: 21 Global Step: 210195 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:28:49,856-Speed 128.79 samples/sec Loss 0.3374 Epoch: 21 Global Step: 210200 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:29:09,865-Speed 127.95 samples/sec Loss 0.3444 Epoch: 21 Global Step: 210205 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:29:30,116-Speed 126.42 samples/sec Loss 0.4209 Epoch: 21 Global Step: 210210 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:29:49,785-Speed 130.16 samples/sec Loss 0.3947 Epoch: 21 Global Step: 210215 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:30:09,706-Speed 128.51 samples/sec Loss 0.3666 Epoch: 21 Global Step: 210220 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:30:29,592-Speed 128.74 samples/sec Loss 0.3460 Epoch: 21 Global Step: 210225 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:30:49,505-Speed 128.57 samples/sec Loss 0.3144 Epoch: 21 Global Step: 210230 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:31:09,532-Speed 127.84 samples/sec Loss 0.3892 Epoch: 21 Global Step: 210235 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:31:29,556-Speed 127.85 samples/sec Loss 0.4268 Epoch: 21 Global Step: 210240 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:31:49,133-Speed 130.77 samples/sec Loss 0.3819 Epoch: 21 Global Step: 210245 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:32:09,165-Speed 127.80 samples/sec Loss 0.3648 Epoch: 21 Global Step: 210250 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:32:28,617-Speed 131.61 samples/sec Loss 0.4051 Epoch: 21 Global Step: 210255 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:32:48,545-Speed 128.47 samples/sec Loss 0.3281 Epoch: 21 Global Step: 210260 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:33:08,529-Speed 128.11 samples/sec Loss 0.3126 Epoch: 21 Global Step: 210265 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:33:28,360-Speed 129.10 samples/sec Loss 0.3927 Epoch: 21 Global Step: 210270 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:33:48,527-Speed 126.94 samples/sec Loss 0.3747 Epoch: 21 Global Step: 210275 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:34:08,707-Speed 126.87 samples/sec Loss 0.3465 Epoch: 21 Global Step: 210280 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:34:28,705-Speed 128.02 samples/sec Loss 0.4326 Epoch: 21 Global Step: 210285 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:34:48,514-Speed 129.24 samples/sec Loss 0.3986 Epoch: 21 Global Step: 210290 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:35:08,401-Speed 128.73 samples/sec Loss 0.4593 Epoch: 21 Global Step: 210295 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:35:28,522-Speed 127.23 samples/sec Loss 0.3836 Epoch: 21 Global Step: 210300 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:35:48,621-Speed 127.38 samples/sec Loss 0.3864 Epoch: 21 Global Step: 210305 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:36:08,642-Speed 127.87 samples/sec Loss 0.4463 Epoch: 21 Global Step: 210310 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:36:28,494-Speed 128.97 samples/sec Loss 0.3547 Epoch: 21 Global Step: 210315 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:36:48,466-Speed 128.18 samples/sec Loss 0.3355 Epoch: 21 Global Step: 210320 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:37:08,498-Speed 127.81 samples/sec Loss 0.3577 Epoch: 21 Global Step: 210325 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:37:28,441-Speed 128.37 samples/sec Loss 0.3826 Epoch: 21 Global Step: 210330 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:37:48,179-Speed 129.70 samples/sec Loss 0.3587 Epoch: 21 Global Step: 210335 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:38:08,150-Speed 128.19 samples/sec Loss 0.4400 Epoch: 21 Global Step: 210340 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:38:28,387-Speed 126.51 samples/sec Loss 0.3670 Epoch: 21 Global Step: 210345 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:38:48,405-Speed 127.89 samples/sec Loss 0.3951 Epoch: 21 Global Step: 210350 Fp16 Grad Scale: 4096 Required: 47 hours Training: 2021-06-17 07:39:08,313-Speed 128.60 samples/sec Loss 0.4022 Epoch: 21 Global Step: 210355 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:39:28,068-Speed 129.59 samples/sec Loss 0.3816 Epoch: 21 Global Step: 210360 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:39:48,192-Speed 127.22 samples/sec Loss 0.3783 Epoch: 21 Global Step: 210365 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:40:08,299-Speed 127.32 samples/sec Loss 0.3598 Epoch: 21 Global Step: 210370 Fp16 Grad Scale: 8192 Required: 47 hours Training: 2021-06-17 07:40:27,790-Speed 131.35 samples/sec Loss 0.3585 Epoch: 21 Global Step: 210375 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:40:47,651-Speed 128.90 samples/sec Loss 0.3908 Epoch: 21 Global Step: 210380 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:41:07,633-Speed 128.13 samples/sec Loss 0.3360 Epoch: 21 Global Step: 210385 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:41:27,398-Speed 129.52 samples/sec Loss 0.4028 Epoch: 21 Global Step: 210390 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:41:47,444-Speed 127.72 samples/sec Loss 0.2991 Epoch: 21 Global Step: 210395 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:42:07,331-Speed 128.74 samples/sec Loss 0.4068 Epoch: 21 Global Step: 210400 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:42:26,961-Speed 130.42 samples/sec Loss 0.3580 Epoch: 21 Global Step: 210405 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:42:46,942-Speed 128.13 samples/sec Loss 0.3302 Epoch: 21 Global Step: 210410 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:43:06,874-Speed 128.44 samples/sec Loss 0.4152 Epoch: 21 Global Step: 210415 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:43:26,793-Speed 128.53 samples/sec Loss 0.3923 Epoch: 21 Global Step: 210420 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:43:46,546-Speed 129.60 samples/sec Loss 0.4300 Epoch: 21 Global Step: 210425 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:44:06,482-Speed 128.42 samples/sec Loss 0.3607 Epoch: 21 Global Step: 210430 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:44:26,359-Speed 128.80 samples/sec Loss 0.3799 Epoch: 21 Global Step: 210435 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:44:46,538-Speed 126.87 samples/sec Loss 0.3855 Epoch: 21 Global Step: 210440 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:45:06,420-Speed 128.77 samples/sec Loss 0.4402 Epoch: 21 Global Step: 210445 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:45:26,271-Speed 128.97 samples/sec Loss 0.4104 Epoch: 21 Global Step: 210450 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:45:46,199-Speed 128.47 samples/sec Loss 0.4226 Epoch: 21 Global Step: 210455 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:46:06,273-Speed 127.53 samples/sec Loss 0.3644 Epoch: 21 Global Step: 210460 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:46:25,999-Speed 129.79 samples/sec Loss 0.3919 Epoch: 21 Global Step: 210465 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:46:45,734-Speed 129.72 samples/sec Loss 0.3863 Epoch: 21 Global Step: 210470 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:47:05,644-Speed 128.59 samples/sec Loss 0.4219 Epoch: 21 Global Step: 210475 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:47:25,535-Speed 128.71 samples/sec Loss 0.4540 Epoch: 21 Global Step: 210480 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:47:45,727-Speed 126.78 samples/sec Loss 0.3955 Epoch: 21 Global Step: 210485 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:48:05,469-Speed 129.68 samples/sec Loss 0.3335 Epoch: 21 Global Step: 210490 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:48:25,259-Speed 129.36 samples/sec Loss 0.3349 Epoch: 21 Global Step: 210495 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:48:45,166-Speed 128.61 samples/sec Loss 0.3384 Epoch: 21 Global Step: 210500 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:49:05,258-Speed 127.43 samples/sec Loss 0.3447 Epoch: 21 Global Step: 210505 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:49:24,890-Speed 130.40 samples/sec Loss 0.3910 Epoch: 21 Global Step: 210510 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:49:44,266-Speed 132.12 samples/sec Loss 0.3959 Epoch: 21 Global Step: 210515 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:50:04,232-Speed 128.23 samples/sec Loss 0.3943 Epoch: 21 Global Step: 210520 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:50:23,724-Speed 131.34 samples/sec Loss 0.3849 Epoch: 21 Global Step: 210525 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:50:43,638-Speed 128.56 samples/sec Loss 0.3731 Epoch: 21 Global Step: 210530 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:51:03,246-Speed 130.57 samples/sec Loss 0.4491 Epoch: 21 Global Step: 210535 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:51:22,842-Speed 130.64 samples/sec Loss 0.4007 Epoch: 21 Global Step: 210540 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:51:42,375-Speed 131.07 samples/sec Loss 0.3741 Epoch: 21 Global Step: 210545 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:52:02,151-Speed 129.46 samples/sec Loss 0.4222 Epoch: 21 Global Step: 210550 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:52:22,128-Speed 128.15 samples/sec Loss 0.3494 Epoch: 21 Global Step: 210555 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:52:41,911-Speed 129.42 samples/sec Loss 0.4026 Epoch: 21 Global Step: 210560 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:53:01,807-Speed 128.67 samples/sec Loss 0.4470 Epoch: 21 Global Step: 210565 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:53:21,509-Speed 129.94 samples/sec Loss 0.4483 Epoch: 21 Global Step: 210570 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:53:41,792-Speed 126.22 samples/sec Loss 0.3574 Epoch: 21 Global Step: 210575 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:54:01,618-Speed 129.13 samples/sec Loss 0.3700 Epoch: 21 Global Step: 210580 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:54:21,472-Speed 128.94 samples/sec Loss 0.3502 Epoch: 21 Global Step: 210585 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:54:41,274-Speed 129.29 samples/sec Loss 0.3924 Epoch: 21 Global Step: 210590 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:55:01,123-Speed 128.98 samples/sec Loss 0.3725 Epoch: 21 Global Step: 210595 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:55:20,736-Speed 130.53 samples/sec Loss 0.3636 Epoch: 21 Global Step: 210600 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:55:40,512-Speed 129.46 samples/sec Loss 0.3046 Epoch: 21 Global Step: 210605 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:56:00,398-Speed 128.74 samples/sec Loss 0.3550 Epoch: 21 Global Step: 210610 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:56:20,112-Speed 129.86 samples/sec Loss 0.3521 Epoch: 21 Global Step: 210615 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:56:40,904-Speed 123.13 samples/sec Loss 0.4252 Epoch: 21 Global Step: 210620 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:57:01,511-Speed 124.23 samples/sec Loss 0.3466 Epoch: 21 Global Step: 210625 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:57:21,851-Speed 125.86 samples/sec Loss 0.3259 Epoch: 21 Global Step: 210630 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:57:41,803-Speed 128.32 samples/sec Loss 0.3714 Epoch: 21 Global Step: 210635 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:58:01,654-Speed 128.97 samples/sec Loss 0.3553 Epoch: 21 Global Step: 210640 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:58:21,338-Speed 130.06 samples/sec Loss 0.3655 Epoch: 21 Global Step: 210645 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:58:41,131-Speed 129.35 samples/sec Loss 0.4803 Epoch: 21 Global Step: 210650 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 07:59:00,541-Speed 131.90 samples/sec Loss 0.4481 Epoch: 21 Global Step: 210655 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 07:59:19,931-Speed 132.03 samples/sec Loss 0.3261 Epoch: 21 Global Step: 210660 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 07:59:39,570-Speed 130.36 samples/sec Loss 0.3721 Epoch: 21 Global Step: 210665 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 07:59:59,519-Speed 128.33 samples/sec Loss 0.3801 Epoch: 21 Global Step: 210670 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:00:19,382-Speed 128.89 samples/sec Loss 0.3804 Epoch: 21 Global Step: 210675 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:00:38,972-Speed 130.68 samples/sec Loss 0.3451 Epoch: 21 Global Step: 210680 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:00:58,782-Speed 129.24 samples/sec Loss 0.4032 Epoch: 21 Global Step: 210685 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:01:18,402-Speed 130.48 samples/sec Loss 0.4661 Epoch: 21 Global Step: 210690 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:01:38,111-Speed 129.89 samples/sec Loss 0.3178 Epoch: 21 Global Step: 210695 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:01:57,677-Speed 130.85 samples/sec Loss 0.3212 Epoch: 21 Global Step: 210700 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:02:17,382-Speed 129.92 samples/sec Loss 0.3508 Epoch: 21 Global Step: 210705 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:02:37,040-Speed 130.24 samples/sec Loss 0.3754 Epoch: 21 Global Step: 210710 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:02:56,993-Speed 128.31 samples/sec Loss 0.3678 Epoch: 21 Global Step: 210715 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:03:16,884-Speed 128.71 samples/sec Loss 0.3398 Epoch: 21 Global Step: 210720 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:03:36,510-Speed 130.45 samples/sec Loss 0.4221 Epoch: 21 Global Step: 210725 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:03:56,491-Speed 128.13 samples/sec Loss 0.4379 Epoch: 21 Global Step: 210730 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:04:16,076-Speed 130.71 samples/sec Loss 0.3497 Epoch: 21 Global Step: 210735 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:04:35,767-Speed 130.02 samples/sec Loss 0.4015 Epoch: 21 Global Step: 210740 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:04:55,377-Speed 130.55 samples/sec Loss 0.3444 Epoch: 21 Global Step: 210745 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:05:15,112-Speed 129.73 samples/sec Loss 0.3661 Epoch: 21 Global Step: 210750 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:05:34,790-Speed 130.10 samples/sec Loss 0.4100 Epoch: 21 Global Step: 210755 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:05:54,725-Speed 128.42 samples/sec Loss 0.4300 Epoch: 21 Global Step: 210760 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:06:14,601-Speed 128.80 samples/sec Loss 0.4060 Epoch: 21 Global Step: 210765 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:06:34,145-Speed 130.99 samples/sec Loss 0.4459 Epoch: 21 Global Step: 210770 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:06:54,065-Speed 128.52 samples/sec Loss 0.3623 Epoch: 21 Global Step: 210775 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:07:13,567-Speed 131.27 samples/sec Loss 0.3876 Epoch: 21 Global Step: 210780 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:07:33,048-Speed 131.42 samples/sec Loss 0.4337 Epoch: 21 Global Step: 210785 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:07:52,774-Speed 129.79 samples/sec Loss 0.4104 Epoch: 21 Global Step: 210790 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:08:12,494-Speed 129.82 samples/sec Loss 0.3793 Epoch: 21 Global Step: 210795 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:08:32,419-Speed 128.50 samples/sec Loss 0.3575 Epoch: 21 Global Step: 210800 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:08:52,081-Speed 130.20 samples/sec Loss 0.4314 Epoch: 21 Global Step: 210805 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:09:12,296-Speed 126.65 samples/sec Loss 0.3511 Epoch: 21 Global Step: 210810 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:09:32,066-Speed 129.49 samples/sec Loss 0.3694 Epoch: 21 Global Step: 210815 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:09:52,024-Speed 128.27 samples/sec Loss 0.3512 Epoch: 21 Global Step: 210820 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:10:11,860-Speed 129.07 samples/sec Loss 0.4059 Epoch: 21 Global Step: 210825 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:10:31,769-Speed 128.59 samples/sec Loss 0.4028 Epoch: 21 Global Step: 210830 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:10:51,497-Speed 129.77 samples/sec Loss 0.4114 Epoch: 21 Global Step: 210835 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:11:11,498-Speed 128.00 samples/sec Loss 0.3951 Epoch: 21 Global Step: 210840 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:11:31,237-Speed 129.70 samples/sec Loss 0.3267 Epoch: 21 Global Step: 210845 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:11:51,209-Speed 128.19 samples/sec Loss 0.2941 Epoch: 21 Global Step: 210850 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:12:11,226-Speed 127.90 samples/sec Loss 0.3568 Epoch: 21 Global Step: 210855 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:12:31,082-Speed 128.94 samples/sec Loss 0.3392 Epoch: 21 Global Step: 210860 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:12:50,949-Speed 128.86 samples/sec Loss 0.3356 Epoch: 21 Global Step: 210865 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:13:11,049-Speed 127.36 samples/sec Loss 0.4491 Epoch: 21 Global Step: 210870 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:13:30,814-Speed 129.53 samples/sec Loss 0.3999 Epoch: 21 Global Step: 210875 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:13:50,698-Speed 128.75 samples/sec Loss 0.3483 Epoch: 21 Global Step: 210880 Fp16 Grad Scale: 2048 Required: 46 hours Training: 2021-06-17 08:14:10,463-Speed 129.53 samples/sec Loss 0.3337 Epoch: 21 Global Step: 210885 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:14:30,195-Speed 129.74 samples/sec Loss 0.3169 Epoch: 21 Global Step: 210890 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:14:49,984-Speed 129.37 samples/sec Loss 0.4535 Epoch: 21 Global Step: 210895 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:15:09,978-Speed 128.04 samples/sec Loss 0.3681 Epoch: 21 Global Step: 210900 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:15:29,803-Speed 129.14 samples/sec Loss 0.3614 Epoch: 21 Global Step: 210905 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:15:49,615-Speed 129.22 samples/sec Loss 0.3246 Epoch: 21 Global Step: 210910 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:16:09,602-Speed 128.09 samples/sec Loss 0.4010 Epoch: 21 Global Step: 210915 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:16:29,469-Speed 128.86 samples/sec Loss 0.3483 Epoch: 21 Global Step: 210920 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:16:49,711-Speed 126.48 samples/sec Loss 0.3394 Epoch: 21 Global Step: 210925 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:17:09,522-Speed 129.23 samples/sec Loss 0.3928 Epoch: 21 Global Step: 210930 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:17:29,373-Speed 128.97 samples/sec Loss 0.3849 Epoch: 21 Global Step: 210935 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:17:49,192-Speed 129.17 samples/sec Loss 0.3096 Epoch: 21 Global Step: 210940 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:18:09,264-Speed 127.55 samples/sec Loss 0.3800 Epoch: 21 Global Step: 210945 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:18:29,212-Speed 128.34 samples/sec Loss 0.4080 Epoch: 21 Global Step: 210950 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:18:49,263-Speed 127.68 samples/sec Loss 0.3363 Epoch: 21 Global Step: 210955 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:19:09,300-Speed 127.77 samples/sec Loss 0.3460 Epoch: 21 Global Step: 210960 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:19:29,045-Speed 129.66 samples/sec Loss 0.4148 Epoch: 21 Global Step: 210965 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:19:48,961-Speed 128.54 samples/sec Loss 0.4046 Epoch: 21 Global Step: 210970 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:20:08,726-Speed 129.53 samples/sec Loss 0.3506 Epoch: 21 Global Step: 210975 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:20:28,520-Speed 129.34 samples/sec Loss 0.4344 Epoch: 21 Global Step: 210980 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:20:48,307-Speed 129.39 samples/sec Loss 0.4334 Epoch: 21 Global Step: 210985 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:21:08,418-Speed 127.30 samples/sec Loss 0.3090 Epoch: 21 Global Step: 210990 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:21:28,340-Speed 128.51 samples/sec Loss 0.3409 Epoch: 21 Global Step: 210995 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:21:48,082-Speed 129.68 samples/sec Loss 0.4002 Epoch: 21 Global Step: 211000 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:22:08,121-Speed 127.76 samples/sec Loss 0.3284 Epoch: 21 Global Step: 211005 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:22:27,897-Speed 129.46 samples/sec Loss 0.3551 Epoch: 21 Global Step: 211010 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:22:48,204-Speed 126.07 samples/sec Loss 0.4192 Epoch: 21 Global Step: 211015 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:23:07,922-Speed 129.84 samples/sec Loss 0.4026 Epoch: 21 Global Step: 211020 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:23:27,844-Speed 128.51 samples/sec Loss 0.3675 Epoch: 21 Global Step: 211025 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:23:47,481-Speed 130.37 samples/sec Loss 0.3644 Epoch: 21 Global Step: 211030 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:24:07,635-Speed 127.03 samples/sec Loss 0.4088 Epoch: 21 Global Step: 211035 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:24:27,526-Speed 128.71 samples/sec Loss 0.4348 Epoch: 21 Global Step: 211040 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:24:47,288-Speed 129.55 samples/sec Loss 0.3552 Epoch: 21 Global Step: 211045 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:25:07,396-Speed 127.32 samples/sec Loss 0.3358 Epoch: 21 Global Step: 211050 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:25:27,161-Speed 129.53 samples/sec Loss 0.4000 Epoch: 21 Global Step: 211055 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:25:47,159-Speed 128.02 samples/sec Loss 0.3648 Epoch: 21 Global Step: 211060 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:26:06,886-Speed 129.78 samples/sec Loss 0.3805 Epoch: 21 Global Step: 211065 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:26:26,367-Speed 131.42 samples/sec Loss 0.3655 Epoch: 21 Global Step: 211070 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:26:46,113-Speed 129.65 samples/sec Loss 0.4156 Epoch: 21 Global Step: 211075 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:27:06,317-Speed 126.72 samples/sec Loss 0.3189 Epoch: 21 Global Step: 211080 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:27:26,181-Speed 128.88 samples/sec Loss 0.4050 Epoch: 21 Global Step: 211085 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:27:45,904-Speed 129.81 samples/sec Loss 0.4144 Epoch: 21 Global Step: 211090 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:28:06,116-Speed 126.66 samples/sec Loss 0.2906 Epoch: 21 Global Step: 211095 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:28:25,962-Speed 129.00 samples/sec Loss 0.4090 Epoch: 21 Global Step: 211100 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:28:45,975-Speed 127.92 samples/sec Loss 0.3651 Epoch: 21 Global Step: 211105 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:29:05,692-Speed 129.84 samples/sec Loss 0.3590 Epoch: 21 Global Step: 211110 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:29:25,830-Speed 127.13 samples/sec Loss 0.3478 Epoch: 21 Global Step: 211115 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:29:47,674-Speed 117.20 samples/sec Loss 0.3977 Epoch: 21 Global Step: 211120 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:30:08,057-Speed 125.60 samples/sec Loss 0.4426 Epoch: 21 Global Step: 211125 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:30:27,892-Speed 129.07 samples/sec Loss 0.3918 Epoch: 21 Global Step: 211130 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:30:47,886-Speed 128.05 samples/sec Loss 0.4120 Epoch: 21 Global Step: 211135 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:31:08,089-Speed 126.72 samples/sec Loss 0.3471 Epoch: 21 Global Step: 211140 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:31:28,097-Speed 127.96 samples/sec Loss 0.3815 Epoch: 21 Global Step: 211145 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:31:48,013-Speed 128.54 samples/sec Loss 0.2955 Epoch: 21 Global Step: 211150 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:32:07,901-Speed 128.72 samples/sec Loss 0.3684 Epoch: 21 Global Step: 211155 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:32:27,687-Speed 129.40 samples/sec Loss 0.3957 Epoch: 21 Global Step: 211160 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:32:47,636-Speed 128.33 samples/sec Loss 0.3929 Epoch: 21 Global Step: 211165 Fp16 Grad Scale: 4096 Required: 46 hours Training: 2021-06-17 08:33:07,781-Speed 127.09 samples/sec Loss 0.3450 Epoch: 21 Global Step: 211170 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:33:27,397-Speed 130.52 samples/sec Loss 0.3786 Epoch: 21 Global Step: 211175 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:33:47,287-Speed 128.71 samples/sec Loss 0.3868 Epoch: 21 Global Step: 211180 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:34:07,254-Speed 128.22 samples/sec Loss 0.3590 Epoch: 21 Global Step: 211185 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:34:27,241-Speed 128.09 samples/sec Loss 0.3235 Epoch: 21 Global Step: 211190 Fp16 Grad Scale: 8192 Required: 46 hours Training: 2021-06-17 08:34:46,951-Speed 129.89 samples/sec Loss 0.3585 Epoch: 21 Global Step: 211195 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:35:06,817-Speed 128.87 samples/sec Loss 0.3684 Epoch: 21 Global Step: 211200 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:35:26,543-Speed 129.79 samples/sec Loss 0.3403 Epoch: 21 Global Step: 211205 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:35:46,858-Speed 126.02 samples/sec Loss 0.3374 Epoch: 21 Global Step: 211210 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:36:06,945-Speed 127.46 samples/sec Loss 0.3394 Epoch: 21 Global Step: 211215 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:36:26,260-Speed 132.54 samples/sec Loss 0.4455 Epoch: 21 Global Step: 211220 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:36:46,130-Speed 128.84 samples/sec Loss 0.4138 Epoch: 21 Global Step: 211225 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:37:06,405-Speed 126.27 samples/sec Loss 0.4573 Epoch: 21 Global Step: 211230 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:37:26,183-Speed 129.44 samples/sec Loss 0.3063 Epoch: 21 Global Step: 211235 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:37:45,812-Speed 130.43 samples/sec Loss 0.3920 Epoch: 21 Global Step: 211240 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:38:05,786-Speed 128.17 samples/sec Loss 0.3490 Epoch: 21 Global Step: 211245 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:38:25,450-Speed 130.20 samples/sec Loss 0.4451 Epoch: 21 Global Step: 211250 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:38:45,445-Speed 128.04 samples/sec Loss 0.3715 Epoch: 21 Global Step: 211255 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:39:05,200-Speed 129.59 samples/sec Loss 0.4419 Epoch: 21 Global Step: 211260 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:39:24,857-Speed 130.24 samples/sec Loss 0.3881 Epoch: 21 Global Step: 211265 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:39:45,004-Speed 127.07 samples/sec Loss 0.3711 Epoch: 21 Global Step: 211270 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:40:05,144-Speed 127.11 samples/sec Loss 0.3511 Epoch: 21 Global Step: 211275 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:40:24,872-Speed 129.77 samples/sec Loss 0.4270 Epoch: 21 Global Step: 211280 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:40:44,566-Speed 130.00 samples/sec Loss 0.4147 Epoch: 21 Global Step: 211285 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:41:04,450-Speed 128.76 samples/sec Loss 0.4069 Epoch: 21 Global Step: 211290 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:41:24,185-Speed 129.73 samples/sec Loss 0.3860 Epoch: 21 Global Step: 211295 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:41:44,115-Speed 128.45 samples/sec Loss 0.2681 Epoch: 21 Global Step: 211300 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:42:03,899-Speed 129.41 samples/sec Loss 0.3914 Epoch: 21 Global Step: 211305 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:42:23,609-Speed 129.89 samples/sec Loss 0.3410 Epoch: 21 Global Step: 211310 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:42:43,779-Speed 126.93 samples/sec Loss 0.3722 Epoch: 21 Global Step: 211315 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:43:03,945-Speed 126.95 samples/sec Loss 0.3397 Epoch: 21 Global Step: 211320 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:43:23,082-Speed 133.78 samples/sec Loss 0.3685 Epoch: 21 Global Step: 211325 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:43:42,965-Speed 128.76 samples/sec Loss 0.3438 Epoch: 21 Global Step: 211330 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:44:02,766-Speed 129.29 samples/sec Loss 0.4236 Epoch: 21 Global Step: 211335 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:44:22,496-Speed 129.76 samples/sec Loss 0.3948 Epoch: 21 Global Step: 211340 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:44:42,426-Speed 128.46 samples/sec Loss 0.3767 Epoch: 21 Global Step: 211345 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:45:02,193-Speed 129.51 samples/sec Loss 0.4477 Epoch: 21 Global Step: 211350 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:45:22,014-Speed 129.16 samples/sec Loss 0.3656 Epoch: 21 Global Step: 211355 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:45:41,996-Speed 128.12 samples/sec Loss 0.3440 Epoch: 21 Global Step: 211360 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:46:01,774-Speed 129.45 samples/sec Loss 0.3449 Epoch: 21 Global Step: 211365 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:46:21,686-Speed 128.57 samples/sec Loss 0.4423 Epoch: 21 Global Step: 211370 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:46:41,695-Speed 127.95 samples/sec Loss 0.4081 Epoch: 21 Global Step: 211375 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:47:01,407-Speed 129.88 samples/sec Loss 0.3770 Epoch: 21 Global Step: 211380 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:47:21,156-Speed 129.63 samples/sec Loss 0.3148 Epoch: 21 Global Step: 211385 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:47:41,087-Speed 128.45 samples/sec Loss 0.3465 Epoch: 21 Global Step: 211390 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:48:00,918-Speed 129.10 samples/sec Loss 0.3894 Epoch: 21 Global Step: 211395 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:48:20,711-Speed 129.34 samples/sec Loss 0.3627 Epoch: 21 Global Step: 211400 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:48:40,642-Speed 128.45 samples/sec Loss 0.4029 Epoch: 21 Global Step: 211405 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:49:00,363-Speed 129.81 samples/sec Loss 0.3665 Epoch: 21 Global Step: 211410 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:49:20,323-Speed 128.26 samples/sec Loss 0.3480 Epoch: 21 Global Step: 211415 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:49:40,172-Speed 128.98 samples/sec Loss 0.4206 Epoch: 21 Global Step: 211420 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:50:00,134-Speed 128.25 samples/sec Loss 0.3747 Epoch: 21 Global Step: 211425 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:50:19,855-Speed 129.82 samples/sec Loss 0.3347 Epoch: 21 Global Step: 211430 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:50:39,737-Speed 128.77 samples/sec Loss 0.3369 Epoch: 21 Global Step: 211435 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:50:59,567-Speed 129.10 samples/sec Loss 0.3498 Epoch: 21 Global Step: 211440 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:51:19,329-Speed 129.55 samples/sec Loss 0.3668 Epoch: 21 Global Step: 211445 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:51:39,376-Speed 127.71 samples/sec Loss 0.3336 Epoch: 21 Global Step: 211450 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:51:59,381-Speed 127.97 samples/sec Loss 0.3979 Epoch: 21 Global Step: 211455 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:52:19,277-Speed 128.68 samples/sec Loss 0.3634 Epoch: 21 Global Step: 211460 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:52:39,312-Speed 127.78 samples/sec Loss 0.3511 Epoch: 21 Global Step: 211465 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:52:59,017-Speed 129.92 samples/sec Loss 0.3959 Epoch: 21 Global Step: 211470 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:53:18,874-Speed 128.93 samples/sec Loss 0.3732 Epoch: 21 Global Step: 211475 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:53:38,748-Speed 128.82 samples/sec Loss 0.3445 Epoch: 21 Global Step: 211480 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:53:58,556-Speed 129.25 samples/sec Loss 0.3269 Epoch: 21 Global Step: 211485 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 08:54:17,907-Speed 132.30 samples/sec Loss 0.3622 Epoch: 21 Global Step: 211490 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:54:37,911-Speed 127.98 samples/sec Loss 0.4456 Epoch: 21 Global Step: 211495 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:54:57,657-Speed 129.65 samples/sec Loss 0.3506 Epoch: 21 Global Step: 211500 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:55:17,593-Speed 128.42 samples/sec Loss 0.3371 Epoch: 21 Global Step: 211505 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:55:37,600-Speed 127.96 samples/sec Loss 0.3774 Epoch: 21 Global Step: 211510 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:55:57,268-Speed 130.16 samples/sec Loss 0.3791 Epoch: 21 Global Step: 211515 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:56:17,093-Speed 129.14 samples/sec Loss 0.4295 Epoch: 21 Global Step: 211520 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:56:36,934-Speed 129.03 samples/sec Loss 0.4038 Epoch: 21 Global Step: 211525 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:56:56,740-Speed 129.26 samples/sec Loss 0.4412 Epoch: 21 Global Step: 211530 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:57:16,530-Speed 129.37 samples/sec Loss 0.3206 Epoch: 21 Global Step: 211535 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:57:36,498-Speed 128.21 samples/sec Loss 0.4063 Epoch: 21 Global Step: 211540 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:57:56,415-Speed 128.54 samples/sec Loss 0.3500 Epoch: 21 Global Step: 211545 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:58:16,180-Speed 129.52 samples/sec Loss 0.3504 Epoch: 21 Global Step: 211550 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:58:36,267-Speed 127.46 samples/sec Loss 0.3456 Epoch: 21 Global Step: 211555 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:58:55,983-Speed 129.85 samples/sec Loss 0.4342 Epoch: 21 Global Step: 211560 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:59:15,900-Speed 128.54 samples/sec Loss 0.3036 Epoch: 21 Global Step: 211565 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:59:35,598-Speed 129.97 samples/sec Loss 0.3967 Epoch: 21 Global Step: 211570 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 08:59:55,455-Speed 128.93 samples/sec Loss 0.3987 Epoch: 21 Global Step: 211575 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:00:15,290-Speed 129.07 samples/sec Loss 0.3937 Epoch: 21 Global Step: 211580 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:00:35,302-Speed 127.93 samples/sec Loss 0.3789 Epoch: 21 Global Step: 211585 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:00:55,226-Speed 128.49 samples/sec Loss 0.3866 Epoch: 21 Global Step: 211590 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:01:15,015-Speed 129.37 samples/sec Loss 0.4071 Epoch: 21 Global Step: 211595 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:01:35,249-Speed 126.53 samples/sec Loss 0.4285 Epoch: 21 Global Step: 211600 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:01:54,920-Speed 130.15 samples/sec Loss 0.3193 Epoch: 21 Global Step: 211605 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:02:14,825-Speed 128.62 samples/sec Loss 0.3481 Epoch: 21 Global Step: 211610 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:02:34,832-Speed 127.96 samples/sec Loss 0.4947 Epoch: 21 Global Step: 211615 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:02:54,731-Speed 128.66 samples/sec Loss 0.3463 Epoch: 21 Global Step: 211620 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:03:14,538-Speed 129.25 samples/sec Loss 0.3410 Epoch: 21 Global Step: 211625 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:03:34,528-Speed 128.07 samples/sec Loss 0.4789 Epoch: 21 Global Step: 211630 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:03:54,420-Speed 128.70 samples/sec Loss 0.3690 Epoch: 21 Global Step: 211635 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:04:14,154-Speed 129.73 samples/sec Loss 0.3434 Epoch: 21 Global Step: 211640 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:04:34,228-Speed 127.54 samples/sec Loss 0.2973 Epoch: 21 Global Step: 211645 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:04:53,890-Speed 130.21 samples/sec Loss 0.4011 Epoch: 21 Global Step: 211650 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:05:13,779-Speed 128.72 samples/sec Loss 0.3016 Epoch: 21 Global Step: 211655 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:05:33,473-Speed 130.00 samples/sec Loss 0.3934 Epoch: 21 Global Step: 211660 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:05:53,380-Speed 128.60 samples/sec Loss 0.3576 Epoch: 21 Global Step: 211665 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:06:13,084-Speed 129.93 samples/sec Loss 0.3988 Epoch: 21 Global Step: 211670 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:06:33,128-Speed 127.72 samples/sec Loss 0.3776 Epoch: 21 Global Step: 211675 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:06:53,054-Speed 128.48 samples/sec Loss 0.4085 Epoch: 21 Global Step: 211680 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:07:12,766-Speed 129.88 samples/sec Loss 0.3234 Epoch: 21 Global Step: 211685 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:07:32,838-Speed 127.55 samples/sec Loss 0.3859 Epoch: 21 Global Step: 211690 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:07:52,733-Speed 128.68 samples/sec Loss 0.3096 Epoch: 21 Global Step: 211695 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:08:12,462-Speed 129.77 samples/sec Loss 0.3382 Epoch: 21 Global Step: 211700 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:08:32,208-Speed 129.65 samples/sec Loss 0.3949 Epoch: 21 Global Step: 211705 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:08:52,145-Speed 128.41 samples/sec Loss 0.3141 Epoch: 21 Global Step: 211710 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:09:11,876-Speed 129.75 samples/sec Loss 0.5221 Epoch: 21 Global Step: 211715 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:09:32,068-Speed 126.79 samples/sec Loss 0.3848 Epoch: 21 Global Step: 211720 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:09:51,752-Speed 130.06 samples/sec Loss 0.3423 Epoch: 21 Global Step: 211725 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:10:11,499-Speed 129.65 samples/sec Loss 0.3133 Epoch: 21 Global Step: 211730 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:10:31,534-Speed 127.78 samples/sec Loss 0.3800 Epoch: 21 Global Step: 211735 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:10:51,420-Speed 128.74 samples/sec Loss 0.2954 Epoch: 21 Global Step: 211740 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:11:11,140-Speed 129.82 samples/sec Loss 0.4175 Epoch: 21 Global Step: 211745 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:11:31,047-Speed 128.60 samples/sec Loss 0.2888 Epoch: 21 Global Step: 211750 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:11:50,409-Speed 132.22 samples/sec Loss 0.4105 Epoch: 21 Global Step: 211755 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:12:10,179-Speed 129.50 samples/sec Loss 0.3512 Epoch: 21 Global Step: 211760 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:12:30,087-Speed 128.60 samples/sec Loss 0.3398 Epoch: 21 Global Step: 211765 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:12:49,827-Speed 129.69 samples/sec Loss 0.4176 Epoch: 21 Global Step: 211770 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:13:09,653-Speed 129.13 samples/sec Loss 0.3094 Epoch: 21 Global Step: 211775 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:13:29,912-Speed 126.37 samples/sec Loss 0.4004 Epoch: 21 Global Step: 211780 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:13:49,727-Speed 129.20 samples/sec Loss 0.3221 Epoch: 21 Global Step: 211785 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:14:09,269-Speed 131.01 samples/sec Loss 0.3773 Epoch: 21 Global Step: 211790 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:14:28,866-Speed 130.64 samples/sec Loss 0.3951 Epoch: 21 Global Step: 211795 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:14:48,413-Speed 130.97 samples/sec Loss 0.3342 Epoch: 21 Global Step: 211800 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:15:08,036-Speed 130.47 samples/sec Loss 0.4035 Epoch: 21 Global Step: 211805 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:15:27,224-Speed 133.43 samples/sec Loss 0.3259 Epoch: 21 Global Step: 211810 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:15:46,781-Speed 130.90 samples/sec Loss 0.3567 Epoch: 21 Global Step: 211815 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:16:06,716-Speed 128.42 samples/sec Loss 0.2940 Epoch: 21 Global Step: 211820 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:16:26,448-Speed 129.75 samples/sec Loss 0.3869 Epoch: 21 Global Step: 211825 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:16:46,151-Speed 129.94 samples/sec Loss 0.3938 Epoch: 21 Global Step: 211830 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:17:06,410-Speed 126.37 samples/sec Loss 0.4092 Epoch: 21 Global Step: 211835 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:17:26,351-Speed 128.39 samples/sec Loss 0.4034 Epoch: 21 Global Step: 211840 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:17:46,068-Speed 129.84 samples/sec Loss 0.3698 Epoch: 21 Global Step: 211845 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:18:05,744-Speed 130.11 samples/sec Loss 0.3477 Epoch: 21 Global Step: 211850 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:18:25,512-Speed 129.51 samples/sec Loss 0.3268 Epoch: 21 Global Step: 211855 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:18:45,293-Speed 129.42 samples/sec Loss 0.3898 Epoch: 21 Global Step: 211860 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:19:05,120-Speed 129.12 samples/sec Loss 0.3952 Epoch: 21 Global Step: 211865 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:19:24,857-Speed 129.72 samples/sec Loss 0.3693 Epoch: 21 Global Step: 211870 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:19:44,519-Speed 130.21 samples/sec Loss 0.3471 Epoch: 21 Global Step: 211875 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:20:04,767-Speed 126.44 samples/sec Loss 0.5165 Epoch: 21 Global Step: 211880 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:20:24,781-Speed 127.91 samples/sec Loss 0.3286 Epoch: 21 Global Step: 211885 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:20:44,497-Speed 129.85 samples/sec Loss 0.3510 Epoch: 21 Global Step: 211890 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:21:04,157-Speed 130.22 samples/sec Loss 0.3809 Epoch: 21 Global Step: 211895 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:21:23,902-Speed 129.66 samples/sec Loss 0.3820 Epoch: 21 Global Step: 211900 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:21:43,642-Speed 129.69 samples/sec Loss 0.3683 Epoch: 21 Global Step: 211905 Fp16 Grad Scale: 2048 Required: 45 hours Training: 2021-06-17 09:22:03,599-Speed 128.29 samples/sec Loss 0.3512 Epoch: 21 Global Step: 211910 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:22:23,313-Speed 129.86 samples/sec Loss 0.3828 Epoch: 21 Global Step: 211915 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:22:42,975-Speed 130.21 samples/sec Loss 0.4472 Epoch: 21 Global Step: 211920 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:23:03,211-Speed 126.51 samples/sec Loss 0.3998 Epoch: 21 Global Step: 211925 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:23:23,180-Speed 128.20 samples/sec Loss 0.3787 Epoch: 21 Global Step: 211930 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:23:42,829-Speed 130.29 samples/sec Loss 0.3800 Epoch: 21 Global Step: 211935 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:24:03,087-Speed 126.38 samples/sec Loss 0.3527 Epoch: 21 Global Step: 211940 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:24:22,691-Speed 130.60 samples/sec Loss 0.2784 Epoch: 21 Global Step: 211945 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:24:42,400-Speed 129.89 samples/sec Loss 0.3052 Epoch: 21 Global Step: 211950 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:25:02,361-Speed 128.26 samples/sec Loss 0.3810 Epoch: 21 Global Step: 211955 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:25:22,097-Speed 129.71 samples/sec Loss 0.3716 Epoch: 21 Global Step: 211960 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:25:41,894-Speed 129.32 samples/sec Loss 0.2910 Epoch: 21 Global Step: 211965 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:26:02,520-Speed 124.12 samples/sec Loss 0.3875 Epoch: 21 Global Step: 211970 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:26:22,499-Speed 128.14 samples/sec Loss 0.3642 Epoch: 21 Global Step: 211975 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:26:42,221-Speed 129.81 samples/sec Loss 0.3734 Epoch: 21 Global Step: 211980 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:27:02,040-Speed 129.17 samples/sec Loss 0.4017 Epoch: 21 Global Step: 211985 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:27:21,651-Speed 130.54 samples/sec Loss 0.3986 Epoch: 21 Global Step: 211990 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:27:41,539-Speed 128.73 samples/sec Loss 0.3085 Epoch: 21 Global Step: 211995 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:28:01,300-Speed 129.56 samples/sec Loss 0.4168 Epoch: 21 Global Step: 212000 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:28:21,090-Speed 129.36 samples/sec Loss 0.3632 Epoch: 21 Global Step: 212005 Fp16 Grad Scale: 4096 Required: 45 hours Training: 2021-06-17 09:28:40,899-Speed 129.24 samples/sec Loss 0.3605 Epoch: 21 Global Step: 212010 Fp16 Grad Scale: 8192 Required: 45 hours Training: 2021-06-17 09:29:01,170-Speed 126.30 samples/sec Loss 0.3872 Epoch: 21 Global Step: 212015 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:29:20,527-Speed 132.25 samples/sec Loss 0.3867 Epoch: 21 Global Step: 212020 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:29:40,290-Speed 129.55 samples/sec Loss 0.3548 Epoch: 21 Global Step: 212025 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:30:00,065-Speed 129.46 samples/sec Loss 0.4582 Epoch: 21 Global Step: 212030 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:30:19,685-Speed 130.48 samples/sec Loss 0.4241 Epoch: 21 Global Step: 212035 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:30:39,598-Speed 128.57 samples/sec Loss 0.3905 Epoch: 21 Global Step: 212040 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:30:59,381-Speed 129.41 samples/sec Loss 0.3402 Epoch: 21 Global Step: 212045 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:31:19,135-Speed 129.60 samples/sec Loss 0.3308 Epoch: 21 Global Step: 212050 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:31:38,902-Speed 129.51 samples/sec Loss 0.3882 Epoch: 21 Global Step: 212055 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:31:59,196-Speed 126.15 samples/sec Loss 0.4201 Epoch: 21 Global Step: 212060 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:32:19,135-Speed 128.40 samples/sec Loss 0.3690 Epoch: 21 Global Step: 212065 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:32:38,913-Speed 129.44 samples/sec Loss 0.4515 Epoch: 21 Global Step: 212070 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:32:58,668-Speed 129.59 samples/sec Loss 0.3741 Epoch: 21 Global Step: 212075 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:33:18,280-Speed 130.54 samples/sec Loss 0.4332 Epoch: 21 Global Step: 212080 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:33:38,245-Speed 128.23 samples/sec Loss 0.4130 Epoch: 21 Global Step: 212085 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:33:57,983-Speed 129.71 samples/sec Loss 0.4004 Epoch: 21 Global Step: 212090 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:34:17,788-Speed 129.26 samples/sec Loss 0.4043 Epoch: 21 Global Step: 212095 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:34:37,465-Speed 130.11 samples/sec Loss 0.3629 Epoch: 21 Global Step: 212100 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:34:57,857-Speed 125.54 samples/sec Loss 0.3863 Epoch: 21 Global Step: 212105 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:35:17,643-Speed 129.39 samples/sec Loss 0.3795 Epoch: 21 Global Step: 212110 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:35:37,337-Speed 129.99 samples/sec Loss 0.4094 Epoch: 21 Global Step: 212115 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:35:57,051-Speed 129.87 samples/sec Loss 0.3697 Epoch: 21 Global Step: 212120 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:36:16,751-Speed 129.95 samples/sec Loss 0.3941 Epoch: 21 Global Step: 212125 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:36:36,771-Speed 127.88 samples/sec Loss 0.3715 Epoch: 21 Global Step: 212130 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:36:56,533-Speed 129.55 samples/sec Loss 0.4318 Epoch: 21 Global Step: 212135 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:37:16,257-Speed 129.80 samples/sec Loss 0.3156 Epoch: 21 Global Step: 212140 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:37:35,942-Speed 130.05 samples/sec Loss 0.3950 Epoch: 21 Global Step: 212145 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:37:56,297-Speed 125.77 samples/sec Loss 0.4065 Epoch: 21 Global Step: 212150 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:38:16,039-Speed 129.68 samples/sec Loss 0.3119 Epoch: 21 Global Step: 212155 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:38:35,408-Speed 132.18 samples/sec Loss 0.4323 Epoch: 21 Global Step: 212160 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:38:54,935-Speed 131.10 samples/sec Loss 0.4291 Epoch: 21 Global Step: 212165 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:39:14,374-Speed 131.70 samples/sec Loss 0.3356 Epoch: 21 Global Step: 212170 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:39:33,984-Speed 130.56 samples/sec Loss 0.3796 Epoch: 21 Global Step: 212175 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:39:53,693-Speed 129.89 samples/sec Loss 0.3638 Epoch: 21 Global Step: 212180 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:40:13,297-Speed 130.60 samples/sec Loss 0.3602 Epoch: 21 Global Step: 212185 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:40:32,992-Speed 129.99 samples/sec Loss 0.3778 Epoch: 21 Global Step: 212190 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:40:52,970-Speed 128.15 samples/sec Loss 0.3523 Epoch: 21 Global Step: 212195 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:41:12,760-Speed 129.36 samples/sec Loss 0.4372 Epoch: 21 Global Step: 212200 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:41:32,626-Speed 128.87 samples/sec Loss 0.3784 Epoch: 21 Global Step: 212205 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:41:52,243-Speed 130.51 samples/sec Loss 0.3305 Epoch: 21 Global Step: 212210 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:42:11,957-Speed 129.86 samples/sec Loss 0.3716 Epoch: 21 Global Step: 212215 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:42:32,177-Speed 126.62 samples/sec Loss 0.3314 Epoch: 21 Global Step: 212220 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:42:51,851-Speed 130.13 samples/sec Loss 0.3387 Epoch: 21 Global Step: 212225 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:43:11,719-Speed 128.86 samples/sec Loss 0.3766 Epoch: 21 Global Step: 212230 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:43:31,351-Speed 130.40 samples/sec Loss 0.3649 Epoch: 21 Global Step: 212235 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:43:51,510-Speed 127.00 samples/sec Loss 0.3929 Epoch: 21 Global Step: 212240 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:44:11,323-Speed 129.21 samples/sec Loss 0.3537 Epoch: 21 Global Step: 212245 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:44:31,136-Speed 129.21 samples/sec Loss 0.3477 Epoch: 21 Global Step: 212250 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:44:50,829-Speed 130.00 samples/sec Loss 0.4036 Epoch: 21 Global Step: 212255 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:45:10,492-Speed 130.20 samples/sec Loss 0.3638 Epoch: 21 Global Step: 212260 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:45:30,734-Speed 126.48 samples/sec Loss 0.3186 Epoch: 21 Global Step: 212265 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:45:50,417-Speed 130.07 samples/sec Loss 0.3281 Epoch: 21 Global Step: 212270 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:46:10,291-Speed 128.82 samples/sec Loss 0.4330 Epoch: 21 Global Step: 212275 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:46:29,925-Speed 130.39 samples/sec Loss 0.3909 Epoch: 21 Global Step: 212280 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:46:50,050-Speed 127.21 samples/sec Loss 0.3880 Epoch: 21 Global Step: 212285 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:47:09,815-Speed 129.53 samples/sec Loss 0.3254 Epoch: 21 Global Step: 212290 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:47:29,742-Speed 128.47 samples/sec Loss 0.3825 Epoch: 21 Global Step: 212295 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:47:49,453-Speed 129.88 samples/sec Loss 0.3330 Epoch: 21 Global Step: 212300 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:48:09,236-Speed 129.41 samples/sec Loss 0.3945 Epoch: 21 Global Step: 212305 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:48:29,300-Speed 127.60 samples/sec Loss 0.3307 Epoch: 21 Global Step: 212310 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:48:49,028-Speed 129.77 samples/sec Loss 0.3577 Epoch: 21 Global Step: 212315 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:49:08,871-Speed 129.02 samples/sec Loss 0.3504 Epoch: 21 Global Step: 212320 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:49:28,497-Speed 130.44 samples/sec Loss 0.3400 Epoch: 21 Global Step: 212325 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:49:47,994-Speed 131.31 samples/sec Loss 0.4002 Epoch: 21 Global Step: 212330 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:50:07,724-Speed 129.76 samples/sec Loss 0.3680 Epoch: 21 Global Step: 212335 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:50:27,544-Speed 129.17 samples/sec Loss 0.3866 Epoch: 21 Global Step: 212340 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:50:47,410-Speed 128.87 samples/sec Loss 0.3407 Epoch: 21 Global Step: 212345 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:51:06,985-Speed 130.79 samples/sec Loss 0.3922 Epoch: 21 Global Step: 212350 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:51:26,880-Speed 128.68 samples/sec Loss 0.3602 Epoch: 21 Global Step: 212355 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:51:46,705-Speed 129.14 samples/sec Loss 0.3793 Epoch: 21 Global Step: 212360 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:52:06,158-Speed 131.61 samples/sec Loss 0.4064 Epoch: 21 Global Step: 212365 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:52:25,786-Speed 130.43 samples/sec Loss 0.4109 Epoch: 21 Global Step: 212370 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:52:45,474-Speed 130.04 samples/sec Loss 0.3528 Epoch: 21 Global Step: 212375 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:53:05,182-Speed 129.90 samples/sec Loss 0.3364 Epoch: 21 Global Step: 212380 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:53:25,100-Speed 128.54 samples/sec Loss 0.4896 Epoch: 21 Global Step: 212385 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:53:44,948-Speed 128.98 samples/sec Loss 0.3710 Epoch: 21 Global Step: 212390 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:54:04,551-Speed 130.60 samples/sec Loss 0.3393 Epoch: 21 Global Step: 212395 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:54:24,348-Speed 129.32 samples/sec Loss 0.4020 Epoch: 21 Global Step: 212400 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:54:44,262-Speed 128.56 samples/sec Loss 0.3316 Epoch: 21 Global Step: 212405 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:55:03,677-Speed 131.86 samples/sec Loss 0.3238 Epoch: 21 Global Step: 212410 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:55:23,319-Speed 130.34 samples/sec Loss 0.3516 Epoch: 21 Global Step: 212415 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:55:42,964-Speed 130.32 samples/sec Loss 0.3521 Epoch: 21 Global Step: 212420 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:56:02,770-Speed 129.26 samples/sec Loss 0.3865 Epoch: 21 Global Step: 212425 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:56:22,568-Speed 129.31 samples/sec Loss 0.4121 Epoch: 21 Global Step: 212430 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:56:42,352-Speed 129.41 samples/sec Loss 0.3968 Epoch: 21 Global Step: 212435 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:57:02,407-Speed 127.65 samples/sec Loss 0.3116 Epoch: 21 Global Step: 212440 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:57:22,306-Speed 128.65 samples/sec Loss 0.3746 Epoch: 21 Global Step: 212445 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:57:42,219-Speed 128.57 samples/sec Loss 0.4025 Epoch: 21 Global Step: 212450 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 09:58:01,200-Speed 134.88 samples/sec Loss 0.4189 Epoch: 21 Global Step: 212455 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:58:20,850-Speed 130.28 samples/sec Loss 0.4031 Epoch: 21 Global Step: 212460 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:58:40,429-Speed 130.76 samples/sec Loss 0.3455 Epoch: 21 Global Step: 212465 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:59:00,160-Speed 129.75 samples/sec Loss 0.3533 Epoch: 21 Global Step: 212470 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:59:19,975-Speed 129.20 samples/sec Loss 0.3233 Epoch: 21 Global Step: 212475 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:59:39,724-Speed 129.63 samples/sec Loss 0.3358 Epoch: 21 Global Step: 212480 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 09:59:59,284-Speed 130.89 samples/sec Loss 0.3500 Epoch: 21 Global Step: 212485 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:00:19,143-Speed 128.92 samples/sec Loss 0.3650 Epoch: 21 Global Step: 212490 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:00:39,107-Speed 128.23 samples/sec Loss 0.3431 Epoch: 21 Global Step: 212495 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:00:58,654-Speed 130.98 samples/sec Loss 0.3506 Epoch: 21 Global Step: 212500 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:01:18,329-Speed 130.12 samples/sec Loss 0.3911 Epoch: 21 Global Step: 212505 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:01:37,890-Speed 130.88 samples/sec Loss 0.3972 Epoch: 21 Global Step: 212510 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:01:57,589-Speed 129.97 samples/sec Loss 0.3094 Epoch: 21 Global Step: 212515 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:02:17,393-Speed 129.27 samples/sec Loss 0.4715 Epoch: 21 Global Step: 212520 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:02:37,180-Speed 129.38 samples/sec Loss 0.2948 Epoch: 21 Global Step: 212525 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:02:56,759-Speed 130.76 samples/sec Loss 0.3064 Epoch: 21 Global Step: 212530 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:03:16,539-Speed 129.43 samples/sec Loss 0.3840 Epoch: 21 Global Step: 212535 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:03:36,507-Speed 128.21 samples/sec Loss 0.3799 Epoch: 21 Global Step: 212540 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:03:56,082-Speed 130.79 samples/sec Loss 0.2843 Epoch: 21 Global Step: 212545 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:04:15,748-Speed 130.18 samples/sec Loss 0.3789 Epoch: 21 Global Step: 212550 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:04:35,366-Speed 130.50 samples/sec Loss 0.4288 Epoch: 21 Global Step: 212555 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:04:55,073-Speed 129.91 samples/sec Loss 0.3979 Epoch: 21 Global Step: 212560 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:05:14,849-Speed 129.46 samples/sec Loss 0.3832 Epoch: 21 Global Step: 212565 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:05:34,590-Speed 129.69 samples/sec Loss 0.3347 Epoch: 21 Global Step: 212570 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:05:54,163-Speed 130.80 samples/sec Loss 0.3324 Epoch: 21 Global Step: 212575 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:06:14,021-Speed 128.92 samples/sec Loss 0.3647 Epoch: 21 Global Step: 212580 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:06:33,997-Speed 128.16 samples/sec Loss 0.3593 Epoch: 21 Global Step: 212585 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:06:53,485-Speed 131.37 samples/sec Loss 0.3485 Epoch: 21 Global Step: 212590 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:07:13,145-Speed 130.22 samples/sec Loss 0.4375 Epoch: 21 Global Step: 212595 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:07:32,745-Speed 130.63 samples/sec Loss 0.3701 Epoch: 21 Global Step: 212600 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:07:52,470-Speed 129.79 samples/sec Loss 0.3264 Epoch: 21 Global Step: 212605 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:08:12,277-Speed 129.26 samples/sec Loss 0.3582 Epoch: 21 Global Step: 212610 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:08:32,044-Speed 129.51 samples/sec Loss 0.3653 Epoch: 21 Global Step: 212615 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:08:51,576-Speed 131.07 samples/sec Loss 0.3972 Epoch: 21 Global Step: 212620 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:09:11,365-Speed 129.37 samples/sec Loss 0.4139 Epoch: 21 Global Step: 212625 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:09:31,374-Speed 127.95 samples/sec Loss 0.3710 Epoch: 21 Global Step: 212630 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:09:50,958-Speed 130.73 samples/sec Loss 0.3950 Epoch: 21 Global Step: 212635 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:10:10,839-Speed 128.77 samples/sec Loss 0.4338 Epoch: 21 Global Step: 212640 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:10:30,277-Speed 131.71 samples/sec Loss 0.3992 Epoch: 21 Global Step: 212645 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:10:50,031-Speed 129.60 samples/sec Loss 0.3786 Epoch: 21 Global Step: 212650 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:11:09,792-Speed 129.56 samples/sec Loss 0.2989 Epoch: 21 Global Step: 212655 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:11:29,712-Speed 128.52 samples/sec Loss 0.3240 Epoch: 21 Global Step: 212660 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:11:49,126-Speed 131.87 samples/sec Loss 0.3510 Epoch: 21 Global Step: 212665 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:12:08,518-Speed 132.02 samples/sec Loss 0.3824 Epoch: 21 Global Step: 212670 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:12:28,085-Speed 130.85 samples/sec Loss 0.4010 Epoch: 21 Global Step: 212675 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:12:48,193-Speed 127.31 samples/sec Loss 0.4366 Epoch: 21 Global Step: 212680 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:13:08,101-Speed 128.60 samples/sec Loss 0.3283 Epoch: 21 Global Step: 212685 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:13:27,940-Speed 129.05 samples/sec Loss 0.3600 Epoch: 21 Global Step: 212690 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:13:47,605-Speed 130.19 samples/sec Loss 0.3330 Epoch: 21 Global Step: 212695 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:14:07,682-Speed 127.51 samples/sec Loss 0.4030 Epoch: 21 Global Step: 212700 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:14:27,552-Speed 128.84 samples/sec Loss 0.4268 Epoch: 21 Global Step: 212705 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:14:47,255-Speed 129.94 samples/sec Loss 0.4082 Epoch: 21 Global Step: 212710 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:15:07,053-Speed 129.31 samples/sec Loss 0.3053 Epoch: 21 Global Step: 212715 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:15:26,867-Speed 129.21 samples/sec Loss 0.4245 Epoch: 21 Global Step: 212720 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:15:46,749-Speed 128.77 samples/sec Loss 0.3335 Epoch: 21 Global Step: 212725 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:16:06,647-Speed 128.66 samples/sec Loss 0.3518 Epoch: 21 Global Step: 212730 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:16:26,346-Speed 129.96 samples/sec Loss 0.3774 Epoch: 21 Global Step: 212735 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:16:46,475-Speed 127.18 samples/sec Loss 0.3993 Epoch: 21 Global Step: 212740 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:17:06,438-Speed 128.25 samples/sec Loss 0.3960 Epoch: 21 Global Step: 212745 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:17:26,404-Speed 128.22 samples/sec Loss 0.3566 Epoch: 21 Global Step: 212750 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:17:46,115-Speed 129.89 samples/sec Loss 0.3461 Epoch: 21 Global Step: 212755 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:18:05,953-Speed 129.05 samples/sec Loss 0.3488 Epoch: 21 Global Step: 212760 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:18:25,810-Speed 128.93 samples/sec Loss 0.3596 Epoch: 21 Global Step: 212765 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:18:45,520-Speed 129.89 samples/sec Loss 0.3360 Epoch: 21 Global Step: 212770 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:19:05,405-Speed 128.75 samples/sec Loss 0.3843 Epoch: 21 Global Step: 212775 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:19:25,191-Speed 129.40 samples/sec Loss 0.3886 Epoch: 21 Global Step: 212780 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:19:45,345-Speed 127.02 samples/sec Loss 0.3304 Epoch: 21 Global Step: 212785 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:20:05,253-Speed 128.60 samples/sec Loss 0.4625 Epoch: 21 Global Step: 212790 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:20:25,211-Speed 128.28 samples/sec Loss 0.3814 Epoch: 21 Global Step: 212795 Fp16 Grad Scale: 8192 Required: 44 hours Training: 2021-06-17 10:20:44,496-Speed 132.75 samples/sec Loss 0.3456 Epoch: 21 Global Step: 212800 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:21:04,406-Speed 128.59 samples/sec Loss 0.3611 Epoch: 21 Global Step: 212805 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:21:24,187-Speed 129.42 samples/sec Loss 0.3974 Epoch: 21 Global Step: 212810 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:21:44,009-Speed 129.16 samples/sec Loss 0.3166 Epoch: 21 Global Step: 212815 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:22:03,850-Speed 129.03 samples/sec Loss 0.3806 Epoch: 21 Global Step: 212820 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:22:23,584-Speed 129.74 samples/sec Loss 0.3704 Epoch: 21 Global Step: 212825 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:22:43,800-Speed 126.63 samples/sec Loss 0.3485 Epoch: 21 Global Step: 212830 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:23:03,420-Speed 130.49 samples/sec Loss 0.2791 Epoch: 21 Global Step: 212835 Fp16 Grad Scale: 4096 Required: 44 hours Training: 2021-06-17 10:23:23,035-Speed 130.52 samples/sec Loss 0.3745 Epoch: 21 Global Step: 212840 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:23:42,748-Speed 129.87 samples/sec Loss 0.3869 Epoch: 21 Global Step: 212845 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:24:02,387-Speed 130.36 samples/sec Loss 0.3493 Epoch: 21 Global Step: 212850 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:24:22,169-Speed 129.42 samples/sec Loss 0.3768 Epoch: 21 Global Step: 212855 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:24:41,967-Speed 129.31 samples/sec Loss 0.4197 Epoch: 21 Global Step: 212860 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:25:01,719-Speed 129.62 samples/sec Loss 0.2997 Epoch: 21 Global Step: 212865 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:25:21,332-Speed 130.53 samples/sec Loss 0.3463 Epoch: 21 Global Step: 212870 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:25:42,010-Speed 123.81 samples/sec Loss 0.3499 Epoch: 21 Global Step: 212875 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:26:01,932-Speed 128.51 samples/sec Loss 0.3565 Epoch: 21 Global Step: 212880 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:26:21,729-Speed 129.32 samples/sec Loss 0.3803 Epoch: 21 Global Step: 212885 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:26:41,524-Speed 129.33 samples/sec Loss 0.3699 Epoch: 21 Global Step: 212890 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:27:01,155-Speed 130.41 samples/sec Loss 0.3338 Epoch: 21 Global Step: 212895 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:27:20,903-Speed 129.64 samples/sec Loss 0.3622 Epoch: 21 Global Step: 212900 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:27:40,287-Speed 132.08 samples/sec Loss 0.3663 Epoch: 21 Global Step: 212905 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:27:59,948-Speed 130.22 samples/sec Loss 0.3777 Epoch: 21 Global Step: 212910 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:28:19,657-Speed 129.89 samples/sec Loss 0.3182 Epoch: 21 Global Step: 212915 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:28:39,470-Speed 129.21 samples/sec Loss 0.3234 Epoch: 21 Global Step: 212920 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:28:59,113-Speed 130.34 samples/sec Loss 0.3497 Epoch: 21 Global Step: 212925 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:29:18,717-Speed 130.60 samples/sec Loss 0.3658 Epoch: 21 Global Step: 212930 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:29:38,444-Speed 129.78 samples/sec Loss 0.3457 Epoch: 21 Global Step: 212935 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:29:58,065-Speed 130.48 samples/sec Loss 0.3802 Epoch: 21 Global Step: 212940 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:30:17,863-Speed 129.31 samples/sec Loss 0.3677 Epoch: 21 Global Step: 212945 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:30:37,647-Speed 129.41 samples/sec Loss 0.3777 Epoch: 21 Global Step: 212950 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:30:57,467-Speed 129.16 samples/sec Loss 0.4428 Epoch: 21 Global Step: 212955 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:31:17,266-Speed 129.31 samples/sec Loss 0.4499 Epoch: 21 Global Step: 212960 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:31:37,120-Speed 128.95 samples/sec Loss 0.3687 Epoch: 21 Global Step: 212965 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:31:56,901-Speed 129.42 samples/sec Loss 0.3751 Epoch: 21 Global Step: 212970 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:32:16,478-Speed 130.77 samples/sec Loss 0.4368 Epoch: 21 Global Step: 212975 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:32:36,200-Speed 129.81 samples/sec Loss 0.3992 Epoch: 21 Global Step: 212980 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:32:55,860-Speed 130.22 samples/sec Loss 0.3704 Epoch: 21 Global Step: 212985 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:33:15,665-Speed 129.27 samples/sec Loss 0.4107 Epoch: 21 Global Step: 212990 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:33:35,521-Speed 128.94 samples/sec Loss 0.4048 Epoch: 21 Global Step: 212995 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:33:55,336-Speed 129.20 samples/sec Loss 0.3498 Epoch: 21 Global Step: 213000 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:34:14,729-Speed 132.01 samples/sec Loss 0.3256 Epoch: 21 Global Step: 213005 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:34:34,535-Speed 129.26 samples/sec Loss 0.3485 Epoch: 21 Global Step: 213010 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:34:54,495-Speed 128.26 samples/sec Loss 0.3925 Epoch: 21 Global Step: 213015 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:35:14,479-Speed 128.11 samples/sec Loss 0.4080 Epoch: 21 Global Step: 213020 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:35:34,294-Speed 129.20 samples/sec Loss 0.3553 Epoch: 21 Global Step: 213025 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:35:54,245-Speed 128.33 samples/sec Loss 0.3459 Epoch: 21 Global Step: 213030 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:36:14,052-Speed 129.25 samples/sec Loss 0.3012 Epoch: 21 Global Step: 213035 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:36:34,138-Speed 127.46 samples/sec Loss 0.3172 Epoch: 21 Global Step: 213040 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:36:54,362-Speed 126.59 samples/sec Loss 0.3533 Epoch: 21 Global Step: 213045 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:37:14,169-Speed 129.26 samples/sec Loss 0.4189 Epoch: 21 Global Step: 213050 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:37:33,881-Speed 129.87 samples/sec Loss 0.4109 Epoch: 21 Global Step: 213055 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:37:53,824-Speed 128.37 samples/sec Loss 0.3906 Epoch: 21 Global Step: 213060 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:38:13,911-Speed 127.45 samples/sec Loss 0.3587 Epoch: 21 Global Step: 213065 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:38:33,819-Speed 128.60 samples/sec Loss 0.3298 Epoch: 21 Global Step: 213070 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:38:53,660-Speed 129.03 samples/sec Loss 0.3407 Epoch: 21 Global Step: 213075 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:39:13,503-Speed 129.02 samples/sec Loss 0.4409 Epoch: 21 Global Step: 213080 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:39:33,559-Speed 127.65 samples/sec Loss 0.3648 Epoch: 21 Global Step: 213085 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:39:53,418-Speed 128.92 samples/sec Loss 0.3596 Epoch: 21 Global Step: 213090 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:40:13,278-Speed 128.91 samples/sec Loss 0.4000 Epoch: 21 Global Step: 213095 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:40:33,145-Speed 128.86 samples/sec Loss 0.3096 Epoch: 21 Global Step: 213100 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:40:52,923-Speed 129.44 samples/sec Loss 0.3752 Epoch: 21 Global Step: 213105 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:41:13,053-Speed 127.18 samples/sec Loss 0.4160 Epoch: 21 Global Step: 213110 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:41:33,645-Speed 124.33 samples/sec Loss 0.3695 Epoch: 21 Global Step: 213115 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:41:53,415-Speed 129.49 samples/sec Loss 0.3571 Epoch: 21 Global Step: 213120 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:42:13,357-Speed 128.38 samples/sec Loss 0.4229 Epoch: 21 Global Step: 213125 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:42:33,690-Speed 125.91 samples/sec Loss 0.3042 Epoch: 21 Global Step: 213130 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:42:53,463-Speed 129.48 samples/sec Loss 0.4445 Epoch: 21 Global Step: 213135 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:43:13,374-Speed 128.57 samples/sec Loss 0.3359 Epoch: 21 Global Step: 213140 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:43:33,416-Speed 127.74 samples/sec Loss 0.4439 Epoch: 21 Global Step: 213145 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:43:53,187-Speed 129.49 samples/sec Loss 0.4385 Epoch: 21 Global Step: 213150 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:44:13,192-Speed 127.97 samples/sec Loss 0.3676 Epoch: 21 Global Step: 213155 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:44:33,087-Speed 128.68 samples/sec Loss 0.4075 Epoch: 21 Global Step: 213160 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:44:52,705-Speed 130.50 samples/sec Loss 0.4032 Epoch: 21 Global Step: 213165 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:45:12,630-Speed 128.49 samples/sec Loss 0.4217 Epoch: 21 Global Step: 213170 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:45:32,623-Speed 128.05 samples/sec Loss 0.3513 Epoch: 21 Global Step: 213175 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:45:52,390-Speed 129.51 samples/sec Loss 0.3863 Epoch: 21 Global Step: 213180 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:46:12,150-Speed 129.56 samples/sec Loss 0.4401 Epoch: 21 Global Step: 213185 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:46:31,986-Speed 129.06 samples/sec Loss 0.3947 Epoch: 21 Global Step: 213190 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:46:51,697-Speed 129.88 samples/sec Loss 0.3278 Epoch: 21 Global Step: 213195 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:47:11,762-Speed 127.59 samples/sec Loss 0.3703 Epoch: 21 Global Step: 213200 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:47:31,593-Speed 129.10 samples/sec Loss 0.4657 Epoch: 21 Global Step: 213205 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:47:51,212-Speed 130.49 samples/sec Loss 0.4006 Epoch: 21 Global Step: 213210 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:48:11,199-Speed 128.09 samples/sec Loss 0.3573 Epoch: 21 Global Step: 213215 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:48:31,046-Speed 129.00 samples/sec Loss 0.3814 Epoch: 21 Global Step: 213220 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:48:50,813-Speed 129.51 samples/sec Loss 0.4052 Epoch: 21 Global Step: 213225 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:49:10,551-Speed 129.70 samples/sec Loss 0.3911 Epoch: 21 Global Step: 213230 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:49:30,367-Speed 129.20 samples/sec Loss 0.3345 Epoch: 21 Global Step: 213235 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:49:50,144-Speed 129.45 samples/sec Loss 0.4067 Epoch: 21 Global Step: 213240 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:50:09,673-Speed 131.09 samples/sec Loss 0.3587 Epoch: 21 Global Step: 213245 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:50:29,543-Speed 128.84 samples/sec Loss 0.3561 Epoch: 21 Global Step: 213250 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:50:49,017-Speed 131.47 samples/sec Loss 0.4489 Epoch: 21 Global Step: 213255 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:51:08,973-Speed 128.29 samples/sec Loss 0.3531 Epoch: 21 Global Step: 213260 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:51:28,466-Speed 131.34 samples/sec Loss 0.3258 Epoch: 21 Global Step: 213265 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:51:48,317-Speed 128.97 samples/sec Loss 0.4177 Epoch: 21 Global Step: 213270 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:52:07,893-Speed 130.78 samples/sec Loss 0.3405 Epoch: 21 Global Step: 213275 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:52:27,393-Speed 131.29 samples/sec Loss 0.3390 Epoch: 21 Global Step: 213280 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:52:47,219-Speed 129.13 samples/sec Loss 0.3777 Epoch: 21 Global Step: 213285 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:53:07,598-Speed 125.62 samples/sec Loss 0.3864 Epoch: 21 Global Step: 213290 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:53:27,529-Speed 128.45 samples/sec Loss 0.3335 Epoch: 21 Global Step: 213295 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:53:47,680-Speed 127.05 samples/sec Loss 0.3999 Epoch: 21 Global Step: 213300 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:54:07,912-Speed 126.54 samples/sec Loss 0.4367 Epoch: 21 Global Step: 213305 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:54:28,270-Speed 125.76 samples/sec Loss 0.4104 Epoch: 21 Global Step: 213310 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:54:48,645-Speed 125.65 samples/sec Loss 0.3700 Epoch: 21 Global Step: 213315 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:55:08,502-Speed 128.93 samples/sec Loss 0.3661 Epoch: 21 Global Step: 213320 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:55:28,379-Speed 128.80 samples/sec Loss 0.4447 Epoch: 21 Global Step: 213325 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:55:48,068-Speed 130.03 samples/sec Loss 0.3401 Epoch: 21 Global Step: 213330 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:56:08,478-Speed 125.43 samples/sec Loss 0.3876 Epoch: 21 Global Step: 213335 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:56:28,263-Speed 129.39 samples/sec Loss 0.3274 Epoch: 21 Global Step: 213340 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:56:48,109-Speed 129.00 samples/sec Loss 0.3101 Epoch: 21 Global Step: 213345 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:57:07,922-Speed 129.22 samples/sec Loss 0.3321 Epoch: 21 Global Step: 213350 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:57:27,943-Speed 127.87 samples/sec Loss 0.4025 Epoch: 21 Global Step: 213355 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:57:47,730-Speed 129.38 samples/sec Loss 0.3550 Epoch: 21 Global Step: 213360 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:58:07,481-Speed 129.62 samples/sec Loss 0.3321 Epoch: 21 Global Step: 213365 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:58:27,687-Speed 126.70 samples/sec Loss 0.4236 Epoch: 21 Global Step: 213370 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 10:58:46,907-Speed 133.20 samples/sec Loss 0.3662 Epoch: 21 Global Step: 213375 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:59:07,574-Speed 123.87 samples/sec Loss 0.3261 Epoch: 21 Global Step: 213380 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:59:27,324-Speed 129.63 samples/sec Loss 0.3419 Epoch: 21 Global Step: 213385 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 10:59:47,191-Speed 128.86 samples/sec Loss 0.3865 Epoch: 21 Global Step: 213390 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:00:07,120-Speed 128.46 samples/sec Loss 0.3648 Epoch: 21 Global Step: 213395 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:00:26,932-Speed 129.23 samples/sec Loss 0.3018 Epoch: 21 Global Step: 213400 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:00:46,672-Speed 129.69 samples/sec Loss 0.4358 Epoch: 21 Global Step: 213405 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:01:06,498-Speed 129.13 samples/sec Loss 0.3911 Epoch: 21 Global Step: 213410 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:01:26,363-Speed 128.87 samples/sec Loss 0.3404 Epoch: 21 Global Step: 213415 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:01:46,324-Speed 128.26 samples/sec Loss 0.4097 Epoch: 21 Global Step: 213420 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:02:06,617-Speed 126.16 samples/sec Loss 0.3488 Epoch: 21 Global Step: 213425 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:02:26,392-Speed 129.46 samples/sec Loss 0.3707 Epoch: 21 Global Step: 213430 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:02:46,264-Speed 128.84 samples/sec Loss 0.3825 Epoch: 21 Global Step: 213435 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:03:06,128-Speed 128.88 samples/sec Loss 0.3926 Epoch: 21 Global Step: 213440 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:03:25,954-Speed 129.13 samples/sec Loss 0.3599 Epoch: 21 Global Step: 213445 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:03:45,672-Speed 129.84 samples/sec Loss 0.3396 Epoch: 21 Global Step: 213450 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:04:05,510-Speed 129.05 samples/sec Loss 0.3811 Epoch: 21 Global Step: 213455 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:04:25,352-Speed 129.03 samples/sec Loss 0.4310 Epoch: 21 Global Step: 213460 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:04:45,314-Speed 128.25 samples/sec Loss 0.4437 Epoch: 21 Global Step: 213465 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:05:05,670-Speed 125.77 samples/sec Loss 0.3758 Epoch: 21 Global Step: 213470 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:05:25,742-Speed 127.55 samples/sec Loss 0.3526 Epoch: 21 Global Step: 213475 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:05:45,588-Speed 129.00 samples/sec Loss 0.4310 Epoch: 21 Global Step: 213480 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:06:05,467-Speed 128.79 samples/sec Loss 0.3728 Epoch: 21 Global Step: 213485 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:06:25,343-Speed 128.81 samples/sec Loss 0.3356 Epoch: 21 Global Step: 213490 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:06:45,202-Speed 128.92 samples/sec Loss 0.3258 Epoch: 21 Global Step: 213495 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:07:05,281-Speed 127.50 samples/sec Loss 0.4013 Epoch: 21 Global Step: 213500 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:07:25,153-Speed 128.83 samples/sec Loss 0.3767 Epoch: 21 Global Step: 213505 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:07:45,100-Speed 128.35 samples/sec Loss 0.3768 Epoch: 21 Global Step: 213510 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:08:05,311-Speed 126.67 samples/sec Loss 0.3432 Epoch: 21 Global Step: 213515 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:08:25,241-Speed 128.45 samples/sec Loss 0.3187 Epoch: 21 Global Step: 213520 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:08:45,038-Speed 129.32 samples/sec Loss 0.3112 Epoch: 21 Global Step: 213525 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:09:04,909-Speed 128.83 samples/sec Loss 0.3871 Epoch: 21 Global Step: 213530 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:09:24,122-Speed 133.25 samples/sec Loss 0.4069 Epoch: 21 Global Step: 213535 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:09:43,672-Speed 130.95 samples/sec Loss 0.3887 Epoch: 21 Global Step: 213540 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:10:03,256-Speed 130.73 samples/sec Loss 0.3798 Epoch: 21 Global Step: 213545 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:10:22,889-Speed 130.40 samples/sec Loss 0.3990 Epoch: 21 Global Step: 213550 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:10:42,670-Speed 129.42 samples/sec Loss 0.3796 Epoch: 21 Global Step: 213555 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:11:02,453-Speed 129.41 samples/sec Loss 0.3439 Epoch: 21 Global Step: 213560 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:11:22,337-Speed 128.76 samples/sec Loss 0.3649 Epoch: 21 Global Step: 213565 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:11:42,007-Speed 130.15 samples/sec Loss 0.3234 Epoch: 21 Global Step: 213570 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:12:02,019-Speed 127.93 samples/sec Loss 0.3749 Epoch: 21 Global Step: 213575 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:12:21,623-Speed 130.59 samples/sec Loss 0.3510 Epoch: 21 Global Step: 213580 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:12:41,739-Speed 127.27 samples/sec Loss 0.3858 Epoch: 21 Global Step: 213585 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:13:01,568-Speed 129.11 samples/sec Loss 0.3548 Epoch: 21 Global Step: 213590 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:13:21,348-Speed 129.43 samples/sec Loss 0.3558 Epoch: 21 Global Step: 213595 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:13:41,156-Speed 129.25 samples/sec Loss 0.3055 Epoch: 21 Global Step: 213600 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:14:01,121-Speed 128.23 samples/sec Loss 0.4088 Epoch: 21 Global Step: 213605 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:14:21,011-Speed 128.71 samples/sec Loss 0.4185 Epoch: 21 Global Step: 213610 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:14:40,692-Speed 130.08 samples/sec Loss 0.3033 Epoch: 21 Global Step: 213615 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:15:00,705-Speed 127.92 samples/sec Loss 0.3705 Epoch: 21 Global Step: 213620 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:15:20,324-Speed 130.50 samples/sec Loss 0.3947 Epoch: 21 Global Step: 213625 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:15:40,430-Speed 127.33 samples/sec Loss 0.3372 Epoch: 21 Global Step: 213630 Fp16 Grad Scale: 4096 Required: 43 hours Training: 2021-06-17 11:16:00,273-Speed 129.03 samples/sec Loss 0.3995 Epoch: 21 Global Step: 213635 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:16:20,067-Speed 129.34 samples/sec Loss 0.4089 Epoch: 21 Global Step: 213640 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:16:39,861-Speed 129.34 samples/sec Loss 0.4028 Epoch: 21 Global Step: 213645 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:17:00,055-Speed 126.78 samples/sec Loss 0.3694 Epoch: 21 Global Step: 213650 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:17:19,797-Speed 129.68 samples/sec Loss 0.3904 Epoch: 21 Global Step: 213655 Fp16 Grad Scale: 8192 Required: 43 hours Training: 2021-06-17 11:17:39,042-Speed 133.03 samples/sec Loss 0.3399 Epoch: 21 Global Step: 213660 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:17:58,975-Speed 128.43 samples/sec Loss 0.4601 Epoch: 21 Global Step: 213665 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:18:18,404-Speed 131.77 samples/sec Loss 0.3553 Epoch: 21 Global Step: 213670 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:18:38,302-Speed 128.66 samples/sec Loss 0.2952 Epoch: 21 Global Step: 213675 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:18:57,738-Speed 131.72 samples/sec Loss 0.3244 Epoch: 21 Global Step: 213680 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:19:17,336-Speed 130.63 samples/sec Loss 0.2991 Epoch: 21 Global Step: 213685 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:19:36,931-Speed 130.66 samples/sec Loss 0.3990 Epoch: 21 Global Step: 213690 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:19:56,652-Speed 129.82 samples/sec Loss 0.3988 Epoch: 21 Global Step: 213695 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:20:16,554-Speed 128.63 samples/sec Loss 0.4112 Epoch: 21 Global Step: 213700 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:20:36,165-Speed 130.55 samples/sec Loss 0.3820 Epoch: 21 Global Step: 213705 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:20:56,121-Speed 128.29 samples/sec Loss 0.3930 Epoch: 21 Global Step: 213710 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:21:15,750-Speed 130.42 samples/sec Loss 0.2919 Epoch: 21 Global Step: 213715 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:21:36,116-Speed 125.71 samples/sec Loss 0.3627 Epoch: 21 Global Step: 213720 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:21:55,707-Speed 130.68 samples/sec Loss 0.4405 Epoch: 21 Global Step: 213725 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:22:15,980-Speed 126.28 samples/sec Loss 0.3655 Epoch: 21 Global Step: 213730 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:22:35,705-Speed 129.79 samples/sec Loss 0.3884 Epoch: 21 Global Step: 213735 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:22:55,606-Speed 128.65 samples/sec Loss 0.4449 Epoch: 21 Global Step: 213740 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:23:15,359-Speed 129.61 samples/sec Loss 0.3608 Epoch: 21 Global Step: 213745 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:23:35,031-Speed 130.14 samples/sec Loss 0.3194 Epoch: 21 Global Step: 213750 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:23:54,953-Speed 128.51 samples/sec Loss 0.3872 Epoch: 21 Global Step: 213755 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:24:14,665-Speed 129.88 samples/sec Loss 0.3149 Epoch: 21 Global Step: 213760 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:24:35,033-Speed 125.70 samples/sec Loss 0.3760 Epoch: 21 Global Step: 213765 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:24:54,689-Speed 130.24 samples/sec Loss 0.3424 Epoch: 21 Global Step: 213770 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:25:14,625-Speed 128.42 samples/sec Loss 0.3005 Epoch: 21 Global Step: 213775 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:25:34,369-Speed 129.67 samples/sec Loss 0.3692 Epoch: 21 Global Step: 213780 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:25:54,607-Speed 126.50 samples/sec Loss 0.3958 Epoch: 21 Global Step: 213785 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:26:14,376-Speed 129.51 samples/sec Loss 0.4128 Epoch: 21 Global Step: 213790 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:26:34,552-Speed 126.89 samples/sec Loss 0.2691 Epoch: 21 Global Step: 213795 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:26:54,526-Speed 128.17 samples/sec Loss 0.4383 Epoch: 21 Global Step: 213800 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:27:14,241-Speed 129.85 samples/sec Loss 0.3489 Epoch: 21 Global Step: 213805 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:27:35,285-Speed 121.66 samples/sec Loss 0.3719 Epoch: 21 Global Step: 213810 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:27:55,327-Speed 127.74 samples/sec Loss 0.4763 Epoch: 21 Global Step: 213815 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:28:15,074-Speed 129.65 samples/sec Loss 0.3846 Epoch: 21 Global Step: 213820 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:28:34,842-Speed 129.51 samples/sec Loss 0.3581 Epoch: 21 Global Step: 213825 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:28:54,597-Speed 129.59 samples/sec Loss 0.4668 Epoch: 21 Global Step: 213830 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:29:14,344-Speed 129.65 samples/sec Loss 0.3089 Epoch: 21 Global Step: 213835 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:29:34,281-Speed 128.41 samples/sec Loss 0.4234 Epoch: 21 Global Step: 213840 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:29:54,088-Speed 129.26 samples/sec Loss 0.3600 Epoch: 21 Global Step: 213845 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:30:13,807-Speed 129.83 samples/sec Loss 0.3574 Epoch: 21 Global Step: 213850 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:30:34,094-Speed 126.20 samples/sec Loss 0.3735 Epoch: 21 Global Step: 213855 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:30:53,902-Speed 129.24 samples/sec Loss 0.3836 Epoch: 21 Global Step: 213860 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:31:13,692-Speed 129.36 samples/sec Loss 0.3640 Epoch: 21 Global Step: 213865 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:31:33,454-Speed 129.55 samples/sec Loss 0.3241 Epoch: 21 Global Step: 213870 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:31:53,210-Speed 129.58 samples/sec Loss 0.4563 Epoch: 21 Global Step: 213875 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:32:12,994-Speed 129.40 samples/sec Loss 0.3047 Epoch: 21 Global Step: 213880 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:32:32,944-Speed 128.33 samples/sec Loss 0.3779 Epoch: 21 Global Step: 213885 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:32:52,771-Speed 129.12 samples/sec Loss 0.3524 Epoch: 21 Global Step: 213890 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:33:13,819-Speed 121.63 samples/sec Loss 0.3588 Epoch: 21 Global Step: 213895 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:33:34,663-Speed 122.82 samples/sec Loss 0.4209 Epoch: 21 Global Step: 213900 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:33:54,093-Speed 131.76 samples/sec Loss 0.3389 Epoch: 21 Global Step: 213905 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:34:13,898-Speed 129.27 samples/sec Loss 0.3761 Epoch: 21 Global Step: 213910 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:34:33,649-Speed 129.62 samples/sec Loss 0.3678 Epoch: 21 Global Step: 213915 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:34:53,529-Speed 128.78 samples/sec Loss 0.5324 Epoch: 21 Global Step: 213920 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:35:13,278-Speed 129.64 samples/sec Loss 0.4127 Epoch: 21 Global Step: 213925 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:35:33,081-Speed 129.28 samples/sec Loss 0.3592 Epoch: 21 Global Step: 213930 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:35:52,858-Speed 129.45 samples/sec Loss 0.3650 Epoch: 21 Global Step: 213935 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:36:12,576-Speed 129.83 samples/sec Loss 0.3429 Epoch: 21 Global Step: 213940 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:36:32,858-Speed 126.23 samples/sec Loss 0.3086 Epoch: 21 Global Step: 213945 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:36:52,763-Speed 128.62 samples/sec Loss 0.3370 Epoch: 21 Global Step: 213950 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:37:12,474-Speed 129.88 samples/sec Loss 0.4265 Epoch: 21 Global Step: 213955 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:37:32,380-Speed 128.61 samples/sec Loss 0.4044 Epoch: 21 Global Step: 213960 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:37:51,994-Speed 130.52 samples/sec Loss 0.3176 Epoch: 21 Global Step: 213965 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:38:11,761-Speed 129.52 samples/sec Loss 0.3922 Epoch: 21 Global Step: 213970 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:38:31,632-Speed 128.84 samples/sec Loss 0.3642 Epoch: 21 Global Step: 213975 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:38:51,426-Speed 129.34 samples/sec Loss 0.4825 Epoch: 21 Global Step: 213980 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:39:11,270-Speed 129.01 samples/sec Loss 0.3555 Epoch: 21 Global Step: 213985 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:39:31,525-Speed 126.39 samples/sec Loss 0.3638 Epoch: 21 Global Step: 213990 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:39:51,386-Speed 128.91 samples/sec Loss 0.3188 Epoch: 21 Global Step: 213995 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:40:11,241-Speed 128.94 samples/sec Loss 0.3771 Epoch: 21 Global Step: 214000 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:40:31,324-Speed 127.48 samples/sec Loss 0.3920 Epoch: 21 Global Step: 214005 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:40:50,972-Speed 130.30 samples/sec Loss 0.4105 Epoch: 21 Global Step: 214010 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:41:10,781-Speed 129.24 samples/sec Loss 0.4550 Epoch: 21 Global Step: 214015 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:41:30,603-Speed 129.16 samples/sec Loss 0.4059 Epoch: 21 Global Step: 214020 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:41:50,381-Speed 129.44 samples/sec Loss 0.4214 Epoch: 21 Global Step: 214025 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:42:10,188-Speed 129.25 samples/sec Loss 0.4120 Epoch: 21 Global Step: 214030 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:42:30,362-Speed 126.90 samples/sec Loss 0.3183 Epoch: 21 Global Step: 214035 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:42:50,225-Speed 128.89 samples/sec Loss 0.3430 Epoch: 21 Global Step: 214040 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:43:10,031-Speed 129.26 samples/sec Loss 0.3369 Epoch: 21 Global Step: 214045 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:43:29,840-Speed 129.24 samples/sec Loss 0.3682 Epoch: 21 Global Step: 214050 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:43:49,626-Speed 129.39 samples/sec Loss 0.3411 Epoch: 21 Global Step: 214055 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:44:09,459-Speed 129.09 samples/sec Loss 0.3251 Epoch: 21 Global Step: 214060 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:44:29,159-Speed 129.96 samples/sec Loss 0.3853 Epoch: 21 Global Step: 214065 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:44:48,946-Speed 129.38 samples/sec Loss 0.3593 Epoch: 21 Global Step: 214070 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:45:08,838-Speed 128.70 samples/sec Loss 0.4158 Epoch: 21 Global Step: 214075 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:45:29,099-Speed 126.35 samples/sec Loss 0.3258 Epoch: 21 Global Step: 214080 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:45:48,760-Speed 130.21 samples/sec Loss 0.3386 Epoch: 21 Global Step: 214085 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:46:08,568-Speed 129.25 samples/sec Loss 0.3391 Epoch: 21 Global Step: 214090 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:46:28,364-Speed 129.33 samples/sec Loss 0.3442 Epoch: 21 Global Step: 214095 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:46:47,974-Speed 130.55 samples/sec Loss 0.3352 Epoch: 21 Global Step: 214100 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:47:07,891-Speed 128.54 samples/sec Loss 0.3599 Epoch: 21 Global Step: 214105 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:47:27,555-Speed 130.19 samples/sec Loss 0.3712 Epoch: 21 Global Step: 214110 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:47:47,410-Speed 128.94 samples/sec Loss 0.4697 Epoch: 21 Global Step: 214115 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:48:07,175-Speed 129.53 samples/sec Loss 0.4386 Epoch: 21 Global Step: 214120 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:48:26,656-Speed 131.41 samples/sec Loss 0.3781 Epoch: 21 Global Step: 214125 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:48:46,173-Speed 131.18 samples/sec Loss 0.3652 Epoch: 21 Global Step: 214130 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:49:05,833-Speed 130.22 samples/sec Loss 0.3913 Epoch: 21 Global Step: 214135 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:49:25,458-Speed 130.45 samples/sec Loss 0.3533 Epoch: 21 Global Step: 214140 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:49:45,181-Speed 129.80 samples/sec Loss 0.3361 Epoch: 21 Global Step: 214145 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:50:04,788-Speed 130.58 samples/sec Loss 0.3818 Epoch: 21 Global Step: 214150 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:50:24,477-Speed 130.03 samples/sec Loss 0.3447 Epoch: 21 Global Step: 214155 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:50:44,306-Speed 129.11 samples/sec Loss 0.3225 Epoch: 21 Global Step: 214160 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:51:04,064-Speed 129.58 samples/sec Loss 0.3326 Epoch: 21 Global Step: 214165 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:51:24,169-Speed 127.33 samples/sec Loss 0.3495 Epoch: 21 Global Step: 214170 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:51:43,851-Speed 130.07 samples/sec Loss 0.3478 Epoch: 21 Global Step: 214175 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:52:03,618-Speed 129.52 samples/sec Loss 0.3253 Epoch: 21 Global Step: 214180 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:52:23,562-Speed 128.37 samples/sec Loss 0.3367 Epoch: 21 Global Step: 214185 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:52:43,168-Speed 130.58 samples/sec Loss 0.3009 Epoch: 21 Global Step: 214190 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:53:03,346-Speed 126.88 samples/sec Loss 0.3546 Epoch: 21 Global Step: 214195 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:53:23,064-Speed 129.84 samples/sec Loss 0.4045 Epoch: 21 Global Step: 214200 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:53:43,000-Speed 128.42 samples/sec Loss 0.3835 Epoch: 21 Global Step: 214205 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:54:02,665-Speed 130.19 samples/sec Loss 0.3560 Epoch: 21 Global Step: 214210 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:54:22,976-Speed 126.05 samples/sec Loss 0.3938 Epoch: 21 Global Step: 214215 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:54:42,705-Speed 129.76 samples/sec Loss 0.3570 Epoch: 21 Global Step: 214220 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 11:55:02,645-Speed 128.39 samples/sec Loss 0.3300 Epoch: 21 Global Step: 214225 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:55:22,435-Speed 129.36 samples/sec Loss 0.3214 Epoch: 21 Global Step: 214230 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:55:42,089-Speed 130.26 samples/sec Loss 0.3788 Epoch: 21 Global Step: 214235 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:56:02,245-Speed 127.02 samples/sec Loss 0.3611 Epoch: 21 Global Step: 214240 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:56:21,974-Speed 129.77 samples/sec Loss 0.3413 Epoch: 21 Global Step: 214245 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:56:41,939-Speed 128.23 samples/sec Loss 0.3149 Epoch: 21 Global Step: 214250 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:57:01,648-Speed 129.90 samples/sec Loss 0.3936 Epoch: 21 Global Step: 214255 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:57:21,743-Speed 127.40 samples/sec Loss 0.4041 Epoch: 21 Global Step: 214260 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:57:41,461-Speed 129.83 samples/sec Loss 0.3031 Epoch: 21 Global Step: 214265 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:58:01,363-Speed 128.64 samples/sec Loss 0.3467 Epoch: 21 Global Step: 214270 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:58:21,164-Speed 129.29 samples/sec Loss 0.3723 Epoch: 21 Global Step: 214275 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:58:40,876-Speed 129.87 samples/sec Loss 0.3564 Epoch: 21 Global Step: 214280 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:59:00,996-Speed 127.24 samples/sec Loss 0.4871 Epoch: 21 Global Step: 214285 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:59:20,833-Speed 129.06 samples/sec Loss 0.3626 Epoch: 21 Global Step: 214290 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 11:59:40,804-Speed 128.19 samples/sec Loss 0.3566 Epoch: 21 Global Step: 214295 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 12:00:00,053-Speed 133.00 samples/sec Loss 0.5083 Epoch: 21 Global Step: 214300 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:00:19,918-Speed 128.87 samples/sec Loss 0.3773 Epoch: 21 Global Step: 214305 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:00:39,724-Speed 129.26 samples/sec Loss 0.3237 Epoch: 21 Global Step: 214310 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:00:59,599-Speed 128.82 samples/sec Loss 0.3159 Epoch: 21 Global Step: 214315 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:01:19,402-Speed 129.28 samples/sec Loss 0.4063 Epoch: 21 Global Step: 214320 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:01:39,117-Speed 129.86 samples/sec Loss 0.3665 Epoch: 21 Global Step: 214325 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:01:58,936-Speed 129.17 samples/sec Loss 0.4468 Epoch: 21 Global Step: 214330 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:02:18,699-Speed 129.54 samples/sec Loss 0.3932 Epoch: 21 Global Step: 214335 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:02:38,322-Speed 130.47 samples/sec Loss 0.3331 Epoch: 21 Global Step: 214340 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:02:58,066-Speed 129.67 samples/sec Loss 0.3575 Epoch: 21 Global Step: 214345 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:03:17,662-Speed 130.65 samples/sec Loss 0.3689 Epoch: 21 Global Step: 214350 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:03:37,421-Speed 129.57 samples/sec Loss 0.3458 Epoch: 21 Global Step: 214355 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:03:57,362-Speed 128.39 samples/sec Loss 0.4764 Epoch: 21 Global Step: 214360 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:04:17,166-Speed 129.27 samples/sec Loss 0.3533 Epoch: 21 Global Step: 214365 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:04:36,903-Speed 129.71 samples/sec Loss 0.3980 Epoch: 21 Global Step: 214370 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:04:56,721-Speed 129.18 samples/sec Loss 0.3700 Epoch: 21 Global Step: 214375 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:05:16,516-Speed 129.33 samples/sec Loss 0.3573 Epoch: 21 Global Step: 214380 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:05:36,110-Speed 130.66 samples/sec Loss 0.3742 Epoch: 21 Global Step: 214385 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:05:56,061-Speed 128.32 samples/sec Loss 0.3639 Epoch: 21 Global Step: 214390 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:06:15,653-Speed 130.67 samples/sec Loss 0.4081 Epoch: 21 Global Step: 214395 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:06:35,373-Speed 129.82 samples/sec Loss 0.4318 Epoch: 21 Global Step: 214400 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 12:06:55,165-Speed 129.36 samples/sec Loss 0.3838 Epoch: 21 Global Step: 214405 Fp16 Grad Scale: 8192 Required: 42 hours Training: 2021-06-17 12:07:14,542-Speed 132.12 samples/sec Loss 0.3613 Epoch: 21 Global Step: 214410 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:07:34,240-Speed 129.97 samples/sec Loss 0.4094 Epoch: 21 Global Step: 214415 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:07:54,023-Speed 129.41 samples/sec Loss 0.3958 Epoch: 21 Global Step: 214420 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:08:13,962-Speed 128.40 samples/sec Loss 0.3417 Epoch: 21 Global Step: 214425 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:08:33,568-Speed 130.58 samples/sec Loss 0.3317 Epoch: 21 Global Step: 214430 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:08:53,332-Speed 129.54 samples/sec Loss 0.3469 Epoch: 21 Global Step: 214435 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:09:12,935-Speed 130.61 samples/sec Loss 0.3084 Epoch: 21 Global Step: 214440 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:09:32,622-Speed 130.04 samples/sec Loss 0.3124 Epoch: 21 Global Step: 214445 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:09:52,436-Speed 129.21 samples/sec Loss 0.3778 Epoch: 21 Global Step: 214450 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:10:12,431-Speed 128.04 samples/sec Loss 0.4316 Epoch: 21 Global Step: 214455 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:10:31,867-Speed 131.72 samples/sec Loss 0.3258 Epoch: 21 Global Step: 214460 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:10:51,611-Speed 129.67 samples/sec Loss 0.3339 Epoch: 21 Global Step: 214465 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:11:11,487-Speed 128.81 samples/sec Loss 0.3932 Epoch: 21 Global Step: 214470 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:11:31,090-Speed 130.60 samples/sec Loss 0.3594 Epoch: 21 Global Step: 214475 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:11:50,876-Speed 129.40 samples/sec Loss 0.3550 Epoch: 21 Global Step: 214480 Fp16 Grad Scale: 4096 Required: 42 hours Training: 2021-06-17 12:12:10,462-Speed 130.71 samples/sec Loss 0.4078 Epoch: 21 Global Step: 214485 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:12:30,184-Speed 129.81 samples/sec Loss 0.3429 Epoch: 21 Global Step: 214490 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:12:49,983-Speed 129.31 samples/sec Loss 0.3688 Epoch: 21 Global Step: 214495 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:13:09,897-Speed 128.56 samples/sec Loss 0.4344 Epoch: 21 Global Step: 214500 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:13:29,328-Speed 131.75 samples/sec Loss 0.3646 Epoch: 21 Global Step: 214505 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:13:49,124-Speed 129.33 samples/sec Loss 0.4315 Epoch: 21 Global Step: 214510 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:14:09,502-Speed 125.63 samples/sec Loss 0.3517 Epoch: 21 Global Step: 214515 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:14:29,020-Speed 131.17 samples/sec Loss 0.3603 Epoch: 21 Global Step: 214520 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:14:48,980-Speed 128.27 samples/sec Loss 0.3330 Epoch: 21 Global Step: 214525 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:15:08,442-Speed 131.55 samples/sec Loss 0.4405 Epoch: 21 Global Step: 214530 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:15:28,210-Speed 129.51 samples/sec Loss 0.3720 Epoch: 21 Global Step: 214535 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:15:48,027-Speed 129.19 samples/sec Loss 0.3733 Epoch: 21 Global Step: 214540 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:16:07,931-Speed 128.62 samples/sec Loss 0.3255 Epoch: 21 Global Step: 214545 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:16:27,345-Speed 131.86 samples/sec Loss 0.3548 Epoch: 21 Global Step: 214550 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:16:47,154-Speed 129.24 samples/sec Loss 0.3567 Epoch: 21 Global Step: 214555 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:17:07,003-Speed 128.98 samples/sec Loss 0.4179 Epoch: 21 Global Step: 214560 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:17:26,264-Speed 132.91 samples/sec Loss 0.3682 Epoch: 21 Global Step: 214565 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:17:46,258-Speed 128.05 samples/sec Loss 0.3663 Epoch: 21 Global Step: 214570 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:18:05,886-Speed 130.44 samples/sec Loss 0.4048 Epoch: 21 Global Step: 214575 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:18:25,652-Speed 129.52 samples/sec Loss 0.3925 Epoch: 21 Global Step: 214580 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:18:45,642-Speed 128.08 samples/sec Loss 0.3560 Epoch: 21 Global Step: 214585 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:19:05,618-Speed 128.16 samples/sec Loss 0.4388 Epoch: 21 Global Step: 214590 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:19:25,357-Speed 129.70 samples/sec Loss 0.4011 Epoch: 21 Global Step: 214595 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:19:45,157-Speed 129.30 samples/sec Loss 0.3668 Epoch: 21 Global Step: 214600 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:20:04,866-Speed 129.89 samples/sec Loss 0.4038 Epoch: 21 Global Step: 214605 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:20:24,573-Speed 129.91 samples/sec Loss 0.3222 Epoch: 21 Global Step: 214610 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:20:44,393-Speed 129.17 samples/sec Loss 0.3207 Epoch: 21 Global Step: 214615 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:21:04,175-Speed 129.42 samples/sec Loss 0.4156 Epoch: 21 Global Step: 214620 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:21:24,282-Speed 127.32 samples/sec Loss 0.3740 Epoch: 21 Global Step: 214625 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:21:44,184-Speed 128.64 samples/sec Loss 0.3892 Epoch: 21 Global Step: 214630 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:22:03,989-Speed 129.27 samples/sec Loss 0.3710 Epoch: 21 Global Step: 214635 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:22:23,754-Speed 129.52 samples/sec Loss 0.4638 Epoch: 21 Global Step: 214640 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:22:43,557-Speed 129.28 samples/sec Loss 0.3669 Epoch: 21 Global Step: 214645 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:23:03,289-Speed 129.74 samples/sec Loss 0.2968 Epoch: 21 Global Step: 214650 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:23:23,081-Speed 129.35 samples/sec Loss 0.3472 Epoch: 21 Global Step: 214655 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:23:42,850-Speed 129.50 samples/sec Loss 0.3749 Epoch: 21 Global Step: 214660 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:24:02,597-Speed 129.65 samples/sec Loss 0.4092 Epoch: 21 Global Step: 214665 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:24:22,792-Speed 126.77 samples/sec Loss 0.3359 Epoch: 21 Global Step: 214670 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:24:42,734-Speed 128.38 samples/sec Loss 0.3994 Epoch: 21 Global Step: 214675 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:25:02,494-Speed 129.56 samples/sec Loss 0.3566 Epoch: 21 Global Step: 214680 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:25:22,276-Speed 129.42 samples/sec Loss 0.3675 Epoch: 21 Global Step: 214685 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:25:42,186-Speed 128.58 samples/sec Loss 0.4004 Epoch: 21 Global Step: 214690 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:26:02,069-Speed 128.76 samples/sec Loss 0.3176 Epoch: 21 Global Step: 214695 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:26:21,973-Speed 128.63 samples/sec Loss 0.3840 Epoch: 21 Global Step: 214700 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:26:41,639-Speed 130.18 samples/sec Loss 0.3652 Epoch: 21 Global Step: 214705 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:27:01,474-Speed 129.07 samples/sec Loss 0.3769 Epoch: 21 Global Step: 214710 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:27:21,644-Speed 126.93 samples/sec Loss 0.3988 Epoch: 21 Global Step: 214715 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:27:41,655-Speed 127.94 samples/sec Loss 0.4025 Epoch: 21 Global Step: 214720 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:28:01,315-Speed 130.22 samples/sec Loss 0.3590 Epoch: 21 Global Step: 214725 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:28:21,245-Speed 128.46 samples/sec Loss 0.3624 Epoch: 21 Global Step: 214730 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:28:40,969-Speed 129.79 samples/sec Loss 0.3507 Epoch: 21 Global Step: 214735 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:29:00,693-Speed 129.80 samples/sec Loss 0.4308 Epoch: 21 Global Step: 214740 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:29:20,496-Speed 129.28 samples/sec Loss 0.3435 Epoch: 21 Global Step: 214745 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:29:39,779-Speed 132.77 samples/sec Loss 0.3554 Epoch: 21 Global Step: 214750 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:29:59,684-Speed 128.62 samples/sec Loss 0.3151 Epoch: 21 Global Step: 214755 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:30:19,749-Speed 127.59 samples/sec Loss 0.2777 Epoch: 21 Global Step: 214760 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:30:39,446-Speed 129.97 samples/sec Loss 0.3614 Epoch: 21 Global Step: 214765 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:30:59,336-Speed 128.72 samples/sec Loss 0.3914 Epoch: 21 Global Step: 214770 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:31:18,958-Speed 130.47 samples/sec Loss 0.2945 Epoch: 21 Global Step: 214775 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:31:38,831-Speed 128.82 samples/sec Loss 0.2997 Epoch: 21 Global Step: 214780 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:31:58,563-Speed 129.75 samples/sec Loss 0.3064 Epoch: 21 Global Step: 214785 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:32:18,487-Speed 128.50 samples/sec Loss 0.3599 Epoch: 21 Global Step: 214790 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:32:38,349-Speed 128.89 samples/sec Loss 0.3935 Epoch: 21 Global Step: 214795 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:32:57,876-Speed 131.11 samples/sec Loss 0.2926 Epoch: 21 Global Step: 214800 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:33:17,718-Speed 129.03 samples/sec Loss 0.3278 Epoch: 21 Global Step: 214805 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:33:37,523-Speed 129.26 samples/sec Loss 0.3953 Epoch: 21 Global Step: 214810 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:33:56,983-Speed 131.56 samples/sec Loss 0.2823 Epoch: 21 Global Step: 214815 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:34:16,966-Speed 128.12 samples/sec Loss 0.3828 Epoch: 21 Global Step: 214820 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:34:36,500-Speed 131.06 samples/sec Loss 0.3395 Epoch: 21 Global Step: 214825 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:34:56,820-Speed 125.99 samples/sec Loss 0.3557 Epoch: 21 Global Step: 214830 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:35:16,617-Speed 129.32 samples/sec Loss 0.3295 Epoch: 21 Global Step: 214835 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:35:36,435-Speed 129.18 samples/sec Loss 0.4284 Epoch: 21 Global Step: 214840 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:35:55,903-Speed 131.50 samples/sec Loss 0.3171 Epoch: 21 Global Step: 214845 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:36:15,683-Speed 129.43 samples/sec Loss 0.3354 Epoch: 21 Global Step: 214850 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:36:35,563-Speed 128.78 samples/sec Loss 0.3564 Epoch: 21 Global Step: 214855 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:36:55,006-Speed 131.67 samples/sec Loss 0.3632 Epoch: 21 Global Step: 214860 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:37:14,977-Speed 128.19 samples/sec Loss 0.3816 Epoch: 21 Global Step: 214865 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:37:34,515-Speed 131.03 samples/sec Loss 0.2951 Epoch: 21 Global Step: 214870 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:37:54,242-Speed 129.78 samples/sec Loss 0.4368 Epoch: 21 Global Step: 214875 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:38:14,007-Speed 129.53 samples/sec Loss 0.2805 Epoch: 21 Global Step: 214880 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:38:33,365-Speed 132.25 samples/sec Loss 0.4162 Epoch: 21 Global Step: 214885 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:38:52,788-Speed 131.81 samples/sec Loss 0.3202 Epoch: 21 Global Step: 214890 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:39:12,698-Speed 128.59 samples/sec Loss 0.3020 Epoch: 21 Global Step: 214895 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:39:32,401-Speed 129.93 samples/sec Loss 0.3342 Epoch: 21 Global Step: 214900 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:39:51,887-Speed 131.39 samples/sec Loss 0.3520 Epoch: 21 Global Step: 214905 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:40:11,882-Speed 128.04 samples/sec Loss 0.3094 Epoch: 21 Global Step: 214910 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:40:31,513-Speed 130.41 samples/sec Loss 0.3616 Epoch: 21 Global Step: 214915 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:40:51,224-Speed 129.89 samples/sec Loss 0.3421 Epoch: 21 Global Step: 214920 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:41:11,022-Speed 129.31 samples/sec Loss 0.4127 Epoch: 21 Global Step: 214925 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:41:30,883-Speed 128.90 samples/sec Loss 0.3614 Epoch: 21 Global Step: 214930 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:41:50,302-Speed 131.84 samples/sec Loss 0.3542 Epoch: 21 Global Step: 214935 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:42:10,235-Speed 128.44 samples/sec Loss 0.3620 Epoch: 21 Global Step: 214940 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:42:29,830-Speed 130.65 samples/sec Loss 0.3518 Epoch: 21 Global Step: 214945 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:42:49,591-Speed 129.55 samples/sec Loss 0.3512 Epoch: 21 Global Step: 214950 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:43:09,532-Speed 128.38 samples/sec Loss 0.3222 Epoch: 21 Global Step: 214955 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:43:29,088-Speed 130.91 samples/sec Loss 0.3591 Epoch: 21 Global Step: 214960 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:43:48,872-Speed 129.41 samples/sec Loss 0.4487 Epoch: 21 Global Step: 214965 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:44:08,521-Speed 130.30 samples/sec Loss 0.4532 Epoch: 21 Global Step: 214970 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:44:28,506-Speed 128.10 samples/sec Loss 0.3973 Epoch: 21 Global Step: 214975 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:44:48,069-Speed 130.87 samples/sec Loss 0.3140 Epoch: 21 Global Step: 214980 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:45:07,977-Speed 128.60 samples/sec Loss 0.3479 Epoch: 21 Global Step: 214985 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:45:27,585-Speed 130.57 samples/sec Loss 0.4020 Epoch: 21 Global Step: 214990 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:45:47,121-Speed 131.05 samples/sec Loss 0.4168 Epoch: 21 Global Step: 214995 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:46:07,025-Speed 128.63 samples/sec Loss 0.3624 Epoch: 21 Global Step: 215000 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:46:26,579-Speed 130.92 samples/sec Loss 0.4041 Epoch: 21 Global Step: 215005 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:46:46,393-Speed 129.21 samples/sec Loss 0.3538 Epoch: 21 Global Step: 215010 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:47:06,194-Speed 129.30 samples/sec Loss 0.3741 Epoch: 21 Global Step: 215015 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:47:26,080-Speed 128.73 samples/sec Loss 0.3784 Epoch: 21 Global Step: 215020 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:47:45,651-Speed 130.82 samples/sec Loss 0.4039 Epoch: 21 Global Step: 215025 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:48:05,489-Speed 129.05 samples/sec Loss 0.3416 Epoch: 21 Global Step: 215030 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:48:25,106-Speed 130.51 samples/sec Loss 0.3520 Epoch: 21 Global Step: 215035 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:48:44,649-Speed 131.00 samples/sec Loss 0.3468 Epoch: 21 Global Step: 215040 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:49:04,593-Speed 128.37 samples/sec Loss 0.3787 Epoch: 21 Global Step: 215045 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:49:24,153-Speed 130.89 samples/sec Loss 0.3052 Epoch: 21 Global Step: 215050 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:49:43,496-Speed 132.36 samples/sec Loss 0.3724 Epoch: 21 Global Step: 215055 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:50:03,352-Speed 128.94 samples/sec Loss 0.4036 Epoch: 21 Global Step: 215060 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:50:23,231-Speed 128.78 samples/sec Loss 0.3271 Epoch: 21 Global Step: 215065 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:50:43,031-Speed 129.30 samples/sec Loss 0.4310 Epoch: 21 Global Step: 215070 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:51:02,940-Speed 128.59 samples/sec Loss 0.3449 Epoch: 21 Global Step: 215075 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:51:22,550-Speed 130.56 samples/sec Loss 0.3777 Epoch: 21 Global Step: 215080 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:51:42,200-Speed 130.28 samples/sec Loss 0.4062 Epoch: 21 Global Step: 215085 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:52:02,073-Speed 128.83 samples/sec Loss 0.3666 Epoch: 21 Global Step: 215090 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:52:21,862-Speed 129.37 samples/sec Loss 0.3375 Epoch: 21 Global Step: 215095 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:52:41,894-Speed 127.80 samples/sec Loss 0.4093 Epoch: 21 Global Step: 215100 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:53:01,835-Speed 128.39 samples/sec Loss 0.3179 Epoch: 21 Global Step: 215105 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:53:21,711-Speed 128.81 samples/sec Loss 0.4062 Epoch: 21 Global Step: 215110 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:53:41,559-Speed 128.99 samples/sec Loss 0.3820 Epoch: 21 Global Step: 215115 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:54:01,532-Speed 128.18 samples/sec Loss 0.3512 Epoch: 21 Global Step: 215120 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:54:21,159-Speed 130.44 samples/sec Loss 0.3515 Epoch: 21 Global Step: 215125 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:54:41,052-Speed 128.70 samples/sec Loss 0.4064 Epoch: 21 Global Step: 215130 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:55:00,836-Speed 129.41 samples/sec Loss 0.3241 Epoch: 21 Global Step: 215135 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:55:20,667-Speed 129.10 samples/sec Loss 0.4289 Epoch: 21 Global Step: 215140 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:55:40,630-Speed 128.24 samples/sec Loss 0.3871 Epoch: 21 Global Step: 215145 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:56:00,637-Speed 127.96 samples/sec Loss 0.3878 Epoch: 21 Global Step: 215150 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 12:56:20,437-Speed 129.30 samples/sec Loss 0.3582 Epoch: 21 Global Step: 215155 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:56:40,359-Speed 128.51 samples/sec Loss 0.3415 Epoch: 21 Global Step: 215160 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:57:00,142-Speed 129.41 samples/sec Loss 0.4087 Epoch: 21 Global Step: 215165 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:57:19,766-Speed 130.46 samples/sec Loss 0.3639 Epoch: 21 Global Step: 215170 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:57:39,700-Speed 128.43 samples/sec Loss 0.3670 Epoch: 21 Global Step: 215175 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:57:59,453-Speed 129.61 samples/sec Loss 0.4049 Epoch: 21 Global Step: 215180 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:58:19,206-Speed 129.61 samples/sec Loss 0.3453 Epoch: 21 Global Step: 215185 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:58:39,248-Speed 127.74 samples/sec Loss 0.3479 Epoch: 21 Global Step: 215190 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:58:59,321-Speed 127.54 samples/sec Loss 0.3292 Epoch: 21 Global Step: 215195 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:59:19,062-Speed 129.69 samples/sec Loss 0.3648 Epoch: 21 Global Step: 215200 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:59:38,816-Speed 129.60 samples/sec Loss 0.3574 Epoch: 21 Global Step: 215205 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 12:59:58,679-Speed 128.89 samples/sec Loss 0.4216 Epoch: 21 Global Step: 215210 Fp16 Grad Scale: 8192 Required: 41 hours Training: 2021-06-17 13:00:17,872-Speed 133.39 samples/sec Loss 0.4409 Epoch: 21 Global Step: 215215 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:00:37,812-Speed 128.39 samples/sec Loss 0.3469 Epoch: 21 Global Step: 215220 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:00:57,552-Speed 129.70 samples/sec Loss 0.3773 Epoch: 21 Global Step: 215225 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:01:17,437-Speed 128.75 samples/sec Loss 0.3830 Epoch: 21 Global Step: 215230 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:01:37,377-Speed 128.39 samples/sec Loss 0.3443 Epoch: 21 Global Step: 215235 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:01:57,144-Speed 129.52 samples/sec Loss 0.3701 Epoch: 21 Global Step: 215240 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:02:16,967-Speed 129.15 samples/sec Loss 0.3237 Epoch: 21 Global Step: 215245 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:02:36,753-Speed 129.39 samples/sec Loss 0.3865 Epoch: 21 Global Step: 215250 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:02:56,377-Speed 130.46 samples/sec Loss 0.4074 Epoch: 21 Global Step: 215255 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:03:16,080-Speed 129.93 samples/sec Loss 0.3618 Epoch: 21 Global Step: 215260 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:03:35,879-Speed 129.30 samples/sec Loss 0.3277 Epoch: 21 Global Step: 215265 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:03:55,617-Speed 129.70 samples/sec Loss 0.3461 Epoch: 21 Global Step: 215270 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:04:15,320-Speed 129.94 samples/sec Loss 0.3705 Epoch: 21 Global Step: 215275 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:04:35,159-Speed 129.05 samples/sec Loss 0.3410 Epoch: 21 Global Step: 215280 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:04:54,962-Speed 129.28 samples/sec Loss 0.3856 Epoch: 21 Global Step: 215285 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:05:14,518-Speed 130.92 samples/sec Loss 0.4230 Epoch: 21 Global Step: 215290 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:05:34,268-Speed 129.62 samples/sec Loss 0.4195 Epoch: 21 Global Step: 215295 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:05:53,871-Speed 130.60 samples/sec Loss 0.3588 Epoch: 21 Global Step: 215300 Fp16 Grad Scale: 4096 Required: 41 hours Training: 2021-06-17 13:06:13,576-Speed 129.92 samples/sec Loss 0.4156 Epoch: 21 Global Step: 215305 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:06:33,376-Speed 129.30 samples/sec Loss 0.3904 Epoch: 21 Global Step: 215310 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:06:53,348-Speed 128.18 samples/sec Loss 0.3581 Epoch: 21 Global Step: 215315 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:07:12,779-Speed 131.76 samples/sec Loss 0.3674 Epoch: 21 Global Step: 215320 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:07:32,583-Speed 129.27 samples/sec Loss 0.3524 Epoch: 21 Global Step: 215325 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:07:52,461-Speed 128.79 samples/sec Loss 0.3441 Epoch: 21 Global Step: 215330 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:08:11,954-Speed 131.34 samples/sec Loss 0.3595 Epoch: 21 Global Step: 215335 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:08:31,779-Speed 129.13 samples/sec Loss 0.4069 Epoch: 21 Global Step: 215340 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:08:51,349-Speed 130.82 samples/sec Loss 0.3695 Epoch: 21 Global Step: 215345 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:09:11,128-Speed 129.44 samples/sec Loss 0.3295 Epoch: 21 Global Step: 215350 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:09:30,926-Speed 129.32 samples/sec Loss 0.4159 Epoch: 21 Global Step: 215355 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:09:50,815-Speed 128.72 samples/sec Loss 0.3502 Epoch: 21 Global Step: 215360 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:10:10,249-Speed 131.73 samples/sec Loss 0.3478 Epoch: 21 Global Step: 215365 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:10:30,020-Speed 129.49 samples/sec Loss 0.3429 Epoch: 21 Global Step: 215370 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:10:49,921-Speed 128.64 samples/sec Loss 0.3387 Epoch: 21 Global Step: 215375 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:11:09,450-Speed 131.09 samples/sec Loss 0.3335 Epoch: 21 Global Step: 215380 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:11:29,346-Speed 128.68 samples/sec Loss 0.3607 Epoch: 21 Global Step: 215385 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:11:48,786-Speed 131.69 samples/sec Loss 0.3452 Epoch: 21 Global Step: 215390 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:12:08,522-Speed 129.72 samples/sec Loss 0.4197 Epoch: 21 Global Step: 215395 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:12:28,276-Speed 129.60 samples/sec Loss 0.3521 Epoch: 21 Global Step: 215400 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:12:48,233-Speed 128.28 samples/sec Loss 0.4192 Epoch: 21 Global Step: 215405 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:13:07,654-Speed 131.83 samples/sec Loss 0.3930 Epoch: 21 Global Step: 215410 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:13:27,378-Speed 129.79 samples/sec Loss 0.4039 Epoch: 21 Global Step: 215415 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:13:47,238-Speed 128.91 samples/sec Loss 0.3352 Epoch: 21 Global Step: 215420 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:14:06,681-Speed 131.68 samples/sec Loss 0.3554 Epoch: 21 Global Step: 215425 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:14:26,592-Speed 128.58 samples/sec Loss 0.3364 Epoch: 21 Global Step: 215430 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:14:46,182-Speed 130.68 samples/sec Loss 0.4402 Epoch: 21 Global Step: 215435 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:15:06,084-Speed 128.64 samples/sec Loss 0.3789 Epoch: 21 Global Step: 215440 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:15:25,871-Speed 129.39 samples/sec Loss 0.3601 Epoch: 21 Global Step: 215445 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:15:45,786-Speed 128.55 samples/sec Loss 0.3714 Epoch: 21 Global Step: 215450 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:16:05,255-Speed 131.49 samples/sec Loss 0.4213 Epoch: 21 Global Step: 215455 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:16:25,146-Speed 128.71 samples/sec Loss 0.3790 Epoch: 21 Global Step: 215460 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:16:44,914-Speed 129.51 samples/sec Loss 0.3977 Epoch: 21 Global Step: 215465 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:17:04,440-Speed 131.12 samples/sec Loss 0.4307 Epoch: 21 Global Step: 215470 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:17:24,240-Speed 129.30 samples/sec Loss 0.4220 Epoch: 21 Global Step: 215475 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:17:43,769-Speed 131.09 samples/sec Loss 0.4067 Epoch: 21 Global Step: 215480 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:18:03,652-Speed 128.77 samples/sec Loss 0.3171 Epoch: 21 Global Step: 215485 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:18:23,461-Speed 129.24 samples/sec Loss 0.3943 Epoch: 21 Global Step: 215490 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:18:43,313-Speed 128.96 samples/sec Loss 0.3719 Epoch: 21 Global Step: 215495 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:19:02,736-Speed 131.80 samples/sec Loss 0.4018 Epoch: 21 Global Step: 215500 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:19:22,647-Speed 128.58 samples/sec Loss 0.4507 Epoch: 21 Global Step: 215505 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:19:42,368-Speed 129.82 samples/sec Loss 0.3321 Epoch: 21 Global Step: 215510 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:20:01,980-Speed 130.54 samples/sec Loss 0.4112 Epoch: 21 Global Step: 215515 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:20:21,600-Speed 130.49 samples/sec Loss 0.3756 Epoch: 21 Global Step: 215520 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:20:41,136-Speed 131.05 samples/sec Loss 0.3738 Epoch: 21 Global Step: 215525 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:21:01,050-Speed 128.56 samples/sec Loss 0.3468 Epoch: 21 Global Step: 215530 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:21:21,042-Speed 128.06 samples/sec Loss 0.3547 Epoch: 21 Global Step: 215535 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:21:41,143-Speed 127.36 samples/sec Loss 0.3022 Epoch: 21 Global Step: 215540 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:22:00,611-Speed 131.50 samples/sec Loss 0.3547 Epoch: 21 Global Step: 215545 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:22:20,466-Speed 128.94 samples/sec Loss 0.3598 Epoch: 21 Global Step: 215550 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:22:40,206-Speed 129.69 samples/sec Loss 0.3814 Epoch: 21 Global Step: 215555 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:23:00,071-Speed 128.88 samples/sec Loss 0.3777 Epoch: 21 Global Step: 215560 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:23:19,676-Speed 130.59 samples/sec Loss 0.3395 Epoch: 21 Global Step: 215565 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:23:38,757-Speed 134.18 samples/sec Loss 0.3170 Epoch: 21 Global Step: 215570 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:23:58,639-Speed 128.77 samples/sec Loss 0.3872 Epoch: 21 Global Step: 215575 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:24:18,550-Speed 128.57 samples/sec Loss 0.3391 Epoch: 21 Global Step: 215580 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:24:38,272-Speed 129.81 samples/sec Loss 0.3905 Epoch: 21 Global Step: 215585 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:24:57,754-Speed 131.41 samples/sec Loss 0.3850 Epoch: 21 Global Step: 215590 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:25:17,677-Speed 128.50 samples/sec Loss 0.3753 Epoch: 21 Global Step: 215595 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:25:37,430-Speed 129.61 samples/sec Loss 0.3782 Epoch: 21 Global Step: 215600 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:25:57,195-Speed 129.53 samples/sec Loss 0.3768 Epoch: 21 Global Step: 215605 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:26:16,805-Speed 130.55 samples/sec Loss 0.3425 Epoch: 21 Global Step: 215610 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:26:36,381-Speed 130.78 samples/sec Loss 0.4067 Epoch: 21 Global Step: 215615 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:26:56,345-Speed 128.24 samples/sec Loss 0.3520 Epoch: 21 Global Step: 215620 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:27:16,457-Speed 127.30 samples/sec Loss 0.3935 Epoch: 21 Global Step: 215625 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:27:36,139-Speed 130.08 samples/sec Loss 0.4037 Epoch: 21 Global Step: 215630 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:27:55,740-Speed 130.61 samples/sec Loss 0.4159 Epoch: 21 Global Step: 215635 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:28:15,625-Speed 128.75 samples/sec Loss 0.3821 Epoch: 21 Global Step: 215640 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:28:35,371-Speed 129.65 samples/sec Loss 0.4188 Epoch: 21 Global Step: 215645 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:28:54,928-Speed 130.90 samples/sec Loss 0.3547 Epoch: 21 Global Step: 215650 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:29:14,553-Speed 130.45 samples/sec Loss 0.3400 Epoch: 21 Global Step: 215655 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:29:34,179-Speed 130.45 samples/sec Loss 0.3425 Epoch: 21 Global Step: 215660 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:29:53,930-Speed 129.62 samples/sec Loss 0.3018 Epoch: 21 Global Step: 215665 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:30:13,897-Speed 128.22 samples/sec Loss 0.3990 Epoch: 21 Global Step: 215670 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:30:33,506-Speed 130.55 samples/sec Loss 0.4152 Epoch: 21 Global Step: 215675 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:30:53,134-Speed 130.43 samples/sec Loss 0.4009 Epoch: 21 Global Step: 215680 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:31:12,988-Speed 128.95 samples/sec Loss 0.3768 Epoch: 21 Global Step: 215685 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:31:32,740-Speed 129.61 samples/sec Loss 0.3194 Epoch: 21 Global Step: 215690 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:31:52,501-Speed 129.56 samples/sec Loss 0.3875 Epoch: 21 Global Step: 215695 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:32:12,114-Speed 130.53 samples/sec Loss 0.2715 Epoch: 21 Global Step: 215700 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:32:31,720-Speed 130.57 samples/sec Loss 0.3784 Epoch: 21 Global Step: 215705 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:32:51,066-Speed 132.34 samples/sec Loss 0.4193 Epoch: 21 Global Step: 215710 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:33:11,155-Speed 127.44 samples/sec Loss 0.4391 Epoch: 21 Global Step: 215715 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:33:30,796-Speed 130.34 samples/sec Loss 0.3966 Epoch: 21 Global Step: 215720 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:33:50,648-Speed 128.96 samples/sec Loss 0.3763 Epoch: 21 Global Step: 215725 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:34:10,520-Speed 128.83 samples/sec Loss 0.3473 Epoch: 21 Global Step: 215730 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:34:30,293-Speed 129.48 samples/sec Loss 0.4217 Epoch: 21 Global Step: 215735 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:34:50,052-Speed 129.57 samples/sec Loss 0.3571 Epoch: 21 Global Step: 215740 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:35:09,665-Speed 130.54 samples/sec Loss 0.3108 Epoch: 21 Global Step: 215745 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:35:29,775-Speed 127.30 samples/sec Loss 0.3373 Epoch: 21 Global Step: 215750 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:35:49,779-Speed 127.98 samples/sec Loss 0.3788 Epoch: 21 Global Step: 215755 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:36:09,841-Speed 127.61 samples/sec Loss 0.3225 Epoch: 21 Global Step: 215760 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:36:29,480-Speed 130.36 samples/sec Loss 0.3984 Epoch: 21 Global Step: 215765 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:36:49,723-Speed 126.47 samples/sec Loss 0.3306 Epoch: 21 Global Step: 215770 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:37:09,446-Speed 129.80 samples/sec Loss 0.3970 Epoch: 21 Global Step: 215775 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:37:29,301-Speed 128.94 samples/sec Loss 0.3454 Epoch: 21 Global Step: 215780 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:37:49,054-Speed 129.60 samples/sec Loss 0.3746 Epoch: 21 Global Step: 215785 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:38:08,737-Speed 130.07 samples/sec Loss 0.3439 Epoch: 21 Global Step: 215790 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:38:28,645-Speed 128.60 samples/sec Loss 0.3586 Epoch: 21 Global Step: 215795 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:38:48,649-Speed 127.98 samples/sec Loss 0.3988 Epoch: 21 Global Step: 215800 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:39:08,672-Speed 127.86 samples/sec Loss 0.3249 Epoch: 21 Global Step: 215805 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:39:28,262-Speed 130.68 samples/sec Loss 0.3083 Epoch: 21 Global Step: 215810 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:39:48,166-Speed 128.62 samples/sec Loss 0.4441 Epoch: 21 Global Step: 215815 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:40:07,924-Speed 129.58 samples/sec Loss 0.3425 Epoch: 21 Global Step: 215820 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:40:27,751-Speed 129.12 samples/sec Loss 0.3926 Epoch: 21 Global Step: 215825 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:40:47,572-Speed 129.16 samples/sec Loss 0.3453 Epoch: 21 Global Step: 215830 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:41:07,213-Speed 130.35 samples/sec Loss 0.3465 Epoch: 21 Global Step: 215835 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:41:27,111-Speed 128.66 samples/sec Loss 0.3033 Epoch: 21 Global Step: 215840 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:41:47,259-Speed 127.06 samples/sec Loss 0.3608 Epoch: 21 Global Step: 215845 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:42:07,593-Speed 125.91 samples/sec Loss 0.3072 Epoch: 21 Global Step: 215850 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:42:27,334-Speed 129.68 samples/sec Loss 0.2999 Epoch: 21 Global Step: 215855 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:42:47,200-Speed 128.87 samples/sec Loss 0.3724 Epoch: 21 Global Step: 215860 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:43:06,925-Speed 129.80 samples/sec Loss 0.3431 Epoch: 21 Global Step: 215865 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:43:26,912-Speed 128.09 samples/sec Loss 0.3473 Epoch: 21 Global Step: 215870 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:43:46,637-Speed 129.79 samples/sec Loss 0.3231 Epoch: 21 Global Step: 215875 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:44:06,377-Speed 129.69 samples/sec Loss 0.3351 Epoch: 21 Global Step: 215880 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:44:26,193-Speed 129.20 samples/sec Loss 0.3657 Epoch: 21 Global Step: 215885 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:44:46,314-Speed 127.23 samples/sec Loss 0.4285 Epoch: 21 Global Step: 215890 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:45:06,369-Speed 127.66 samples/sec Loss 0.3042 Epoch: 21 Global Step: 215895 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:45:26,163-Speed 129.34 samples/sec Loss 0.4839 Epoch: 21 Global Step: 215900 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:45:45,901-Speed 129.70 samples/sec Loss 0.3397 Epoch: 21 Global Step: 215905 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:46:05,652-Speed 129.62 samples/sec Loss 0.3156 Epoch: 21 Global Step: 215910 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:46:25,581-Speed 128.47 samples/sec Loss 0.3483 Epoch: 21 Global Step: 215915 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:46:45,280-Speed 129.96 samples/sec Loss 0.4355 Epoch: 21 Global Step: 215920 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:47:05,168-Speed 128.73 samples/sec Loss 0.3467 Epoch: 21 Global Step: 215925 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:47:24,878-Speed 129.89 samples/sec Loss 0.3589 Epoch: 21 Global Step: 215930 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:47:45,011-Speed 127.16 samples/sec Loss 0.3989 Epoch: 21 Global Step: 215935 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:48:05,015-Speed 127.98 samples/sec Loss 0.2501 Epoch: 21 Global Step: 215940 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:48:24,887-Speed 128.83 samples/sec Loss 0.3209 Epoch: 21 Global Step: 215945 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:48:44,616-Speed 129.77 samples/sec Loss 0.3505 Epoch: 21 Global Step: 215950 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:49:04,506-Speed 128.71 samples/sec Loss 0.3952 Epoch: 21 Global Step: 215955 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:49:24,305-Speed 129.31 samples/sec Loss 0.3416 Epoch: 21 Global Step: 215960 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:49:44,048-Speed 129.67 samples/sec Loss 0.3249 Epoch: 21 Global Step: 215965 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:50:03,916-Speed 128.85 samples/sec Loss 0.3454 Epoch: 21 Global Step: 215970 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:50:23,603-Speed 130.05 samples/sec Loss 0.3495 Epoch: 21 Global Step: 215975 Fp16 Grad Scale: 8192 Required: 40 hours Training: 2021-06-17 13:50:43,404-Speed 129.29 samples/sec Loss 0.3839 Epoch: 21 Global Step: 215980 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:51:03,134-Speed 129.75 samples/sec Loss 0.4026 Epoch: 21 Global Step: 215985 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:51:22,877-Speed 129.67 samples/sec Loss 0.3846 Epoch: 21 Global Step: 215990 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:51:42,738-Speed 128.91 samples/sec Loss 0.4181 Epoch: 21 Global Step: 215995 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:52:02,825-Speed 127.45 samples/sec Loss 0.3319 Epoch: 21 Global Step: 216000 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:52:22,331-Speed 131.25 samples/sec Loss 0.3702 Epoch: 21 Global Step: 216005 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:52:42,380-Speed 127.70 samples/sec Loss 0.2810 Epoch: 21 Global Step: 216010 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:53:02,203-Speed 129.15 samples/sec Loss 0.3661 Epoch: 21 Global Step: 216015 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:53:21,813-Speed 130.55 samples/sec Loss 0.3804 Epoch: 21 Global Step: 216020 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:53:41,612-Speed 129.31 samples/sec Loss 0.3408 Epoch: 21 Global Step: 216025 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:54:01,400-Speed 129.38 samples/sec Loss 0.3457 Epoch: 21 Global Step: 216030 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:54:21,037-Speed 130.37 samples/sec Loss 0.3592 Epoch: 21 Global Step: 216035 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:54:40,748-Speed 129.89 samples/sec Loss 0.3691 Epoch: 21 Global Step: 216040 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:55:00,300-Speed 130.94 samples/sec Loss 0.3687 Epoch: 21 Global Step: 216045 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:55:20,062-Speed 129.54 samples/sec Loss 0.3065 Epoch: 21 Global Step: 216050 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:55:39,981-Speed 128.53 samples/sec Loss 0.3869 Epoch: 21 Global Step: 216055 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:55:59,710-Speed 129.76 samples/sec Loss 0.3229 Epoch: 21 Global Step: 216060 Fp16 Grad Scale: 4096 Required: 40 hours Training: 2021-06-17 13:56:18,822-Speed 133.96 samples/sec Loss 0.3961 Epoch: 21 Global Step: 216065 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:56:38,669-Speed 128.99 samples/sec Loss 0.3498 Epoch: 21 Global Step: 216070 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:56:58,439-Speed 129.50 samples/sec Loss 0.3344 Epoch: 21 Global Step: 216075 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:57:18,062-Speed 130.47 samples/sec Loss 0.3298 Epoch: 21 Global Step: 216080 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:57:37,766-Speed 129.93 samples/sec Loss 0.3137 Epoch: 21 Global Step: 216085 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:57:57,356-Speed 130.69 samples/sec Loss 0.3483 Epoch: 21 Global Step: 216090 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:58:16,985-Speed 130.43 samples/sec Loss 0.3607 Epoch: 21 Global Step: 216095 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:58:36,950-Speed 128.23 samples/sec Loss 0.3451 Epoch: 21 Global Step: 216100 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:58:56,654-Speed 129.92 samples/sec Loss 0.4025 Epoch: 21 Global Step: 216105 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:59:16,240-Speed 130.71 samples/sec Loss 0.3453 Epoch: 21 Global Step: 216110 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:59:36,094-Speed 128.95 samples/sec Loss 0.3696 Epoch: 21 Global Step: 216115 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 13:59:55,883-Speed 129.37 samples/sec Loss 0.3807 Epoch: 21 Global Step: 216120 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 14:00:15,489-Speed 130.58 samples/sec Loss 0.3528 Epoch: 21 Global Step: 216125 Fp16 Grad Scale: 2048 Required: 40 hours Training: 2021-06-17 14:00:35,176-Speed 130.04 samples/sec Loss 0.3392 Epoch: 21 Global Step: 216130 Fp16 Grad Scale: 2048 Required: 39 hours Training: 2021-06-17 14:00:54,712-Speed 131.05 samples/sec Loss 0.3375 Epoch: 21 Global Step: 216135 Fp16 Grad Scale: 2048 Required: 39 hours Training: 2021-06-17 14:01:14,396-Speed 130.06 samples/sec Loss 0.3875 Epoch: 21 Global Step: 216140 Fp16 Grad Scale: 2048 Required: 39 hours Training: 2021-06-17 14:01:34,386-Speed 128.07 samples/sec Loss 0.3984 Epoch: 21 Global Step: 216145 Fp16 Grad Scale: 2048 Required: 39 hours Training: 2021-06-17 14:01:54,123-Speed 129.71 samples/sec Loss 0.4596 Epoch: 21 Global Step: 216150 Fp16 Grad Scale: 2048 Required: 39 hours Training: 2021-06-17 14:02:13,714-Speed 130.68 samples/sec Loss 0.3560 Epoch: 21 Global Step: 216155 Fp16 Grad Scale: 2048 Required: 39 hours Training: 2021-06-17 14:02:33,553-Speed 129.04 samples/sec Loss 0.3264 Epoch: 21 Global Step: 216160 Fp16 Grad Scale: 2048 Required: 39 hours Training: 2021-06-17 14:02:53,334-Speed 129.43 samples/sec Loss 0.3955 Epoch: 21 Global Step: 216165 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:03:12,941-Speed 130.57 samples/sec Loss 0.3799 Epoch: 21 Global Step: 216170 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:03:32,677-Speed 129.72 samples/sec Loss 0.3467 Epoch: 21 Global Step: 216175 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:03:52,221-Speed 130.99 samples/sec Loss 0.3471 Epoch: 21 Global Step: 216180 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:04:11,905-Speed 130.06 samples/sec Loss 0.3387 Epoch: 21 Global Step: 216185 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:04:31,887-Speed 128.12 samples/sec Loss 0.3896 Epoch: 21 Global Step: 216190 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:04:51,609-Speed 129.81 samples/sec Loss 0.3531 Epoch: 21 Global Step: 216195 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:05:11,197-Speed 130.70 samples/sec Loss 0.3649 Epoch: 21 Global Step: 216200 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:05:30,959-Speed 129.54 samples/sec Loss 0.3743 Epoch: 21 Global Step: 216205 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:05:50,696-Speed 129.72 samples/sec Loss 0.3633 Epoch: 21 Global Step: 216210 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:06:10,305-Speed 130.55 samples/sec Loss 0.4216 Epoch: 21 Global Step: 216215 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:06:30,041-Speed 129.72 samples/sec Loss 0.4150 Epoch: 21 Global Step: 216220 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:06:49,565-Speed 131.13 samples/sec Loss 0.4331 Epoch: 21 Global Step: 216225 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:07:09,311-Speed 129.66 samples/sec Loss 0.3533 Epoch: 21 Global Step: 216230 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:07:29,224-Speed 128.56 samples/sec Loss 0.3938 Epoch: 21 Global Step: 216235 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:07:48,963-Speed 129.70 samples/sec Loss 0.3475 Epoch: 21 Global Step: 216240 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:08:08,554-Speed 130.68 samples/sec Loss 0.3071 Epoch: 21 Global Step: 216245 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:08:28,320-Speed 129.52 samples/sec Loss 0.3771 Epoch: 21 Global Step: 216250 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:08:48,079-Speed 129.57 samples/sec Loss 0.3439 Epoch: 21 Global Step: 216255 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:09:07,977-Speed 128.66 samples/sec Loss 0.4048 Epoch: 21 Global Step: 216260 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:09:27,727-Speed 129.63 samples/sec Loss 0.4179 Epoch: 21 Global Step: 216265 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:09:47,337-Speed 130.55 samples/sec Loss 0.3948 Epoch: 21 Global Step: 216270 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:10:06,956-Speed 130.49 samples/sec Loss 0.4257 Epoch: 21 Global Step: 216275 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:10:26,357-Speed 131.96 samples/sec Loss 0.3687 Epoch: 21 Global Step: 216280 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:10:46,076-Speed 129.83 samples/sec Loss 0.3440 Epoch: 21 Global Step: 216285 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:11:05,946-Speed 128.84 samples/sec Loss 0.3857 Epoch: 21 Global Step: 216290 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:11:25,558-Speed 130.54 samples/sec Loss 0.3594 Epoch: 21 Global Step: 216295 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:11:45,355-Speed 129.32 samples/sec Loss 0.3500 Epoch: 21 Global Step: 216300 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:12:05,412-Speed 127.64 samples/sec Loss 0.3054 Epoch: 21 Global Step: 216305 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:12:25,255-Speed 129.02 samples/sec Loss 0.3387 Epoch: 21 Global Step: 216310 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:12:45,023-Speed 129.51 samples/sec Loss 0.4280 Epoch: 21 Global Step: 216315 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:13:04,728-Speed 129.93 samples/sec Loss 0.4347 Epoch: 21 Global Step: 216320 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:13:24,804-Speed 127.53 samples/sec Loss 0.3177 Epoch: 21 Global Step: 216325 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:13:44,550-Speed 129.65 samples/sec Loss 0.3378 Epoch: 21 Global Step: 216330 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:14:04,428-Speed 128.79 samples/sec Loss 0.3412 Epoch: 21 Global Step: 216335 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:14:24,042-Speed 130.52 samples/sec Loss 0.3656 Epoch: 21 Global Step: 216340 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:14:43,834-Speed 129.35 samples/sec Loss 0.3343 Epoch: 21 Global Step: 216345 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:15:03,870-Speed 127.78 samples/sec Loss 0.4072 Epoch: 21 Global Step: 216350 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:15:23,768-Speed 128.66 samples/sec Loss 0.4541 Epoch: 21 Global Step: 216355 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:15:43,425-Speed 130.24 samples/sec Loss 0.3900 Epoch: 21 Global Step: 216360 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:16:03,105-Speed 130.09 samples/sec Loss 0.4733 Epoch: 21 Global Step: 216365 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:16:23,160-Speed 127.65 samples/sec Loss 0.3214 Epoch: 21 Global Step: 216370 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:16:43,086-Speed 128.48 samples/sec Loss 0.4154 Epoch: 21 Global Step: 216375 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:17:02,744-Speed 130.23 samples/sec Loss 0.3216 Epoch: 21 Global Step: 216380 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:17:22,633-Speed 128.72 samples/sec Loss 0.3589 Epoch: 21 Global Step: 216385 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:17:42,281-Speed 130.30 samples/sec Loss 0.3872 Epoch: 21 Global Step: 216390 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:18:02,307-Speed 127.84 samples/sec Loss 0.3720 Epoch: 21 Global Step: 216395 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:18:22,181-Speed 128.82 samples/sec Loss 0.3422 Epoch: 21 Global Step: 216400 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:18:41,842-Speed 130.22 samples/sec Loss 0.3617 Epoch: 21 Global Step: 216405 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:19:01,729-Speed 128.73 samples/sec Loss 0.3694 Epoch: 21 Global Step: 216410 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:19:21,751-Speed 127.87 samples/sec Loss 0.3903 Epoch: 21 Global Step: 216415 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:19:41,608-Speed 128.92 samples/sec Loss 0.3066 Epoch: 21 Global Step: 216420 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:20:00,868-Speed 132.93 samples/sec Loss 0.4060 Epoch: 21 Global Step: 216425 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:20:20,686-Speed 129.18 samples/sec Loss 0.3626 Epoch: 21 Global Step: 216430 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:20:40,121-Speed 131.73 samples/sec Loss 0.3856 Epoch: 21 Global Step: 216435 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:20:59,949-Speed 129.11 samples/sec Loss 0.2985 Epoch: 21 Global Step: 216440 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:21:19,508-Speed 130.90 samples/sec Loss 0.3563 Epoch: 21 Global Step: 216445 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:21:38,989-Speed 131.41 samples/sec Loss 0.4252 Epoch: 21 Global Step: 216450 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:21:58,528-Speed 131.03 samples/sec Loss 0.3204 Epoch: 21 Global Step: 216455 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:22:18,257-Speed 129.76 samples/sec Loss 0.4570 Epoch: 21 Global Step: 216460 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:22:38,338-Speed 127.49 samples/sec Loss 0.3720 Epoch: 21 Global Step: 216465 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:22:58,068-Speed 129.76 samples/sec Loss 0.3365 Epoch: 21 Global Step: 216470 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:23:17,767-Speed 129.96 samples/sec Loss 0.3855 Epoch: 21 Global Step: 216475 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:23:37,620-Speed 128.96 samples/sec Loss 0.4381 Epoch: 21 Global Step: 216480 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:23:58,078-Speed 125.14 samples/sec Loss 0.3861 Epoch: 21 Global Step: 216485 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:24:17,805-Speed 129.78 samples/sec Loss 0.3015 Epoch: 21 Global Step: 216490 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:24:37,457-Speed 130.27 samples/sec Loss 0.3803 Epoch: 21 Global Step: 216495 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:24:57,291-Speed 129.08 samples/sec Loss 0.3633 Epoch: 21 Global Step: 216500 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:25:16,934-Speed 130.33 samples/sec Loss 0.3602 Epoch: 21 Global Step: 216505 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:25:36,825-Speed 128.71 samples/sec Loss 0.4690 Epoch: 21 Global Step: 216510 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:25:56,583-Speed 129.58 samples/sec Loss 0.2882 Epoch: 21 Global Step: 216515 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:26:16,948-Speed 125.71 samples/sec Loss 0.3081 Epoch: 21 Global Step: 216520 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:26:36,659-Speed 129.89 samples/sec Loss 0.4540 Epoch: 21 Global Step: 216525 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:26:57,001-Speed 125.85 samples/sec Loss 0.3924 Epoch: 21 Global Step: 216530 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:27:16,726-Speed 129.79 samples/sec Loss 0.3902 Epoch: 21 Global Step: 216535 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:27:36,539-Speed 129.22 samples/sec Loss 0.3470 Epoch: 21 Global Step: 216540 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:27:56,307-Speed 129.51 samples/sec Loss 0.3852 Epoch: 21 Global Step: 216545 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:28:16,041-Speed 129.73 samples/sec Loss 0.3738 Epoch: 21 Global Step: 216550 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:28:35,903-Speed 128.90 samples/sec Loss 0.3896 Epoch: 21 Global Step: 216555 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:28:55,699-Speed 129.32 samples/sec Loss 0.3261 Epoch: 21 Global Step: 216560 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:29:15,518-Speed 129.18 samples/sec Loss 0.3451 Epoch: 21 Global Step: 216565 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:29:35,220-Speed 129.94 samples/sec Loss 0.3416 Epoch: 21 Global Step: 216570 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:29:55,470-Speed 126.42 samples/sec Loss 0.2870 Epoch: 21 Global Step: 216575 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:30:15,261-Speed 129.36 samples/sec Loss 0.3129 Epoch: 21 Global Step: 216580 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:30:35,063-Speed 129.28 samples/sec Loss 0.4498 Epoch: 21 Global Step: 216585 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:30:54,543-Speed 131.43 samples/sec Loss 0.3537 Epoch: 21 Global Step: 216590 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:31:14,227-Speed 130.06 samples/sec Loss 0.3495 Epoch: 21 Global Step: 216595 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:31:34,106-Speed 128.79 samples/sec Loss 0.3435 Epoch: 21 Global Step: 216600 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:31:53,867-Speed 129.56 samples/sec Loss 0.3579 Epoch: 21 Global Step: 216605 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:32:13,675-Speed 129.25 samples/sec Loss 0.3679 Epoch: 21 Global Step: 216610 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:32:33,384-Speed 129.89 samples/sec Loss 0.3678 Epoch: 21 Global Step: 216615 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:32:53,682-Speed 126.13 samples/sec Loss 0.3870 Epoch: 21 Global Step: 216620 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:33:13,501-Speed 129.18 samples/sec Loss 0.3319 Epoch: 21 Global Step: 216625 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:33:33,350-Speed 128.98 samples/sec Loss 0.3640 Epoch: 21 Global Step: 216630 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:33:53,076-Speed 129.78 samples/sec Loss 0.3655 Epoch: 21 Global Step: 216635 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:34:12,727-Speed 130.28 samples/sec Loss 0.3383 Epoch: 21 Global Step: 216640 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:34:32,651-Speed 128.50 samples/sec Loss 0.3413 Epoch: 21 Global Step: 216645 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:34:52,333-Speed 130.07 samples/sec Loss 0.4012 Epoch: 21 Global Step: 216650 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:35:12,167-Speed 129.08 samples/sec Loss 0.4027 Epoch: 21 Global Step: 216655 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:35:31,890-Speed 129.81 samples/sec Loss 0.2892 Epoch: 21 Global Step: 216660 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:35:52,133-Speed 126.47 samples/sec Loss 0.4003 Epoch: 21 Global Step: 216665 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:36:11,959-Speed 129.13 samples/sec Loss 0.3938 Epoch: 21 Global Step: 216670 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:36:31,728-Speed 129.50 samples/sec Loss 0.3355 Epoch: 21 Global Step: 216675 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:36:51,434-Speed 129.92 samples/sec Loss 0.3884 Epoch: 21 Global Step: 216680 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:37:11,312-Speed 128.79 samples/sec Loss 0.4589 Epoch: 21 Global Step: 216685 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:37:31,052-Speed 129.69 samples/sec Loss 0.3532 Epoch: 21 Global Step: 216690 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:37:50,741-Speed 130.03 samples/sec Loss 0.3374 Epoch: 21 Global Step: 216695 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:38:10,606-Speed 128.88 samples/sec Loss 0.4096 Epoch: 21 Global Step: 216700 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:38:30,317-Speed 129.88 samples/sec Loss 0.4128 Epoch: 21 Global Step: 216705 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:38:50,868-Speed 124.58 samples/sec Loss 0.3076 Epoch: 21 Global Step: 216710 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:39:10,117-Speed 133.00 samples/sec Loss 0.3255 Epoch: 21 Global Step: 216715 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:39:29,721-Speed 130.59 samples/sec Loss 0.3088 Epoch: 21 Global Step: 216720 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:39:49,307-Speed 130.71 samples/sec Loss 0.3368 Epoch: 21 Global Step: 216725 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:40:08,913-Speed 130.58 samples/sec Loss 0.3803 Epoch: 21 Global Step: 216730 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:40:28,533-Speed 130.49 samples/sec Loss 0.3620 Epoch: 21 Global Step: 216735 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:40:48,173-Speed 130.35 samples/sec Loss 0.3747 Epoch: 21 Global Step: 216740 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:41:07,791-Speed 130.50 samples/sec Loss 0.4076 Epoch: 21 Global Step: 216745 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:41:27,485-Speed 129.99 samples/sec Loss 0.3983 Epoch: 21 Global Step: 216750 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:41:47,563-Speed 127.51 samples/sec Loss 0.3645 Epoch: 21 Global Step: 216755 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:42:07,335-Speed 129.48 samples/sec Loss 0.3897 Epoch: 21 Global Step: 216760 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:42:27,113-Speed 129.44 samples/sec Loss 0.3945 Epoch: 21 Global Step: 216765 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:42:46,837-Speed 129.80 samples/sec Loss 0.3569 Epoch: 21 Global Step: 216770 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:43:06,732-Speed 128.68 samples/sec Loss 0.3825 Epoch: 21 Global Step: 216775 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:43:26,768-Speed 127.78 samples/sec Loss 0.3002 Epoch: 21 Global Step: 216780 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:43:46,492-Speed 129.80 samples/sec Loss 0.3459 Epoch: 21 Global Step: 216785 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:44:06,340-Speed 128.99 samples/sec Loss 0.3307 Epoch: 21 Global Step: 216790 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:44:26,060-Speed 129.82 samples/sec Loss 0.3870 Epoch: 21 Global Step: 216795 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:44:46,322-Speed 126.36 samples/sec Loss 0.3236 Epoch: 21 Global Step: 216800 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:45:06,051-Speed 129.76 samples/sec Loss 0.3270 Epoch: 21 Global Step: 216805 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:45:25,943-Speed 128.70 samples/sec Loss 0.4467 Epoch: 21 Global Step: 216810 Fp16 Grad Scale: 4096 Required: 39 hours Training: 2021-06-17 14:45:45,755-Speed 129.22 samples/sec Loss 0.3917 Epoch: 21 Global Step: 216815 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:46:05,455-Speed 129.96 samples/sec Loss 0.3880 Epoch: 21 Global Step: 216820 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:46:25,602-Speed 127.07 samples/sec Loss 0.3392 Epoch: 21 Global Step: 216825 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:46:45,481-Speed 128.78 samples/sec Loss 0.3355 Epoch: 21 Global Step: 216830 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:47:05,346-Speed 128.88 samples/sec Loss 0.3524 Epoch: 21 Global Step: 216835 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:47:25,049-Speed 129.94 samples/sec Loss 0.3419 Epoch: 21 Global Step: 216840 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:47:45,210-Speed 126.98 samples/sec Loss 0.4200 Epoch: 21 Global Step: 216845 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:48:04,829-Speed 130.49 samples/sec Loss 0.3520 Epoch: 21 Global Step: 216850 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:48:24,734-Speed 128.62 samples/sec Loss 0.3779 Epoch: 21 Global Step: 216855 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:48:44,376-Speed 130.34 samples/sec Loss 0.3794 Epoch: 21 Global Step: 216860 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:49:04,274-Speed 128.66 samples/sec Loss 0.4051 Epoch: 21 Global Step: 216865 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:49:24,258-Speed 128.10 samples/sec Loss 0.3211 Epoch: 21 Global Step: 216870 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:49:43,899-Speed 130.35 samples/sec Loss 0.4235 Epoch: 21 Global Step: 216875 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:50:03,780-Speed 128.77 samples/sec Loss 0.4156 Epoch: 21 Global Step: 216880 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:50:23,490-Speed 129.89 samples/sec Loss 0.4328 Epoch: 21 Global Step: 216885 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:50:43,638-Speed 127.07 samples/sec Loss 0.4037 Epoch: 21 Global Step: 216890 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:51:03,356-Speed 129.83 samples/sec Loss 0.3378 Epoch: 21 Global Step: 216895 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:51:23,109-Speed 129.61 samples/sec Loss 0.4104 Epoch: 21 Global Step: 216900 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:51:42,858-Speed 129.63 samples/sec Loss 0.3481 Epoch: 21 Global Step: 216905 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:52:02,693-Speed 129.07 samples/sec Loss 0.4140 Epoch: 21 Global Step: 216910 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:52:22,720-Speed 127.83 samples/sec Loss 0.3711 Epoch: 21 Global Step: 216915 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:52:42,384-Speed 130.19 samples/sec Loss 0.3815 Epoch: 21 Global Step: 216920 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:53:02,242-Speed 128.92 samples/sec Loss 0.3892 Epoch: 21 Global Step: 216925 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:53:21,956-Speed 129.86 samples/sec Loss 0.4318 Epoch: 21 Global Step: 216930 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:53:42,115-Speed 127.00 samples/sec Loss 0.3730 Epoch: 21 Global Step: 216935 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:54:01,793-Speed 130.11 samples/sec Loss 0.3113 Epoch: 21 Global Step: 216940 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:54:21,831-Speed 127.76 samples/sec Loss 0.3881 Epoch: 21 Global Step: 216945 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:54:41,543-Speed 129.87 samples/sec Loss 0.4123 Epoch: 21 Global Step: 216950 Fp16 Grad Scale: 8192 Required: 39 hours Training: 2021-06-17 14:55:01,427-Speed 128.76 samples/sec Loss 0.3900 Epoch: 21 Global Step: 216955 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 14:55:21,473-Speed 127.71 samples/sec Loss 0.3585 Epoch: 21 Global Step: 216960 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 14:55:41,314-Speed 129.03 samples/sec Loss 0.4456 Epoch: 21 Global Step: 216965 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 14:56:00,618-Speed 132.62 samples/sec Loss 0.3870 Epoch: 21 Global Step: 216970 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 14:56:20,376-Speed 129.58 samples/sec Loss 0.3288 Epoch: 21 Global Step: 216975 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 14:56:39,967-Speed 130.68 samples/sec Loss 0.3419 Epoch: 21 Global Step: 216980 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 14:56:59,666-Speed 129.97 samples/sec Loss 0.3493 Epoch: 21 Global Step: 216985 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 14:57:19,447-Speed 129.42 samples/sec Loss 0.4085 Epoch: 21 Global Step: 216990 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 14:57:39,204-Speed 129.58 samples/sec Loss 0.4281 Epoch: 21 Global Step: 216995 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 14:57:58,830-Speed 130.45 samples/sec Loss 0.3964 Epoch: 21 Global Step: 217000 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 14:58:18,228-Speed 131.98 samples/sec Loss 0.4130 Epoch: 21 Global Step: 217005 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 14:58:38,146-Speed 128.53 samples/sec Loss 0.3534 Epoch: 21 Global Step: 217010 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 14:58:57,657-Speed 131.22 samples/sec Loss 0.3375 Epoch: 21 Global Step: 217015 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 14:59:17,494-Speed 129.05 samples/sec Loss 0.3076 Epoch: 21 Global Step: 217020 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 14:59:37,042-Speed 130.97 samples/sec Loss 0.3291 Epoch: 21 Global Step: 217025 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 14:59:56,796-Speed 129.61 samples/sec Loss 0.3646 Epoch: 21 Global Step: 217030 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:00:16,592-Speed 129.32 samples/sec Loss 0.4100 Epoch: 21 Global Step: 217035 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:00:36,439-Speed 128.99 samples/sec Loss 0.3257 Epoch: 21 Global Step: 217040 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:00:55,985-Speed 130.98 samples/sec Loss 0.3523 Epoch: 21 Global Step: 217045 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:01:15,744-Speed 129.57 samples/sec Loss 0.3997 Epoch: 21 Global Step: 217050 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:01:35,610-Speed 128.87 samples/sec Loss 0.3273 Epoch: 21 Global Step: 217055 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:01:55,073-Speed 131.54 samples/sec Loss 0.2960 Epoch: 21 Global Step: 217060 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:02:15,039-Speed 128.22 samples/sec Loss 0.3500 Epoch: 21 Global Step: 217065 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:02:34,582-Speed 131.00 samples/sec Loss 0.4158 Epoch: 21 Global Step: 217070 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:02:54,272-Speed 130.02 samples/sec Loss 0.4249 Epoch: 21 Global Step: 217075 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:03:14,037-Speed 129.53 samples/sec Loss 0.3840 Epoch: 21 Global Step: 217080 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:03:33,916-Speed 128.78 samples/sec Loss 0.3574 Epoch: 21 Global Step: 217085 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:03:53,344-Speed 131.77 samples/sec Loss 0.2923 Epoch: 21 Global Step: 217090 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:04:13,206-Speed 128.90 samples/sec Loss 0.3174 Epoch: 21 Global Step: 217095 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:04:33,070-Speed 128.88 samples/sec Loss 0.3404 Epoch: 21 Global Step: 217100 Fp16 Grad Scale: 2048 Required: 38 hours Training: 2021-06-17 15:04:52,567-Speed 131.31 samples/sec Loss 0.3575 Epoch: 21 Global Step: 217105 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:05:12,513-Speed 128.36 samples/sec Loss 0.3631 Epoch: 21 Global Step: 217110 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:05:31,946-Speed 131.74 samples/sec Loss 0.3434 Epoch: 21 Global Step: 217115 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:05:51,675-Speed 129.76 samples/sec Loss 0.4490 Epoch: 21 Global Step: 217120 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:06:11,462-Speed 129.38 samples/sec Loss 0.3820 Epoch: 21 Global Step: 217125 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:06:31,294-Speed 129.09 samples/sec Loss 0.2957 Epoch: 21 Global Step: 217130 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:06:50,852-Speed 130.90 samples/sec Loss 0.3300 Epoch: 21 Global Step: 217135 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:07:10,643-Speed 129.36 samples/sec Loss 0.4278 Epoch: 21 Global Step: 217140 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:07:30,481-Speed 129.05 samples/sec Loss 0.3815 Epoch: 21 Global Step: 217145 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:07:50,178-Speed 129.98 samples/sec Loss 0.3431 Epoch: 21 Global Step: 217150 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:08:10,093-Speed 128.56 samples/sec Loss 0.3356 Epoch: 21 Global Step: 217155 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:08:29,586-Speed 131.33 samples/sec Loss 0.2969 Epoch: 21 Global Step: 217160 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:08:49,376-Speed 129.37 samples/sec Loss 0.3946 Epoch: 21 Global Step: 217165 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:09:09,168-Speed 129.36 samples/sec Loss 0.3238 Epoch: 21 Global Step: 217170 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:09:29,010-Speed 129.02 samples/sec Loss 0.3235 Epoch: 21 Global Step: 217175 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:09:48,505-Speed 131.32 samples/sec Loss 0.4187 Epoch: 21 Global Step: 217180 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:10:08,378-Speed 128.83 samples/sec Loss 0.3556 Epoch: 21 Global Step: 217185 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:10:28,202-Speed 129.14 samples/sec Loss 0.3659 Epoch: 21 Global Step: 217190 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:10:47,736-Speed 131.06 samples/sec Loss 0.3280 Epoch: 21 Global Step: 217195 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:11:07,548-Speed 129.22 samples/sec Loss 0.4060 Epoch: 21 Global Step: 217200 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:11:27,110-Speed 130.87 samples/sec Loss 0.3990 Epoch: 21 Global Step: 217205 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:11:46,866-Speed 129.59 samples/sec Loss 0.3353 Epoch: 21 Global Step: 217210 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:12:06,785-Speed 128.53 samples/sec Loss 0.3885 Epoch: 21 Global Step: 217215 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:12:26,511-Speed 129.78 samples/sec Loss 0.3302 Epoch: 21 Global Step: 217220 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:12:46,231-Speed 129.83 samples/sec Loss 0.3464 Epoch: 21 Global Step: 217225 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:13:06,088-Speed 128.93 samples/sec Loss 0.3494 Epoch: 21 Global Step: 217230 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:13:25,897-Speed 129.24 samples/sec Loss 0.3299 Epoch: 21 Global Step: 217235 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:13:45,352-Speed 131.59 samples/sec Loss 0.3840 Epoch: 21 Global Step: 217240 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:14:05,190-Speed 129.05 samples/sec Loss 0.4022 Epoch: 21 Global Step: 217245 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:14:24,763-Speed 130.80 samples/sec Loss 0.3875 Epoch: 21 Global Step: 217250 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:14:44,487-Speed 129.80 samples/sec Loss 0.3918 Epoch: 21 Global Step: 217255 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:15:04,283-Speed 129.33 samples/sec Loss 0.3304 Epoch: 21 Global Step: 217260 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:15:24,068-Speed 129.40 samples/sec Loss 0.3006 Epoch: 21 Global Step: 217265 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:15:43,566-Speed 131.30 samples/sec Loss 0.3676 Epoch: 21 Global Step: 217270 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:16:03,537-Speed 128.19 samples/sec Loss 0.4124 Epoch: 21 Global Step: 217275 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:16:23,430-Speed 128.70 samples/sec Loss 0.3080 Epoch: 21 Global Step: 217280 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:16:42,848-Speed 131.85 samples/sec Loss 0.4026 Epoch: 21 Global Step: 217285 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:17:02,553-Speed 129.93 samples/sec Loss 0.3490 Epoch: 21 Global Step: 217290 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:17:21,676-Speed 133.89 samples/sec Loss 0.4292 Epoch: 21 Global Step: 217295 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:17:41,625-Speed 128.33 samples/sec Loss 0.3584 Epoch: 21 Global Step: 217300 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:18:01,728-Speed 127.35 samples/sec Loss 0.3250 Epoch: 21 Global Step: 217305 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:18:21,489-Speed 129.56 samples/sec Loss 0.3514 Epoch: 21 Global Step: 217310 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:18:41,177-Speed 130.04 samples/sec Loss 0.3433 Epoch: 21 Global Step: 217315 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:19:01,561-Speed 125.59 samples/sec Loss 0.3231 Epoch: 21 Global Step: 217320 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:19:21,435-Speed 128.82 samples/sec Loss 0.2866 Epoch: 21 Global Step: 217325 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:19:41,196-Speed 129.55 samples/sec Loss 0.3436 Epoch: 21 Global Step: 217330 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:20:00,877-Speed 130.09 samples/sec Loss 0.3268 Epoch: 21 Global Step: 217335 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:20:20,796-Speed 128.52 samples/sec Loss 0.3827 Epoch: 21 Global Step: 217340 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:20:40,735-Speed 128.40 samples/sec Loss 0.4466 Epoch: 21 Global Step: 217345 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:21:00,781-Speed 127.71 samples/sec Loss 0.3146 Epoch: 21 Global Step: 217350 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:21:20,578-Speed 129.32 samples/sec Loss 0.3595 Epoch: 21 Global Step: 217355 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:21:40,297-Speed 129.83 samples/sec Loss 0.4067 Epoch: 21 Global Step: 217360 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:22:00,089-Speed 129.35 samples/sec Loss 0.2833 Epoch: 21 Global Step: 217365 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:22:20,031-Speed 128.38 samples/sec Loss 0.3739 Epoch: 21 Global Step: 217370 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:22:39,653-Speed 130.47 samples/sec Loss 0.3158 Epoch: 21 Global Step: 217375 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:22:59,610-Speed 128.29 samples/sec Loss 0.3372 Epoch: 21 Global Step: 217380 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:23:19,315-Speed 129.92 samples/sec Loss 0.3924 Epoch: 21 Global Step: 217385 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:23:39,549-Speed 126.52 samples/sec Loss 0.3980 Epoch: 21 Global Step: 217390 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:23:59,411-Speed 128.90 samples/sec Loss 0.4018 Epoch: 21 Global Step: 217395 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:24:19,193-Speed 129.42 samples/sec Loss 0.3858 Epoch: 21 Global Step: 217400 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:24:38,904-Speed 129.88 samples/sec Loss 0.3522 Epoch: 21 Global Step: 217405 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:24:58,787-Speed 128.76 samples/sec Loss 0.3405 Epoch: 21 Global Step: 217410 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:25:18,511-Speed 129.80 samples/sec Loss 0.3630 Epoch: 21 Global Step: 217415 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:25:38,146-Speed 130.38 samples/sec Loss 0.3577 Epoch: 21 Global Step: 217420 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:25:58,054-Speed 128.60 samples/sec Loss 0.3340 Epoch: 21 Global Step: 217425 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:26:17,741-Speed 130.04 samples/sec Loss 0.3005 Epoch: 21 Global Step: 217430 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:26:37,935-Speed 126.78 samples/sec Loss 0.4198 Epoch: 21 Global Step: 217435 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:26:57,834-Speed 128.66 samples/sec Loss 0.3874 Epoch: 21 Global Step: 217440 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:27:17,634-Speed 129.30 samples/sec Loss 0.3165 Epoch: 21 Global Step: 217445 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:27:37,344-Speed 129.89 samples/sec Loss 0.3364 Epoch: 21 Global Step: 217450 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:27:57,254-Speed 128.58 samples/sec Loss 0.4447 Epoch: 21 Global Step: 217455 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:28:17,019-Speed 129.53 samples/sec Loss 0.3794 Epoch: 21 Global Step: 217460 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:28:36,644-Speed 130.46 samples/sec Loss 0.3309 Epoch: 21 Global Step: 217465 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:28:56,462-Speed 129.18 samples/sec Loss 0.4160 Epoch: 21 Global Step: 217470 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:29:16,142-Speed 130.09 samples/sec Loss 0.3771 Epoch: 21 Global Step: 217475 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:29:35,559-Speed 131.85 samples/sec Loss 0.3891 Epoch: 21 Global Step: 217480 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:29:55,562-Speed 127.98 samples/sec Loss 0.3731 Epoch: 21 Global Step: 217485 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:30:15,299-Speed 129.71 samples/sec Loss 0.3868 Epoch: 21 Global Step: 217490 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:30:35,064-Speed 129.53 samples/sec Loss 0.3536 Epoch: 21 Global Step: 217495 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:30:55,220-Speed 127.02 samples/sec Loss 0.3715 Epoch: 21 Global Step: 217500 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:31:14,755-Speed 131.06 samples/sec Loss 0.3500 Epoch: 21 Global Step: 217505 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:31:34,693-Speed 128.40 samples/sec Loss 0.3299 Epoch: 21 Global Step: 217510 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:31:54,399-Speed 129.92 samples/sec Loss 0.3688 Epoch: 21 Global Step: 217515 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:32:14,069-Speed 130.15 samples/sec Loss 0.4021 Epoch: 21 Global Step: 217520 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:32:34,145-Speed 127.52 samples/sec Loss 0.3722 Epoch: 21 Global Step: 217525 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:32:53,777-Speed 130.41 samples/sec Loss 0.4047 Epoch: 21 Global Step: 217530 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:33:13,386-Speed 130.56 samples/sec Loss 0.3837 Epoch: 21 Global Step: 217535 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:33:33,120-Speed 129.73 samples/sec Loss 0.3860 Epoch: 21 Global Step: 217540 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:33:52,727-Speed 130.58 samples/sec Loss 0.3713 Epoch: 21 Global Step: 217545 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:34:12,540-Speed 129.22 samples/sec Loss 0.4020 Epoch: 21 Global Step: 217550 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:34:32,488-Speed 128.34 samples/sec Loss 0.4117 Epoch: 21 Global Step: 217555 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:34:52,111-Speed 130.46 samples/sec Loss 0.3628 Epoch: 21 Global Step: 217560 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:35:11,875-Speed 129.54 samples/sec Loss 0.2915 Epoch: 21 Global Step: 217565 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:35:31,697-Speed 129.15 samples/sec Loss 0.4237 Epoch: 21 Global Step: 217570 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:35:51,355-Speed 130.24 samples/sec Loss 0.3612 Epoch: 21 Global Step: 217575 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:36:10,918-Speed 130.87 samples/sec Loss 0.3541 Epoch: 21 Global Step: 217580 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:36:30,690-Speed 129.48 samples/sec Loss 0.3896 Epoch: 21 Global Step: 217585 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:36:50,295-Speed 130.59 samples/sec Loss 0.3245 Epoch: 21 Global Step: 217590 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:37:09,609-Speed 132.56 samples/sec Loss 0.3455 Epoch: 21 Global Step: 217595 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:37:29,418-Speed 129.24 samples/sec Loss 0.4019 Epoch: 21 Global Step: 217600 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:37:49,133-Speed 129.85 samples/sec Loss 0.3570 Epoch: 21 Global Step: 217605 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:38:08,884-Speed 129.62 samples/sec Loss 0.4183 Epoch: 21 Global Step: 217610 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:38:28,760-Speed 128.81 samples/sec Loss 0.4407 Epoch: 21 Global Step: 217615 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:38:48,425-Speed 130.19 samples/sec Loss 0.2957 Epoch: 21 Global Step: 217620 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:39:07,981-Speed 130.91 samples/sec Loss 0.3730 Epoch: 21 Global Step: 217625 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:39:27,722-Speed 129.69 samples/sec Loss 0.3667 Epoch: 21 Global Step: 217630 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:39:47,329-Speed 130.57 samples/sec Loss 0.3855 Epoch: 21 Global Step: 217635 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:40:07,076-Speed 129.65 samples/sec Loss 0.3939 Epoch: 21 Global Step: 217640 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:40:26,875-Speed 129.30 samples/sec Loss 0.3167 Epoch: 21 Global Step: 217645 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:40:46,618-Speed 129.67 samples/sec Loss 0.3875 Epoch: 21 Global Step: 217650 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:41:06,729-Speed 127.30 samples/sec Loss 0.3314 Epoch: 21 Global Step: 217655 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:41:26,617-Speed 128.73 samples/sec Loss 0.3546 Epoch: 21 Global Step: 217660 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:41:46,237-Speed 130.49 samples/sec Loss 0.3410 Epoch: 21 Global Step: 217665 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:42:05,869-Speed 130.41 samples/sec Loss 0.3427 Epoch: 21 Global Step: 217670 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:42:25,610-Speed 129.69 samples/sec Loss 0.3361 Epoch: 21 Global Step: 217675 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:42:45,203-Speed 130.66 samples/sec Loss 0.4306 Epoch: 21 Global Step: 217680 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:43:04,907-Speed 129.94 samples/sec Loss 0.3611 Epoch: 21 Global Step: 217685 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:43:24,723-Speed 129.19 samples/sec Loss 0.3292 Epoch: 21 Global Step: 217690 Fp16 Grad Scale: 4096 Required: 38 hours Training: 2021-06-17 15:43:44,505-Speed 129.42 samples/sec Loss 0.3033 Epoch: 21 Global Step: 217695 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:44:04,213-Speed 129.90 samples/sec Loss 0.3571 Epoch: 21 Global Step: 217700 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:44:24,043-Speed 129.10 samples/sec Loss 0.3918 Epoch: 21 Global Step: 217705 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:44:43,665-Speed 130.47 samples/sec Loss 0.4075 Epoch: 21 Global Step: 217710 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:45:03,364-Speed 129.96 samples/sec Loss 0.4310 Epoch: 21 Global Step: 217715 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:45:23,161-Speed 129.32 samples/sec Loss 0.3716 Epoch: 21 Global Step: 217720 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:45:42,849-Speed 130.04 samples/sec Loss 0.3618 Epoch: 21 Global Step: 217725 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:46:02,573-Speed 129.79 samples/sec Loss 0.3685 Epoch: 21 Global Step: 217730 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:46:22,381-Speed 129.25 samples/sec Loss 0.2780 Epoch: 21 Global Step: 217735 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:46:42,126-Speed 129.66 samples/sec Loss 0.3701 Epoch: 21 Global Step: 217740 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:47:01,766-Speed 130.35 samples/sec Loss 0.3527 Epoch: 21 Global Step: 217745 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:47:21,638-Speed 128.83 samples/sec Loss 0.3353 Epoch: 21 Global Step: 217750 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:47:41,252-Speed 130.53 samples/sec Loss 0.3454 Epoch: 21 Global Step: 217755 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:48:00,848-Speed 130.64 samples/sec Loss 0.3171 Epoch: 21 Global Step: 217760 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:48:20,625-Speed 129.45 samples/sec Loss 0.3220 Epoch: 21 Global Step: 217765 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:48:40,232-Speed 130.57 samples/sec Loss 0.3982 Epoch: 21 Global Step: 217770 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:48:59,912-Speed 130.09 samples/sec Loss 0.3964 Epoch: 21 Global Step: 217775 Fp16 Grad Scale: 8192 Required: 38 hours Training: 2021-06-17 15:49:19,734-Speed 129.16 samples/sec Loss 0.3798 Epoch: 21 Global Step: 217780 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:49:39,462-Speed 129.77 samples/sec Loss 0.3279 Epoch: 21 Global Step: 217785 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:49:59,051-Speed 130.70 samples/sec Loss 0.4836 Epoch: 21 Global Step: 217790 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:50:18,897-Speed 129.00 samples/sec Loss 0.3328 Epoch: 21 Global Step: 217795 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:50:38,737-Speed 129.04 samples/sec Loss 0.3370 Epoch: 21 Global Step: 217800 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:50:58,415-Speed 130.10 samples/sec Loss 0.3117 Epoch: 21 Global Step: 217805 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:51:18,160-Speed 129.66 samples/sec Loss 0.3747 Epoch: 21 Global Step: 217810 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:51:37,760-Speed 130.62 samples/sec Loss 0.3645 Epoch: 21 Global Step: 217815 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:51:57,372-Speed 130.53 samples/sec Loss 0.3294 Epoch: 21 Global Step: 217820 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:52:17,185-Speed 129.21 samples/sec Loss 0.3634 Epoch: 21 Global Step: 217825 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:52:36,894-Speed 129.90 samples/sec Loss 0.4698 Epoch: 21 Global Step: 217830 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:52:56,652-Speed 129.58 samples/sec Loss 0.4082 Epoch: 21 Global Step: 217835 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:53:16,460-Speed 129.25 samples/sec Loss 0.3646 Epoch: 21 Global Step: 217840 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:53:36,206-Speed 129.66 samples/sec Loss 0.3733 Epoch: 21 Global Step: 217845 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 15:53:55,628-Speed 131.81 samples/sec Loss 0.3678 Epoch: 21 Global Step: 217850 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:54:15,358-Speed 129.76 samples/sec Loss 0.3638 Epoch: 21 Global Step: 217855 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:54:35,336-Speed 128.15 samples/sec Loss 0.3809 Epoch: 21 Global Step: 217860 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:54:54,969-Speed 130.40 samples/sec Loss 0.3548 Epoch: 21 Global Step: 217865 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:55:15,050-Speed 127.49 samples/sec Loss 0.3433 Epoch: 21 Global Step: 217870 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:55:34,820-Speed 129.49 samples/sec Loss 0.3697 Epoch: 21 Global Step: 217875 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:55:54,637-Speed 129.19 samples/sec Loss 0.3013 Epoch: 21 Global Step: 217880 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:56:14,422-Speed 129.40 samples/sec Loss 0.4453 Epoch: 21 Global Step: 217885 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:56:34,314-Speed 128.70 samples/sec Loss 0.3361 Epoch: 21 Global Step: 217890 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:56:54,026-Speed 129.88 samples/sec Loss 0.2939 Epoch: 21 Global Step: 217895 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:57:13,836-Speed 129.24 samples/sec Loss 0.3298 Epoch: 21 Global Step: 217900 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:57:33,701-Speed 128.87 samples/sec Loss 0.3344 Epoch: 21 Global Step: 217905 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:57:53,781-Speed 127.50 samples/sec Loss 0.3165 Epoch: 21 Global Step: 217910 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:58:13,860-Speed 127.50 samples/sec Loss 0.3864 Epoch: 21 Global Step: 217915 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:58:33,585-Speed 129.79 samples/sec Loss 0.3327 Epoch: 21 Global Step: 217920 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:58:53,544-Speed 128.27 samples/sec Loss 0.4109 Epoch: 21 Global Step: 217925 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:59:13,319-Speed 129.46 samples/sec Loss 0.3668 Epoch: 21 Global Step: 217930 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:59:33,243-Speed 128.50 samples/sec Loss 0.4349 Epoch: 21 Global Step: 217935 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 15:59:52,952-Speed 129.90 samples/sec Loss 0.4354 Epoch: 21 Global Step: 217940 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:00:12,689-Speed 129.71 samples/sec Loss 0.3354 Epoch: 21 Global Step: 217945 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:00:32,475-Speed 129.39 samples/sec Loss 0.3582 Epoch: 21 Global Step: 217950 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:00:52,539-Speed 127.60 samples/sec Loss 0.4177 Epoch: 21 Global Step: 217955 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:01:12,612-Speed 127.54 samples/sec Loss 0.4215 Epoch: 21 Global Step: 217960 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:01:31,713-Speed 134.03 samples/sec Loss 0.3397 Epoch: 21 Global Step: 217965 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:01:51,712-Speed 128.01 samples/sec Loss 0.3651 Epoch: 21 Global Step: 217970 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:02:11,490-Speed 129.45 samples/sec Loss 0.3382 Epoch: 21 Global Step: 217975 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:02:31,386-Speed 128.67 samples/sec Loss 0.3278 Epoch: 21 Global Step: 217980 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:02:51,097-Speed 129.88 samples/sec Loss 0.3880 Epoch: 21 Global Step: 217985 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:03:10,976-Speed 128.78 samples/sec Loss 0.4364 Epoch: 21 Global Step: 217990 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:03:30,603-Speed 130.44 samples/sec Loss 0.3700 Epoch: 21 Global Step: 217995 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:03:50,492-Speed 128.72 samples/sec Loss 0.3751 Epoch: 21 Global Step: 218000 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:04:10,357-Speed 128.88 samples/sec Loss 0.3606 Epoch: 21 Global Step: 218005 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:04:30,052-Speed 129.99 samples/sec Loss 0.4198 Epoch: 21 Global Step: 218010 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:04:49,909-Speed 128.93 samples/sec Loss 0.3991 Epoch: 21 Global Step: 218015 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:05:09,509-Speed 130.62 samples/sec Loss 0.3167 Epoch: 21 Global Step: 218020 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:05:29,178-Speed 130.16 samples/sec Loss 0.3485 Epoch: 21 Global Step: 218025 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:05:48,969-Speed 129.36 samples/sec Loss 0.3871 Epoch: 21 Global Step: 218030 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:06:08,882-Speed 128.56 samples/sec Loss 0.3614 Epoch: 21 Global Step: 218035 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:06:28,300-Speed 131.84 samples/sec Loss 0.3159 Epoch: 21 Global Step: 218040 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:06:48,273-Speed 128.18 samples/sec Loss 0.3392 Epoch: 21 Global Step: 218045 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:07:07,997-Speed 129.80 samples/sec Loss 0.3957 Epoch: 21 Global Step: 218050 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:07:27,531-Speed 131.06 samples/sec Loss 0.3728 Epoch: 21 Global Step: 218055 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:07:47,322-Speed 129.36 samples/sec Loss 0.4695 Epoch: 21 Global Step: 218060 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:08:06,923-Speed 130.61 samples/sec Loss 0.3331 Epoch: 21 Global Step: 218065 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:08:26,742-Speed 129.17 samples/sec Loss 0.4264 Epoch: 21 Global Step: 218070 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:08:46,494-Speed 129.62 samples/sec Loss 0.3770 Epoch: 21 Global Step: 218075 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:09:06,372-Speed 128.79 samples/sec Loss 0.3537 Epoch: 21 Global Step: 218080 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:09:25,801-Speed 131.77 samples/sec Loss 0.4434 Epoch: 21 Global Step: 218085 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:09:45,716-Speed 128.55 samples/sec Loss 0.3310 Epoch: 21 Global Step: 218090 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:10:05,324-Speed 130.57 samples/sec Loss 0.4131 Epoch: 21 Global Step: 218095 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:10:24,882-Speed 130.90 samples/sec Loss 0.4158 Epoch: 21 Global Step: 218100 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:10:44,746-Speed 128.88 samples/sec Loss 0.3576 Epoch: 21 Global Step: 218105 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:11:04,347-Speed 130.61 samples/sec Loss 0.4230 Epoch: 21 Global Step: 218110 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:11:24,135-Speed 129.38 samples/sec Loss 0.3995 Epoch: 21 Global Step: 218115 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:11:43,975-Speed 129.04 samples/sec Loss 0.3571 Epoch: 21 Global Step: 218120 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:12:03,844-Speed 128.85 samples/sec Loss 0.4475 Epoch: 21 Global Step: 218125 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:12:23,238-Speed 132.00 samples/sec Loss 0.4371 Epoch: 21 Global Step: 218130 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:12:43,221-Speed 128.11 samples/sec Loss 0.3075 Epoch: 21 Global Step: 218135 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:13:02,933-Speed 129.88 samples/sec Loss 0.3362 Epoch: 21 Global Step: 218140 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:13:22,494-Speed 130.88 samples/sec Loss 0.4410 Epoch: 21 Global Step: 218145 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:13:42,255-Speed 129.55 samples/sec Loss 0.3726 Epoch: 21 Global Step: 218150 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:14:01,874-Speed 130.50 samples/sec Loss 0.3820 Epoch: 21 Global Step: 218155 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:14:21,880-Speed 127.97 samples/sec Loss 0.3809 Epoch: 21 Global Step: 218160 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:14:41,556-Speed 130.11 samples/sec Loss 0.3289 Epoch: 21 Global Step: 218165 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:15:01,430-Speed 128.82 samples/sec Loss 0.3289 Epoch: 21 Global Step: 218170 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:15:20,840-Speed 131.90 samples/sec Loss 0.3492 Epoch: 21 Global Step: 218175 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:15:40,919-Speed 127.50 samples/sec Loss 0.3512 Epoch: 21 Global Step: 218180 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:16:00,671-Speed 129.61 samples/sec Loss 0.3345 Epoch: 21 Global Step: 218185 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:16:20,212-Speed 131.02 samples/sec Loss 0.3808 Epoch: 21 Global Step: 218190 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:16:40,081-Speed 128.85 samples/sec Loss 0.3925 Epoch: 21 Global Step: 218195 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:16:59,661-Speed 130.75 samples/sec Loss 0.4307 Epoch: 21 Global Step: 218200 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:17:20,015-Speed 125.78 samples/sec Loss 0.4564 Epoch: 21 Global Step: 218205 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:17:39,765-Speed 129.62 samples/sec Loss 0.2848 Epoch: 21 Global Step: 218210 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:17:59,644-Speed 128.79 samples/sec Loss 0.3403 Epoch: 21 Global Step: 218215 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:18:19,098-Speed 131.59 samples/sec Loss 0.3568 Epoch: 21 Global Step: 218220 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:18:39,019-Speed 128.51 samples/sec Loss 0.3879 Epoch: 21 Global Step: 218225 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:18:58,740-Speed 129.82 samples/sec Loss 0.4401 Epoch: 21 Global Step: 218230 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:19:18,346-Speed 130.58 samples/sec Loss 0.3395 Epoch: 21 Global Step: 218235 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:19:38,119-Speed 129.48 samples/sec Loss 0.3031 Epoch: 21 Global Step: 218240 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:19:57,228-Speed 133.97 samples/sec Loss 0.4153 Epoch: 21 Global Step: 218245 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:20:17,164-Speed 128.42 samples/sec Loss 0.3331 Epoch: 21 Global Step: 218250 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:20:37,363-Speed 126.74 samples/sec Loss 0.3983 Epoch: 21 Global Step: 218255 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:20:57,056-Speed 130.00 samples/sec Loss 0.4079 Epoch: 21 Global Step: 218260 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:21:16,810-Speed 129.59 samples/sec Loss 0.3786 Epoch: 21 Global Step: 218265 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:21:36,682-Speed 128.84 samples/sec Loss 0.3689 Epoch: 21 Global Step: 218270 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:21:56,869-Speed 126.82 samples/sec Loss 0.3255 Epoch: 21 Global Step: 218275 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:22:17,089-Speed 126.61 samples/sec Loss 0.2778 Epoch: 21 Global Step: 218280 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:22:36,795-Speed 129.92 samples/sec Loss 0.4419 Epoch: 21 Global Step: 218285 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:22:56,619-Speed 129.14 samples/sec Loss 0.3898 Epoch: 21 Global Step: 218290 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:23:16,418-Speed 129.31 samples/sec Loss 0.2909 Epoch: 21 Global Step: 218295 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:23:36,514-Speed 127.40 samples/sec Loss 0.4099 Epoch: 21 Global Step: 218300 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:23:56,281-Speed 129.52 samples/sec Loss 0.3314 Epoch: 21 Global Step: 218305 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:24:16,170-Speed 128.72 samples/sec Loss 0.3424 Epoch: 21 Global Step: 218310 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:24:35,892-Speed 129.81 samples/sec Loss 0.3881 Epoch: 21 Global Step: 218315 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:24:55,751-Speed 128.92 samples/sec Loss 0.4135 Epoch: 21 Global Step: 218320 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:25:15,960-Speed 126.68 samples/sec Loss 0.4705 Epoch: 21 Global Step: 218325 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:25:35,761-Speed 129.29 samples/sec Loss 0.3255 Epoch: 21 Global Step: 218330 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:25:55,653-Speed 128.71 samples/sec Loss 0.3289 Epoch: 21 Global Step: 218335 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:26:15,367-Speed 129.86 samples/sec Loss 0.4253 Epoch: 21 Global Step: 218340 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:26:35,495-Speed 127.19 samples/sec Loss 0.3966 Epoch: 21 Global Step: 218345 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:26:55,217-Speed 129.81 samples/sec Loss 0.3517 Epoch: 21 Global Step: 218350 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:27:15,389-Speed 126.92 samples/sec Loss 0.4449 Epoch: 21 Global Step: 218355 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:27:35,035-Speed 130.31 samples/sec Loss 0.3853 Epoch: 21 Global Step: 218360 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:27:54,855-Speed 129.17 samples/sec Loss 0.3546 Epoch: 21 Global Step: 218365 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:28:14,968-Speed 127.29 samples/sec Loss 0.4596 Epoch: 21 Global Step: 218370 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:28:34,725-Speed 129.58 samples/sec Loss 0.3921 Epoch: 21 Global Step: 218375 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:28:54,528-Speed 129.28 samples/sec Loss 0.4513 Epoch: 21 Global Step: 218380 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:29:14,254-Speed 129.79 samples/sec Loss 0.3792 Epoch: 21 Global Step: 218385 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:29:34,338-Speed 127.47 samples/sec Loss 0.4119 Epoch: 21 Global Step: 218390 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:29:54,121-Speed 129.41 samples/sec Loss 0.3684 Epoch: 21 Global Step: 218395 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:30:13,986-Speed 128.88 samples/sec Loss 0.3348 Epoch: 21 Global Step: 218400 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:30:33,732-Speed 129.65 samples/sec Loss 0.4031 Epoch: 21 Global Step: 218405 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:30:53,590-Speed 128.92 samples/sec Loss 0.3831 Epoch: 21 Global Step: 218410 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:31:13,109-Speed 131.16 samples/sec Loss 0.3191 Epoch: 21 Global Step: 218415 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:31:33,139-Speed 127.82 samples/sec Loss 0.4540 Epoch: 21 Global Step: 218420 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:31:52,665-Speed 131.11 samples/sec Loss 0.3833 Epoch: 21 Global Step: 218425 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:32:12,304-Speed 130.36 samples/sec Loss 0.4520 Epoch: 21 Global Step: 218430 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:32:31,925-Speed 130.48 samples/sec Loss 0.3917 Epoch: 21 Global Step: 218435 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:32:51,695-Speed 129.50 samples/sec Loss 0.3701 Epoch: 21 Global Step: 218440 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:33:11,476-Speed 129.43 samples/sec Loss 0.3961 Epoch: 21 Global Step: 218445 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:33:31,303-Speed 129.12 samples/sec Loss 0.3804 Epoch: 21 Global Step: 218450 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:33:50,936-Speed 130.40 samples/sec Loss 0.3470 Epoch: 21 Global Step: 218455 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:34:10,718-Speed 129.41 samples/sec Loss 0.3112 Epoch: 21 Global Step: 218460 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:34:30,676-Speed 128.28 samples/sec Loss 0.3491 Epoch: 21 Global Step: 218465 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:34:50,199-Speed 131.13 samples/sec Loss 0.3724 Epoch: 21 Global Step: 218470 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:35:10,020-Speed 129.16 samples/sec Loss 0.4107 Epoch: 21 Global Step: 218475 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:35:29,557-Speed 131.04 samples/sec Loss 0.3797 Epoch: 21 Global Step: 218480 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:35:49,267-Speed 129.90 samples/sec Loss 0.3614 Epoch: 21 Global Step: 218485 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:36:09,054-Speed 129.38 samples/sec Loss 0.3027 Epoch: 21 Global Step: 218490 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:36:29,054-Speed 128.01 samples/sec Loss 0.3524 Epoch: 21 Global Step: 218495 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:36:48,496-Speed 131.68 samples/sec Loss 0.3831 Epoch: 21 Global Step: 218500 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:37:08,256-Speed 129.56 samples/sec Loss 0.4244 Epoch: 21 Global Step: 218505 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:37:28,116-Speed 128.91 samples/sec Loss 0.3233 Epoch: 21 Global Step: 218510 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:37:47,706-Speed 130.69 samples/sec Loss 0.4017 Epoch: 21 Global Step: 218515 Fp16 Grad Scale: 8192 Required: 37 hours Training: 2021-06-17 16:38:07,101-Speed 132.00 samples/sec Loss 0.3551 Epoch: 21 Global Step: 218520 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:38:26,770-Speed 130.16 samples/sec Loss 0.3572 Epoch: 21 Global Step: 218525 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:38:46,488-Speed 129.84 samples/sec Loss 0.3139 Epoch: 21 Global Step: 218530 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:39:06,286-Speed 129.31 samples/sec Loss 0.3679 Epoch: 21 Global Step: 218535 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:39:26,300-Speed 127.92 samples/sec Loss 0.3481 Epoch: 21 Global Step: 218540 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:39:45,710-Speed 131.90 samples/sec Loss 0.3264 Epoch: 21 Global Step: 218545 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:40:05,532-Speed 129.16 samples/sec Loss 0.3677 Epoch: 21 Global Step: 218550 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:40:25,392-Speed 128.91 samples/sec Loss 0.3517 Epoch: 21 Global Step: 218555 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:40:45,009-Speed 130.50 samples/sec Loss 0.3199 Epoch: 21 Global Step: 218560 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:41:04,914-Speed 128.62 samples/sec Loss 0.3469 Epoch: 21 Global Step: 218565 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:41:24,446-Speed 131.07 samples/sec Loss 0.3539 Epoch: 21 Global Step: 218570 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:41:44,160-Speed 129.87 samples/sec Loss 0.3305 Epoch: 21 Global Step: 218575 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:42:03,945-Speed 129.40 samples/sec Loss 0.3731 Epoch: 21 Global Step: 218580 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:42:23,698-Speed 129.60 samples/sec Loss 0.4099 Epoch: 21 Global Step: 218585 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:42:43,337-Speed 130.36 samples/sec Loss 0.3288 Epoch: 21 Global Step: 218590 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:43:03,059-Speed 129.81 samples/sec Loss 0.4017 Epoch: 21 Global Step: 218595 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:43:22,935-Speed 128.80 samples/sec Loss 0.3250 Epoch: 21 Global Step: 218600 Fp16 Grad Scale: 4096 Required: 37 hours Training: 2021-06-17 16:43:42,491-Speed 130.92 samples/sec Loss 0.4107 Epoch: 21 Global Step: 218605 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:44:02,374-Speed 128.76 samples/sec Loss 0.3401 Epoch: 21 Global Step: 218610 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:44:21,900-Speed 131.12 samples/sec Loss 0.4139 Epoch: 21 Global Step: 218615 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:44:41,611-Speed 129.88 samples/sec Loss 0.3063 Epoch: 21 Global Step: 218620 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:45:01,390-Speed 129.44 samples/sec Loss 0.3825 Epoch: 21 Global Step: 218625 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:45:21,229-Speed 129.04 samples/sec Loss 0.3088 Epoch: 21 Global Step: 218630 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:45:40,775-Speed 130.98 samples/sec Loss 0.4193 Epoch: 21 Global Step: 218635 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:46:00,169-Speed 132.01 samples/sec Loss 0.3943 Epoch: 21 Global Step: 218640 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:46:19,760-Speed 130.68 samples/sec Loss 0.3379 Epoch: 21 Global Step: 218645 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:46:39,854-Speed 127.41 samples/sec Loss 0.3881 Epoch: 21 Global Step: 218650 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:46:59,748-Speed 128.69 samples/sec Loss 0.3471 Epoch: 21 Global Step: 218655 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:47:19,673-Speed 128.49 samples/sec Loss 0.3647 Epoch: 21 Global Step: 218660 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:47:39,317-Speed 130.33 samples/sec Loss 0.4054 Epoch: 21 Global Step: 218665 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:47:59,360-Speed 127.73 samples/sec Loss 0.4325 Epoch: 21 Global Step: 218670 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:48:19,359-Speed 128.01 samples/sec Loss 0.3838 Epoch: 21 Global Step: 218675 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:48:39,054-Speed 129.99 samples/sec Loss 0.4303 Epoch: 21 Global Step: 218680 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:48:58,781-Speed 129.78 samples/sec Loss 0.3460 Epoch: 21 Global Step: 218685 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:49:18,590-Speed 129.24 samples/sec Loss 0.3749 Epoch: 21 Global Step: 218690 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:49:38,268-Speed 130.10 samples/sec Loss 0.3379 Epoch: 21 Global Step: 218695 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:49:58,159-Speed 128.71 samples/sec Loss 0.4037 Epoch: 21 Global Step: 218700 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:50:17,870-Speed 129.89 samples/sec Loss 0.3603 Epoch: 21 Global Step: 218705 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:50:37,830-Speed 128.26 samples/sec Loss 0.4610 Epoch: 21 Global Step: 218710 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:50:57,879-Speed 127.69 samples/sec Loss 0.4017 Epoch: 21 Global Step: 218715 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:51:17,787-Speed 128.60 samples/sec Loss 0.3659 Epoch: 21 Global Step: 218720 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:51:37,492-Speed 129.92 samples/sec Loss 0.3894 Epoch: 21 Global Step: 218725 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:51:57,254-Speed 129.55 samples/sec Loss 0.3413 Epoch: 21 Global Step: 218730 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:52:17,124-Speed 128.84 samples/sec Loss 0.3284 Epoch: 21 Global Step: 218735 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 16:52:36,826-Speed 129.94 samples/sec Loss 0.2970 Epoch: 21 Global Step: 218740 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:52:56,796-Speed 128.20 samples/sec Loss 0.3705 Epoch: 21 Global Step: 218745 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:53:16,416-Speed 130.49 samples/sec Loss 0.3015 Epoch: 21 Global Step: 218750 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:53:36,727-Speed 126.04 samples/sec Loss 0.3456 Epoch: 21 Global Step: 218755 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:53:56,771-Speed 127.73 samples/sec Loss 0.3334 Epoch: 21 Global Step: 218760 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:54:16,731-Speed 128.27 samples/sec Loss 0.3823 Epoch: 21 Global Step: 218765 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:54:36,387-Speed 130.24 samples/sec Loss 0.4268 Epoch: 21 Global Step: 218770 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:54:56,059-Speed 130.15 samples/sec Loss 0.3915 Epoch: 21 Global Step: 218775 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:55:15,915-Speed 128.93 samples/sec Loss 0.4171 Epoch: 21 Global Step: 218780 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:55:35,638-Speed 129.81 samples/sec Loss 0.3955 Epoch: 21 Global Step: 218785 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:55:55,618-Speed 128.13 samples/sec Loss 0.3210 Epoch: 21 Global Step: 218790 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:56:15,241-Speed 130.47 samples/sec Loss 0.3445 Epoch: 21 Global Step: 218795 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:56:35,236-Speed 128.03 samples/sec Loss 0.3370 Epoch: 21 Global Step: 218800 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:56:55,315-Speed 127.51 samples/sec Loss 0.3420 Epoch: 21 Global Step: 218805 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:57:15,230-Speed 128.55 samples/sec Loss 0.3839 Epoch: 21 Global Step: 218810 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:57:34,849-Speed 130.49 samples/sec Loss 0.3759 Epoch: 21 Global Step: 218815 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:57:54,605-Speed 129.59 samples/sec Loss 0.4038 Epoch: 21 Global Step: 218820 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:58:14,459-Speed 128.95 samples/sec Loss 0.3371 Epoch: 21 Global Step: 218825 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:58:34,105-Speed 130.32 samples/sec Loss 0.3488 Epoch: 21 Global Step: 218830 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:58:54,178-Speed 127.54 samples/sec Loss 0.3378 Epoch: 21 Global Step: 218835 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:59:13,794-Speed 130.51 samples/sec Loss 0.4171 Epoch: 21 Global Step: 218840 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:59:33,864-Speed 127.56 samples/sec Loss 0.3564 Epoch: 21 Global Step: 218845 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 16:59:53,965-Speed 127.37 samples/sec Loss 0.3968 Epoch: 21 Global Step: 218850 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:00:13,860-Speed 128.68 samples/sec Loss 0.3511 Epoch: 21 Global Step: 218855 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:00:33,477-Speed 130.50 samples/sec Loss 0.3728 Epoch: 21 Global Step: 218860 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:00:53,345-Speed 128.86 samples/sec Loss 0.3618 Epoch: 21 Global Step: 218865 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:01:13,018-Speed 130.13 samples/sec Loss 0.3253 Epoch: 21 Global Step: 218870 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:01:32,725-Speed 129.91 samples/sec Loss 0.3921 Epoch: 21 Global Step: 218875 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:01:52,643-Speed 128.54 samples/sec Loss 0.4038 Epoch: 21 Global Step: 218880 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:02:12,258-Speed 130.52 samples/sec Loss 0.4355 Epoch: 21 Global Step: 218885 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:02:32,312-Speed 127.66 samples/sec Loss 0.3267 Epoch: 21 Global Step: 218890 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:02:51,759-Speed 131.65 samples/sec Loss 0.4702 Epoch: 21 Global Step: 218895 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:03:11,641-Speed 128.76 samples/sec Loss 0.4194 Epoch: 21 Global Step: 218900 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:03:31,580-Speed 128.40 samples/sec Loss 0.4259 Epoch: 21 Global Step: 218905 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:03:51,480-Speed 128.65 samples/sec Loss 0.4525 Epoch: 21 Global Step: 218910 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:04:11,204-Speed 129.79 samples/sec Loss 0.3422 Epoch: 21 Global Step: 218915 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:04:31,030-Speed 129.13 samples/sec Loss 0.3139 Epoch: 21 Global Step: 218920 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:04:50,860-Speed 129.10 samples/sec Loss 0.3250 Epoch: 21 Global Step: 218925 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:05:10,469-Speed 130.56 samples/sec Loss 0.3651 Epoch: 21 Global Step: 218930 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:05:30,362-Speed 128.69 samples/sec Loss 0.3787 Epoch: 21 Global Step: 218935 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:05:50,134-Speed 129.48 samples/sec Loss 0.3664 Epoch: 21 Global Step: 218940 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:06:09,747-Speed 130.53 samples/sec Loss 0.3787 Epoch: 21 Global Step: 218945 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:06:29,467-Speed 129.83 samples/sec Loss 0.3826 Epoch: 21 Global Step: 218950 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:06:49,073-Speed 130.58 samples/sec Loss 0.4534 Epoch: 21 Global Step: 218955 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:07:08,975-Speed 128.63 samples/sec Loss 0.3522 Epoch: 21 Global Step: 218960 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:07:28,838-Speed 128.89 samples/sec Loss 0.3847 Epoch: 21 Global Step: 218965 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:07:48,674-Speed 129.06 samples/sec Loss 0.3332 Epoch: 21 Global Step: 218970 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:08:08,102-Speed 131.78 samples/sec Loss 0.3539 Epoch: 21 Global Step: 218975 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:08:28,275-Speed 126.91 samples/sec Loss 0.3647 Epoch: 21 Global Step: 218980 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:08:48,097-Speed 129.15 samples/sec Loss 0.3941 Epoch: 21 Global Step: 218985 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:09:07,571-Speed 131.47 samples/sec Loss 0.3209 Epoch: 21 Global Step: 218990 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:09:27,316-Speed 129.66 samples/sec Loss 0.3535 Epoch: 21 Global Step: 218995 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:09:47,032-Speed 129.85 samples/sec Loss 0.3526 Epoch: 21 Global Step: 219000 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:10:06,995-Speed 128.25 samples/sec Loss 0.3892 Epoch: 21 Global Step: 219005 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:10:26,858-Speed 128.89 samples/sec Loss 0.4482 Epoch: 21 Global Step: 219010 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:10:46,725-Speed 128.86 samples/sec Loss 0.2858 Epoch: 21 Global Step: 219015 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:11:06,205-Speed 131.43 samples/sec Loss 0.2950 Epoch: 21 Global Step: 219020 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:11:26,121-Speed 128.55 samples/sec Loss 0.3103 Epoch: 21 Global Step: 219025 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:11:45,817-Speed 129.98 samples/sec Loss 0.3700 Epoch: 21 Global Step: 219030 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:12:05,354-Speed 131.04 samples/sec Loss 0.3636 Epoch: 21 Global Step: 219035 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:12:25,060-Speed 129.92 samples/sec Loss 0.4638 Epoch: 21 Global Step: 219040 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:12:44,506-Speed 131.65 samples/sec Loss 0.3413 Epoch: 21 Global Step: 219045 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:13:04,394-Speed 128.73 samples/sec Loss 0.3351 Epoch: 21 Global Step: 219050 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:13:24,129-Speed 129.72 samples/sec Loss 0.3580 Epoch: 21 Global Step: 219055 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:13:44,063-Speed 128.43 samples/sec Loss 0.3594 Epoch: 21 Global Step: 219060 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:14:03,487-Speed 131.80 samples/sec Loss 0.4208 Epoch: 21 Global Step: 219065 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:14:23,374-Speed 128.73 samples/sec Loss 0.3296 Epoch: 21 Global Step: 219070 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:14:43,071-Speed 129.98 samples/sec Loss 0.3025 Epoch: 21 Global Step: 219075 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:15:02,680-Speed 130.56 samples/sec Loss 0.3311 Epoch: 21 Global Step: 219080 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:15:22,450-Speed 129.50 samples/sec Loss 0.4367 Epoch: 21 Global Step: 219085 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:15:42,005-Speed 130.92 samples/sec Loss 0.3006 Epoch: 21 Global Step: 219090 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:16:01,934-Speed 128.46 samples/sec Loss 0.3635 Epoch: 21 Global Step: 219095 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:16:21,702-Speed 129.51 samples/sec Loss 0.3027 Epoch: 21 Global Step: 219100 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:16:41,590-Speed 128.73 samples/sec Loss 0.3814 Epoch: 21 Global Step: 219105 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:17:01,008-Speed 131.84 samples/sec Loss 0.4042 Epoch: 21 Global Step: 219110 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:17:20,907-Speed 128.66 samples/sec Loss 0.3420 Epoch: 21 Global Step: 219115 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:17:40,635-Speed 129.77 samples/sec Loss 0.3735 Epoch: 21 Global Step: 219120 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:18:00,243-Speed 130.57 samples/sec Loss 0.4129 Epoch: 21 Global Step: 219125 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:18:20,014-Speed 129.49 samples/sec Loss 0.3336 Epoch: 21 Global Step: 219130 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:18:39,553-Speed 131.03 samples/sec Loss 0.3391 Epoch: 21 Global Step: 219135 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:18:59,406-Speed 128.96 samples/sec Loss 0.3655 Epoch: 21 Global Step: 219140 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:19:19,327-Speed 128.51 samples/sec Loss 0.4967 Epoch: 21 Global Step: 219145 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:19:38,954-Speed 130.44 samples/sec Loss 0.4339 Epoch: 21 Global Step: 219150 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:19:58,484-Speed 131.09 samples/sec Loss 0.3495 Epoch: 21 Global Step: 219155 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:20:18,361-Speed 128.80 samples/sec Loss 0.3445 Epoch: 21 Global Step: 219160 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:20:38,081-Speed 129.83 samples/sec Loss 0.4188 Epoch: 21 Global Step: 219165 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:20:57,685-Speed 130.59 samples/sec Loss 0.3499 Epoch: 21 Global Step: 219170 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:21:16,842-Speed 133.64 samples/sec Loss 0.3069 Epoch: 21 Global Step: 219175 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:21:36,441-Speed 130.63 samples/sec Loss 0.3201 Epoch: 21 Global Step: 219180 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:21:56,315-Speed 128.82 samples/sec Loss 0.3100 Epoch: 21 Global Step: 219185 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:22:16,227-Speed 128.57 samples/sec Loss 0.3341 Epoch: 21 Global Step: 219190 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:22:35,843-Speed 130.51 samples/sec Loss 0.3446 Epoch: 21 Global Step: 219195 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:22:55,283-Speed 131.70 samples/sec Loss 0.3934 Epoch: 21 Global Step: 219200 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:23:15,126-Speed 129.02 samples/sec Loss 0.3554 Epoch: 21 Global Step: 219205 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:24:02,629-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-17 17:24:17,860-Speed 40.81 samples/sec Loss 0.3253 Epoch: 22 Global Step: 219210 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:24:37,458-Speed 130.64 samples/sec Loss 0.3159 Epoch: 22 Global Step: 219215 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:24:57,192-Speed 129.73 samples/sec Loss 0.2775 Epoch: 22 Global Step: 219220 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:25:16,823-Speed 130.41 samples/sec Loss 0.3177 Epoch: 22 Global Step: 219225 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:25:36,813-Speed 128.07 samples/sec Loss 0.2910 Epoch: 22 Global Step: 219230 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:25:56,779-Speed 128.22 samples/sec Loss 0.3014 Epoch: 22 Global Step: 219235 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:26:16,420-Speed 130.34 samples/sec Loss 0.3389 Epoch: 22 Global Step: 219240 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:26:36,574-Speed 127.03 samples/sec Loss 0.3279 Epoch: 22 Global Step: 219245 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:26:56,479-Speed 128.62 samples/sec Loss 0.2694 Epoch: 22 Global Step: 219250 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:27:16,898-Speed 125.38 samples/sec Loss 0.3455 Epoch: 22 Global Step: 219255 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:27:36,620-Speed 129.81 samples/sec Loss 0.3386 Epoch: 22 Global Step: 219260 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:27:56,366-Speed 129.66 samples/sec Loss 0.3639 Epoch: 22 Global Step: 219265 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:28:16,260-Speed 128.69 samples/sec Loss 0.3321 Epoch: 22 Global Step: 219270 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:28:36,842-Speed 124.39 samples/sec Loss 0.3415 Epoch: 22 Global Step: 219275 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:28:56,580-Speed 129.70 samples/sec Loss 0.3336 Epoch: 22 Global Step: 219280 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:29:16,429-Speed 128.98 samples/sec Loss 0.3484 Epoch: 22 Global Step: 219285 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:29:36,318-Speed 128.72 samples/sec Loss 0.4152 Epoch: 22 Global Step: 219290 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:29:56,336-Speed 127.89 samples/sec Loss 0.2769 Epoch: 22 Global Step: 219295 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:30:16,225-Speed 128.72 samples/sec Loss 0.3558 Epoch: 22 Global Step: 219300 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:30:35,869-Speed 130.33 samples/sec Loss 0.2626 Epoch: 22 Global Step: 219305 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:30:55,190-Speed 132.51 samples/sec Loss 0.3645 Epoch: 22 Global Step: 219310 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:31:14,910-Speed 129.82 samples/sec Loss 0.2993 Epoch: 22 Global Step: 219315 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:31:35,080-Speed 126.93 samples/sec Loss 0.2895 Epoch: 22 Global Step: 219320 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:31:54,804-Speed 129.80 samples/sec Loss 0.3856 Epoch: 22 Global Step: 219325 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:32:14,763-Speed 128.26 samples/sec Loss 0.2989 Epoch: 22 Global Step: 219330 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:32:34,383-Speed 130.49 samples/sec Loss 0.3695 Epoch: 22 Global Step: 219335 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:32:54,163-Speed 129.43 samples/sec Loss 0.3544 Epoch: 22 Global Step: 219340 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:33:14,034-Speed 128.84 samples/sec Loss 0.3335 Epoch: 22 Global Step: 219345 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:33:34,044-Speed 127.94 samples/sec Loss 0.3657 Epoch: 22 Global Step: 219350 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:33:53,840-Speed 129.33 samples/sec Loss 0.3877 Epoch: 22 Global Step: 219355 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:34:13,550-Speed 129.89 samples/sec Loss 0.3500 Epoch: 22 Global Step: 219360 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:34:33,629-Speed 127.50 samples/sec Loss 0.3043 Epoch: 22 Global Step: 219365 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:34:53,364-Speed 129.73 samples/sec Loss 0.3204 Epoch: 22 Global Step: 219370 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:35:13,224-Speed 128.91 samples/sec Loss 0.2712 Epoch: 22 Global Step: 219375 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:35:33,060-Speed 129.07 samples/sec Loss 0.3312 Epoch: 22 Global Step: 219380 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:35:52,918-Speed 128.92 samples/sec Loss 0.4299 Epoch: 22 Global Step: 219385 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:36:12,701-Speed 129.41 samples/sec Loss 0.4512 Epoch: 22 Global Step: 219390 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:36:32,784-Speed 127.48 samples/sec Loss 0.3340 Epoch: 22 Global Step: 219395 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:36:52,674-Speed 128.72 samples/sec Loss 0.3146 Epoch: 22 Global Step: 219400 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:37:12,340-Speed 130.18 samples/sec Loss 0.3548 Epoch: 22 Global Step: 219405 Fp16 Grad Scale: 4096 Required: 36 hours Training: 2021-06-17 17:37:32,454-Speed 127.28 samples/sec Loss 0.3492 Epoch: 22 Global Step: 219410 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:37:52,263-Speed 129.24 samples/sec Loss 0.3304 Epoch: 22 Global Step: 219415 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:38:12,129-Speed 128.87 samples/sec Loss 0.2992 Epoch: 22 Global Step: 219420 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:38:31,848-Speed 129.83 samples/sec Loss 0.3700 Epoch: 22 Global Step: 219425 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:38:51,779-Speed 128.45 samples/sec Loss 0.3032 Epoch: 22 Global Step: 219430 Fp16 Grad Scale: 8192 Required: 36 hours Training: 2021-06-17 17:39:11,526-Speed 129.65 samples/sec Loss 0.3016 Epoch: 22 Global Step: 219435 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:39:31,777-Speed 126.42 samples/sec Loss 0.4133 Epoch: 22 Global Step: 219440 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:39:51,525-Speed 129.64 samples/sec Loss 0.3654 Epoch: 22 Global Step: 219445 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:40:11,212-Speed 130.04 samples/sec Loss 0.2768 Epoch: 22 Global Step: 219450 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:40:31,328-Speed 127.27 samples/sec Loss 0.2782 Epoch: 22 Global Step: 219455 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:40:51,303-Speed 128.16 samples/sec Loss 0.3699 Epoch: 22 Global Step: 219460 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:41:11,013-Speed 129.89 samples/sec Loss 0.3475 Epoch: 22 Global Step: 219465 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:41:30,769-Speed 129.59 samples/sec Loss 0.3619 Epoch: 22 Global Step: 219470 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:41:50,666-Speed 128.67 samples/sec Loss 0.3591 Epoch: 22 Global Step: 219475 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:42:10,426-Speed 129.56 samples/sec Loss 0.2995 Epoch: 22 Global Step: 219480 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:42:30,662-Speed 126.51 samples/sec Loss 0.3091 Epoch: 22 Global Step: 219485 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:42:50,383-Speed 129.82 samples/sec Loss 0.3591 Epoch: 22 Global Step: 219490 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:43:10,186-Speed 129.29 samples/sec Loss 0.2833 Epoch: 22 Global Step: 219495 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:43:30,931-Speed 123.41 samples/sec Loss 0.3902 Epoch: 22 Global Step: 219500 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:43:50,865-Speed 128.43 samples/sec Loss 0.3345 Epoch: 22 Global Step: 219505 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:44:10,899-Speed 127.79 samples/sec Loss 0.2570 Epoch: 22 Global Step: 219510 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:44:30,823-Speed 128.49 samples/sec Loss 0.2766 Epoch: 22 Global Step: 219515 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:44:50,498-Speed 130.13 samples/sec Loss 0.3283 Epoch: 22 Global Step: 219520 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:45:10,324-Speed 129.12 samples/sec Loss 0.3158 Epoch: 22 Global Step: 219525 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:45:30,543-Speed 126.62 samples/sec Loss 0.3080 Epoch: 22 Global Step: 219530 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:45:50,273-Speed 129.76 samples/sec Loss 0.3756 Epoch: 22 Global Step: 219535 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:46:10,116-Speed 129.02 samples/sec Loss 0.3495 Epoch: 22 Global Step: 219540 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:46:30,090-Speed 128.18 samples/sec Loss 0.2524 Epoch: 22 Global Step: 219545 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:46:49,989-Speed 128.65 samples/sec Loss 0.3811 Epoch: 22 Global Step: 219550 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:47:09,765-Speed 129.46 samples/sec Loss 0.2987 Epoch: 22 Global Step: 219555 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:47:29,769-Speed 127.98 samples/sec Loss 0.3285 Epoch: 22 Global Step: 219560 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:47:49,584-Speed 129.20 samples/sec Loss 0.2872 Epoch: 22 Global Step: 219565 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:48:09,440-Speed 128.93 samples/sec Loss 0.2678 Epoch: 22 Global Step: 219570 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:48:29,490-Speed 127.69 samples/sec Loss 0.2349 Epoch: 22 Global Step: 219575 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:48:49,303-Speed 129.21 samples/sec Loss 0.4283 Epoch: 22 Global Step: 219580 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:49:09,196-Speed 128.70 samples/sec Loss 0.3646 Epoch: 22 Global Step: 219585 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:49:29,215-Speed 127.89 samples/sec Loss 0.2957 Epoch: 22 Global Step: 219590 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:49:49,023-Speed 129.25 samples/sec Loss 0.3604 Epoch: 22 Global Step: 219595 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:50:08,835-Speed 129.22 samples/sec Loss 0.3284 Epoch: 22 Global Step: 219600 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:50:28,723-Speed 128.73 samples/sec Loss 0.3867 Epoch: 22 Global Step: 219605 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:50:48,490-Speed 129.52 samples/sec Loss 0.3307 Epoch: 22 Global Step: 219610 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:51:08,330-Speed 129.04 samples/sec Loss 0.3473 Epoch: 22 Global Step: 219615 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:51:27,929-Speed 130.63 samples/sec Loss 0.3665 Epoch: 22 Global Step: 219620 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:51:47,826-Speed 128.67 samples/sec Loss 0.3263 Epoch: 22 Global Step: 219625 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:52:07,558-Speed 129.74 samples/sec Loss 0.3666 Epoch: 22 Global Step: 219630 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:52:27,268-Speed 129.89 samples/sec Loss 0.3214 Epoch: 22 Global Step: 219635 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:52:46,897-Speed 130.43 samples/sec Loss 0.3800 Epoch: 22 Global Step: 219640 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:53:06,661-Speed 129.53 samples/sec Loss 0.3620 Epoch: 22 Global Step: 219645 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:53:26,255-Speed 130.66 samples/sec Loss 0.3260 Epoch: 22 Global Step: 219650 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:53:45,798-Speed 131.01 samples/sec Loss 0.3443 Epoch: 22 Global Step: 219655 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:54:05,864-Speed 127.58 samples/sec Loss 0.3116 Epoch: 22 Global Step: 219660 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:54:25,862-Speed 128.02 samples/sec Loss 0.3004 Epoch: 22 Global Step: 219665 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:54:45,468-Speed 130.57 samples/sec Loss 0.3597 Epoch: 22 Global Step: 219670 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:55:05,228-Speed 129.56 samples/sec Loss 0.2964 Epoch: 22 Global Step: 219675 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:55:24,870-Speed 130.34 samples/sec Loss 0.3606 Epoch: 22 Global Step: 219680 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:55:44,665-Speed 129.33 samples/sec Loss 0.4308 Epoch: 22 Global Step: 219685 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:56:04,264-Speed 130.63 samples/sec Loss 0.3497 Epoch: 22 Global Step: 219690 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:56:24,132-Speed 128.86 samples/sec Loss 0.3584 Epoch: 22 Global Step: 219695 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:56:43,588-Speed 131.58 samples/sec Loss 0.3137 Epoch: 22 Global Step: 219700 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:57:03,705-Speed 127.26 samples/sec Loss 0.3478 Epoch: 22 Global Step: 219705 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:57:23,556-Speed 128.97 samples/sec Loss 0.3201 Epoch: 22 Global Step: 219710 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:57:43,142-Speed 130.71 samples/sec Loss 0.3274 Epoch: 22 Global Step: 219715 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 17:58:02,846-Speed 129.93 samples/sec Loss 0.3211 Epoch: 22 Global Step: 219720 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:58:22,463-Speed 130.50 samples/sec Loss 0.3557 Epoch: 22 Global Step: 219725 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:58:42,179-Speed 129.85 samples/sec Loss 0.2995 Epoch: 22 Global Step: 219730 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:59:01,729-Speed 130.95 samples/sec Loss 0.3454 Epoch: 22 Global Step: 219735 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:59:21,593-Speed 128.88 samples/sec Loss 0.3153 Epoch: 22 Global Step: 219740 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 17:59:41,153-Speed 130.89 samples/sec Loss 0.2868 Epoch: 22 Global Step: 219745 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:00:01,295-Speed 127.10 samples/sec Loss 0.3552 Epoch: 22 Global Step: 219750 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:00:21,123-Speed 129.11 samples/sec Loss 0.3411 Epoch: 22 Global Step: 219755 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:00:40,731-Speed 130.57 samples/sec Loss 0.3870 Epoch: 22 Global Step: 219760 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:01:00,477-Speed 129.65 samples/sec Loss 0.3096 Epoch: 22 Global Step: 219765 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:01:20,081-Speed 130.60 samples/sec Loss 0.2556 Epoch: 22 Global Step: 219770 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:01:40,423-Speed 125.85 samples/sec Loss 0.2875 Epoch: 22 Global Step: 219775 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:02:00,030-Speed 130.58 samples/sec Loss 0.3551 Epoch: 22 Global Step: 219780 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:02:19,746-Speed 129.86 samples/sec Loss 0.3497 Epoch: 22 Global Step: 219785 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:02:39,295-Speed 130.96 samples/sec Loss 0.3534 Epoch: 22 Global Step: 219790 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:02:59,391-Speed 127.39 samples/sec Loss 0.3496 Epoch: 22 Global Step: 219795 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:03:19,402-Speed 127.94 samples/sec Loss 0.2894 Epoch: 22 Global Step: 219800 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:03:38,853-Speed 131.62 samples/sec Loss 0.3400 Epoch: 22 Global Step: 219805 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:03:58,560-Speed 129.91 samples/sec Loss 0.3390 Epoch: 22 Global Step: 219810 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:04:18,221-Speed 130.21 samples/sec Loss 0.3644 Epoch: 22 Global Step: 219815 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:04:37,993-Speed 129.48 samples/sec Loss 0.3899 Epoch: 22 Global Step: 219820 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:04:57,573-Speed 130.75 samples/sec Loss 0.3595 Epoch: 22 Global Step: 219825 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:05:17,331-Speed 129.58 samples/sec Loss 0.3012 Epoch: 22 Global Step: 219830 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:05:36,887-Speed 130.91 samples/sec Loss 0.3001 Epoch: 22 Global Step: 219835 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:05:57,375-Speed 124.96 samples/sec Loss 0.3726 Epoch: 22 Global Step: 219840 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:06:17,560-Speed 126.83 samples/sec Loss 0.3297 Epoch: 22 Global Step: 219845 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:06:37,183-Speed 130.47 samples/sec Loss 0.3517 Epoch: 22 Global Step: 219850 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:06:56,973-Speed 129.37 samples/sec Loss 0.3651 Epoch: 22 Global Step: 219855 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:07:16,161-Speed 133.42 samples/sec Loss 0.3529 Epoch: 22 Global Step: 219860 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:07:36,077-Speed 128.55 samples/sec Loss 0.3207 Epoch: 22 Global Step: 219865 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:07:55,722-Speed 130.32 samples/sec Loss 0.3110 Epoch: 22 Global Step: 219870 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:08:15,433-Speed 129.89 samples/sec Loss 0.3597 Epoch: 22 Global Step: 219875 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:08:34,986-Speed 130.93 samples/sec Loss 0.3527 Epoch: 22 Global Step: 219880 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:08:55,236-Speed 126.43 samples/sec Loss 0.3105 Epoch: 22 Global Step: 219885 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:09:15,724-Speed 124.96 samples/sec Loss 0.3055 Epoch: 22 Global Step: 219890 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:09:35,244-Speed 131.15 samples/sec Loss 0.2822 Epoch: 22 Global Step: 219895 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:09:55,070-Speed 129.13 samples/sec Loss 0.3155 Epoch: 22 Global Step: 219900 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:10:14,754-Speed 130.06 samples/sec Loss 0.3441 Epoch: 22 Global Step: 219905 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:10:34,393-Speed 130.36 samples/sec Loss 0.3639 Epoch: 22 Global Step: 219910 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:10:54,734-Speed 125.86 samples/sec Loss 0.2914 Epoch: 22 Global Step: 219915 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:11:14,898-Speed 126.97 samples/sec Loss 0.3407 Epoch: 22 Global Step: 219920 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:11:34,870-Speed 128.19 samples/sec Loss 0.3088 Epoch: 22 Global Step: 219925 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:11:55,488-Speed 124.17 samples/sec Loss 0.3464 Epoch: 22 Global Step: 219930 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:12:17,459-Speed 116.52 samples/sec Loss 0.3268 Epoch: 22 Global Step: 219935 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:12:37,122-Speed 130.20 samples/sec Loss 0.3415 Epoch: 22 Global Step: 219940 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:12:56,900-Speed 129.44 samples/sec Loss 0.3421 Epoch: 22 Global Step: 219945 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:13:16,500-Speed 130.62 samples/sec Loss 0.3688 Epoch: 22 Global Step: 219950 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:13:36,115-Speed 130.52 samples/sec Loss 0.3103 Epoch: 22 Global Step: 219955 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 18:13:55,679-Speed 130.86 samples/sec Loss 0.3669 Epoch: 22 Global Step: 219960 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:14:15,521-Speed 129.03 samples/sec Loss 0.2960 Epoch: 22 Global Step: 219965 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:14:35,106-Speed 130.72 samples/sec Loss 0.3024 Epoch: 22 Global Step: 219970 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:14:55,201-Speed 127.40 samples/sec Loss 0.2921 Epoch: 22 Global Step: 219975 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:15:15,221-Speed 127.88 samples/sec Loss 0.2961 Epoch: 22 Global Step: 219980 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:15:34,758-Speed 131.04 samples/sec Loss 0.3050 Epoch: 22 Global Step: 219985 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:15:54,438-Speed 130.09 samples/sec Loss 0.3189 Epoch: 22 Global Step: 219990 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:16:14,060-Speed 130.47 samples/sec Loss 0.3933 Epoch: 22 Global Step: 219995 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:16:33,781-Speed 129.82 samples/sec Loss 0.2725 Epoch: 22 Global Step: 220000 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 18:31:19,568-[lfw][220000]XNorm: 21.696123 Training: 2021-06-17 18:31:19,568-[lfw][220000]Accuracy-Flip: 0.99850+-0.00189 Training: 2021-06-17 18:31:19,568-[lfw][220000]Accuracy-Highest: 0.99867 Training: 2021-06-17 18:48:37,203-[cfp_fp][220000]XNorm: 23.471388 Training: 2021-06-17 18:48:37,203-[cfp_fp][220000]Accuracy-Flip: 0.99157+-0.00316 Training: 2021-06-17 18:48:37,203-[cfp_fp][220000]Accuracy-Highest: 0.99229 Training: 2021-06-17 19:03:23,377-[agedb_30][220000]XNorm: 22.823973 Training: 2021-06-17 19:03:23,377-[agedb_30][220000]Accuracy-Flip: 0.98550+-0.00548 Training: 2021-06-17 19:03:23,377-[agedb_30][220000]Accuracy-Highest: 0.98667 Training: 2021-06-17 19:03:43,053-Speed 0.90 samples/sec Loss 0.3601 Epoch: 22 Global Step: 220005 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:04:02,757-Speed 129.93 samples/sec Loss 0.3736 Epoch: 22 Global Step: 220010 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:04:22,333-Speed 130.78 samples/sec Loss 0.3849 Epoch: 22 Global Step: 220015 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:04:42,364-Speed 127.81 samples/sec Loss 0.3425 Epoch: 22 Global Step: 220020 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:05:02,088-Speed 129.79 samples/sec Loss 0.3706 Epoch: 22 Global Step: 220025 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:05:21,569-Speed 131.42 samples/sec Loss 0.3311 Epoch: 22 Global Step: 220030 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:05:41,570-Speed 128.00 samples/sec Loss 0.3604 Epoch: 22 Global Step: 220035 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:06:01,013-Speed 131.67 samples/sec Loss 0.3408 Epoch: 22 Global Step: 220040 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:06:20,813-Speed 129.30 samples/sec Loss 0.2855 Epoch: 22 Global Step: 220045 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:06:40,356-Speed 130.99 samples/sec Loss 0.3618 Epoch: 22 Global Step: 220050 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:07:00,056-Speed 129.96 samples/sec Loss 0.3062 Epoch: 22 Global Step: 220055 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:07:19,648-Speed 130.68 samples/sec Loss 0.3262 Epoch: 22 Global Step: 220060 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:07:39,635-Speed 128.09 samples/sec Loss 0.3538 Epoch: 22 Global Step: 220065 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:07:59,257-Speed 130.47 samples/sec Loss 0.4440 Epoch: 22 Global Step: 220070 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:08:18,796-Speed 131.02 samples/sec Loss 0.3641 Epoch: 22 Global Step: 220075 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:08:38,681-Speed 128.75 samples/sec Loss 0.3098 Epoch: 22 Global Step: 220080 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:08:58,154-Speed 131.47 samples/sec Loss 0.3426 Epoch: 22 Global Step: 220085 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:09:18,038-Speed 128.75 samples/sec Loss 0.3082 Epoch: 22 Global Step: 220090 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:09:37,572-Speed 131.06 samples/sec Loss 0.3556 Epoch: 22 Global Step: 220095 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:09:57,288-Speed 129.86 samples/sec Loss 0.3365 Epoch: 22 Global Step: 220100 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:10:16,918-Speed 130.42 samples/sec Loss 0.2845 Epoch: 22 Global Step: 220105 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:10:36,917-Speed 128.01 samples/sec Loss 0.3480 Epoch: 22 Global Step: 220110 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:10:56,774-Speed 128.93 samples/sec Loss 0.3078 Epoch: 22 Global Step: 220115 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:11:16,314-Speed 131.02 samples/sec Loss 0.2945 Epoch: 22 Global Step: 220120 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:11:36,224-Speed 128.58 samples/sec Loss 0.2857 Epoch: 22 Global Step: 220125 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:11:55,692-Speed 131.51 samples/sec Loss 0.3226 Epoch: 22 Global Step: 220130 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:12:15,464-Speed 129.48 samples/sec Loss 0.2931 Epoch: 22 Global Step: 220135 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:12:35,029-Speed 130.86 samples/sec Loss 0.3521 Epoch: 22 Global Step: 220140 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:12:54,849-Speed 129.17 samples/sec Loss 0.3155 Epoch: 22 Global Step: 220145 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:13:14,506-Speed 130.24 samples/sec Loss 0.4110 Epoch: 22 Global Step: 220150 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:13:34,327-Speed 129.16 samples/sec Loss 0.3599 Epoch: 22 Global Step: 220155 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:13:54,155-Speed 129.12 samples/sec Loss 0.2674 Epoch: 22 Global Step: 220160 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:14:14,361-Speed 126.70 samples/sec Loss 0.3154 Epoch: 22 Global Step: 220165 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:14:34,295-Speed 128.43 samples/sec Loss 0.3165 Epoch: 22 Global Step: 220170 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:14:53,735-Speed 131.69 samples/sec Loss 0.4094 Epoch: 22 Global Step: 220175 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:15:13,603-Speed 128.86 samples/sec Loss 0.3067 Epoch: 22 Global Step: 220180 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:15:33,275-Speed 130.14 samples/sec Loss 0.3075 Epoch: 22 Global Step: 220185 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:15:52,995-Speed 129.83 samples/sec Loss 0.4524 Epoch: 22 Global Step: 220190 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:16:12,602-Speed 130.57 samples/sec Loss 0.2972 Epoch: 22 Global Step: 220195 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:16:32,406-Speed 129.28 samples/sec Loss 0.3177 Epoch: 22 Global Step: 220200 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:16:52,447-Speed 127.75 samples/sec Loss 0.3349 Epoch: 22 Global Step: 220205 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:17:11,902-Speed 131.59 samples/sec Loss 0.2684 Epoch: 22 Global Step: 220210 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:17:31,813-Speed 128.58 samples/sec Loss 0.2983 Epoch: 22 Global Step: 220215 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:17:51,314-Speed 131.29 samples/sec Loss 0.2827 Epoch: 22 Global Step: 220220 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:18:11,186-Speed 128.83 samples/sec Loss 0.2984 Epoch: 22 Global Step: 220225 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:18:30,787-Speed 130.61 samples/sec Loss 0.3411 Epoch: 22 Global Step: 220230 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:18:50,476-Speed 130.03 samples/sec Loss 0.4263 Epoch: 22 Global Step: 220235 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:19:09,636-Speed 133.62 samples/sec Loss 0.2718 Epoch: 22 Global Step: 220240 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:19:29,229-Speed 130.67 samples/sec Loss 0.3126 Epoch: 22 Global Step: 220245 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:19:49,118-Speed 128.72 samples/sec Loss 0.3773 Epoch: 22 Global Step: 220250 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:20:08,357-Speed 133.07 samples/sec Loss 0.3487 Epoch: 22 Global Step: 220255 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:20:28,266-Speed 128.59 samples/sec Loss 0.3466 Epoch: 22 Global Step: 220260 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:20:47,917-Speed 130.28 samples/sec Loss 0.3964 Epoch: 22 Global Step: 220265 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:21:07,425-Speed 131.23 samples/sec Loss 0.2934 Epoch: 22 Global Step: 220270 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:21:27,026-Speed 130.61 samples/sec Loss 0.3664 Epoch: 22 Global Step: 220275 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:21:46,760-Speed 129.73 samples/sec Loss 0.3184 Epoch: 22 Global Step: 220280 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:22:06,386-Speed 130.45 samples/sec Loss 0.3715 Epoch: 22 Global Step: 220285 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:22:26,086-Speed 129.96 samples/sec Loss 0.3421 Epoch: 22 Global Step: 220290 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:22:45,638-Speed 130.94 samples/sec Loss 0.3429 Epoch: 22 Global Step: 220295 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:23:05,496-Speed 128.93 samples/sec Loss 0.4590 Epoch: 22 Global Step: 220300 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:23:25,396-Speed 128.65 samples/sec Loss 0.3225 Epoch: 22 Global Step: 220305 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:23:45,281-Speed 128.74 samples/sec Loss 0.3370 Epoch: 22 Global Step: 220310 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:24:04,774-Speed 131.34 samples/sec Loss 0.3310 Epoch: 22 Global Step: 220315 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:24:24,666-Speed 128.71 samples/sec Loss 0.3257 Epoch: 22 Global Step: 220320 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:24:44,394-Speed 129.77 samples/sec Loss 0.3412 Epoch: 22 Global Step: 220325 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:25:04,023-Speed 130.42 samples/sec Loss 0.3159 Epoch: 22 Global Step: 220330 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:25:23,664-Speed 130.35 samples/sec Loss 0.3610 Epoch: 22 Global Step: 220335 Fp16 Grad Scale: 4096 Required: 35 hours Training: 2021-06-17 19:25:43,281-Speed 130.51 samples/sec Loss 0.3481 Epoch: 22 Global Step: 220340 Fp16 Grad Scale: 8192 Required: 35 hours Training: 2021-06-17 19:26:03,023-Speed 129.68 samples/sec Loss 0.3998 Epoch: 22 Global Step: 220345 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:26:22,949-Speed 128.49 samples/sec Loss 0.3767 Epoch: 22 Global Step: 220350 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:26:42,738-Speed 129.37 samples/sec Loss 0.3236 Epoch: 22 Global Step: 220355 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:27:02,240-Speed 131.27 samples/sec Loss 0.3668 Epoch: 22 Global Step: 220360 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:27:21,684-Speed 131.67 samples/sec Loss 0.3488 Epoch: 22 Global Step: 220365 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:27:41,419-Speed 129.73 samples/sec Loss 0.2544 Epoch: 22 Global Step: 220370 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:28:01,066-Speed 130.31 samples/sec Loss 0.3533 Epoch: 22 Global Step: 220375 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:28:20,868-Speed 129.29 samples/sec Loss 0.3529 Epoch: 22 Global Step: 220380 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:28:40,466-Speed 130.63 samples/sec Loss 0.3690 Epoch: 22 Global Step: 220385 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:29:00,228-Speed 129.55 samples/sec Loss 0.3694 Epoch: 22 Global Step: 220390 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:29:20,130-Speed 128.64 samples/sec Loss 0.3731 Epoch: 22 Global Step: 220395 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:29:39,847-Speed 129.84 samples/sec Loss 0.3106 Epoch: 22 Global Step: 220400 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:29:59,479-Speed 130.41 samples/sec Loss 0.3081 Epoch: 22 Global Step: 220405 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:30:19,385-Speed 128.61 samples/sec Loss 0.3761 Epoch: 22 Global Step: 220410 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:30:39,114-Speed 129.77 samples/sec Loss 0.2935 Epoch: 22 Global Step: 220415 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:30:58,712-Speed 130.63 samples/sec Loss 0.3111 Epoch: 22 Global Step: 220420 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:31:18,443-Speed 129.75 samples/sec Loss 0.3969 Epoch: 22 Global Step: 220425 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:31:38,040-Speed 130.64 samples/sec Loss 0.3291 Epoch: 22 Global Step: 220430 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:31:57,796-Speed 129.59 samples/sec Loss 0.2933 Epoch: 22 Global Step: 220435 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:32:17,681-Speed 128.75 samples/sec Loss 0.2947 Epoch: 22 Global Step: 220440 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:32:37,418-Speed 129.71 samples/sec Loss 0.3608 Epoch: 22 Global Step: 220445 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:32:57,135-Speed 129.85 samples/sec Loss 0.3886 Epoch: 22 Global Step: 220450 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:33:16,961-Speed 129.13 samples/sec Loss 0.3127 Epoch: 22 Global Step: 220455 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:33:36,710-Speed 129.63 samples/sec Loss 0.3600 Epoch: 22 Global Step: 220460 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:33:56,286-Speed 130.78 samples/sec Loss 0.3392 Epoch: 22 Global Step: 220465 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:34:16,035-Speed 129.64 samples/sec Loss 0.3445 Epoch: 22 Global Step: 220470 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:34:35,693-Speed 130.24 samples/sec Loss 0.2907 Epoch: 22 Global Step: 220475 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:34:55,452-Speed 129.56 samples/sec Loss 0.3490 Epoch: 22 Global Step: 220480 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:35:15,237-Speed 129.40 samples/sec Loss 0.3637 Epoch: 22 Global Step: 220485 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:35:34,993-Speed 129.59 samples/sec Loss 0.3570 Epoch: 22 Global Step: 220490 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:35:54,740-Speed 129.64 samples/sec Loss 0.3898 Epoch: 22 Global Step: 220495 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:36:14,531-Speed 129.36 samples/sec Loss 0.3075 Epoch: 22 Global Step: 220500 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:36:34,285-Speed 129.60 samples/sec Loss 0.2957 Epoch: 22 Global Step: 220505 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:36:53,912-Speed 130.44 samples/sec Loss 0.3058 Epoch: 22 Global Step: 220510 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:37:13,713-Speed 129.30 samples/sec Loss 0.3749 Epoch: 22 Global Step: 220515 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:37:33,321-Speed 130.57 samples/sec Loss 0.3065 Epoch: 22 Global Step: 220520 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:37:53,020-Speed 129.97 samples/sec Loss 0.3510 Epoch: 22 Global Step: 220525 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:38:12,805-Speed 129.40 samples/sec Loss 0.3472 Epoch: 22 Global Step: 220530 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:38:32,561-Speed 129.59 samples/sec Loss 0.3673 Epoch: 22 Global Step: 220535 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:38:52,322-Speed 129.56 samples/sec Loss 0.3499 Epoch: 22 Global Step: 220540 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:39:12,123-Speed 129.29 samples/sec Loss 0.3168 Epoch: 22 Global Step: 220545 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:39:31,476-Speed 132.29 samples/sec Loss 0.3501 Epoch: 22 Global Step: 220550 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:39:51,193-Speed 129.84 samples/sec Loss 0.3044 Epoch: 22 Global Step: 220555 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:40:10,966-Speed 129.48 samples/sec Loss 0.3887 Epoch: 22 Global Step: 220560 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:40:30,570-Speed 130.59 samples/sec Loss 0.3927 Epoch: 22 Global Step: 220565 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:40:50,295-Speed 129.79 samples/sec Loss 0.3773 Epoch: 22 Global Step: 220570 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:41:10,096-Speed 129.30 samples/sec Loss 0.2824 Epoch: 22 Global Step: 220575 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:41:29,791-Speed 129.98 samples/sec Loss 0.3135 Epoch: 22 Global Step: 220580 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:41:49,460-Speed 130.16 samples/sec Loss 0.3158 Epoch: 22 Global Step: 220585 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:42:09,268-Speed 129.25 samples/sec Loss 0.2714 Epoch: 22 Global Step: 220590 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:42:29,181-Speed 128.56 samples/sec Loss 0.3523 Epoch: 22 Global Step: 220595 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:42:48,708-Speed 131.11 samples/sec Loss 0.2746 Epoch: 22 Global Step: 220600 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:43:08,490-Speed 129.42 samples/sec Loss 0.3380 Epoch: 22 Global Step: 220605 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:43:28,113-Speed 130.47 samples/sec Loss 0.3794 Epoch: 22 Global Step: 220610 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:43:47,835-Speed 129.81 samples/sec Loss 0.2723 Epoch: 22 Global Step: 220615 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:44:07,654-Speed 129.18 samples/sec Loss 0.3621 Epoch: 22 Global Step: 220620 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:44:27,387-Speed 129.74 samples/sec Loss 0.3076 Epoch: 22 Global Step: 220625 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:44:47,187-Speed 129.30 samples/sec Loss 0.3603 Epoch: 22 Global Step: 220630 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:45:07,005-Speed 129.18 samples/sec Loss 0.3290 Epoch: 22 Global Step: 220635 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:45:26,663-Speed 130.23 samples/sec Loss 0.3335 Epoch: 22 Global Step: 220640 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:45:46,234-Speed 130.82 samples/sec Loss 0.2980 Epoch: 22 Global Step: 220645 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:46:06,043-Speed 129.24 samples/sec Loss 0.2736 Epoch: 22 Global Step: 220650 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:46:25,637-Speed 130.66 samples/sec Loss 0.3244 Epoch: 22 Global Step: 220655 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:46:45,335-Speed 129.97 samples/sec Loss 0.3894 Epoch: 22 Global Step: 220660 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:47:05,137-Speed 129.29 samples/sec Loss 0.3403 Epoch: 22 Global Step: 220665 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:47:24,885-Speed 129.64 samples/sec Loss 0.3237 Epoch: 22 Global Step: 220670 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:47:44,610-Speed 129.79 samples/sec Loss 0.3916 Epoch: 22 Global Step: 220675 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:48:04,316-Speed 129.92 samples/sec Loss 0.2901 Epoch: 22 Global Step: 220680 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:48:24,020-Speed 129.93 samples/sec Loss 0.3275 Epoch: 22 Global Step: 220685 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:48:43,564-Speed 131.00 samples/sec Loss 0.3311 Epoch: 22 Global Step: 220690 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:49:03,354-Speed 129.37 samples/sec Loss 0.3587 Epoch: 22 Global Step: 220695 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:49:22,996-Speed 130.34 samples/sec Loss 0.3571 Epoch: 22 Global Step: 220700 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:49:42,706-Speed 129.89 samples/sec Loss 0.3307 Epoch: 22 Global Step: 220705 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:50:02,499-Speed 129.34 samples/sec Loss 0.3720 Epoch: 22 Global Step: 220710 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:50:22,239-Speed 129.69 samples/sec Loss 0.3238 Epoch: 22 Global Step: 220715 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:50:41,985-Speed 129.66 samples/sec Loss 0.3007 Epoch: 22 Global Step: 220720 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:51:01,722-Speed 129.71 samples/sec Loss 0.3639 Epoch: 22 Global Step: 220725 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:51:20,885-Speed 133.60 samples/sec Loss 0.3978 Epoch: 22 Global Step: 220730 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:51:40,589-Speed 129.93 samples/sec Loss 0.3200 Epoch: 22 Global Step: 220735 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:52:00,332-Speed 129.67 samples/sec Loss 0.3327 Epoch: 22 Global Step: 220740 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:52:19,940-Speed 130.57 samples/sec Loss 0.3254 Epoch: 22 Global Step: 220745 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:52:39,400-Speed 131.56 samples/sec Loss 0.3481 Epoch: 22 Global Step: 220750 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:52:59,063-Speed 130.20 samples/sec Loss 0.3282 Epoch: 22 Global Step: 220755 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:53:18,598-Speed 131.05 samples/sec Loss 0.2734 Epoch: 22 Global Step: 220760 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:53:38,356-Speed 129.58 samples/sec Loss 0.3267 Epoch: 22 Global Step: 220765 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:53:58,242-Speed 128.74 samples/sec Loss 0.3303 Epoch: 22 Global Step: 220770 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:54:17,855-Speed 130.53 samples/sec Loss 0.3255 Epoch: 22 Global Step: 220775 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:54:38,220-Speed 125.72 samples/sec Loss 0.3536 Epoch: 22 Global Step: 220780 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:54:58,195-Speed 128.16 samples/sec Loss 0.4090 Epoch: 22 Global Step: 220785 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:55:17,956-Speed 129.56 samples/sec Loss 0.2992 Epoch: 22 Global Step: 220790 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:55:37,690-Speed 129.73 samples/sec Loss 0.3726 Epoch: 22 Global Step: 220795 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:55:57,562-Speed 128.84 samples/sec Loss 0.3548 Epoch: 22 Global Step: 220800 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:56:17,286-Speed 129.80 samples/sec Loss 0.3070 Epoch: 22 Global Step: 220805 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:56:37,176-Speed 128.71 samples/sec Loss 0.2933 Epoch: 22 Global Step: 220810 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:56:56,905-Speed 129.77 samples/sec Loss 0.2727 Epoch: 22 Global Step: 220815 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:57:16,566-Speed 130.21 samples/sec Loss 0.3540 Epoch: 22 Global Step: 220820 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:57:36,916-Speed 125.80 samples/sec Loss 0.3160 Epoch: 22 Global Step: 220825 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 19:57:56,900-Speed 128.11 samples/sec Loss 0.3087 Epoch: 22 Global Step: 220830 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:58:16,623-Speed 129.80 samples/sec Loss 0.3627 Epoch: 22 Global Step: 220835 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:58:36,281-Speed 130.23 samples/sec Loss 0.3427 Epoch: 22 Global Step: 220840 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:58:56,211-Speed 128.46 samples/sec Loss 0.2855 Epoch: 22 Global Step: 220845 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:59:15,938-Speed 129.78 samples/sec Loss 0.3823 Epoch: 22 Global Step: 220850 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:59:35,911-Speed 128.18 samples/sec Loss 0.3192 Epoch: 22 Global Step: 220855 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 19:59:55,690-Speed 129.44 samples/sec Loss 0.3274 Epoch: 22 Global Step: 220860 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:00:15,455-Speed 129.53 samples/sec Loss 0.3354 Epoch: 22 Global Step: 220865 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:00:35,832-Speed 125.64 samples/sec Loss 0.3249 Epoch: 22 Global Step: 220870 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:00:55,798-Speed 128.22 samples/sec Loss 0.3590 Epoch: 22 Global Step: 220875 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:01:15,511-Speed 129.87 samples/sec Loss 0.3639 Epoch: 22 Global Step: 220880 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:01:35,224-Speed 129.87 samples/sec Loss 0.3530 Epoch: 22 Global Step: 220885 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:01:55,070-Speed 129.00 samples/sec Loss 0.3808 Epoch: 22 Global Step: 220890 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:02:14,783-Speed 129.87 samples/sec Loss 0.3369 Epoch: 22 Global Step: 220895 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:02:34,650-Speed 128.86 samples/sec Loss 0.3396 Epoch: 22 Global Step: 220900 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:02:54,384-Speed 129.73 samples/sec Loss 0.3676 Epoch: 22 Global Step: 220905 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:03:14,068-Speed 130.07 samples/sec Loss 0.4603 Epoch: 22 Global Step: 220910 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:03:34,409-Speed 125.86 samples/sec Loss 0.3256 Epoch: 22 Global Step: 220915 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:03:54,249-Speed 129.04 samples/sec Loss 0.3958 Epoch: 22 Global Step: 220920 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:04:14,142-Speed 128.69 samples/sec Loss 0.3121 Epoch: 22 Global Step: 220925 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:04:34,051-Speed 128.59 samples/sec Loss 0.3139 Epoch: 22 Global Step: 220930 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:04:54,112-Speed 127.62 samples/sec Loss 0.2852 Epoch: 22 Global Step: 220935 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:05:13,871-Speed 129.57 samples/sec Loss 0.3357 Epoch: 22 Global Step: 220940 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:05:33,669-Speed 129.31 samples/sec Loss 0.3420 Epoch: 22 Global Step: 220945 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:05:53,461-Speed 129.35 samples/sec Loss 0.3504 Epoch: 22 Global Step: 220950 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:06:13,255-Speed 129.34 samples/sec Loss 0.3093 Epoch: 22 Global Step: 220955 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:06:33,481-Speed 126.57 samples/sec Loss 0.3534 Epoch: 22 Global Step: 220960 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:06:53,322-Speed 129.03 samples/sec Loss 0.3177 Epoch: 22 Global Step: 220965 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:07:13,106-Speed 129.40 samples/sec Loss 0.3171 Epoch: 22 Global Step: 220970 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:07:33,033-Speed 128.47 samples/sec Loss 0.3411 Epoch: 22 Global Step: 220975 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:07:52,747-Speed 129.87 samples/sec Loss 0.3163 Epoch: 22 Global Step: 220980 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:08:12,443-Speed 129.98 samples/sec Loss 0.3410 Epoch: 22 Global Step: 220985 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:08:32,230-Speed 129.39 samples/sec Loss 0.3243 Epoch: 22 Global Step: 220990 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:08:52,034-Speed 129.28 samples/sec Loss 0.3401 Epoch: 22 Global Step: 220995 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:09:11,943-Speed 128.59 samples/sec Loss 0.3225 Epoch: 22 Global Step: 221000 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:09:32,065-Speed 127.23 samples/sec Loss 0.3693 Epoch: 22 Global Step: 221005 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:09:51,941-Speed 128.80 samples/sec Loss 0.3354 Epoch: 22 Global Step: 221010 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:10:11,640-Speed 129.96 samples/sec Loss 0.3202 Epoch: 22 Global Step: 221015 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:10:31,600-Speed 128.27 samples/sec Loss 0.4246 Epoch: 22 Global Step: 221020 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:10:51,306-Speed 129.91 samples/sec Loss 0.3328 Epoch: 22 Global Step: 221025 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:11:10,952-Speed 130.32 samples/sec Loss 0.3049 Epoch: 22 Global Step: 221030 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:11:30,931-Speed 128.14 samples/sec Loss 0.3619 Epoch: 22 Global Step: 221035 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:11:50,655-Speed 129.80 samples/sec Loss 0.3823 Epoch: 22 Global Step: 221040 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:12:10,553-Speed 128.66 samples/sec Loss 0.2702 Epoch: 22 Global Step: 221045 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:12:30,003-Speed 131.62 samples/sec Loss 0.3349 Epoch: 22 Global Step: 221050 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:12:50,085-Speed 127.49 samples/sec Loss 0.3251 Epoch: 22 Global Step: 221055 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:13:09,803-Speed 129.83 samples/sec Loss 0.3183 Epoch: 22 Global Step: 221060 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:13:29,705-Speed 128.64 samples/sec Loss 0.3030 Epoch: 22 Global Step: 221065 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:13:49,390-Speed 130.06 samples/sec Loss 0.3038 Epoch: 22 Global Step: 221070 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:14:09,089-Speed 129.96 samples/sec Loss 0.3901 Epoch: 22 Global Step: 221075 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:14:28,911-Speed 129.16 samples/sec Loss 0.3402 Epoch: 22 Global Step: 221080 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:14:48,686-Speed 129.46 samples/sec Loss 0.3379 Epoch: 22 Global Step: 221085 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:15:08,585-Speed 128.66 samples/sec Loss 0.3601 Epoch: 22 Global Step: 221090 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:15:28,647-Speed 127.61 samples/sec Loss 0.3105 Epoch: 22 Global Step: 221095 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:15:48,492-Speed 129.00 samples/sec Loss 0.2898 Epoch: 22 Global Step: 221100 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:16:08,090-Speed 130.63 samples/sec Loss 0.3593 Epoch: 22 Global Step: 221105 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:16:28,053-Speed 128.25 samples/sec Loss 0.3292 Epoch: 22 Global Step: 221110 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:16:47,740-Speed 130.04 samples/sec Loss 0.3071 Epoch: 22 Global Step: 221115 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:17:07,464-Speed 129.79 samples/sec Loss 0.3229 Epoch: 22 Global Step: 221120 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:17:27,250-Speed 129.40 samples/sec Loss 0.3045 Epoch: 22 Global Step: 221125 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:17:46,921-Speed 130.15 samples/sec Loss 0.3813 Epoch: 22 Global Step: 221130 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:18:07,040-Speed 127.25 samples/sec Loss 0.2806 Epoch: 22 Global Step: 221135 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:18:27,302-Speed 126.35 samples/sec Loss 0.4573 Epoch: 22 Global Step: 221140 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:18:47,008-Speed 129.91 samples/sec Loss 0.3210 Epoch: 22 Global Step: 221145 Fp16 Grad Scale: 4096 Required: 34 hours Training: 2021-06-17 20:19:06,644-Speed 130.38 samples/sec Loss 0.3187 Epoch: 22 Global Step: 221150 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:19:26,661-Speed 127.90 samples/sec Loss 0.3292 Epoch: 22 Global Step: 221155 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:19:46,270-Speed 130.56 samples/sec Loss 0.3359 Epoch: 22 Global Step: 221160 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:20:06,056-Speed 129.40 samples/sec Loss 0.3603 Epoch: 22 Global Step: 221165 Fp16 Grad Scale: 8192 Required: 34 hours Training: 2021-06-17 20:20:25,670-Speed 130.52 samples/sec Loss 0.3840 Epoch: 22 Global Step: 221170 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:20:45,391-Speed 129.82 samples/sec Loss 0.3605 Epoch: 22 Global Step: 221175 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:21:05,353-Speed 128.25 samples/sec Loss 0.3536 Epoch: 22 Global Step: 221180 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:21:25,484-Speed 127.17 samples/sec Loss 0.3097 Epoch: 22 Global Step: 221185 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:21:45,402-Speed 128.53 samples/sec Loss 0.3194 Epoch: 22 Global Step: 221190 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:22:05,118-Speed 129.85 samples/sec Loss 0.3204 Epoch: 22 Global Step: 221195 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:22:25,095-Speed 128.15 samples/sec Loss 0.4092 Epoch: 22 Global Step: 221200 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:22:44,752-Speed 130.24 samples/sec Loss 0.3599 Epoch: 22 Global Step: 221205 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:23:04,553-Speed 129.29 samples/sec Loss 0.2572 Epoch: 22 Global Step: 221210 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:23:24,280-Speed 129.78 samples/sec Loss 0.3274 Epoch: 22 Global Step: 221215 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:23:44,042-Speed 129.55 samples/sec Loss 0.3783 Epoch: 22 Global Step: 221220 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:24:04,058-Speed 127.91 samples/sec Loss 0.4143 Epoch: 22 Global Step: 221225 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:24:24,117-Speed 127.63 samples/sec Loss 0.3456 Epoch: 22 Global Step: 221230 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:24:43,823-Speed 129.92 samples/sec Loss 0.3461 Epoch: 22 Global Step: 221235 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:25:03,572-Speed 129.63 samples/sec Loss 0.3545 Epoch: 22 Global Step: 221240 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:25:23,505-Speed 128.44 samples/sec Loss 0.3059 Epoch: 22 Global Step: 221245 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:25:43,170-Speed 130.19 samples/sec Loss 0.3672 Epoch: 22 Global Step: 221250 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:26:02,955-Speed 129.40 samples/sec Loss 0.3380 Epoch: 22 Global Step: 221255 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:26:22,671-Speed 129.85 samples/sec Loss 0.3238 Epoch: 22 Global Step: 221260 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:26:42,564-Speed 128.70 samples/sec Loss 0.3296 Epoch: 22 Global Step: 221265 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:27:02,289-Speed 129.79 samples/sec Loss 0.3885 Epoch: 22 Global Step: 221270 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:27:22,540-Speed 126.42 samples/sec Loss 0.3392 Epoch: 22 Global Step: 221275 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:27:42,229-Speed 130.02 samples/sec Loss 0.3258 Epoch: 22 Global Step: 221280 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:28:01,981-Speed 129.61 samples/sec Loss 0.3687 Epoch: 22 Global Step: 221285 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:28:21,912-Speed 128.45 samples/sec Loss 0.3231 Epoch: 22 Global Step: 221290 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:28:41,563-Speed 130.28 samples/sec Loss 0.3591 Epoch: 22 Global Step: 221295 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:29:01,344-Speed 129.43 samples/sec Loss 0.3398 Epoch: 22 Global Step: 221300 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:29:21,076-Speed 129.75 samples/sec Loss 0.2667 Epoch: 22 Global Step: 221305 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:29:41,111-Speed 127.78 samples/sec Loss 0.3827 Epoch: 22 Global Step: 221310 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:30:00,844-Speed 129.74 samples/sec Loss 0.3228 Epoch: 22 Global Step: 221315 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:30:21,107-Speed 126.35 samples/sec Loss 0.2999 Epoch: 22 Global Step: 221320 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:30:40,824-Speed 129.84 samples/sec Loss 0.3370 Epoch: 22 Global Step: 221325 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:31:00,765-Speed 128.39 samples/sec Loss 0.3640 Epoch: 22 Global Step: 221330 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:31:20,500-Speed 129.73 samples/sec Loss 0.3091 Epoch: 22 Global Step: 221335 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:31:40,462-Speed 128.25 samples/sec Loss 0.3056 Epoch: 22 Global Step: 221340 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:32:00,224-Speed 129.55 samples/sec Loss 0.3619 Epoch: 22 Global Step: 221345 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:32:19,952-Speed 129.77 samples/sec Loss 0.3465 Epoch: 22 Global Step: 221350 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:32:39,851-Speed 128.66 samples/sec Loss 0.3451 Epoch: 22 Global Step: 221355 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:32:59,599-Speed 129.64 samples/sec Loss 0.2608 Epoch: 22 Global Step: 221360 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:33:19,919-Speed 125.99 samples/sec Loss 0.2805 Epoch: 22 Global Step: 221365 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:33:39,660-Speed 129.69 samples/sec Loss 0.3083 Epoch: 22 Global Step: 221370 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:33:59,542-Speed 128.76 samples/sec Loss 0.3194 Epoch: 22 Global Step: 221375 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:34:19,281-Speed 129.70 samples/sec Loss 0.3475 Epoch: 22 Global Step: 221380 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:34:38,664-Speed 132.08 samples/sec Loss 0.2993 Epoch: 22 Global Step: 221385 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:34:58,383-Speed 129.83 samples/sec Loss 0.3035 Epoch: 22 Global Step: 221390 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:35:18,115-Speed 129.75 samples/sec Loss 0.3612 Epoch: 22 Global Step: 221395 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:35:38,029-Speed 128.56 samples/sec Loss 0.2746 Epoch: 22 Global Step: 221400 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:35:57,768-Speed 129.71 samples/sec Loss 0.3254 Epoch: 22 Global Step: 221405 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:36:18,040-Speed 126.29 samples/sec Loss 0.3361 Epoch: 22 Global Step: 221410 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:36:37,795-Speed 129.59 samples/sec Loss 0.2959 Epoch: 22 Global Step: 221415 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:36:57,553-Speed 129.57 samples/sec Loss 0.3287 Epoch: 22 Global Step: 221420 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:37:17,290-Speed 129.71 samples/sec Loss 0.3376 Epoch: 22 Global Step: 221425 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:37:37,362-Speed 127.55 samples/sec Loss 0.3634 Epoch: 22 Global Step: 221430 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:37:57,044-Speed 130.08 samples/sec Loss 0.3896 Epoch: 22 Global Step: 221435 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:38:17,065-Speed 127.87 samples/sec Loss 0.3067 Epoch: 22 Global Step: 221440 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:38:36,886-Speed 129.17 samples/sec Loss 0.3903 Epoch: 22 Global Step: 221445 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:38:56,655-Speed 129.50 samples/sec Loss 0.2968 Epoch: 22 Global Step: 221450 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:39:17,120-Speed 125.10 samples/sec Loss 0.2627 Epoch: 22 Global Step: 221455 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:39:37,045-Speed 128.48 samples/sec Loss 0.3219 Epoch: 22 Global Step: 221460 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:39:56,697-Speed 130.27 samples/sec Loss 0.3115 Epoch: 22 Global Step: 221465 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:40:16,580-Speed 128.76 samples/sec Loss 0.2608 Epoch: 22 Global Step: 221470 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:40:36,518-Speed 128.40 samples/sec Loss 0.3432 Epoch: 22 Global Step: 221475 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:40:56,219-Speed 129.95 samples/sec Loss 0.3285 Epoch: 22 Global Step: 221480 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:41:16,048-Speed 129.11 samples/sec Loss 0.3043 Epoch: 22 Global Step: 221485 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:41:35,746-Speed 129.97 samples/sec Loss 0.3855 Epoch: 22 Global Step: 221490 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:41:55,492-Speed 129.65 samples/sec Loss 0.3142 Epoch: 22 Global Step: 221495 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:42:15,920-Speed 125.32 samples/sec Loss 0.2635 Epoch: 22 Global Step: 221500 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:42:35,856-Speed 128.42 samples/sec Loss 0.3221 Epoch: 22 Global Step: 221505 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:42:55,532-Speed 130.11 samples/sec Loss 0.3635 Epoch: 22 Global Step: 221510 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:43:15,303-Speed 129.49 samples/sec Loss 0.3714 Epoch: 22 Global Step: 221515 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:43:35,248-Speed 128.36 samples/sec Loss 0.3186 Epoch: 22 Global Step: 221520 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:43:54,967-Speed 129.83 samples/sec Loss 0.3386 Epoch: 22 Global Step: 221525 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:44:14,796-Speed 129.11 samples/sec Loss 0.3499 Epoch: 22 Global Step: 221530 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:44:34,445-Speed 130.29 samples/sec Loss 0.3069 Epoch: 22 Global Step: 221535 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:44:54,207-Speed 129.55 samples/sec Loss 0.2801 Epoch: 22 Global Step: 221540 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:45:14,657-Speed 125.19 samples/sec Loss 0.3062 Epoch: 22 Global Step: 221545 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:45:34,920-Speed 126.34 samples/sec Loss 0.2972 Epoch: 22 Global Step: 221550 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:45:54,627-Speed 129.91 samples/sec Loss 0.2571 Epoch: 22 Global Step: 221555 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:46:14,512-Speed 128.75 samples/sec Loss 0.3135 Epoch: 22 Global Step: 221560 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:46:34,320-Speed 129.25 samples/sec Loss 0.2560 Epoch: 22 Global Step: 221565 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:46:54,083-Speed 129.54 samples/sec Loss 0.2858 Epoch: 22 Global Step: 221570 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:47:13,843-Speed 129.57 samples/sec Loss 0.3745 Epoch: 22 Global Step: 221575 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:47:33,576-Speed 129.74 samples/sec Loss 0.3809 Epoch: 22 Global Step: 221580 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:47:53,488-Speed 128.57 samples/sec Loss 0.3618 Epoch: 22 Global Step: 221585 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 20:48:13,175-Speed 130.04 samples/sec Loss 0.2865 Epoch: 22 Global Step: 221590 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:48:32,782-Speed 130.57 samples/sec Loss 0.2698 Epoch: 22 Global Step: 221595 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:48:52,913-Speed 127.17 samples/sec Loss 0.3500 Epoch: 22 Global Step: 221600 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:49:12,783-Speed 128.84 samples/sec Loss 0.2871 Epoch: 22 Global Step: 221605 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:49:32,544-Speed 129.55 samples/sec Loss 0.3560 Epoch: 22 Global Step: 221610 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:49:52,400-Speed 128.93 samples/sec Loss 0.3800 Epoch: 22 Global Step: 221615 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:50:11,936-Speed 131.05 samples/sec Loss 0.3866 Epoch: 22 Global Step: 221620 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:50:31,845-Speed 128.59 samples/sec Loss 0.2757 Epoch: 22 Global Step: 221625 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:50:51,185-Speed 132.38 samples/sec Loss 0.3702 Epoch: 22 Global Step: 221630 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:51:10,950-Speed 129.53 samples/sec Loss 0.3175 Epoch: 22 Global Step: 221635 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:51:30,774-Speed 129.15 samples/sec Loss 0.3577 Epoch: 22 Global Step: 221640 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:51:50,395-Speed 130.48 samples/sec Loss 0.3677 Epoch: 22 Global Step: 221645 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:52:10,787-Speed 125.55 samples/sec Loss 0.3491 Epoch: 22 Global Step: 221650 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:52:30,470-Speed 130.07 samples/sec Loss 0.3010 Epoch: 22 Global Step: 221655 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:52:50,852-Speed 125.60 samples/sec Loss 0.4113 Epoch: 22 Global Step: 221660 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:53:10,462-Speed 130.55 samples/sec Loss 0.2708 Epoch: 22 Global Step: 221665 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:53:30,346-Speed 128.75 samples/sec Loss 0.3593 Epoch: 22 Global Step: 221670 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:53:50,118-Speed 129.49 samples/sec Loss 0.3302 Epoch: 22 Global Step: 221675 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:54:09,875-Speed 129.58 samples/sec Loss 0.3333 Epoch: 22 Global Step: 221680 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:54:29,716-Speed 129.03 samples/sec Loss 0.3252 Epoch: 22 Global Step: 221685 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:54:49,334-Speed 130.50 samples/sec Loss 0.3568 Epoch: 22 Global Step: 221690 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:55:09,242-Speed 128.60 samples/sec Loss 0.3222 Epoch: 22 Global Step: 221695 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:55:29,034-Speed 129.35 samples/sec Loss 0.3730 Epoch: 22 Global Step: 221700 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:55:49,385-Speed 125.80 samples/sec Loss 0.3329 Epoch: 22 Global Step: 221705 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:56:09,043-Speed 130.23 samples/sec Loss 0.3603 Epoch: 22 Global Step: 221710 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:56:28,921-Speed 128.79 samples/sec Loss 0.3001 Epoch: 22 Global Step: 221715 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:56:48,643-Speed 129.81 samples/sec Loss 0.2753 Epoch: 22 Global Step: 221720 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:57:08,540-Speed 128.67 samples/sec Loss 0.3001 Epoch: 22 Global Step: 221725 Fp16 Grad Scale: 2048 Required: 33 hours Training: 2021-06-17 20:57:28,247-Speed 129.91 samples/sec Loss 0.3099 Epoch: 22 Global Step: 221730 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:57:48,013-Speed 129.52 samples/sec Loss 0.2821 Epoch: 22 Global Step: 221735 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:58:07,901-Speed 128.73 samples/sec Loss 0.4007 Epoch: 22 Global Step: 221740 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:58:27,664-Speed 129.54 samples/sec Loss 0.3133 Epoch: 22 Global Step: 221745 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:58:47,913-Speed 126.43 samples/sec Loss 0.3648 Epoch: 22 Global Step: 221750 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:59:07,581-Speed 130.17 samples/sec Loss 0.2819 Epoch: 22 Global Step: 221755 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:59:27,455-Speed 128.81 samples/sec Loss 0.3079 Epoch: 22 Global Step: 221760 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 20:59:47,248-Speed 129.35 samples/sec Loss 0.3739 Epoch: 22 Global Step: 221765 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:00:07,134-Speed 128.75 samples/sec Loss 0.3228 Epoch: 22 Global Step: 221770 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:00:26,735-Speed 130.61 samples/sec Loss 0.3647 Epoch: 22 Global Step: 221775 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:00:46,520-Speed 129.40 samples/sec Loss 0.3839 Epoch: 22 Global Step: 221780 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:01:06,394-Speed 128.82 samples/sec Loss 0.3110 Epoch: 22 Global Step: 221785 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:01:26,118-Speed 129.79 samples/sec Loss 0.3400 Epoch: 22 Global Step: 221790 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:01:46,458-Speed 125.87 samples/sec Loss 0.3263 Epoch: 22 Global Step: 221795 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:02:06,253-Speed 129.33 samples/sec Loss 0.2753 Epoch: 22 Global Step: 221800 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:02:25,990-Speed 129.71 samples/sec Loss 0.3042 Epoch: 22 Global Step: 221805 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:02:45,806-Speed 129.20 samples/sec Loss 0.2888 Epoch: 22 Global Step: 221810 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:03:05,713-Speed 128.61 samples/sec Loss 0.3173 Epoch: 22 Global Step: 221815 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:03:25,320-Speed 130.57 samples/sec Loss 0.2768 Epoch: 22 Global Step: 221820 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:03:45,125-Speed 129.26 samples/sec Loss 0.3212 Epoch: 22 Global Step: 221825 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:04:04,880-Speed 129.60 samples/sec Loss 0.3248 Epoch: 22 Global Step: 221830 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 21:04:24,683-Speed 129.28 samples/sec Loss 0.3262 Epoch: 22 Global Step: 221835 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 21:04:44,978-Speed 126.15 samples/sec Loss 0.3057 Epoch: 22 Global Step: 221840 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 21:05:04,771-Speed 129.35 samples/sec Loss 0.3302 Epoch: 22 Global Step: 221845 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 21:05:24,501-Speed 129.76 samples/sec Loss 0.3174 Epoch: 22 Global Step: 221850 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 21:05:43,832-Speed 132.43 samples/sec Loss 0.3416 Epoch: 22 Global Step: 221855 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:06:03,839-Speed 127.97 samples/sec Loss 0.3068 Epoch: 22 Global Step: 221860 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:06:23,471-Speed 130.41 samples/sec Loss 0.3602 Epoch: 22 Global Step: 221865 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:06:43,121-Speed 130.28 samples/sec Loss 0.3380 Epoch: 22 Global Step: 221870 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:07:03,014-Speed 128.70 samples/sec Loss 0.3112 Epoch: 22 Global Step: 221875 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:07:22,723-Speed 129.89 samples/sec Loss 0.2625 Epoch: 22 Global Step: 221880 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:07:43,100-Speed 125.64 samples/sec Loss 0.3164 Epoch: 22 Global Step: 221885 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:08:02,754-Speed 130.26 samples/sec Loss 0.3287 Epoch: 22 Global Step: 221890 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:08:22,627-Speed 128.82 samples/sec Loss 0.4214 Epoch: 22 Global Step: 221895 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:08:42,337-Speed 129.90 samples/sec Loss 0.3798 Epoch: 22 Global Step: 221900 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:09:02,263-Speed 128.48 samples/sec Loss 0.3354 Epoch: 22 Global Step: 221905 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:09:21,985-Speed 129.81 samples/sec Loss 0.3191 Epoch: 22 Global Step: 221910 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:09:41,643-Speed 130.24 samples/sec Loss 0.3616 Epoch: 22 Global Step: 221915 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:10:01,535-Speed 128.70 samples/sec Loss 0.3085 Epoch: 22 Global Step: 221920 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:10:21,273-Speed 129.71 samples/sec Loss 0.3574 Epoch: 22 Global Step: 221925 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:10:41,635-Speed 125.73 samples/sec Loss 0.2965 Epoch: 22 Global Step: 221930 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:11:01,443-Speed 129.24 samples/sec Loss 0.2853 Epoch: 22 Global Step: 221935 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:11:21,182-Speed 129.70 samples/sec Loss 0.3804 Epoch: 22 Global Step: 221940 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:11:40,999-Speed 129.18 samples/sec Loss 0.3520 Epoch: 22 Global Step: 221945 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:12:00,920-Speed 128.52 samples/sec Loss 0.3175 Epoch: 22 Global Step: 221950 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:12:20,638-Speed 129.84 samples/sec Loss 0.3196 Epoch: 22 Global Step: 221955 Fp16 Grad Scale: 8192 Required: 33 hours Training: 2021-06-17 21:12:40,001-Speed 132.22 samples/sec Loss 0.2923 Epoch: 22 Global Step: 221960 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:12:59,724-Speed 129.81 samples/sec Loss 0.3311 Epoch: 22 Global Step: 221965 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:13:19,729-Speed 127.98 samples/sec Loss 0.3019 Epoch: 22 Global Step: 221970 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:13:39,605-Speed 128.81 samples/sec Loss 0.3521 Epoch: 22 Global Step: 221975 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:13:59,268-Speed 130.20 samples/sec Loss 0.2693 Epoch: 22 Global Step: 221980 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:14:19,042-Speed 129.47 samples/sec Loss 0.3412 Epoch: 22 Global Step: 221985 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:14:38,728-Speed 130.05 samples/sec Loss 0.3218 Epoch: 22 Global Step: 221990 Fp16 Grad Scale: 4096 Required: 33 hours Training: 2021-06-17 21:14:58,611-Speed 128.76 samples/sec Loss 0.3144 Epoch: 22 Global Step: 221995 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:15:18,756-Speed 127.08 samples/sec Loss 0.3053 Epoch: 22 Global Step: 222000 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:15:38,525-Speed 129.50 samples/sec Loss 0.3375 Epoch: 22 Global Step: 222005 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:15:58,265-Speed 129.69 samples/sec Loss 0.3603 Epoch: 22 Global Step: 222010 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:16:18,136-Speed 128.84 samples/sec Loss 0.3280 Epoch: 22 Global Step: 222015 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:16:37,921-Speed 129.40 samples/sec Loss 0.3267 Epoch: 22 Global Step: 222020 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:16:57,732-Speed 129.23 samples/sec Loss 0.2351 Epoch: 22 Global Step: 222025 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:17:17,421-Speed 130.03 samples/sec Loss 0.2567 Epoch: 22 Global Step: 222030 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:17:37,172-Speed 129.63 samples/sec Loss 0.2581 Epoch: 22 Global Step: 222035 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:17:57,084-Speed 128.57 samples/sec Loss 0.2890 Epoch: 22 Global Step: 222040 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:18:17,322-Speed 126.50 samples/sec Loss 0.3392 Epoch: 22 Global Step: 222045 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:18:37,189-Speed 128.86 samples/sec Loss 0.3276 Epoch: 22 Global Step: 222050 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:18:56,937-Speed 129.64 samples/sec Loss 0.2908 Epoch: 22 Global Step: 222055 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:19:16,803-Speed 128.87 samples/sec Loss 0.3366 Epoch: 22 Global Step: 222060 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:19:36,525-Speed 129.81 samples/sec Loss 0.3315 Epoch: 22 Global Step: 222065 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:19:56,427-Speed 128.64 samples/sec Loss 0.2678 Epoch: 22 Global Step: 222070 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:20:15,711-Speed 132.76 samples/sec Loss 0.3918 Epoch: 22 Global Step: 222075 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:20:35,469-Speed 129.58 samples/sec Loss 0.3435 Epoch: 22 Global Step: 222080 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:20:55,268-Speed 129.30 samples/sec Loss 0.3375 Epoch: 22 Global Step: 222085 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:21:15,745-Speed 125.03 samples/sec Loss 0.3741 Epoch: 22 Global Step: 222090 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:21:35,457-Speed 129.88 samples/sec Loss 0.3209 Epoch: 22 Global Step: 222095 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:21:55,273-Speed 129.19 samples/sec Loss 0.3886 Epoch: 22 Global Step: 222100 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:22:15,079-Speed 129.26 samples/sec Loss 0.3204 Epoch: 22 Global Step: 222105 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:22:34,868-Speed 129.37 samples/sec Loss 0.3678 Epoch: 22 Global Step: 222110 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:22:54,775-Speed 128.60 samples/sec Loss 0.3242 Epoch: 22 Global Step: 222115 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:23:14,483-Speed 129.91 samples/sec Loss 0.3229 Epoch: 22 Global Step: 222120 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:23:34,326-Speed 129.01 samples/sec Loss 0.3249 Epoch: 22 Global Step: 222125 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:23:54,039-Speed 129.87 samples/sec Loss 0.3332 Epoch: 22 Global Step: 222130 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:24:14,498-Speed 125.14 samples/sec Loss 0.4384 Epoch: 22 Global Step: 222135 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:24:34,221-Speed 129.80 samples/sec Loss 0.3281 Epoch: 22 Global Step: 222140 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:24:54,032-Speed 129.22 samples/sec Loss 0.3073 Epoch: 22 Global Step: 222145 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:25:13,749-Speed 129.85 samples/sec Loss 0.3182 Epoch: 22 Global Step: 222150 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:25:33,710-Speed 128.26 samples/sec Loss 0.3616 Epoch: 22 Global Step: 222155 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:25:53,532-Speed 129.16 samples/sec Loss 0.3411 Epoch: 22 Global Step: 222160 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:26:13,251-Speed 129.83 samples/sec Loss 0.3478 Epoch: 22 Global Step: 222165 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:26:33,110-Speed 128.92 samples/sec Loss 0.3228 Epoch: 22 Global Step: 222170 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:26:52,833-Speed 129.80 samples/sec Loss 0.2731 Epoch: 22 Global Step: 222175 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:27:13,166-Speed 125.90 samples/sec Loss 0.3667 Epoch: 22 Global Step: 222180 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:27:32,939-Speed 129.48 samples/sec Loss 0.3293 Epoch: 22 Global Step: 222185 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:27:52,704-Speed 129.53 samples/sec Loss 0.3256 Epoch: 22 Global Step: 222190 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:28:12,417-Speed 129.87 samples/sec Loss 0.3774 Epoch: 22 Global Step: 222195 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:28:32,307-Speed 128.72 samples/sec Loss 0.3120 Epoch: 22 Global Step: 222200 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:28:52,080-Speed 129.47 samples/sec Loss 0.3378 Epoch: 22 Global Step: 222205 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:29:11,855-Speed 129.46 samples/sec Loss 0.3446 Epoch: 22 Global Step: 222210 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:29:31,624-Speed 129.51 samples/sec Loss 0.3401 Epoch: 22 Global Step: 222215 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:29:50,947-Speed 132.49 samples/sec Loss 0.3584 Epoch: 22 Global Step: 222220 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:30:11,374-Speed 125.33 samples/sec Loss 0.2855 Epoch: 22 Global Step: 222225 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:30:31,228-Speed 128.95 samples/sec Loss 0.3103 Epoch: 22 Global Step: 222230 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:30:50,950-Speed 129.81 samples/sec Loss 0.3349 Epoch: 22 Global Step: 222235 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:31:10,678-Speed 129.77 samples/sec Loss 0.2811 Epoch: 22 Global Step: 222240 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:31:30,563-Speed 128.75 samples/sec Loss 0.3455 Epoch: 22 Global Step: 222245 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:31:50,352-Speed 129.37 samples/sec Loss 0.3686 Epoch: 22 Global Step: 222250 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:32:10,263-Speed 128.58 samples/sec Loss 0.3908 Epoch: 22 Global Step: 222255 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:32:30,087-Speed 129.16 samples/sec Loss 0.3617 Epoch: 22 Global Step: 222260 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:32:49,826-Speed 129.70 samples/sec Loss 0.3554 Epoch: 22 Global Step: 222265 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:33:10,298-Speed 125.06 samples/sec Loss 0.3076 Epoch: 22 Global Step: 222270 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:33:30,219-Speed 128.51 samples/sec Loss 0.2688 Epoch: 22 Global Step: 222275 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:33:49,846-Speed 130.43 samples/sec Loss 0.3606 Epoch: 22 Global Step: 222280 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:34:09,604-Speed 129.58 samples/sec Loss 0.4140 Epoch: 22 Global Step: 222285 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:34:29,488-Speed 128.76 samples/sec Loss 0.2845 Epoch: 22 Global Step: 222290 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:34:49,245-Speed 129.58 samples/sec Loss 0.3218 Epoch: 22 Global Step: 222295 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:35:09,815-Speed 124.46 samples/sec Loss 0.3391 Epoch: 22 Global Step: 222300 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:35:29,454-Speed 130.36 samples/sec Loss 0.3471 Epoch: 22 Global Step: 222305 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:35:49,249-Speed 129.33 samples/sec Loss 0.2601 Epoch: 22 Global Step: 222310 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:36:09,552-Speed 126.10 samples/sec Loss 0.4136 Epoch: 22 Global Step: 222315 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:36:29,492-Speed 128.38 samples/sec Loss 0.3189 Epoch: 22 Global Step: 222320 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:36:49,113-Speed 130.48 samples/sec Loss 0.3410 Epoch: 22 Global Step: 222325 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:37:09,027-Speed 128.56 samples/sec Loss 0.3213 Epoch: 22 Global Step: 222330 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:37:28,661-Speed 130.40 samples/sec Loss 0.3075 Epoch: 22 Global Step: 222335 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:37:48,609-Speed 128.34 samples/sec Loss 0.3272 Epoch: 22 Global Step: 222340 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:38:08,575-Speed 128.23 samples/sec Loss 0.3268 Epoch: 22 Global Step: 222345 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:38:28,214-Speed 130.36 samples/sec Loss 0.2795 Epoch: 22 Global Step: 222350 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:38:48,175-Speed 128.26 samples/sec Loss 0.3161 Epoch: 22 Global Step: 222355 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:39:08,287-Speed 127.29 samples/sec Loss 0.3412 Epoch: 22 Global Step: 222360 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:39:28,216-Speed 128.46 samples/sec Loss 0.3180 Epoch: 22 Global Step: 222365 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:39:47,833-Speed 130.51 samples/sec Loss 0.3722 Epoch: 22 Global Step: 222370 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:40:07,735-Speed 128.64 samples/sec Loss 0.3494 Epoch: 22 Global Step: 222375 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:40:27,447-Speed 129.88 samples/sec Loss 0.2986 Epoch: 22 Global Step: 222380 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:40:47,326-Speed 128.79 samples/sec Loss 0.3079 Epoch: 22 Global Step: 222385 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:41:07,049-Speed 129.80 samples/sec Loss 0.3540 Epoch: 22 Global Step: 222390 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:41:26,697-Speed 130.30 samples/sec Loss 0.3673 Epoch: 22 Global Step: 222395 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:41:46,603-Speed 128.61 samples/sec Loss 0.3572 Epoch: 22 Global Step: 222400 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:42:06,821-Speed 126.63 samples/sec Loss 0.3475 Epoch: 22 Global Step: 222405 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:42:26,703-Speed 128.76 samples/sec Loss 0.2682 Epoch: 22 Global Step: 222410 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:42:46,358-Speed 130.26 samples/sec Loss 0.2991 Epoch: 22 Global Step: 222415 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:43:06,222-Speed 128.88 samples/sec Loss 0.3068 Epoch: 22 Global Step: 222420 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:43:25,949-Speed 129.77 samples/sec Loss 0.3483 Epoch: 22 Global Step: 222425 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:43:45,791-Speed 129.03 samples/sec Loss 0.3197 Epoch: 22 Global Step: 222430 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:44:05,738-Speed 128.35 samples/sec Loss 0.3979 Epoch: 22 Global Step: 222435 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:44:25,472-Speed 129.74 samples/sec Loss 0.3952 Epoch: 22 Global Step: 222440 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:44:45,445-Speed 128.18 samples/sec Loss 0.2938 Epoch: 22 Global Step: 222445 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:45:05,743-Speed 126.13 samples/sec Loss 0.2876 Epoch: 22 Global Step: 222450 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:45:25,577-Speed 129.08 samples/sec Loss 0.3846 Epoch: 22 Global Step: 222455 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:45:45,345-Speed 129.51 samples/sec Loss 0.3385 Epoch: 22 Global Step: 222460 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:46:05,230-Speed 128.75 samples/sec Loss 0.2898 Epoch: 22 Global Step: 222465 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:46:24,968-Speed 129.70 samples/sec Loss 0.3920 Epoch: 22 Global Step: 222470 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:46:44,887-Speed 128.53 samples/sec Loss 0.3042 Epoch: 22 Global Step: 222475 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:47:04,683-Speed 129.32 samples/sec Loss 0.3161 Epoch: 22 Global Step: 222480 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:47:24,582-Speed 128.66 samples/sec Loss 0.3842 Epoch: 22 Global Step: 222485 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:47:44,363-Speed 129.42 samples/sec Loss 0.3374 Epoch: 22 Global Step: 222490 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:48:04,748-Speed 125.59 samples/sec Loss 0.3218 Epoch: 22 Global Step: 222495 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:48:24,523-Speed 129.46 samples/sec Loss 0.3158 Epoch: 22 Global Step: 222500 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:48:44,421-Speed 128.66 samples/sec Loss 0.3834 Epoch: 22 Global Step: 222505 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:49:04,149-Speed 129.78 samples/sec Loss 0.3529 Epoch: 22 Global Step: 222510 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:49:23,884-Speed 129.72 samples/sec Loss 0.3157 Epoch: 22 Global Step: 222515 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:49:43,729-Speed 129.01 samples/sec Loss 0.3219 Epoch: 22 Global Step: 222520 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:50:03,499-Speed 129.50 samples/sec Loss 0.3179 Epoch: 22 Global Step: 222525 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:50:23,359-Speed 128.91 samples/sec Loss 0.4151 Epoch: 22 Global Step: 222530 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:50:43,063-Speed 129.93 samples/sec Loss 0.3275 Epoch: 22 Global Step: 222535 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:51:03,362-Speed 126.12 samples/sec Loss 0.3627 Epoch: 22 Global Step: 222540 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:51:23,092-Speed 129.76 samples/sec Loss 0.3615 Epoch: 22 Global Step: 222545 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:51:43,000-Speed 128.60 samples/sec Loss 0.3480 Epoch: 22 Global Step: 222550 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:52:02,707-Speed 129.91 samples/sec Loss 0.3178 Epoch: 22 Global Step: 222555 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:52:22,510-Speed 129.28 samples/sec Loss 0.3752 Epoch: 22 Global Step: 222560 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:52:42,341-Speed 129.10 samples/sec Loss 0.3481 Epoch: 22 Global Step: 222565 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:53:02,137-Speed 129.32 samples/sec Loss 0.3810 Epoch: 22 Global Step: 222570 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:53:21,939-Speed 129.29 samples/sec Loss 0.3622 Epoch: 22 Global Step: 222575 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:53:41,676-Speed 129.71 samples/sec Loss 0.3081 Epoch: 22 Global Step: 222580 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:54:02,003-Speed 125.95 samples/sec Loss 0.3459 Epoch: 22 Global Step: 222585 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:54:21,819-Speed 129.19 samples/sec Loss 0.2540 Epoch: 22 Global Step: 222590 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:54:41,578-Speed 129.57 samples/sec Loss 0.3584 Epoch: 22 Global Step: 222595 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:55:01,325-Speed 129.65 samples/sec Loss 0.2814 Epoch: 22 Global Step: 222600 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:55:21,193-Speed 128.85 samples/sec Loss 0.3536 Epoch: 22 Global Step: 222605 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:55:40,795-Speed 130.61 samples/sec Loss 0.3830 Epoch: 22 Global Step: 222610 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:56:00,526-Speed 129.75 samples/sec Loss 0.3576 Epoch: 22 Global Step: 222615 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:56:20,382-Speed 128.94 samples/sec Loss 0.3037 Epoch: 22 Global Step: 222620 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:56:40,189-Speed 129.26 samples/sec Loss 0.2846 Epoch: 22 Global Step: 222625 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:57:00,483-Speed 126.15 samples/sec Loss 0.3325 Epoch: 22 Global Step: 222630 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:57:20,397-Speed 128.56 samples/sec Loss 0.2923 Epoch: 22 Global Step: 222635 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:57:40,097-Speed 129.95 samples/sec Loss 0.3454 Epoch: 22 Global Step: 222640 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:57:59,856-Speed 129.57 samples/sec Loss 0.3158 Epoch: 22 Global Step: 222645 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:58:19,697-Speed 129.03 samples/sec Loss 0.2996 Epoch: 22 Global Step: 222650 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 21:58:38,842-Speed 133.73 samples/sec Loss 0.3040 Epoch: 22 Global Step: 222655 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:58:58,803-Speed 128.26 samples/sec Loss 0.2659 Epoch: 22 Global Step: 222660 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:59:18,503-Speed 129.96 samples/sec Loss 0.3391 Epoch: 22 Global Step: 222665 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:59:38,255-Speed 129.61 samples/sec Loss 0.3166 Epoch: 22 Global Step: 222670 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 21:59:58,127-Speed 128.83 samples/sec Loss 0.3271 Epoch: 22 Global Step: 222675 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:00:17,782-Speed 130.26 samples/sec Loss 0.2899 Epoch: 22 Global Step: 222680 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:00:37,935-Speed 127.03 samples/sec Loss 0.3801 Epoch: 22 Global Step: 222685 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:00:57,646-Speed 129.88 samples/sec Loss 0.3330 Epoch: 22 Global Step: 222690 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:01:17,058-Speed 131.88 samples/sec Loss 0.2921 Epoch: 22 Global Step: 222695 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:01:37,069-Speed 127.94 samples/sec Loss 0.3382 Epoch: 22 Global Step: 222700 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:01:56,986-Speed 128.54 samples/sec Loss 0.3169 Epoch: 22 Global Step: 222705 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:02:16,478-Speed 131.34 samples/sec Loss 0.3389 Epoch: 22 Global Step: 222710 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:02:36,377-Speed 128.65 samples/sec Loss 0.3945 Epoch: 22 Global Step: 222715 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:02:55,813-Speed 131.73 samples/sec Loss 0.3450 Epoch: 22 Global Step: 222720 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:03:16,015-Speed 126.73 samples/sec Loss 0.2953 Epoch: 22 Global Step: 222725 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:03:35,785-Speed 129.49 samples/sec Loss 0.2919 Epoch: 22 Global Step: 222730 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:03:55,705-Speed 128.53 samples/sec Loss 0.3266 Epoch: 22 Global Step: 222735 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:04:15,149-Speed 131.66 samples/sec Loss 0.3090 Epoch: 22 Global Step: 222740 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:04:34,927-Speed 129.44 samples/sec Loss 0.3661 Epoch: 22 Global Step: 222745 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:04:54,812-Speed 128.75 samples/sec Loss 0.3253 Epoch: 22 Global Step: 222750 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:05:14,429-Speed 130.50 samples/sec Loss 0.2865 Epoch: 22 Global Step: 222755 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 22:05:34,262-Speed 129.08 samples/sec Loss 0.3391 Epoch: 22 Global Step: 222760 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 22:05:53,786-Speed 131.13 samples/sec Loss 0.2693 Epoch: 22 Global Step: 222765 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 22:06:13,447-Speed 130.22 samples/sec Loss 0.3544 Epoch: 22 Global Step: 222770 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 22:06:33,248-Speed 129.29 samples/sec Loss 0.3695 Epoch: 22 Global Step: 222775 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 22:06:53,207-Speed 128.27 samples/sec Loss 0.3263 Epoch: 22 Global Step: 222780 Fp16 Grad Scale: 8192 Required: 32 hours Training: 2021-06-17 22:07:12,203-Speed 134.77 samples/sec Loss 0.3992 Epoch: 22 Global Step: 222785 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:07:32,165-Speed 128.25 samples/sec Loss 0.4098 Epoch: 22 Global Step: 222790 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:07:51,880-Speed 129.86 samples/sec Loss 0.3208 Epoch: 22 Global Step: 222795 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:08:11,455-Speed 130.78 samples/sec Loss 0.3757 Epoch: 22 Global Step: 222800 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:08:31,314-Speed 128.92 samples/sec Loss 0.3740 Epoch: 22 Global Step: 222805 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:08:51,046-Speed 129.75 samples/sec Loss 0.3329 Epoch: 22 Global Step: 222810 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:09:10,833-Speed 129.39 samples/sec Loss 0.2668 Epoch: 22 Global Step: 222815 Fp16 Grad Scale: 4096 Required: 32 hours Training: 2021-06-17 22:09:30,589-Speed 129.59 samples/sec Loss 0.2869 Epoch: 22 Global Step: 222820 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:09:50,520-Speed 128.45 samples/sec Loss 0.3946 Epoch: 22 Global Step: 222825 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:10:09,954-Speed 131.73 samples/sec Loss 0.3569 Epoch: 22 Global Step: 222830 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:10:29,840-Speed 128.74 samples/sec Loss 0.4020 Epoch: 22 Global Step: 222835 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:10:49,557-Speed 129.85 samples/sec Loss 0.3329 Epoch: 22 Global Step: 222840 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:11:09,103-Speed 130.98 samples/sec Loss 0.3425 Epoch: 22 Global Step: 222845 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:11:29,098-Speed 128.04 samples/sec Loss 0.2653 Epoch: 22 Global Step: 222850 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:11:48,632-Speed 131.06 samples/sec Loss 0.3901 Epoch: 22 Global Step: 222855 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:12:08,425-Speed 129.34 samples/sec Loss 0.2883 Epoch: 22 Global Step: 222860 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:12:28,164-Speed 129.70 samples/sec Loss 0.3340 Epoch: 22 Global Step: 222865 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:12:48,039-Speed 128.81 samples/sec Loss 0.3077 Epoch: 22 Global Step: 222870 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:13:07,455-Speed 131.86 samples/sec Loss 0.2918 Epoch: 22 Global Step: 222875 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:13:27,370-Speed 128.55 samples/sec Loss 0.3774 Epoch: 22 Global Step: 222880 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:13:47,077-Speed 129.91 samples/sec Loss 0.3505 Epoch: 22 Global Step: 222885 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:14:06,621-Speed 130.99 samples/sec Loss 0.3133 Epoch: 22 Global Step: 222890 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:14:26,522-Speed 128.65 samples/sec Loss 0.3576 Epoch: 22 Global Step: 222895 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:14:46,058-Speed 131.05 samples/sec Loss 0.2990 Epoch: 22 Global Step: 222900 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:15:05,835-Speed 129.45 samples/sec Loss 0.3307 Epoch: 22 Global Step: 222905 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:15:25,512-Speed 130.10 samples/sec Loss 0.3298 Epoch: 22 Global Step: 222910 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:15:45,393-Speed 128.78 samples/sec Loss 0.3704 Epoch: 22 Global Step: 222915 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:16:04,858-Speed 131.52 samples/sec Loss 0.3214 Epoch: 22 Global Step: 222920 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:16:24,642-Speed 129.41 samples/sec Loss 0.3755 Epoch: 22 Global Step: 222925 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:16:44,339-Speed 129.98 samples/sec Loss 0.3056 Epoch: 22 Global Step: 222930 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:17:03,878-Speed 131.03 samples/sec Loss 0.3463 Epoch: 22 Global Step: 222935 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:17:23,795-Speed 128.54 samples/sec Loss 0.3345 Epoch: 22 Global Step: 222940 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:17:42,790-Speed 134.78 samples/sec Loss 0.3479 Epoch: 22 Global Step: 222945 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:18:02,669-Speed 128.78 samples/sec Loss 0.3175 Epoch: 22 Global Step: 222950 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:18:22,430-Speed 129.55 samples/sec Loss 0.4130 Epoch: 22 Global Step: 222955 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:18:42,316-Speed 128.74 samples/sec Loss 0.3305 Epoch: 22 Global Step: 222960 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:19:01,844-Speed 131.10 samples/sec Loss 0.3629 Epoch: 22 Global Step: 222965 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:19:21,736-Speed 128.70 samples/sec Loss 0.2841 Epoch: 22 Global Step: 222970 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:19:41,446-Speed 129.89 samples/sec Loss 0.3656 Epoch: 22 Global Step: 222975 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:20:01,795-Speed 125.81 samples/sec Loss 0.3312 Epoch: 22 Global Step: 222980 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:20:21,665-Speed 128.85 samples/sec Loss 0.3374 Epoch: 22 Global Step: 222985 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:20:41,271-Speed 130.58 samples/sec Loss 0.3307 Epoch: 22 Global Step: 222990 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:21:01,061-Speed 129.37 samples/sec Loss 0.2686 Epoch: 22 Global Step: 222995 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:21:20,821-Speed 129.56 samples/sec Loss 0.3441 Epoch: 22 Global Step: 223000 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:21:40,750-Speed 128.46 samples/sec Loss 0.2639 Epoch: 22 Global Step: 223005 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:22:00,160-Speed 131.90 samples/sec Loss 0.3607 Epoch: 22 Global Step: 223010 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:22:20,139-Speed 128.14 samples/sec Loss 0.3422 Epoch: 22 Global Step: 223015 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:22:39,846-Speed 129.91 samples/sec Loss 0.2696 Epoch: 22 Global Step: 223020 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:22:59,307-Speed 131.56 samples/sec Loss 0.3379 Epoch: 22 Global Step: 223025 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:23:19,198-Speed 128.71 samples/sec Loss 0.3851 Epoch: 22 Global Step: 223030 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:23:38,797-Speed 130.63 samples/sec Loss 0.4070 Epoch: 22 Global Step: 223035 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:23:58,588-Speed 129.36 samples/sec Loss 0.3144 Epoch: 22 Global Step: 223040 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:24:18,326-Speed 129.70 samples/sec Loss 0.3996 Epoch: 22 Global Step: 223045 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:24:38,143-Speed 129.19 samples/sec Loss 0.3137 Epoch: 22 Global Step: 223050 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:24:57,569-Speed 131.79 samples/sec Loss 0.2867 Epoch: 22 Global Step: 223055 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:25:17,398-Speed 129.11 samples/sec Loss 0.3443 Epoch: 22 Global Step: 223060 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:25:37,132-Speed 129.73 samples/sec Loss 0.3827 Epoch: 22 Global Step: 223065 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:25:56,677-Speed 130.99 samples/sec Loss 0.3438 Epoch: 22 Global Step: 223070 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:26:16,555-Speed 128.79 samples/sec Loss 0.2848 Epoch: 22 Global Step: 223075 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:26:36,011-Speed 131.58 samples/sec Loss 0.3570 Epoch: 22 Global Step: 223080 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:26:55,993-Speed 128.12 samples/sec Loss 0.3642 Epoch: 22 Global Step: 223085 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:27:15,678-Speed 130.06 samples/sec Loss 0.2930 Epoch: 22 Global Step: 223090 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:27:35,551-Speed 128.82 samples/sec Loss 0.3473 Epoch: 22 Global Step: 223095 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:27:54,961-Speed 131.90 samples/sec Loss 0.2898 Epoch: 22 Global Step: 223100 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:28:14,881-Speed 128.52 samples/sec Loss 0.3439 Epoch: 22 Global Step: 223105 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:28:34,611-Speed 129.76 samples/sec Loss 0.3717 Epoch: 22 Global Step: 223110 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:28:54,085-Speed 131.46 samples/sec Loss 0.3752 Epoch: 22 Global Step: 223115 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:29:14,041-Speed 128.29 samples/sec Loss 0.3583 Epoch: 22 Global Step: 223120 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:29:33,284-Speed 133.05 samples/sec Loss 0.3952 Epoch: 22 Global Step: 223125 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:29:53,172-Speed 128.73 samples/sec Loss 0.3232 Epoch: 22 Global Step: 223130 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:30:12,704-Speed 131.07 samples/sec Loss 0.2969 Epoch: 22 Global Step: 223135 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:30:32,406-Speed 129.95 samples/sec Loss 0.2803 Epoch: 22 Global Step: 223140 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:30:52,343-Speed 128.41 samples/sec Loss 0.2962 Epoch: 22 Global Step: 223145 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:31:12,257-Speed 128.56 samples/sec Loss 0.3531 Epoch: 22 Global Step: 223150 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:31:32,168-Speed 128.58 samples/sec Loss 0.3750 Epoch: 22 Global Step: 223155 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:31:51,944-Speed 129.45 samples/sec Loss 0.3884 Epoch: 22 Global Step: 223160 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:32:11,784-Speed 129.04 samples/sec Loss 0.4493 Epoch: 22 Global Step: 223165 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:32:31,544-Speed 129.56 samples/sec Loss 0.2711 Epoch: 22 Global Step: 223170 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:32:51,802-Speed 126.38 samples/sec Loss 0.3755 Epoch: 22 Global Step: 223175 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:33:11,515-Speed 129.87 samples/sec Loss 0.3084 Epoch: 22 Global Step: 223180 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:33:31,328-Speed 129.22 samples/sec Loss 0.3672 Epoch: 22 Global Step: 223185 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:33:51,211-Speed 128.76 samples/sec Loss 0.3625 Epoch: 22 Global Step: 223190 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:34:11,064-Speed 128.95 samples/sec Loss 0.3297 Epoch: 22 Global Step: 223195 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:34:30,861-Speed 129.31 samples/sec Loss 0.2963 Epoch: 22 Global Step: 223200 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:34:50,672-Speed 129.23 samples/sec Loss 0.3379 Epoch: 22 Global Step: 223205 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:35:10,430-Speed 129.58 samples/sec Loss 0.2925 Epoch: 22 Global Step: 223210 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:35:30,063-Speed 130.40 samples/sec Loss 0.4009 Epoch: 22 Global Step: 223215 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:35:50,210-Speed 127.07 samples/sec Loss 0.4527 Epoch: 22 Global Step: 223220 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:36:09,868-Speed 130.24 samples/sec Loss 0.3389 Epoch: 22 Global Step: 223225 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:36:29,762-Speed 128.69 samples/sec Loss 0.2806 Epoch: 22 Global Step: 223230 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:36:49,295-Speed 131.07 samples/sec Loss 0.3385 Epoch: 22 Global Step: 223235 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:37:09,128-Speed 129.08 samples/sec Loss 0.3300 Epoch: 22 Global Step: 223240 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:37:28,903-Speed 129.46 samples/sec Loss 0.3312 Epoch: 22 Global Step: 223245 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:37:48,698-Speed 129.33 samples/sec Loss 0.4319 Epoch: 22 Global Step: 223250 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:38:08,450-Speed 129.61 samples/sec Loss 0.2891 Epoch: 22 Global Step: 223255 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:38:28,326-Speed 128.81 samples/sec Loss 0.3487 Epoch: 22 Global Step: 223260 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:38:48,524-Speed 126.75 samples/sec Loss 0.3299 Epoch: 22 Global Step: 223265 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:39:08,317-Speed 129.35 samples/sec Loss 0.2895 Epoch: 22 Global Step: 223270 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:39:28,798-Speed 125.00 samples/sec Loss 0.3467 Epoch: 22 Global Step: 223275 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:39:48,793-Speed 128.04 samples/sec Loss 0.3296 Epoch: 22 Global Step: 223280 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:40:08,522-Speed 129.76 samples/sec Loss 0.3295 Epoch: 22 Global Step: 223285 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:40:28,332-Speed 129.23 samples/sec Loss 0.3191 Epoch: 22 Global Step: 223290 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:40:48,176-Speed 129.02 samples/sec Loss 0.3401 Epoch: 22 Global Step: 223295 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:41:07,943-Speed 129.51 samples/sec Loss 0.3161 Epoch: 22 Global Step: 223300 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:41:27,731-Speed 129.38 samples/sec Loss 0.3378 Epoch: 22 Global Step: 223305 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:41:47,797-Speed 127.58 samples/sec Loss 0.2818 Epoch: 22 Global Step: 223310 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:42:07,604-Speed 129.26 samples/sec Loss 0.3132 Epoch: 22 Global Step: 223315 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:42:27,479-Speed 128.81 samples/sec Loss 0.3229 Epoch: 22 Global Step: 223320 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:42:47,459-Speed 128.13 samples/sec Loss 0.3353 Epoch: 22 Global Step: 223325 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:43:07,419-Speed 128.26 samples/sec Loss 0.3664 Epoch: 22 Global Step: 223330 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:43:27,371-Speed 128.31 samples/sec Loss 0.3284 Epoch: 22 Global Step: 223335 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:43:47,118-Speed 129.64 samples/sec Loss 0.4320 Epoch: 22 Global Step: 223340 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:44:06,902-Speed 129.41 samples/sec Loss 0.3484 Epoch: 22 Global Step: 223345 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:44:26,676-Speed 129.47 samples/sec Loss 0.2985 Epoch: 22 Global Step: 223350 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:44:46,757-Speed 127.49 samples/sec Loss 0.3158 Epoch: 22 Global Step: 223355 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:45:06,640-Speed 128.76 samples/sec Loss 0.2929 Epoch: 22 Global Step: 223360 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:45:26,365-Speed 129.79 samples/sec Loss 0.3421 Epoch: 22 Global Step: 223365 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:45:46,437-Speed 127.55 samples/sec Loss 0.3471 Epoch: 22 Global Step: 223370 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:46:06,147-Speed 129.89 samples/sec Loss 0.3373 Epoch: 22 Global Step: 223375 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:46:25,659-Speed 131.21 samples/sec Loss 0.3360 Epoch: 22 Global Step: 223380 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:46:45,376-Speed 129.84 samples/sec Loss 0.3407 Epoch: 22 Global Step: 223385 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:47:05,152-Speed 129.46 samples/sec Loss 0.3569 Epoch: 22 Global Step: 223390 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:47:24,911-Speed 129.56 samples/sec Loss 0.3620 Epoch: 22 Global Step: 223395 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:47:44,928-Speed 127.90 samples/sec Loss 0.3394 Epoch: 22 Global Step: 223400 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:48:04,827-Speed 128.66 samples/sec Loss 0.2513 Epoch: 22 Global Step: 223405 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:48:24,493-Speed 130.18 samples/sec Loss 0.3030 Epoch: 22 Global Step: 223410 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:48:44,493-Speed 128.01 samples/sec Loss 0.2875 Epoch: 22 Global Step: 223415 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:49:04,246-Speed 129.61 samples/sec Loss 0.3889 Epoch: 22 Global Step: 223420 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:49:24,264-Speed 127.89 samples/sec Loss 0.2783 Epoch: 22 Global Step: 223425 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:49:43,981-Speed 129.84 samples/sec Loss 0.3193 Epoch: 22 Global Step: 223430 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:50:03,841-Speed 128.91 samples/sec Loss 0.3474 Epoch: 22 Global Step: 223435 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:50:23,647-Speed 129.26 samples/sec Loss 0.3442 Epoch: 22 Global Step: 223440 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:50:43,634-Speed 128.09 samples/sec Loss 0.3191 Epoch: 22 Global Step: 223445 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:51:03,551-Speed 128.54 samples/sec Loss 0.3125 Epoch: 22 Global Step: 223450 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:51:23,295-Speed 129.67 samples/sec Loss 0.3643 Epoch: 22 Global Step: 223455 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:51:43,378-Speed 127.48 samples/sec Loss 0.3842 Epoch: 22 Global Step: 223460 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:52:03,181-Speed 129.28 samples/sec Loss 0.3486 Epoch: 22 Global Step: 223465 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:52:23,009-Speed 129.11 samples/sec Loss 0.3519 Epoch: 22 Global Step: 223470 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:52:42,715-Speed 129.92 samples/sec Loss 0.3164 Epoch: 22 Global Step: 223475 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:53:02,640-Speed 128.49 samples/sec Loss 0.3505 Epoch: 22 Global Step: 223480 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:53:22,334-Speed 129.99 samples/sec Loss 0.3028 Epoch: 22 Global Step: 223485 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:53:42,387-Speed 127.67 samples/sec Loss 0.3722 Epoch: 22 Global Step: 223490 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:54:02,272-Speed 128.75 samples/sec Loss 0.3314 Epoch: 22 Global Step: 223495 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:54:22,009-Speed 129.72 samples/sec Loss 0.3116 Epoch: 22 Global Step: 223500 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:54:42,179-Speed 126.93 samples/sec Loss 0.3726 Epoch: 22 Global Step: 223505 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:55:01,909-Speed 129.76 samples/sec Loss 0.3431 Epoch: 22 Global Step: 223510 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:55:21,749-Speed 129.04 samples/sec Loss 0.3575 Epoch: 22 Global Step: 223515 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:55:41,406-Speed 130.24 samples/sec Loss 0.3507 Epoch: 22 Global Step: 223520 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:56:01,363-Speed 128.28 samples/sec Loss 0.3367 Epoch: 22 Global Step: 223525 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:56:21,287-Speed 128.49 samples/sec Loss 0.3119 Epoch: 22 Global Step: 223530 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:56:41,320-Speed 127.79 samples/sec Loss 0.2634 Epoch: 22 Global Step: 223535 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 22:57:00,558-Speed 133.08 samples/sec Loss 0.3223 Epoch: 22 Global Step: 223540 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:57:20,246-Speed 130.04 samples/sec Loss 0.3726 Epoch: 22 Global Step: 223545 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:57:40,710-Speed 125.10 samples/sec Loss 0.3770 Epoch: 22 Global Step: 223550 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:58:00,419-Speed 129.90 samples/sec Loss 0.3546 Epoch: 22 Global Step: 223555 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:58:20,528-Speed 127.31 samples/sec Loss 0.2993 Epoch: 22 Global Step: 223560 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:58:40,199-Speed 130.15 samples/sec Loss 0.3207 Epoch: 22 Global Step: 223565 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:59:00,063-Speed 128.88 samples/sec Loss 0.3017 Epoch: 22 Global Step: 223570 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:59:19,838-Speed 129.46 samples/sec Loss 0.3562 Epoch: 22 Global Step: 223575 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:59:39,728-Speed 128.71 samples/sec Loss 0.3875 Epoch: 22 Global Step: 223580 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 22:59:59,471-Speed 129.67 samples/sec Loss 0.3608 Epoch: 22 Global Step: 223585 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:00:19,433-Speed 128.25 samples/sec Loss 0.3945 Epoch: 22 Global Step: 223590 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:00:39,290-Speed 128.93 samples/sec Loss 0.3416 Epoch: 22 Global Step: 223595 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:00:58,992-Speed 129.95 samples/sec Loss 0.4519 Epoch: 22 Global Step: 223600 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:01:19,149-Speed 127.01 samples/sec Loss 0.3594 Epoch: 22 Global Step: 223605 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:01:38,866-Speed 129.84 samples/sec Loss 0.3607 Epoch: 22 Global Step: 223610 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:01:58,674-Speed 129.25 samples/sec Loss 0.3721 Epoch: 22 Global Step: 223615 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:02:18,319-Speed 130.32 samples/sec Loss 0.3033 Epoch: 22 Global Step: 223620 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:02:38,230-Speed 128.58 samples/sec Loss 0.3562 Epoch: 22 Global Step: 223625 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:02:57,900-Speed 130.15 samples/sec Loss 0.3816 Epoch: 22 Global Step: 223630 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:03:17,937-Speed 127.77 samples/sec Loss 0.3893 Epoch: 22 Global Step: 223635 Fp16 Grad Scale: 4096 Required: 31 hours Training: 2021-06-17 23:03:37,755-Speed 129.19 samples/sec Loss 0.3860 Epoch: 22 Global Step: 223640 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 23:03:57,376-Speed 130.48 samples/sec Loss 0.3670 Epoch: 22 Global Step: 223645 Fp16 Grad Scale: 8192 Required: 31 hours Training: 2021-06-17 23:04:17,546-Speed 126.93 samples/sec Loss 0.2971 Epoch: 22 Global Step: 223650 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:04:37,353-Speed 129.25 samples/sec Loss 0.3361 Epoch: 22 Global Step: 223655 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:04:57,069-Speed 129.85 samples/sec Loss 0.3926 Epoch: 22 Global Step: 223660 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:05:16,398-Speed 132.45 samples/sec Loss 0.3712 Epoch: 22 Global Step: 223665 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:05:36,187-Speed 129.37 samples/sec Loss 0.3046 Epoch: 22 Global Step: 223670 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:05:55,973-Speed 129.39 samples/sec Loss 0.3255 Epoch: 22 Global Step: 223675 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:06:16,052-Speed 127.50 samples/sec Loss 0.3331 Epoch: 22 Global Step: 223680 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:06:35,856-Speed 129.27 samples/sec Loss 0.3903 Epoch: 22 Global Step: 223685 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:06:55,551-Speed 129.99 samples/sec Loss 0.4111 Epoch: 22 Global Step: 223690 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:07:15,642-Speed 127.43 samples/sec Loss 0.3594 Epoch: 22 Global Step: 223695 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:07:35,543-Speed 128.65 samples/sec Loss 0.3872 Epoch: 22 Global Step: 223700 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:07:55,254-Speed 129.88 samples/sec Loss 0.2604 Epoch: 22 Global Step: 223705 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:08:15,020-Speed 129.52 samples/sec Loss 0.3225 Epoch: 22 Global Step: 223710 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:08:34,805-Speed 129.40 samples/sec Loss 0.2754 Epoch: 22 Global Step: 223715 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:08:54,590-Speed 129.40 samples/sec Loss 0.2473 Epoch: 22 Global Step: 223720 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:09:14,754-Speed 126.96 samples/sec Loss 0.2709 Epoch: 22 Global Step: 223725 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:09:34,493-Speed 129.70 samples/sec Loss 0.3568 Epoch: 22 Global Step: 223730 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:09:54,186-Speed 130.00 samples/sec Loss 0.2794 Epoch: 22 Global Step: 223735 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:10:14,311-Speed 127.21 samples/sec Loss 0.2525 Epoch: 22 Global Step: 223740 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:10:34,209-Speed 128.66 samples/sec Loss 0.3370 Epoch: 22 Global Step: 223745 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:10:53,908-Speed 129.96 samples/sec Loss 0.4472 Epoch: 22 Global Step: 223750 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:11:13,912-Speed 127.98 samples/sec Loss 0.3324 Epoch: 22 Global Step: 223755 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:11:33,601-Speed 130.03 samples/sec Loss 0.3369 Epoch: 22 Global Step: 223760 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:11:53,374-Speed 129.48 samples/sec Loss 0.3897 Epoch: 22 Global Step: 223765 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:12:13,441-Speed 127.58 samples/sec Loss 0.2822 Epoch: 22 Global Step: 223770 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:12:32,721-Speed 132.79 samples/sec Loss 0.3360 Epoch: 22 Global Step: 223775 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:12:52,518-Speed 129.32 samples/sec Loss 0.3873 Epoch: 22 Global Step: 223780 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:13:12,523-Speed 127.97 samples/sec Loss 0.3166 Epoch: 22 Global Step: 223785 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:13:32,285-Speed 129.55 samples/sec Loss 0.3085 Epoch: 22 Global Step: 223790 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:13:52,001-Speed 129.85 samples/sec Loss 0.3020 Epoch: 22 Global Step: 223795 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:14:11,890-Speed 128.72 samples/sec Loss 0.3184 Epoch: 22 Global Step: 223800 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:14:31,544-Speed 130.26 samples/sec Loss 0.3413 Epoch: 22 Global Step: 223805 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:14:51,498-Speed 128.30 samples/sec Loss 0.4114 Epoch: 22 Global Step: 223810 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:15:11,501-Speed 127.98 samples/sec Loss 0.3250 Epoch: 22 Global Step: 223815 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:15:31,274-Speed 129.48 samples/sec Loss 0.3422 Epoch: 22 Global Step: 223820 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:15:51,043-Speed 129.50 samples/sec Loss 0.3256 Epoch: 22 Global Step: 223825 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:16:11,181-Speed 127.13 samples/sec Loss 0.2815 Epoch: 22 Global Step: 223830 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:16:30,909-Speed 129.77 samples/sec Loss 0.2970 Epoch: 22 Global Step: 223835 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:16:50,675-Speed 129.53 samples/sec Loss 0.3600 Epoch: 22 Global Step: 223840 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:17:10,537-Speed 128.89 samples/sec Loss 0.3033 Epoch: 22 Global Step: 223845 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:17:30,170-Speed 130.40 samples/sec Loss 0.3289 Epoch: 22 Global Step: 223850 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:17:50,116-Speed 128.35 samples/sec Loss 0.3693 Epoch: 22 Global Step: 223855 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:18:10,134-Speed 127.89 samples/sec Loss 0.3303 Epoch: 22 Global Step: 223860 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:18:29,938-Speed 129.28 samples/sec Loss 0.3032 Epoch: 22 Global Step: 223865 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:18:49,642-Speed 129.93 samples/sec Loss 0.3037 Epoch: 22 Global Step: 223870 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:19:09,803-Speed 126.99 samples/sec Loss 0.3657 Epoch: 22 Global Step: 223875 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:19:28,859-Speed 134.35 samples/sec Loss 0.2910 Epoch: 22 Global Step: 223880 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:19:48,763-Speed 128.62 samples/sec Loss 0.3290 Epoch: 22 Global Step: 223885 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:20:08,645-Speed 128.77 samples/sec Loss 0.3109 Epoch: 22 Global Step: 223890 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:20:28,293-Speed 130.30 samples/sec Loss 0.3235 Epoch: 22 Global Step: 223895 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:20:48,305-Speed 127.93 samples/sec Loss 0.3106 Epoch: 22 Global Step: 223900 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:21:08,002-Speed 129.97 samples/sec Loss 0.3265 Epoch: 22 Global Step: 223905 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:21:27,862-Speed 128.91 samples/sec Loss 0.2809 Epoch: 22 Global Step: 223910 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:21:47,609-Speed 129.65 samples/sec Loss 0.2864 Epoch: 22 Global Step: 223915 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:22:07,505-Speed 128.67 samples/sec Loss 0.2691 Epoch: 22 Global Step: 223920 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:22:27,468-Speed 128.25 samples/sec Loss 0.3586 Epoch: 22 Global Step: 223925 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:22:47,845-Speed 125.64 samples/sec Loss 0.3097 Epoch: 22 Global Step: 223930 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:23:07,575-Speed 129.75 samples/sec Loss 0.3268 Epoch: 22 Global Step: 223935 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:23:27,398-Speed 129.15 samples/sec Loss 0.3590 Epoch: 22 Global Step: 223940 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:23:47,263-Speed 128.88 samples/sec Loss 0.3836 Epoch: 22 Global Step: 223945 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:24:06,989-Speed 129.78 samples/sec Loss 0.3505 Epoch: 22 Global Step: 223950 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:24:26,882-Speed 128.70 samples/sec Loss 0.3164 Epoch: 22 Global Step: 223955 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:24:46,585-Speed 129.93 samples/sec Loss 0.3646 Epoch: 22 Global Step: 223960 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:25:06,442-Speed 128.93 samples/sec Loss 0.3104 Epoch: 22 Global Step: 223965 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:25:26,288-Speed 129.00 samples/sec Loss 0.2540 Epoch: 22 Global Step: 223970 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:25:46,735-Speed 125.21 samples/sec Loss 0.3317 Epoch: 22 Global Step: 223975 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:26:06,510-Speed 129.46 samples/sec Loss 0.3288 Epoch: 22 Global Step: 223980 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:26:26,257-Speed 129.65 samples/sec Loss 0.3661 Epoch: 22 Global Step: 223985 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:26:46,140-Speed 128.76 samples/sec Loss 0.3329 Epoch: 22 Global Step: 223990 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:27:05,743-Speed 130.60 samples/sec Loss 0.3379 Epoch: 22 Global Step: 223995 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:27:25,648-Speed 128.62 samples/sec Loss 0.3709 Epoch: 22 Global Step: 224000 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:27:44,936-Speed 132.73 samples/sec Loss 0.3087 Epoch: 22 Global Step: 224005 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:28:04,837-Speed 128.64 samples/sec Loss 0.3180 Epoch: 22 Global Step: 224010 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:28:24,545-Speed 129.90 samples/sec Loss 0.3850 Epoch: 22 Global Step: 224015 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:28:44,926-Speed 125.61 samples/sec Loss 0.3984 Epoch: 22 Global Step: 224020 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:29:04,627-Speed 129.95 samples/sec Loss 0.3000 Epoch: 22 Global Step: 224025 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:29:24,437-Speed 129.23 samples/sec Loss 0.3125 Epoch: 22 Global Step: 224030 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:29:44,225-Speed 129.38 samples/sec Loss 0.3663 Epoch: 22 Global Step: 224035 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:30:03,886-Speed 130.22 samples/sec Loss 0.3141 Epoch: 22 Global Step: 224040 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:30:23,780-Speed 128.69 samples/sec Loss 0.3554 Epoch: 22 Global Step: 224045 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:30:43,512-Speed 129.75 samples/sec Loss 0.4086 Epoch: 22 Global Step: 224050 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:31:03,354-Speed 129.02 samples/sec Loss 0.4060 Epoch: 22 Global Step: 224055 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:31:23,141-Speed 129.38 samples/sec Loss 0.2807 Epoch: 22 Global Step: 224060 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:31:43,436-Speed 126.14 samples/sec Loss 0.3696 Epoch: 22 Global Step: 224065 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:32:03,150-Speed 129.86 samples/sec Loss 0.3532 Epoch: 22 Global Step: 224070 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:32:23,046-Speed 128.68 samples/sec Loss 0.4292 Epoch: 22 Global Step: 224075 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:32:42,832-Speed 129.39 samples/sec Loss 0.3673 Epoch: 22 Global Step: 224080 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:33:02,479-Speed 130.31 samples/sec Loss 0.3075 Epoch: 22 Global Step: 224085 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:33:22,316-Speed 129.06 samples/sec Loss 0.3267 Epoch: 22 Global Step: 224090 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:33:42,033-Speed 129.84 samples/sec Loss 0.3341 Epoch: 22 Global Step: 224095 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:34:02,061-Speed 127.83 samples/sec Loss 0.4409 Epoch: 22 Global Step: 224100 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:34:21,780-Speed 129.83 samples/sec Loss 0.3119 Epoch: 22 Global Step: 224105 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:34:42,051-Speed 126.29 samples/sec Loss 0.3393 Epoch: 22 Global Step: 224110 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:35:01,819-Speed 129.51 samples/sec Loss 0.3012 Epoch: 22 Global Step: 224115 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:35:21,686-Speed 128.86 samples/sec Loss 0.3045 Epoch: 22 Global Step: 224120 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:35:41,411-Speed 129.80 samples/sec Loss 0.3185 Epoch: 22 Global Step: 224125 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:36:00,778-Speed 132.19 samples/sec Loss 0.3363 Epoch: 22 Global Step: 224130 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:36:20,487-Speed 129.90 samples/sec Loss 0.3110 Epoch: 22 Global Step: 224135 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:36:40,242-Speed 129.59 samples/sec Loss 0.3542 Epoch: 22 Global Step: 224140 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:37:00,109-Speed 128.87 samples/sec Loss 0.3514 Epoch: 22 Global Step: 224145 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:37:19,864-Speed 129.60 samples/sec Loss 0.3139 Epoch: 22 Global Step: 224150 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:37:40,180-Speed 126.01 samples/sec Loss 0.3422 Epoch: 22 Global Step: 224155 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:38:00,008-Speed 129.11 samples/sec Loss 0.3467 Epoch: 22 Global Step: 224160 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:38:19,795-Speed 129.38 samples/sec Loss 0.3378 Epoch: 22 Global Step: 224165 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:38:39,822-Speed 127.83 samples/sec Loss 0.3237 Epoch: 22 Global Step: 224170 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:38:59,582-Speed 129.56 samples/sec Loss 0.2904 Epoch: 22 Global Step: 224175 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:39:19,293-Speed 129.88 samples/sec Loss 0.3416 Epoch: 22 Global Step: 224180 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:39:39,202-Speed 128.59 samples/sec Loss 0.2953 Epoch: 22 Global Step: 224185 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:39:59,242-Speed 127.75 samples/sec Loss 0.2983 Epoch: 22 Global Step: 224190 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:40:19,004-Speed 129.55 samples/sec Loss 0.3605 Epoch: 22 Global Step: 224195 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:40:39,234-Speed 126.55 samples/sec Loss 0.3452 Epoch: 22 Global Step: 224200 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:40:59,139-Speed 128.62 samples/sec Loss 0.4077 Epoch: 22 Global Step: 224205 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:41:18,950-Speed 129.22 samples/sec Loss 0.3212 Epoch: 22 Global Step: 224210 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:41:38,768-Speed 129.18 samples/sec Loss 0.3233 Epoch: 22 Global Step: 224215 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:41:58,459-Speed 130.02 samples/sec Loss 0.3677 Epoch: 22 Global Step: 224220 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:42:18,235-Speed 129.46 samples/sec Loss 0.3300 Epoch: 22 Global Step: 224225 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:42:38,090-Speed 128.94 samples/sec Loss 0.3692 Epoch: 22 Global Step: 224230 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:42:57,876-Speed 129.39 samples/sec Loss 0.3139 Epoch: 22 Global Step: 224235 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:43:17,614-Speed 129.71 samples/sec Loss 0.3384 Epoch: 22 Global Step: 224240 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:43:37,841-Speed 126.57 samples/sec Loss 0.3471 Epoch: 22 Global Step: 224245 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:43:57,791-Speed 128.32 samples/sec Loss 0.3548 Epoch: 22 Global Step: 224250 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:44:17,513-Speed 129.81 samples/sec Loss 0.3155 Epoch: 22 Global Step: 224255 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:44:37,426-Speed 128.57 samples/sec Loss 0.4281 Epoch: 22 Global Step: 224260 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:44:56,596-Speed 133.55 samples/sec Loss 0.4174 Epoch: 22 Global Step: 224265 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:45:16,352-Speed 129.59 samples/sec Loss 0.3212 Epoch: 22 Global Step: 224270 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:45:36,234-Speed 128.77 samples/sec Loss 0.2937 Epoch: 22 Global Step: 224275 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:45:55,984-Speed 129.63 samples/sec Loss 0.3650 Epoch: 22 Global Step: 224280 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:46:15,889-Speed 128.62 samples/sec Loss 0.3640 Epoch: 22 Global Step: 224285 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:46:35,498-Speed 130.56 samples/sec Loss 0.3449 Epoch: 22 Global Step: 224290 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:46:55,387-Speed 128.72 samples/sec Loss 0.3408 Epoch: 22 Global Step: 224295 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:47:15,179-Speed 129.35 samples/sec Loss 0.2817 Epoch: 22 Global Step: 224300 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:47:35,011-Speed 129.09 samples/sec Loss 0.3504 Epoch: 22 Global Step: 224305 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:47:55,836-Speed 122.93 samples/sec Loss 0.3166 Epoch: 22 Global Step: 224310 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:48:16,151-Speed 126.02 samples/sec Loss 0.3178 Epoch: 22 Global Step: 224315 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:48:35,932-Speed 129.42 samples/sec Loss 0.3410 Epoch: 22 Global Step: 224320 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:48:55,749-Speed 129.19 samples/sec Loss 0.3217 Epoch: 22 Global Step: 224325 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:49:15,610-Speed 128.90 samples/sec Loss 0.3241 Epoch: 22 Global Step: 224330 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:49:35,379-Speed 129.50 samples/sec Loss 0.2807 Epoch: 22 Global Step: 224335 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:49:55,266-Speed 128.74 samples/sec Loss 0.3533 Epoch: 22 Global Step: 224340 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:50:14,994-Speed 129.77 samples/sec Loss 0.3296 Epoch: 22 Global Step: 224345 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:50:34,971-Speed 128.15 samples/sec Loss 0.3550 Epoch: 22 Global Step: 224350 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:50:54,692-Speed 129.81 samples/sec Loss 0.3275 Epoch: 22 Global Step: 224355 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:51:15,199-Speed 124.84 samples/sec Loss 0.4374 Epoch: 22 Global Step: 224360 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:51:34,921-Speed 129.81 samples/sec Loss 0.2499 Epoch: 22 Global Step: 224365 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:51:54,802-Speed 128.77 samples/sec Loss 0.3513 Epoch: 22 Global Step: 224370 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:52:14,529-Speed 129.78 samples/sec Loss 0.3897 Epoch: 22 Global Step: 224375 Fp16 Grad Scale: 8192 Required: 30 hours Training: 2021-06-17 23:52:33,819-Speed 132.72 samples/sec Loss 0.3614 Epoch: 22 Global Step: 224380 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:52:53,646-Speed 129.12 samples/sec Loss 0.3424 Epoch: 22 Global Step: 224385 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:53:13,406-Speed 129.56 samples/sec Loss 0.3143 Epoch: 22 Global Step: 224390 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:53:33,200-Speed 129.34 samples/sec Loss 0.3020 Epoch: 22 Global Step: 224395 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:53:53,104-Speed 128.62 samples/sec Loss 0.3690 Epoch: 22 Global Step: 224400 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:54:13,492-Speed 125.57 samples/sec Loss 0.2740 Epoch: 22 Global Step: 224405 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:54:33,259-Speed 129.51 samples/sec Loss 0.3528 Epoch: 22 Global Step: 224410 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:54:53,024-Speed 129.53 samples/sec Loss 0.3037 Epoch: 22 Global Step: 224415 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:55:12,802-Speed 129.45 samples/sec Loss 0.3784 Epoch: 22 Global Step: 224420 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:55:32,704-Speed 128.63 samples/sec Loss 0.3867 Epoch: 22 Global Step: 224425 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:55:52,420-Speed 129.86 samples/sec Loss 0.3977 Epoch: 22 Global Step: 224430 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:56:12,187-Speed 129.52 samples/sec Loss 0.3585 Epoch: 22 Global Step: 224435 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:56:31,947-Speed 129.56 samples/sec Loss 0.2970 Epoch: 22 Global Step: 224440 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:56:51,724-Speed 129.46 samples/sec Loss 0.3886 Epoch: 22 Global Step: 224445 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:57:11,989-Speed 126.33 samples/sec Loss 0.3288 Epoch: 22 Global Step: 224450 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:57:31,761-Speed 129.48 samples/sec Loss 0.3765 Epoch: 22 Global Step: 224455 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:57:51,544-Speed 129.41 samples/sec Loss 0.3287 Epoch: 22 Global Step: 224460 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:58:11,384-Speed 129.04 samples/sec Loss 0.2749 Epoch: 22 Global Step: 224465 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:58:31,284-Speed 128.65 samples/sec Loss 0.3363 Epoch: 22 Global Step: 224470 Fp16 Grad Scale: 4096 Required: 30 hours Training: 2021-06-17 23:58:50,876-Speed 130.67 samples/sec Loss 0.3763 Epoch: 22 Global Step: 224475 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-17 23:59:10,670-Speed 129.34 samples/sec Loss 0.3035 Epoch: 22 Global Step: 224480 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-17 23:59:30,632-Speed 128.25 samples/sec Loss 0.3082 Epoch: 22 Global Step: 224485 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-17 23:59:50,494-Speed 128.90 samples/sec Loss 0.3478 Epoch: 22 Global Step: 224490 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:00:10,867-Speed 125.66 samples/sec Loss 0.4118 Epoch: 22 Global Step: 224495 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:00:30,793-Speed 128.48 samples/sec Loss 0.3668 Epoch: 22 Global Step: 224500 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:00:50,498-Speed 129.93 samples/sec Loss 0.3290 Epoch: 22 Global Step: 224505 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:01:10,302-Speed 129.27 samples/sec Loss 0.3194 Epoch: 22 Global Step: 224510 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:01:30,231-Speed 128.47 samples/sec Loss 0.3140 Epoch: 22 Global Step: 224515 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:01:49,835-Speed 130.59 samples/sec Loss 0.3049 Epoch: 22 Global Step: 224520 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:02:09,581-Speed 129.65 samples/sec Loss 0.3265 Epoch: 22 Global Step: 224525 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:02:29,337-Speed 129.59 samples/sec Loss 0.3146 Epoch: 22 Global Step: 224530 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:02:49,135-Speed 129.31 samples/sec Loss 0.2973 Epoch: 22 Global Step: 224535 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:03:09,409-Speed 126.27 samples/sec Loss 0.3463 Epoch: 22 Global Step: 224540 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:03:29,194-Speed 129.40 samples/sec Loss 0.3104 Epoch: 22 Global Step: 224545 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:03:48,950-Speed 129.58 samples/sec Loss 0.3487 Epoch: 22 Global Step: 224550 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:04:08,738-Speed 129.38 samples/sec Loss 0.3358 Epoch: 22 Global Step: 224555 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:04:28,641-Speed 128.63 samples/sec Loss 0.3040 Epoch: 22 Global Step: 224560 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:04:47,803-Speed 133.61 samples/sec Loss 0.3225 Epoch: 22 Global Step: 224565 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:05:07,465-Speed 130.21 samples/sec Loss 0.2993 Epoch: 22 Global Step: 224570 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:05:27,299-Speed 129.08 samples/sec Loss 0.3167 Epoch: 22 Global Step: 224575 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:05:47,223-Speed 128.50 samples/sec Loss 0.2964 Epoch: 22 Global Step: 224580 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:06:07,432-Speed 126.68 samples/sec Loss 0.3642 Epoch: 22 Global Step: 224585 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:06:27,164-Speed 129.74 samples/sec Loss 0.3063 Epoch: 22 Global Step: 224590 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:06:46,924-Speed 129.56 samples/sec Loss 0.3173 Epoch: 22 Global Step: 224595 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:07:06,723-Speed 129.31 samples/sec Loss 0.3639 Epoch: 22 Global Step: 224600 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:07:26,616-Speed 128.69 samples/sec Loss 0.3412 Epoch: 22 Global Step: 224605 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:07:46,226-Speed 130.56 samples/sec Loss 0.3390 Epoch: 22 Global Step: 224610 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:08:05,951-Speed 129.79 samples/sec Loss 0.3113 Epoch: 22 Global Step: 224615 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:08:25,777-Speed 129.13 samples/sec Loss 0.3365 Epoch: 22 Global Step: 224620 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:08:45,607-Speed 129.11 samples/sec Loss 0.3246 Epoch: 22 Global Step: 224625 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:09:05,838-Speed 126.54 samples/sec Loss 0.3754 Epoch: 22 Global Step: 224630 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:09:25,684-Speed 129.00 samples/sec Loss 0.3871 Epoch: 22 Global Step: 224635 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:09:45,325-Speed 130.35 samples/sec Loss 0.3379 Epoch: 22 Global Step: 224640 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:10:05,159-Speed 129.07 samples/sec Loss 0.4250 Epoch: 22 Global Step: 224645 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:10:24,923-Speed 129.54 samples/sec Loss 0.4453 Epoch: 22 Global Step: 224650 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:10:44,574-Speed 130.28 samples/sec Loss 0.3397 Epoch: 22 Global Step: 224655 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:11:04,363-Speed 129.37 samples/sec Loss 0.3612 Epoch: 22 Global Step: 224660 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:11:24,088-Speed 129.79 samples/sec Loss 0.3614 Epoch: 22 Global Step: 224665 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:11:43,924-Speed 129.07 samples/sec Loss 0.3003 Epoch: 22 Global Step: 224670 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:12:04,175-Speed 126.42 samples/sec Loss 0.3108 Epoch: 22 Global Step: 224675 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:12:23,984-Speed 129.24 samples/sec Loss 0.2782 Epoch: 22 Global Step: 224680 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:12:43,648-Speed 130.19 samples/sec Loss 0.3718 Epoch: 22 Global Step: 224685 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:13:03,447-Speed 129.31 samples/sec Loss 0.3990 Epoch: 22 Global Step: 224690 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:13:23,217-Speed 129.49 samples/sec Loss 0.3502 Epoch: 22 Global Step: 224695 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:13:42,953-Speed 129.72 samples/sec Loss 0.2612 Epoch: 22 Global Step: 224700 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:14:02,750-Speed 129.33 samples/sec Loss 0.3065 Epoch: 22 Global Step: 224705 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:14:22,453-Speed 129.93 samples/sec Loss 0.2823 Epoch: 22 Global Step: 224710 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:14:42,243-Speed 129.36 samples/sec Loss 0.4505 Epoch: 22 Global Step: 224715 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:15:01,678-Speed 131.73 samples/sec Loss 0.3983 Epoch: 22 Global Step: 224720 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:15:21,470-Speed 129.35 samples/sec Loss 0.3259 Epoch: 22 Global Step: 224725 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:15:42,360-Speed 122.55 samples/sec Loss 0.3317 Epoch: 22 Global Step: 224730 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:16:02,332-Speed 128.19 samples/sec Loss 0.2626 Epoch: 22 Global Step: 224735 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:16:22,023-Speed 130.02 samples/sec Loss 0.3234 Epoch: 22 Global Step: 224740 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:16:41,719-Speed 129.98 samples/sec Loss 0.3419 Epoch: 22 Global Step: 224745 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:17:01,485-Speed 129.52 samples/sec Loss 0.3856 Epoch: 22 Global Step: 224750 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:17:21,245-Speed 129.57 samples/sec Loss 0.3197 Epoch: 22 Global Step: 224755 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:17:41,125-Speed 128.78 samples/sec Loss 0.3149 Epoch: 22 Global Step: 224760 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:18:00,829-Speed 129.93 samples/sec Loss 0.2858 Epoch: 22 Global Step: 224765 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:18:20,678-Speed 128.98 samples/sec Loss 0.3278 Epoch: 22 Global Step: 224770 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:18:40,310-Speed 130.41 samples/sec Loss 0.2841 Epoch: 22 Global Step: 224775 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:19:00,194-Speed 128.75 samples/sec Loss 0.3924 Epoch: 22 Global Step: 224780 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:19:19,898-Speed 129.93 samples/sec Loss 0.3942 Epoch: 22 Global Step: 224785 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:19:40,203-Speed 126.08 samples/sec Loss 0.3353 Epoch: 22 Global Step: 224790 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:19:59,924-Speed 129.82 samples/sec Loss 0.2893 Epoch: 22 Global Step: 224795 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:20:19,717-Speed 129.35 samples/sec Loss 0.3644 Epoch: 22 Global Step: 224800 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:20:39,617-Speed 128.65 samples/sec Loss 0.3868 Epoch: 22 Global Step: 224805 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:20:59,248-Speed 130.42 samples/sec Loss 0.3034 Epoch: 22 Global Step: 224810 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:21:19,154-Speed 128.61 samples/sec Loss 0.3959 Epoch: 22 Global Step: 224815 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:21:38,892-Speed 129.70 samples/sec Loss 0.3526 Epoch: 22 Global Step: 224820 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:21:58,806-Speed 128.56 samples/sec Loss 0.3201 Epoch: 22 Global Step: 224825 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:22:18,498-Speed 130.00 samples/sec Loss 0.2554 Epoch: 22 Global Step: 224830 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:22:38,802-Speed 126.09 samples/sec Loss 0.4182 Epoch: 22 Global Step: 224835 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:22:58,532-Speed 129.76 samples/sec Loss 0.3398 Epoch: 22 Global Step: 224840 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:23:18,411-Speed 128.79 samples/sec Loss 0.3023 Epoch: 22 Global Step: 224845 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:23:38,224-Speed 129.24 samples/sec Loss 0.3585 Epoch: 22 Global Step: 224850 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:23:57,886-Speed 130.21 samples/sec Loss 0.2988 Epoch: 22 Global Step: 224855 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:24:17,668-Speed 129.42 samples/sec Loss 0.3542 Epoch: 22 Global Step: 224860 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:24:37,392-Speed 129.80 samples/sec Loss 0.3215 Epoch: 22 Global Step: 224865 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:24:57,301-Speed 128.59 samples/sec Loss 0.3287 Epoch: 22 Global Step: 224870 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:25:17,226-Speed 128.49 samples/sec Loss 0.3679 Epoch: 22 Global Step: 224875 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:25:37,467-Speed 126.48 samples/sec Loss 0.3708 Epoch: 22 Global Step: 224880 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:25:57,247-Speed 129.43 samples/sec Loss 0.2991 Epoch: 22 Global Step: 224885 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:26:17,022-Speed 129.47 samples/sec Loss 0.3972 Epoch: 22 Global Step: 224890 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:26:36,355-Speed 132.42 samples/sec Loss 0.2827 Epoch: 22 Global Step: 224895 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:26:56,119-Speed 129.53 samples/sec Loss 0.2928 Epoch: 22 Global Step: 224900 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:27:15,828-Speed 129.90 samples/sec Loss 0.3308 Epoch: 22 Global Step: 224905 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:27:35,554-Speed 129.78 samples/sec Loss 0.3304 Epoch: 22 Global Step: 224910 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:27:56,613-Speed 121.57 samples/sec Loss 0.3130 Epoch: 22 Global Step: 224915 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:28:16,395-Speed 129.42 samples/sec Loss 0.3436 Epoch: 22 Global Step: 224920 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:28:36,651-Speed 126.38 samples/sec Loss 0.3500 Epoch: 22 Global Step: 224925 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:28:56,362-Speed 129.89 samples/sec Loss 0.3101 Epoch: 22 Global Step: 224930 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:29:16,159-Speed 129.32 samples/sec Loss 0.3466 Epoch: 22 Global Step: 224935 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:29:36,122-Speed 128.24 samples/sec Loss 0.3563 Epoch: 22 Global Step: 224940 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:29:55,860-Speed 129.71 samples/sec Loss 0.3464 Epoch: 22 Global Step: 224945 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:30:16,065-Speed 126.71 samples/sec Loss 0.3335 Epoch: 22 Global Step: 224950 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:30:35,820-Speed 129.60 samples/sec Loss 0.3013 Epoch: 22 Global Step: 224955 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:30:55,638-Speed 129.18 samples/sec Loss 0.2469 Epoch: 22 Global Step: 224960 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:31:15,328-Speed 130.02 samples/sec Loss 0.3538 Epoch: 22 Global Step: 224965 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:31:35,570-Speed 126.48 samples/sec Loss 0.3636 Epoch: 22 Global Step: 224970 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:31:55,281-Speed 129.88 samples/sec Loss 0.3312 Epoch: 22 Global Step: 224975 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:32:15,061-Speed 129.43 samples/sec Loss 0.3155 Epoch: 22 Global Step: 224980 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:32:34,751-Speed 130.02 samples/sec Loss 0.3533 Epoch: 22 Global Step: 224985 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:32:54,555-Speed 129.28 samples/sec Loss 0.3653 Epoch: 22 Global Step: 224990 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:33:14,281-Speed 129.78 samples/sec Loss 0.3041 Epoch: 22 Global Step: 224995 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:33:34,021-Speed 129.70 samples/sec Loss 0.2590 Epoch: 22 Global Step: 225000 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:33:54,259-Speed 126.50 samples/sec Loss 0.3502 Epoch: 22 Global Step: 225005 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:34:13,973-Speed 129.86 samples/sec Loss 0.3515 Epoch: 22 Global Step: 225010 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:34:34,224-Speed 126.42 samples/sec Loss 0.3423 Epoch: 22 Global Step: 225015 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:34:53,975-Speed 129.62 samples/sec Loss 0.3431 Epoch: 22 Global Step: 225020 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:35:13,766-Speed 129.36 samples/sec Loss 0.2993 Epoch: 22 Global Step: 225025 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:35:33,462-Speed 129.98 samples/sec Loss 0.3048 Epoch: 22 Global Step: 225030 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:35:53,381-Speed 128.53 samples/sec Loss 0.3816 Epoch: 22 Global Step: 225035 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:36:13,104-Speed 129.80 samples/sec Loss 0.3388 Epoch: 22 Global Step: 225040 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:36:32,994-Speed 128.72 samples/sec Loss 0.3496 Epoch: 22 Global Step: 225045 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:36:52,704-Speed 129.89 samples/sec Loss 0.3335 Epoch: 22 Global Step: 225050 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:37:12,363-Speed 130.23 samples/sec Loss 0.2484 Epoch: 22 Global Step: 225055 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:37:32,785-Speed 125.36 samples/sec Loss 0.3882 Epoch: 22 Global Step: 225060 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:37:52,617-Speed 129.09 samples/sec Loss 0.3565 Epoch: 22 Global Step: 225065 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:38:12,266-Speed 130.29 samples/sec Loss 0.4131 Epoch: 22 Global Step: 225070 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:38:32,004-Speed 129.71 samples/sec Loss 0.4529 Epoch: 22 Global Step: 225075 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:38:51,943-Speed 128.40 samples/sec Loss 0.3375 Epoch: 22 Global Step: 225080 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:39:11,649-Speed 129.92 samples/sec Loss 0.2541 Epoch: 22 Global Step: 225085 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:39:31,429-Speed 129.43 samples/sec Loss 0.3500 Epoch: 22 Global Step: 225090 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:39:51,121-Speed 130.01 samples/sec Loss 0.3786 Epoch: 22 Global Step: 225095 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:40:10,814-Speed 130.00 samples/sec Loss 0.3451 Epoch: 22 Global Step: 225100 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:40:31,181-Speed 125.70 samples/sec Loss 0.3623 Epoch: 22 Global Step: 225105 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:40:50,980-Speed 129.31 samples/sec Loss 0.4137 Epoch: 22 Global Step: 225110 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:41:10,699-Speed 129.83 samples/sec Loss 0.2981 Epoch: 22 Global Step: 225115 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:41:30,504-Speed 129.27 samples/sec Loss 0.3210 Epoch: 22 Global Step: 225120 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:41:50,364-Speed 128.91 samples/sec Loss 0.2991 Epoch: 22 Global Step: 225125 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:42:10,111-Speed 129.65 samples/sec Loss 0.3250 Epoch: 22 Global Step: 225130 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:42:30,162-Speed 127.69 samples/sec Loss 0.3144 Epoch: 22 Global Step: 225135 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:42:49,882-Speed 129.82 samples/sec Loss 0.3480 Epoch: 22 Global Step: 225140 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:43:09,218-Speed 132.41 samples/sec Loss 0.4259 Epoch: 22 Global Step: 225145 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:43:29,687-Speed 125.07 samples/sec Loss 0.3707 Epoch: 22 Global Step: 225150 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:43:49,439-Speed 129.61 samples/sec Loss 0.3155 Epoch: 22 Global Step: 225155 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:44:09,123-Speed 130.06 samples/sec Loss 0.2951 Epoch: 22 Global Step: 225160 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:44:28,921-Speed 129.31 samples/sec Loss 0.3878 Epoch: 22 Global Step: 225165 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:44:48,553-Speed 130.40 samples/sec Loss 0.2760 Epoch: 22 Global Step: 225170 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:45:08,340-Speed 129.39 samples/sec Loss 0.3178 Epoch: 22 Global Step: 225175 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:45:28,227-Speed 128.74 samples/sec Loss 0.3396 Epoch: 22 Global Step: 225180 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:45:47,960-Speed 129.74 samples/sec Loss 0.3506 Epoch: 22 Global Step: 225185 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:46:07,752-Speed 129.35 samples/sec Loss 0.3629 Epoch: 22 Global Step: 225190 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:46:28,042-Speed 126.18 samples/sec Loss 0.2839 Epoch: 22 Global Step: 225195 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:46:47,828-Speed 129.39 samples/sec Loss 0.3033 Epoch: 22 Global Step: 225200 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:47:07,583-Speed 129.59 samples/sec Loss 0.3013 Epoch: 22 Global Step: 225205 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:47:27,537-Speed 128.30 samples/sec Loss 0.2544 Epoch: 22 Global Step: 225210 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:47:47,247-Speed 129.89 samples/sec Loss 0.3353 Epoch: 22 Global Step: 225215 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:48:07,071-Speed 129.14 samples/sec Loss 0.3512 Epoch: 22 Global Step: 225220 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:48:26,755-Speed 130.07 samples/sec Loss 0.3303 Epoch: 22 Global Step: 225225 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:48:46,485-Speed 129.76 samples/sec Loss 0.2701 Epoch: 22 Global Step: 225230 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:49:06,251-Speed 129.52 samples/sec Loss 0.4017 Epoch: 22 Global Step: 225235 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:49:26,666-Speed 125.41 samples/sec Loss 0.3114 Epoch: 22 Global Step: 225240 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:49:46,314-Speed 130.30 samples/sec Loss 0.3570 Epoch: 22 Global Step: 225245 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:50:06,054-Speed 129.69 samples/sec Loss 0.3291 Epoch: 22 Global Step: 225250 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:50:25,868-Speed 129.21 samples/sec Loss 0.3962 Epoch: 22 Global Step: 225255 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:50:45,589-Speed 129.81 samples/sec Loss 0.3299 Epoch: 22 Global Step: 225260 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:51:05,493-Speed 128.63 samples/sec Loss 0.3321 Epoch: 22 Global Step: 225265 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:51:25,091-Speed 130.63 samples/sec Loss 0.3852 Epoch: 22 Global Step: 225270 Fp16 Grad Scale: 8192 Required: 29 hours Training: 2021-06-18 00:51:44,419-Speed 132.46 samples/sec Loss 0.4040 Epoch: 22 Global Step: 225275 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:52:04,214-Speed 129.33 samples/sec Loss 0.3601 Epoch: 22 Global Step: 225280 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:52:23,998-Speed 129.40 samples/sec Loss 0.3317 Epoch: 22 Global Step: 225285 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:52:43,883-Speed 128.75 samples/sec Loss 0.2978 Epoch: 22 Global Step: 225290 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:53:03,526-Speed 130.33 samples/sec Loss 0.3401 Epoch: 22 Global Step: 225295 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:53:23,858-Speed 125.91 samples/sec Loss 0.3044 Epoch: 22 Global Step: 225300 Fp16 Grad Scale: 4096 Required: 29 hours Training: 2021-06-18 00:53:43,432-Speed 130.79 samples/sec Loss 0.3389 Epoch: 22 Global Step: 225305 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:54:03,481-Speed 127.69 samples/sec Loss 0.3180 Epoch: 22 Global Step: 225310 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:54:23,240-Speed 129.57 samples/sec Loss 0.3307 Epoch: 22 Global Step: 225315 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:54:43,001-Speed 129.55 samples/sec Loss 0.4308 Epoch: 22 Global Step: 225320 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:55:02,704-Speed 129.94 samples/sec Loss 0.3048 Epoch: 22 Global Step: 225325 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:55:22,490-Speed 129.39 samples/sec Loss 0.3947 Epoch: 22 Global Step: 225330 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:55:42,304-Speed 129.21 samples/sec Loss 0.3133 Epoch: 22 Global Step: 225335 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:56:02,095-Speed 129.36 samples/sec Loss 0.3486 Epoch: 22 Global Step: 225340 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:56:21,862-Speed 129.51 samples/sec Loss 0.3257 Epoch: 22 Global Step: 225345 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:56:41,572-Speed 129.89 samples/sec Loss 0.3242 Epoch: 22 Global Step: 225350 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:57:01,916-Speed 125.84 samples/sec Loss 0.3867 Epoch: 22 Global Step: 225355 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:57:21,755-Speed 129.05 samples/sec Loss 0.3708 Epoch: 22 Global Step: 225360 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:57:41,473-Speed 129.83 samples/sec Loss 0.3661 Epoch: 22 Global Step: 225365 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:58:01,221-Speed 129.64 samples/sec Loss 0.3501 Epoch: 22 Global Step: 225370 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:58:21,088-Speed 128.87 samples/sec Loss 0.3270 Epoch: 22 Global Step: 225375 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 00:58:40,777-Speed 130.02 samples/sec Loss 0.3590 Epoch: 22 Global Step: 225380 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 00:59:00,194-Speed 131.85 samples/sec Loss 0.3591 Epoch: 22 Global Step: 225385 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:59:19,963-Speed 129.50 samples/sec Loss 0.4347 Epoch: 22 Global Step: 225390 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 00:59:39,673-Speed 129.89 samples/sec Loss 0.3637 Epoch: 22 Global Step: 225395 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:00:00,013-Speed 125.87 samples/sec Loss 0.3112 Epoch: 22 Global Step: 225400 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:00:20,049-Speed 127.77 samples/sec Loss 0.2974 Epoch: 22 Global Step: 225405 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:00:39,665-Speed 130.51 samples/sec Loss 0.3594 Epoch: 22 Global Step: 225410 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:00:59,416-Speed 129.62 samples/sec Loss 0.4177 Epoch: 22 Global Step: 225415 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:01:19,275-Speed 128.91 samples/sec Loss 0.3209 Epoch: 22 Global Step: 225420 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:01:39,090-Speed 129.21 samples/sec Loss 0.3581 Epoch: 22 Global Step: 225425 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:01:58,996-Speed 128.61 samples/sec Loss 0.3021 Epoch: 22 Global Step: 225430 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:02:18,614-Speed 130.50 samples/sec Loss 0.3001 Epoch: 22 Global Step: 225435 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:02:38,316-Speed 129.94 samples/sec Loss 0.3497 Epoch: 22 Global Step: 225440 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:02:58,718-Speed 125.49 samples/sec Loss 0.2683 Epoch: 22 Global Step: 225445 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:03:18,641-Speed 128.50 samples/sec Loss 0.4016 Epoch: 22 Global Step: 225450 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:03:38,293-Speed 130.27 samples/sec Loss 0.3311 Epoch: 22 Global Step: 225455 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:03:58,024-Speed 129.76 samples/sec Loss 0.3553 Epoch: 22 Global Step: 225460 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:04:17,772-Speed 129.64 samples/sec Loss 0.2748 Epoch: 22 Global Step: 225465 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:04:37,539-Speed 129.51 samples/sec Loss 0.3458 Epoch: 22 Global Step: 225470 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:04:57,456-Speed 128.54 samples/sec Loss 0.3199 Epoch: 22 Global Step: 225475 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:05:16,736-Speed 132.79 samples/sec Loss 0.3395 Epoch: 22 Global Step: 225480 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:05:36,457-Speed 129.81 samples/sec Loss 0.3147 Epoch: 22 Global Step: 225485 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:05:56,891-Speed 125.29 samples/sec Loss 0.4233 Epoch: 22 Global Step: 225490 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:06:16,823-Speed 128.44 samples/sec Loss 0.2844 Epoch: 22 Global Step: 225495 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:06:36,499-Speed 130.11 samples/sec Loss 0.3692 Epoch: 22 Global Step: 225500 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:06:56,238-Speed 129.70 samples/sec Loss 0.2895 Epoch: 22 Global Step: 225505 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:07:15,980-Speed 129.68 samples/sec Loss 0.3634 Epoch: 22 Global Step: 225510 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:07:35,764-Speed 129.40 samples/sec Loss 0.3101 Epoch: 22 Global Step: 225515 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:07:55,671-Speed 128.61 samples/sec Loss 0.3148 Epoch: 22 Global Step: 225520 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:08:15,384-Speed 129.87 samples/sec Loss 0.3614 Epoch: 22 Global Step: 225525 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:08:35,240-Speed 128.93 samples/sec Loss 0.3555 Epoch: 22 Global Step: 225530 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:08:56,222-Speed 122.01 samples/sec Loss 0.2759 Epoch: 22 Global Step: 225535 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:09:16,071-Speed 128.98 samples/sec Loss 0.3663 Epoch: 22 Global Step: 225540 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:09:35,774-Speed 129.93 samples/sec Loss 0.3551 Epoch: 22 Global Step: 225545 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:09:55,658-Speed 128.76 samples/sec Loss 0.3727 Epoch: 22 Global Step: 225550 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:10:15,367-Speed 129.90 samples/sec Loss 0.3798 Epoch: 22 Global Step: 225555 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:10:35,159-Speed 129.35 samples/sec Loss 0.3304 Epoch: 22 Global Step: 225560 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:10:54,906-Speed 129.65 samples/sec Loss 0.3281 Epoch: 22 Global Step: 225565 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:11:14,652-Speed 129.65 samples/sec Loss 0.3290 Epoch: 22 Global Step: 225570 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:11:34,548-Speed 128.68 samples/sec Loss 0.2718 Epoch: 22 Global Step: 225575 Fp16 Grad Scale: 2048 Required: 28 hours Training: 2021-06-18 01:11:54,810-Speed 126.35 samples/sec Loss 0.4050 Epoch: 22 Global Step: 225580 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:12:14,573-Speed 129.53 samples/sec Loss 0.3493 Epoch: 22 Global Step: 225585 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:12:34,298-Speed 129.79 samples/sec Loss 0.3222 Epoch: 22 Global Step: 225590 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:12:54,203-Speed 128.62 samples/sec Loss 0.2860 Epoch: 22 Global Step: 225595 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:13:13,907-Speed 129.93 samples/sec Loss 0.3405 Epoch: 22 Global Step: 225600 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:13:33,779-Speed 128.83 samples/sec Loss 0.3414 Epoch: 22 Global Step: 225605 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:13:53,510-Speed 129.76 samples/sec Loss 0.2801 Epoch: 22 Global Step: 225610 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:14:13,290-Speed 129.43 samples/sec Loss 0.3778 Epoch: 22 Global Step: 225615 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:14:33,202-Speed 128.57 samples/sec Loss 0.3071 Epoch: 22 Global Step: 225620 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:14:53,601-Speed 125.51 samples/sec Loss 0.3927 Epoch: 22 Global Step: 225625 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:15:13,346-Speed 129.66 samples/sec Loss 0.3977 Epoch: 22 Global Step: 225630 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:15:33,085-Speed 129.70 samples/sec Loss 0.2895 Epoch: 22 Global Step: 225635 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:15:52,995-Speed 128.59 samples/sec Loss 0.3208 Epoch: 22 Global Step: 225640 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:16:12,702-Speed 129.91 samples/sec Loss 0.3161 Epoch: 22 Global Step: 225645 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:16:32,592-Speed 128.71 samples/sec Loss 0.4219 Epoch: 22 Global Step: 225650 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:16:52,277-Speed 130.06 samples/sec Loss 0.3204 Epoch: 22 Global Step: 225655 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:17:12,023-Speed 129.66 samples/sec Loss 0.3162 Epoch: 22 Global Step: 225660 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:17:32,737-Speed 123.60 samples/sec Loss 0.3917 Epoch: 22 Global Step: 225665 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:17:53,099-Speed 125.73 samples/sec Loss 0.3479 Epoch: 22 Global Step: 225670 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:18:12,816-Speed 129.85 samples/sec Loss 0.3796 Epoch: 22 Global Step: 225675 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:18:32,541-Speed 129.79 samples/sec Loss 0.3410 Epoch: 22 Global Step: 225680 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:18:52,449-Speed 128.59 samples/sec Loss 0.3357 Epoch: 22 Global Step: 225685 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:19:12,050-Speed 130.62 samples/sec Loss 0.3511 Epoch: 22 Global Step: 225690 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:19:32,014-Speed 128.24 samples/sec Loss 0.4139 Epoch: 22 Global Step: 225695 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:19:51,647-Speed 130.40 samples/sec Loss 0.3073 Epoch: 22 Global Step: 225700 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:20:11,422-Speed 129.47 samples/sec Loss 0.3331 Epoch: 22 Global Step: 225705 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:20:31,281-Speed 128.92 samples/sec Loss 0.3547 Epoch: 22 Global Step: 225710 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:20:51,556-Speed 126.27 samples/sec Loss 0.3849 Epoch: 22 Global Step: 225715 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:21:11,342-Speed 129.39 samples/sec Loss 0.3072 Epoch: 22 Global Step: 225720 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:21:31,023-Speed 130.08 samples/sec Loss 0.3265 Epoch: 22 Global Step: 225725 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:21:50,887-Speed 128.89 samples/sec Loss 0.3366 Epoch: 22 Global Step: 225730 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:22:10,485-Speed 130.63 samples/sec Loss 0.3675 Epoch: 22 Global Step: 225735 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:22:29,945-Speed 131.56 samples/sec Loss 0.3526 Epoch: 22 Global Step: 225740 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:22:49,729-Speed 129.40 samples/sec Loss 0.3839 Epoch: 22 Global Step: 225745 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:23:09,643-Speed 128.56 samples/sec Loss 0.3121 Epoch: 22 Global Step: 225750 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:23:29,355-Speed 129.88 samples/sec Loss 0.3972 Epoch: 22 Global Step: 225755 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:23:49,014-Speed 130.23 samples/sec Loss 0.3779 Epoch: 22 Global Step: 225760 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:24:08,869-Speed 128.94 samples/sec Loss 0.3848 Epoch: 22 Global Step: 225765 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:24:28,640-Speed 129.48 samples/sec Loss 0.3239 Epoch: 22 Global Step: 225770 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:24:48,498-Speed 128.93 samples/sec Loss 0.3209 Epoch: 22 Global Step: 225775 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:25:08,207-Speed 129.90 samples/sec Loss 0.2905 Epoch: 22 Global Step: 225780 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:25:28,469-Speed 126.35 samples/sec Loss 0.3009 Epoch: 22 Global Step: 225785 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:25:48,215-Speed 129.66 samples/sec Loss 0.3563 Epoch: 22 Global Step: 225790 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:26:08,102-Speed 128.73 samples/sec Loss 0.3170 Epoch: 22 Global Step: 225795 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:26:27,802-Speed 129.96 samples/sec Loss 0.2804 Epoch: 22 Global Step: 225800 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:26:47,577-Speed 129.46 samples/sec Loss 0.3390 Epoch: 22 Global Step: 225805 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:27:07,476-Speed 128.66 samples/sec Loss 0.3261 Epoch: 22 Global Step: 225810 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:27:27,267-Speed 129.35 samples/sec Loss 0.3105 Epoch: 22 Global Step: 225815 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:27:47,158-Speed 128.71 samples/sec Loss 0.3452 Epoch: 22 Global Step: 225820 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:28:06,889-Speed 129.76 samples/sec Loss 0.3451 Epoch: 22 Global Step: 225825 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:28:27,215-Speed 125.95 samples/sec Loss 0.4078 Epoch: 22 Global Step: 225830 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:28:46,979-Speed 129.54 samples/sec Loss 0.3421 Epoch: 22 Global Step: 225835 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:29:06,837-Speed 128.92 samples/sec Loss 0.3663 Epoch: 22 Global Step: 225840 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:29:26,013-Speed 133.51 samples/sec Loss 0.3004 Epoch: 22 Global Step: 225845 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:29:45,819-Speed 129.26 samples/sec Loss 0.3463 Epoch: 22 Global Step: 225850 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:30:05,601-Speed 129.42 samples/sec Loss 0.3159 Epoch: 22 Global Step: 225855 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:30:25,377-Speed 129.46 samples/sec Loss 0.3108 Epoch: 22 Global Step: 225860 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:30:45,152-Speed 129.46 samples/sec Loss 0.3821 Epoch: 22 Global Step: 225865 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:31:04,953-Speed 129.29 samples/sec Loss 0.3244 Epoch: 22 Global Step: 225870 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:31:25,279-Speed 125.95 samples/sec Loss 0.4070 Epoch: 22 Global Step: 225875 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:31:45,107-Speed 129.12 samples/sec Loss 0.2979 Epoch: 22 Global Step: 225880 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:32:04,829-Speed 129.81 samples/sec Loss 0.3539 Epoch: 22 Global Step: 225885 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:32:24,567-Speed 129.70 samples/sec Loss 0.3513 Epoch: 22 Global Step: 225890 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:32:44,324-Speed 129.58 samples/sec Loss 0.3590 Epoch: 22 Global Step: 225895 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:33:04,096-Speed 129.49 samples/sec Loss 0.3745 Epoch: 22 Global Step: 225900 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:33:23,877-Speed 129.42 samples/sec Loss 0.3464 Epoch: 22 Global Step: 225905 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:33:43,611-Speed 129.73 samples/sec Loss 0.3147 Epoch: 22 Global Step: 225910 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:34:03,273-Speed 130.21 samples/sec Loss 0.3328 Epoch: 22 Global Step: 225915 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:34:23,529-Speed 126.39 samples/sec Loss 0.3539 Epoch: 22 Global Step: 225920 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:34:43,416-Speed 128.73 samples/sec Loss 0.3461 Epoch: 22 Global Step: 225925 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:35:03,116-Speed 129.96 samples/sec Loss 0.3206 Epoch: 22 Global Step: 225930 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:35:22,914-Speed 129.32 samples/sec Loss 0.3016 Epoch: 22 Global Step: 225935 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:35:42,728-Speed 129.21 samples/sec Loss 0.3201 Epoch: 22 Global Step: 225940 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:36:02,438-Speed 129.89 samples/sec Loss 0.3829 Epoch: 22 Global Step: 225945 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:36:22,324-Speed 128.74 samples/sec Loss 0.3548 Epoch: 22 Global Step: 225950 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:36:42,056-Speed 129.74 samples/sec Loss 0.3437 Epoch: 22 Global Step: 225955 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:37:01,750-Speed 130.00 samples/sec Loss 0.3163 Epoch: 22 Global Step: 225960 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:37:22,071-Speed 125.98 samples/sec Loss 0.3068 Epoch: 22 Global Step: 225965 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:37:41,961-Speed 128.71 samples/sec Loss 0.2930 Epoch: 22 Global Step: 225970 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:38:01,666-Speed 129.92 samples/sec Loss 0.3277 Epoch: 22 Global Step: 225975 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:38:21,502-Speed 129.07 samples/sec Loss 0.4150 Epoch: 22 Global Step: 225980 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:38:41,209-Speed 129.91 samples/sec Loss 0.3293 Epoch: 22 Global Step: 225985 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:39:01,015-Speed 129.26 samples/sec Loss 0.3060 Epoch: 22 Global Step: 225990 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:39:20,870-Speed 128.94 samples/sec Loss 0.3419 Epoch: 22 Global Step: 225995 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:39:40,616-Speed 129.65 samples/sec Loss 0.4113 Epoch: 22 Global Step: 226000 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:40:00,403-Speed 129.38 samples/sec Loss 0.2721 Epoch: 22 Global Step: 226005 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:40:20,688-Speed 126.21 samples/sec Loss 0.3738 Epoch: 22 Global Step: 226010 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:40:40,594-Speed 128.61 samples/sec Loss 0.3480 Epoch: 22 Global Step: 226015 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:41:00,326-Speed 129.75 samples/sec Loss 0.3895 Epoch: 22 Global Step: 226020 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:41:19,682-Speed 132.27 samples/sec Loss 0.3635 Epoch: 22 Global Step: 226025 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:41:39,374-Speed 130.00 samples/sec Loss 0.2670 Epoch: 22 Global Step: 226030 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:41:59,257-Speed 128.76 samples/sec Loss 0.3503 Epoch: 22 Global Step: 226035 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:42:19,071-Speed 129.21 samples/sec Loss 0.2893 Epoch: 22 Global Step: 226040 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:42:38,808-Speed 129.71 samples/sec Loss 0.3329 Epoch: 22 Global Step: 226045 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:42:58,591-Speed 129.41 samples/sec Loss 0.3178 Epoch: 22 Global Step: 226050 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:43:19,031-Speed 125.25 samples/sec Loss 0.3604 Epoch: 22 Global Step: 226055 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:43:38,757-Speed 129.79 samples/sec Loss 0.3257 Epoch: 22 Global Step: 226060 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:43:58,529-Speed 129.48 samples/sec Loss 0.2985 Epoch: 22 Global Step: 226065 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:44:18,284-Speed 129.60 samples/sec Loss 0.3201 Epoch: 22 Global Step: 226070 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:44:38,007-Speed 129.80 samples/sec Loss 0.3336 Epoch: 22 Global Step: 226075 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:44:57,962-Speed 128.30 samples/sec Loss 0.4206 Epoch: 22 Global Step: 226080 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:45:17,681-Speed 129.83 samples/sec Loss 0.3167 Epoch: 22 Global Step: 226085 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:45:37,602-Speed 128.52 samples/sec Loss 0.2705 Epoch: 22 Global Step: 226090 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:45:57,335-Speed 129.74 samples/sec Loss 0.3218 Epoch: 22 Global Step: 226095 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:46:17,774-Speed 125.25 samples/sec Loss 0.3047 Epoch: 22 Global Step: 226100 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:46:37,499-Speed 129.79 samples/sec Loss 0.3268 Epoch: 22 Global Step: 226105 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:46:57,413-Speed 128.56 samples/sec Loss 0.3791 Epoch: 22 Global Step: 226110 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:47:17,109-Speed 129.98 samples/sec Loss 0.3178 Epoch: 22 Global Step: 226115 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:47:36,772-Speed 130.20 samples/sec Loss 0.3280 Epoch: 22 Global Step: 226120 Fp16 Grad Scale: 4096 Required: 28 hours Training: 2021-06-18 01:47:56,697-Speed 128.49 samples/sec Loss 0.2585 Epoch: 22 Global Step: 226125 Fp16 Grad Scale: 8192 Required: 28 hours Training: 2021-06-18 01:48:16,435-Speed 129.71 samples/sec Loss 0.3913 Epoch: 22 Global Step: 226130 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 01:48:36,288-Speed 128.96 samples/sec Loss 0.3185 Epoch: 22 Global Step: 226135 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 01:48:55,983-Speed 129.99 samples/sec Loss 0.3939 Epoch: 22 Global Step: 226140 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 01:49:16,321-Speed 125.88 samples/sec Loss 0.2953 Epoch: 22 Global Step: 226145 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 01:49:36,353-Speed 127.80 samples/sec Loss 0.3921 Epoch: 22 Global Step: 226150 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 01:49:56,146-Speed 129.34 samples/sec Loss 0.4173 Epoch: 22 Global Step: 226155 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 01:50:15,851-Speed 129.92 samples/sec Loss 0.3405 Epoch: 22 Global Step: 226160 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 01:50:35,653-Speed 129.29 samples/sec Loss 0.3357 Epoch: 22 Global Step: 226165 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 01:50:55,439-Speed 129.39 samples/sec Loss 0.3837 Epoch: 22 Global Step: 226170 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 01:51:14,911-Speed 131.48 samples/sec Loss 0.3823 Epoch: 22 Global Step: 226175 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:51:34,725-Speed 129.21 samples/sec Loss 0.4258 Epoch: 22 Global Step: 226180 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:51:54,473-Speed 129.64 samples/sec Loss 0.3259 Epoch: 22 Global Step: 226185 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:52:14,350-Speed 128.80 samples/sec Loss 0.3318 Epoch: 22 Global Step: 226190 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:52:34,037-Speed 130.04 samples/sec Loss 0.3507 Epoch: 22 Global Step: 226195 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:52:53,948-Speed 128.58 samples/sec Loss 0.3272 Epoch: 22 Global Step: 226200 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:53:13,640-Speed 130.01 samples/sec Loss 0.3510 Epoch: 22 Global Step: 226205 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:53:33,481-Speed 129.03 samples/sec Loss 0.4370 Epoch: 22 Global Step: 226210 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:53:53,707-Speed 126.57 samples/sec Loss 0.3054 Epoch: 22 Global Step: 226215 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:54:13,533-Speed 129.13 samples/sec Loss 0.3242 Epoch: 22 Global Step: 226220 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:54:33,273-Speed 129.69 samples/sec Loss 0.3572 Epoch: 22 Global Step: 226225 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:54:53,093-Speed 129.17 samples/sec Loss 0.3096 Epoch: 22 Global Step: 226230 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:55:12,781-Speed 130.03 samples/sec Loss 0.3210 Epoch: 22 Global Step: 226235 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:55:32,603-Speed 129.16 samples/sec Loss 0.3574 Epoch: 22 Global Step: 226240 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:55:52,442-Speed 129.04 samples/sec Loss 0.3916 Epoch: 22 Global Step: 226245 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:56:12,323-Speed 128.77 samples/sec Loss 0.3202 Epoch: 22 Global Step: 226250 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:56:32,171-Speed 128.99 samples/sec Loss 0.3669 Epoch: 22 Global Step: 226255 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:56:52,410-Speed 126.49 samples/sec Loss 0.3943 Epoch: 22 Global Step: 226260 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:57:12,254-Speed 129.01 samples/sec Loss 0.3176 Epoch: 22 Global Step: 226265 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:57:31,966-Speed 129.87 samples/sec Loss 0.3053 Epoch: 22 Global Step: 226270 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:57:51,827-Speed 128.90 samples/sec Loss 0.3068 Epoch: 22 Global Step: 226275 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 01:58:10,993-Speed 133.58 samples/sec Loss 0.4051 Epoch: 22 Global Step: 226280 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:58:30,973-Speed 128.14 samples/sec Loss 0.3068 Epoch: 22 Global Step: 226285 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:58:50,664-Speed 130.01 samples/sec Loss 0.2880 Epoch: 22 Global Step: 226290 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:59:10,418-Speed 129.61 samples/sec Loss 0.3340 Epoch: 22 Global Step: 226295 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:59:30,202-Speed 129.40 samples/sec Loss 0.3742 Epoch: 22 Global Step: 226300 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 01:59:50,535-Speed 125.91 samples/sec Loss 0.3532 Epoch: 22 Global Step: 226305 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:00:10,144-Speed 130.55 samples/sec Loss 0.3269 Epoch: 22 Global Step: 226310 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:00:30,038-Speed 128.69 samples/sec Loss 0.3265 Epoch: 22 Global Step: 226315 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:00:49,664-Speed 130.45 samples/sec Loss 0.3304 Epoch: 22 Global Step: 226320 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:01:09,281-Speed 130.50 samples/sec Loss 0.3704 Epoch: 22 Global Step: 226325 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:01:29,230-Speed 128.34 samples/sec Loss 0.3384 Epoch: 22 Global Step: 226330 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:01:48,833-Speed 130.60 samples/sec Loss 0.3204 Epoch: 22 Global Step: 226335 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:02:08,752-Speed 128.53 samples/sec Loss 0.3295 Epoch: 22 Global Step: 226340 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:02:28,389-Speed 130.38 samples/sec Loss 0.3736 Epoch: 22 Global Step: 226345 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:02:48,735-Speed 125.82 samples/sec Loss 0.3709 Epoch: 22 Global Step: 226350 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:03:08,450-Speed 129.86 samples/sec Loss 0.2931 Epoch: 22 Global Step: 226355 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:03:28,365-Speed 128.55 samples/sec Loss 0.3613 Epoch: 22 Global Step: 226360 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:03:47,945-Speed 130.75 samples/sec Loss 0.3541 Epoch: 22 Global Step: 226365 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:04:07,719-Speed 129.47 samples/sec Loss 0.3042 Epoch: 22 Global Step: 226370 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:04:27,514-Speed 129.33 samples/sec Loss 0.3734 Epoch: 22 Global Step: 226375 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:04:47,189-Speed 130.13 samples/sec Loss 0.2937 Epoch: 22 Global Step: 226380 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:05:07,113-Speed 128.49 samples/sec Loss 0.3105 Epoch: 22 Global Step: 226385 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:05:26,806-Speed 130.00 samples/sec Loss 0.3373 Epoch: 22 Global Step: 226390 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:05:47,170-Speed 125.72 samples/sec Loss 0.2958 Epoch: 22 Global Step: 226395 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:06:06,911-Speed 129.69 samples/sec Loss 0.3482 Epoch: 22 Global Step: 226400 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:06:26,807-Speed 128.67 samples/sec Loss 0.3176 Epoch: 22 Global Step: 226405 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:06:46,552-Speed 129.66 samples/sec Loss 0.3655 Epoch: 22 Global Step: 226410 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:07:06,358-Speed 129.27 samples/sec Loss 0.2768 Epoch: 22 Global Step: 226415 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:07:26,132-Speed 129.47 samples/sec Loss 0.3613 Epoch: 22 Global Step: 226420 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:07:45,864-Speed 129.75 samples/sec Loss 0.3225 Epoch: 22 Global Step: 226425 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:08:05,756-Speed 128.70 samples/sec Loss 0.3154 Epoch: 22 Global Step: 226430 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:08:25,399-Speed 130.33 samples/sec Loss 0.3803 Epoch: 22 Global Step: 226435 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:08:45,742-Speed 125.85 samples/sec Loss 0.4172 Epoch: 22 Global Step: 226440 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:09:05,620-Speed 128.79 samples/sec Loss 0.2896 Epoch: 22 Global Step: 226445 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:09:25,345-Speed 129.79 samples/sec Loss 0.3890 Epoch: 22 Global Step: 226450 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:09:45,014-Speed 130.17 samples/sec Loss 0.3450 Epoch: 22 Global Step: 226455 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:10:04,774-Speed 129.56 samples/sec Loss 0.3507 Epoch: 22 Global Step: 226460 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:10:24,566-Speed 129.35 samples/sec Loss 0.3660 Epoch: 22 Global Step: 226465 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:10:44,437-Speed 128.84 samples/sec Loss 0.4241 Epoch: 22 Global Step: 226470 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:11:04,170-Speed 129.74 samples/sec Loss 0.3093 Epoch: 22 Global Step: 226475 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:11:23,844-Speed 130.13 samples/sec Loss 0.3834 Epoch: 22 Global Step: 226480 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:11:44,138-Speed 126.15 samples/sec Loss 0.3277 Epoch: 22 Global Step: 226485 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:12:04,023-Speed 128.75 samples/sec Loss 0.2870 Epoch: 22 Global Step: 226490 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:12:23,731-Speed 129.90 samples/sec Loss 0.2298 Epoch: 22 Global Step: 226495 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:12:43,471-Speed 129.69 samples/sec Loss 0.3320 Epoch: 22 Global Step: 226500 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:13:03,212-Speed 129.68 samples/sec Loss 0.3578 Epoch: 22 Global Step: 226505 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:13:22,980-Speed 129.51 samples/sec Loss 0.3269 Epoch: 22 Global Step: 226510 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:13:42,856-Speed 128.81 samples/sec Loss 0.2790 Epoch: 22 Global Step: 226515 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:14:02,552-Speed 129.98 samples/sec Loss 0.3293 Epoch: 22 Global Step: 226520 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:14:22,294-Speed 129.68 samples/sec Loss 0.3113 Epoch: 22 Global Step: 226525 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:14:42,158-Speed 128.88 samples/sec Loss 0.3934 Epoch: 22 Global Step: 226530 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:15:02,131-Speed 128.18 samples/sec Loss 0.2995 Epoch: 22 Global Step: 226535 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:15:21,799-Speed 130.16 samples/sec Loss 0.3193 Epoch: 22 Global Step: 226540 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:15:41,557-Speed 129.58 samples/sec Loss 0.3323 Epoch: 22 Global Step: 226545 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:16:01,309-Speed 129.61 samples/sec Loss 0.3471 Epoch: 22 Global Step: 226550 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:16:21,088-Speed 129.44 samples/sec Loss 0.2824 Epoch: 22 Global Step: 226555 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:16:40,996-Speed 128.60 samples/sec Loss 0.3611 Epoch: 22 Global Step: 226560 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:17:00,714-Speed 129.84 samples/sec Loss 0.3668 Epoch: 22 Global Step: 226565 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:17:20,503-Speed 129.37 samples/sec Loss 0.3563 Epoch: 22 Global Step: 226570 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:17:40,754-Speed 126.42 samples/sec Loss 0.2743 Epoch: 22 Global Step: 226575 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:18:00,638-Speed 128.76 samples/sec Loss 0.2886 Epoch: 22 Global Step: 226580 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:18:20,441-Speed 129.27 samples/sec Loss 0.3853 Epoch: 22 Global Step: 226585 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:18:40,349-Speed 128.60 samples/sec Loss 0.3546 Epoch: 22 Global Step: 226590 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:18:59,965-Speed 130.51 samples/sec Loss 0.4017 Epoch: 22 Global Step: 226595 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:19:19,802-Speed 129.08 samples/sec Loss 0.3623 Epoch: 22 Global Step: 226600 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:19:39,648-Speed 129.00 samples/sec Loss 0.3398 Epoch: 22 Global Step: 226605 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:19:59,360-Speed 129.88 samples/sec Loss 0.3634 Epoch: 22 Global Step: 226610 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:20:19,192-Speed 129.09 samples/sec Loss 0.3334 Epoch: 22 Global Step: 226615 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:20:39,351-Speed 126.99 samples/sec Loss 0.4180 Epoch: 22 Global Step: 226620 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:20:59,205-Speed 128.95 samples/sec Loss 0.3260 Epoch: 22 Global Step: 226625 Fp16 Grad Scale: 4096 Required: 27 hours Training: 2021-06-18 02:21:18,945-Speed 129.69 samples/sec Loss 0.3243 Epoch: 22 Global Step: 226630 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:21:38,880-Speed 128.43 samples/sec Loss 0.2940 Epoch: 22 Global Step: 226635 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:21:58,526-Speed 130.31 samples/sec Loss 0.3711 Epoch: 22 Global Step: 226640 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:22:18,488-Speed 128.25 samples/sec Loss 0.2845 Epoch: 22 Global Step: 226645 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:22:38,117-Speed 130.43 samples/sec Loss 0.4387 Epoch: 22 Global Step: 226650 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:22:57,898-Speed 129.42 samples/sec Loss 0.3292 Epoch: 22 Global Step: 226655 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:23:17,762-Speed 128.89 samples/sec Loss 0.3483 Epoch: 22 Global Step: 226660 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:23:38,177-Speed 125.41 samples/sec Loss 0.2911 Epoch: 22 Global Step: 226665 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:23:57,880-Speed 129.93 samples/sec Loss 0.3448 Epoch: 22 Global Step: 226670 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:24:17,566-Speed 130.05 samples/sec Loss 0.2965 Epoch: 22 Global Step: 226675 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:24:37,440-Speed 128.82 samples/sec Loss 0.3460 Epoch: 22 Global Step: 226680 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:24:57,177-Speed 129.72 samples/sec Loss 0.3389 Epoch: 22 Global Step: 226685 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:25:17,141-Speed 128.24 samples/sec Loss 0.3665 Epoch: 22 Global Step: 226690 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:25:36,859-Speed 129.84 samples/sec Loss 0.3585 Epoch: 22 Global Step: 226695 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:25:56,650-Speed 129.36 samples/sec Loss 0.3013 Epoch: 22 Global Step: 226700 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:26:16,648-Speed 128.02 samples/sec Loss 0.3789 Epoch: 22 Global Step: 226705 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:26:37,075-Speed 125.33 samples/sec Loss 0.3574 Epoch: 22 Global Step: 226710 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:26:56,765-Speed 130.02 samples/sec Loss 0.2788 Epoch: 22 Global Step: 226715 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:27:16,437-Speed 130.14 samples/sec Loss 0.3483 Epoch: 22 Global Step: 226720 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:27:36,306-Speed 128.85 samples/sec Loss 0.3016 Epoch: 22 Global Step: 226725 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:27:56,019-Speed 129.87 samples/sec Loss 0.3310 Epoch: 22 Global Step: 226730 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:28:15,982-Speed 128.25 samples/sec Loss 0.3296 Epoch: 22 Global Step: 226735 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:28:35,637-Speed 130.25 samples/sec Loss 0.3230 Epoch: 22 Global Step: 226740 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:28:55,413-Speed 129.46 samples/sec Loss 0.3405 Epoch: 22 Global Step: 226745 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:29:15,125-Speed 129.87 samples/sec Loss 0.4299 Epoch: 22 Global Step: 226750 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:29:35,651-Speed 124.73 samples/sec Loss 0.2970 Epoch: 22 Global Step: 226755 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:29:55,343-Speed 130.02 samples/sec Loss 0.3137 Epoch: 22 Global Step: 226760 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:30:15,141-Speed 129.32 samples/sec Loss 0.2943 Epoch: 22 Global Step: 226765 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:30:34,853-Speed 129.88 samples/sec Loss 0.3622 Epoch: 22 Global Step: 226770 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:30:54,613-Speed 129.56 samples/sec Loss 0.2994 Epoch: 22 Global Step: 226775 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:31:14,536-Speed 128.50 samples/sec Loss 0.2545 Epoch: 22 Global Step: 226780 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:31:34,266-Speed 129.76 samples/sec Loss 0.3412 Epoch: 22 Global Step: 226785 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:31:54,078-Speed 129.22 samples/sec Loss 0.3237 Epoch: 22 Global Step: 226790 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:32:13,734-Speed 130.25 samples/sec Loss 0.3334 Epoch: 22 Global Step: 226795 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:32:34,111-Speed 125.64 samples/sec Loss 0.3299 Epoch: 22 Global Step: 226800 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:32:53,882-Speed 129.49 samples/sec Loss 0.3044 Epoch: 22 Global Step: 226805 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:33:13,651-Speed 129.50 samples/sec Loss 0.2868 Epoch: 22 Global Step: 226810 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:33:34,051-Speed 125.50 samples/sec Loss 0.3346 Epoch: 22 Global Step: 226815 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:33:53,740-Speed 130.03 samples/sec Loss 0.3622 Epoch: 22 Global Step: 226820 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:34:13,776-Speed 127.77 samples/sec Loss 0.3365 Epoch: 22 Global Step: 226825 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:34:33,565-Speed 129.38 samples/sec Loss 0.3444 Epoch: 22 Global Step: 226830 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:34:53,574-Speed 127.95 samples/sec Loss 0.4188 Epoch: 22 Global Step: 226835 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:35:13,230-Speed 130.25 samples/sec Loss 0.3406 Epoch: 22 Global Step: 226840 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:35:33,595-Speed 125.71 samples/sec Loss 0.3771 Epoch: 22 Global Step: 226845 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:35:53,354-Speed 129.56 samples/sec Loss 0.2900 Epoch: 22 Global Step: 226850 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:36:13,178-Speed 129.15 samples/sec Loss 0.3388 Epoch: 22 Global Step: 226855 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:36:32,891-Speed 129.87 samples/sec Loss 0.3598 Epoch: 22 Global Step: 226860 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:36:52,576-Speed 130.06 samples/sec Loss 0.3396 Epoch: 22 Global Step: 226865 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:37:12,450-Speed 128.82 samples/sec Loss 0.3921 Epoch: 22 Global Step: 226870 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:37:32,072-Speed 130.47 samples/sec Loss 0.3321 Epoch: 22 Global Step: 226875 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:37:52,045-Speed 128.18 samples/sec Loss 0.3446 Epoch: 22 Global Step: 226880 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:38:11,768-Speed 129.80 samples/sec Loss 0.3886 Epoch: 22 Global Step: 226885 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:38:32,117-Speed 125.81 samples/sec Loss 0.3660 Epoch: 22 Global Step: 226890 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:38:51,775-Speed 130.23 samples/sec Loss 0.3578 Epoch: 22 Global Step: 226895 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:39:11,697-Speed 128.50 samples/sec Loss 0.3049 Epoch: 22 Global Step: 226900 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:39:31,429-Speed 129.75 samples/sec Loss 0.3009 Epoch: 22 Global Step: 226905 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:39:51,269-Speed 129.04 samples/sec Loss 0.3130 Epoch: 22 Global Step: 226910 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:40:10,978-Speed 129.90 samples/sec Loss 0.3668 Epoch: 22 Global Step: 226915 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:40:30,642-Speed 130.20 samples/sec Loss 0.3559 Epoch: 22 Global Step: 226920 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:40:50,438-Speed 129.33 samples/sec Loss 0.3441 Epoch: 22 Global Step: 226925 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:41:10,189-Speed 129.62 samples/sec Loss 0.3693 Epoch: 22 Global Step: 226930 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:41:30,593-Speed 125.47 samples/sec Loss 0.4089 Epoch: 22 Global Step: 226935 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:41:50,490-Speed 128.67 samples/sec Loss 0.3184 Epoch: 22 Global Step: 226940 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:42:10,089-Speed 130.62 samples/sec Loss 0.3483 Epoch: 22 Global Step: 226945 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:42:29,997-Speed 128.59 samples/sec Loss 0.3505 Epoch: 22 Global Step: 226950 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:42:49,767-Speed 129.50 samples/sec Loss 0.3566 Epoch: 22 Global Step: 226955 Fp16 Grad Scale: 8192 Required: 27 hours Training: 2021-06-18 02:43:09,499-Speed 129.75 samples/sec Loss 0.3290 Epoch: 22 Global Step: 226960 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:43:29,358-Speed 128.92 samples/sec Loss 0.3407 Epoch: 22 Global Step: 226965 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:43:48,986-Speed 130.44 samples/sec Loss 0.3414 Epoch: 22 Global Step: 226970 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:44:08,924-Speed 128.40 samples/sec Loss 0.3153 Epoch: 22 Global Step: 226975 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:44:29,167-Speed 126.47 samples/sec Loss 0.3045 Epoch: 22 Global Step: 226980 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:44:49,445-Speed 126.25 samples/sec Loss 0.3430 Epoch: 22 Global Step: 226985 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:45:09,203-Speed 129.57 samples/sec Loss 0.3496 Epoch: 22 Global Step: 226990 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:45:29,172-Speed 128.21 samples/sec Loss 0.3617 Epoch: 22 Global Step: 226995 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:45:48,862-Speed 130.03 samples/sec Loss 0.4585 Epoch: 22 Global Step: 227000 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:46:08,635-Speed 129.48 samples/sec Loss 0.3511 Epoch: 22 Global Step: 227005 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:46:28,443-Speed 129.25 samples/sec Loss 0.3357 Epoch: 22 Global Step: 227010 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:46:48,077-Speed 130.39 samples/sec Loss 0.3571 Epoch: 22 Global Step: 227015 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:47:08,046-Speed 128.21 samples/sec Loss 0.3180 Epoch: 22 Global Step: 227020 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:47:28,270-Speed 126.59 samples/sec Loss 0.2914 Epoch: 22 Global Step: 227025 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:47:48,043-Speed 129.48 samples/sec Loss 0.3218 Epoch: 22 Global Step: 227030 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:48:07,653-Speed 130.55 samples/sec Loss 0.3322 Epoch: 22 Global Step: 227035 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:48:27,658-Speed 127.97 samples/sec Loss 0.3101 Epoch: 22 Global Step: 227040 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:48:47,308-Speed 130.29 samples/sec Loss 0.3213 Epoch: 22 Global Step: 227045 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:49:07,212-Speed 128.63 samples/sec Loss 0.3187 Epoch: 22 Global Step: 227050 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:49:26,892-Speed 130.08 samples/sec Loss 0.3312 Epoch: 22 Global Step: 227055 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:49:46,553-Speed 130.22 samples/sec Loss 0.3642 Epoch: 22 Global Step: 227060 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:50:06,507-Speed 128.30 samples/sec Loss 0.3518 Epoch: 22 Global Step: 227065 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:50:26,896-Speed 125.56 samples/sec Loss 0.3206 Epoch: 22 Global Step: 227070 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:50:45,955-Speed 134.33 samples/sec Loss 0.3589 Epoch: 22 Global Step: 227075 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:51:05,866-Speed 128.58 samples/sec Loss 0.3166 Epoch: 22 Global Step: 227080 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:51:25,738-Speed 128.83 samples/sec Loss 0.3683 Epoch: 22 Global Step: 227085 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:51:45,434-Speed 129.99 samples/sec Loss 0.3324 Epoch: 22 Global Step: 227090 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:52:05,316-Speed 128.76 samples/sec Loss 0.3334 Epoch: 22 Global Step: 227095 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:52:24,965-Speed 130.29 samples/sec Loss 0.2680 Epoch: 22 Global Step: 227100 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:52:44,856-Speed 128.71 samples/sec Loss 0.2707 Epoch: 22 Global Step: 227105 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:53:04,660-Speed 129.27 samples/sec Loss 0.3615 Epoch: 22 Global Step: 227110 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:53:24,448-Speed 129.38 samples/sec Loss 0.3730 Epoch: 22 Global Step: 227115 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:53:44,240-Speed 129.35 samples/sec Loss 0.2813 Epoch: 22 Global Step: 227120 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:54:04,494-Speed 126.40 samples/sec Loss 0.3116 Epoch: 22 Global Step: 227125 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:54:24,260-Speed 129.52 samples/sec Loss 0.3045 Epoch: 22 Global Step: 227130 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:54:44,011-Speed 129.62 samples/sec Loss 0.3448 Epoch: 22 Global Step: 227135 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:55:03,841-Speed 129.11 samples/sec Loss 0.3663 Epoch: 22 Global Step: 227140 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:55:23,528-Speed 130.04 samples/sec Loss 0.3357 Epoch: 22 Global Step: 227145 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:55:43,442-Speed 128.56 samples/sec Loss 0.3225 Epoch: 22 Global Step: 227150 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:56:03,249-Speed 129.25 samples/sec Loss 0.3494 Epoch: 22 Global Step: 227155 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:56:23,168-Speed 128.52 samples/sec Loss 0.4039 Epoch: 22 Global Step: 227160 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:56:42,877-Speed 129.90 samples/sec Loss 0.2690 Epoch: 22 Global Step: 227165 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:57:03,221-Speed 125.84 samples/sec Loss 0.2814 Epoch: 22 Global Step: 227170 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 02:57:23,062-Speed 129.04 samples/sec Loss 0.3536 Epoch: 22 Global Step: 227175 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:57:42,979-Speed 128.53 samples/sec Loss 0.3543 Epoch: 22 Global Step: 227180 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:58:02,677-Speed 129.97 samples/sec Loss 0.2675 Epoch: 22 Global Step: 227185 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:58:22,421-Speed 129.67 samples/sec Loss 0.3386 Epoch: 22 Global Step: 227190 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:58:42,270-Speed 128.98 samples/sec Loss 0.3168 Epoch: 22 Global Step: 227195 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:59:02,064-Speed 129.34 samples/sec Loss 0.3421 Epoch: 22 Global Step: 227200 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:59:21,973-Speed 128.59 samples/sec Loss 0.3107 Epoch: 22 Global Step: 227205 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 02:59:41,680-Speed 129.91 samples/sec Loss 0.3554 Epoch: 22 Global Step: 227210 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:00:01,962-Speed 126.23 samples/sec Loss 0.3864 Epoch: 22 Global Step: 227215 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:00:21,846-Speed 128.75 samples/sec Loss 0.3489 Epoch: 22 Global Step: 227220 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:00:41,655-Speed 129.24 samples/sec Loss 0.2861 Epoch: 22 Global Step: 227225 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:01:00,976-Speed 132.50 samples/sec Loss 0.3206 Epoch: 22 Global Step: 227230 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:01:20,910-Speed 128.44 samples/sec Loss 0.3907 Epoch: 22 Global Step: 227235 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:01:40,598-Speed 130.03 samples/sec Loss 0.3383 Epoch: 22 Global Step: 227240 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:02:00,500-Speed 128.64 samples/sec Loss 0.3356 Epoch: 22 Global Step: 227245 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:02:20,303-Speed 129.28 samples/sec Loss 0.2882 Epoch: 22 Global Step: 227250 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:02:40,058-Speed 129.59 samples/sec Loss 0.2513 Epoch: 22 Global Step: 227255 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:03:00,282-Speed 126.59 samples/sec Loss 0.3627 Epoch: 22 Global Step: 227260 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:03:20,254-Speed 128.19 samples/sec Loss 0.3130 Epoch: 22 Global Step: 227265 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:03:39,962-Speed 129.90 samples/sec Loss 0.3402 Epoch: 22 Global Step: 227270 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:03:59,849-Speed 128.73 samples/sec Loss 0.4045 Epoch: 22 Global Step: 227275 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:04:19,570-Speed 129.82 samples/sec Loss 0.3334 Epoch: 22 Global Step: 227280 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:04:39,256-Speed 130.05 samples/sec Loss 0.3140 Epoch: 22 Global Step: 227285 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:04:59,050-Speed 129.34 samples/sec Loss 0.2893 Epoch: 22 Global Step: 227290 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:05:18,801-Speed 129.62 samples/sec Loss 0.4170 Epoch: 22 Global Step: 227295 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:05:38,688-Speed 128.74 samples/sec Loss 0.3496 Epoch: 22 Global Step: 227300 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:05:58,801-Speed 127.28 samples/sec Loss 0.4016 Epoch: 22 Global Step: 227305 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:06:18,723-Speed 128.50 samples/sec Loss 0.3711 Epoch: 22 Global Step: 227310 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:06:38,434-Speed 129.88 samples/sec Loss 0.4191 Epoch: 22 Global Step: 227315 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:06:58,343-Speed 128.60 samples/sec Loss 0.3314 Epoch: 22 Global Step: 227320 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:07:18,049-Speed 129.92 samples/sec Loss 0.3493 Epoch: 22 Global Step: 227325 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:07:37,693-Speed 130.32 samples/sec Loss 0.3247 Epoch: 22 Global Step: 227330 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:07:57,546-Speed 128.95 samples/sec Loss 0.3466 Epoch: 22 Global Step: 227335 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:08:17,286-Speed 129.69 samples/sec Loss 0.3140 Epoch: 22 Global Step: 227340 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:08:37,174-Speed 128.73 samples/sec Loss 0.3342 Epoch: 22 Global Step: 227345 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:08:57,345-Speed 126.92 samples/sec Loss 0.3307 Epoch: 22 Global Step: 227350 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:09:17,247-Speed 128.64 samples/sec Loss 0.3492 Epoch: 22 Global Step: 227355 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:09:36,939-Speed 130.01 samples/sec Loss 0.2879 Epoch: 22 Global Step: 227360 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:09:56,848-Speed 128.59 samples/sec Loss 0.3208 Epoch: 22 Global Step: 227365 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:10:16,495-Speed 130.31 samples/sec Loss 0.3421 Epoch: 22 Global Step: 227370 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:10:36,403-Speed 128.61 samples/sec Loss 0.3428 Epoch: 22 Global Step: 227375 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:10:56,029-Speed 130.44 samples/sec Loss 0.2747 Epoch: 22 Global Step: 227380 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:11:15,855-Speed 129.13 samples/sec Loss 0.3191 Epoch: 22 Global Step: 227385 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:11:35,717-Speed 128.89 samples/sec Loss 0.2723 Epoch: 22 Global Step: 227390 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:11:55,966-Speed 126.43 samples/sec Loss 0.3828 Epoch: 22 Global Step: 227395 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:12:15,638-Speed 130.14 samples/sec Loss 0.3752 Epoch: 22 Global Step: 227400 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:12:35,457-Speed 129.17 samples/sec Loss 0.3537 Epoch: 22 Global Step: 227405 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:12:55,303-Speed 129.00 samples/sec Loss 0.3441 Epoch: 22 Global Step: 227410 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:13:14,992-Speed 130.03 samples/sec Loss 0.3195 Epoch: 22 Global Step: 227415 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:13:34,896-Speed 128.62 samples/sec Loss 0.3807 Epoch: 22 Global Step: 227420 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:13:54,503-Speed 130.58 samples/sec Loss 0.3443 Epoch: 22 Global Step: 227425 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:14:14,360-Speed 128.93 samples/sec Loss 0.3695 Epoch: 22 Global Step: 227430 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:14:34,072-Speed 129.88 samples/sec Loss 0.3814 Epoch: 22 Global Step: 227435 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:14:54,354-Speed 126.22 samples/sec Loss 0.2448 Epoch: 22 Global Step: 227440 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:15:14,041-Speed 130.04 samples/sec Loss 0.3534 Epoch: 22 Global Step: 227445 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:15:33,865-Speed 129.14 samples/sec Loss 0.3662 Epoch: 22 Global Step: 227450 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:15:53,730-Speed 128.88 samples/sec Loss 0.3062 Epoch: 22 Global Step: 227455 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:16:13,004-Speed 132.83 samples/sec Loss 0.3622 Epoch: 22 Global Step: 227460 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:16:33,109-Speed 127.33 samples/sec Loss 0.4340 Epoch: 22 Global Step: 227465 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:16:52,707-Speed 130.63 samples/sec Loss 0.4435 Epoch: 22 Global Step: 227470 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:17:12,664-Speed 128.28 samples/sec Loss 0.3352 Epoch: 22 Global Step: 227475 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:17:32,378-Speed 129.86 samples/sec Loss 0.3432 Epoch: 22 Global Step: 227480 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:17:52,788-Speed 125.43 samples/sec Loss 0.3294 Epoch: 22 Global Step: 227485 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:18:12,400-Speed 130.54 samples/sec Loss 0.3232 Epoch: 22 Global Step: 227490 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:18:32,185-Speed 129.40 samples/sec Loss 0.2970 Epoch: 22 Global Step: 227495 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:18:52,012-Speed 129.12 samples/sec Loss 0.3177 Epoch: 22 Global Step: 227500 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:19:11,683-Speed 130.15 samples/sec Loss 0.3628 Epoch: 22 Global Step: 227505 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:19:31,503-Speed 129.17 samples/sec Loss 0.3701 Epoch: 22 Global Step: 227510 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:19:51,147-Speed 130.32 samples/sec Loss 0.4132 Epoch: 22 Global Step: 227515 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:20:11,110-Speed 128.24 samples/sec Loss 0.3331 Epoch: 22 Global Step: 227520 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:20:30,820-Speed 129.89 samples/sec Loss 0.2826 Epoch: 22 Global Step: 227525 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:20:51,126-Speed 126.07 samples/sec Loss 0.3601 Epoch: 22 Global Step: 227530 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:21:10,830-Speed 129.93 samples/sec Loss 0.3891 Epoch: 22 Global Step: 227535 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:21:30,798-Speed 128.21 samples/sec Loss 0.2545 Epoch: 22 Global Step: 227540 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:21:50,525-Speed 129.78 samples/sec Loss 0.2922 Epoch: 22 Global Step: 227545 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:22:10,250-Speed 129.79 samples/sec Loss 0.3570 Epoch: 22 Global Step: 227550 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:22:30,043-Speed 129.35 samples/sec Loss 0.3298 Epoch: 22 Global Step: 227555 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:22:49,723-Speed 130.09 samples/sec Loss 0.2823 Epoch: 22 Global Step: 227560 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:23:09,681-Speed 128.28 samples/sec Loss 0.2751 Epoch: 22 Global Step: 227565 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:23:29,418-Speed 129.71 samples/sec Loss 0.3753 Epoch: 22 Global Step: 227570 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:23:49,681-Speed 126.34 samples/sec Loss 0.3670 Epoch: 22 Global Step: 227575 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:24:09,465-Speed 129.41 samples/sec Loss 0.2895 Epoch: 22 Global Step: 227580 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:24:29,390-Speed 128.49 samples/sec Loss 0.2703 Epoch: 22 Global Step: 227585 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:24:49,095-Speed 129.92 samples/sec Loss 0.3324 Epoch: 22 Global Step: 227590 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:25:09,010-Speed 128.55 samples/sec Loss 0.3559 Epoch: 22 Global Step: 227595 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:25:28,609-Speed 130.63 samples/sec Loss 0.3225 Epoch: 22 Global Step: 227600 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:25:48,331-Speed 129.81 samples/sec Loss 0.3191 Epoch: 22 Global Step: 227605 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:26:08,256-Speed 128.49 samples/sec Loss 0.3015 Epoch: 22 Global Step: 227610 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:26:27,987-Speed 129.75 samples/sec Loss 0.2958 Epoch: 22 Global Step: 227615 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:26:49,082-Speed 121.36 samples/sec Loss 0.3933 Epoch: 22 Global Step: 227620 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:27:09,001-Speed 128.53 samples/sec Loss 0.3373 Epoch: 22 Global Step: 227625 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:27:28,718-Speed 129.84 samples/sec Loss 0.3064 Epoch: 22 Global Step: 227630 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:27:48,466-Speed 129.64 samples/sec Loss 0.2701 Epoch: 22 Global Step: 227635 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:28:07,902-Speed 131.73 samples/sec Loss 0.3352 Epoch: 22 Global Step: 227640 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:28:27,529-Speed 130.44 samples/sec Loss 0.3296 Epoch: 22 Global Step: 227645 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:28:47,365-Speed 129.06 samples/sec Loss 0.2913 Epoch: 22 Global Step: 227650 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:29:07,140-Speed 129.47 samples/sec Loss 0.3458 Epoch: 22 Global Step: 227655 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:29:26,953-Speed 129.22 samples/sec Loss 0.3198 Epoch: 22 Global Step: 227660 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:29:46,693-Speed 129.69 samples/sec Loss 0.3707 Epoch: 22 Global Step: 227665 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:30:06,323-Speed 130.42 samples/sec Loss 0.3036 Epoch: 22 Global Step: 227670 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:30:26,110-Speed 129.38 samples/sec Loss 0.3948 Epoch: 22 Global Step: 227675 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:30:45,960-Speed 128.98 samples/sec Loss 0.3234 Epoch: 22 Global Step: 227680 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:31:05,787-Speed 129.12 samples/sec Loss 0.3937 Epoch: 22 Global Step: 227685 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:31:26,222-Speed 125.28 samples/sec Loss 0.3457 Epoch: 22 Global Step: 227690 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:31:45,945-Speed 129.80 samples/sec Loss 0.2679 Epoch: 22 Global Step: 227695 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:32:05,796-Speed 128.97 samples/sec Loss 0.3610 Epoch: 22 Global Step: 227700 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:32:25,613-Speed 129.19 samples/sec Loss 0.3014 Epoch: 22 Global Step: 227705 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:32:45,386-Speed 129.48 samples/sec Loss 0.3390 Epoch: 22 Global Step: 227710 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:33:05,290-Speed 128.62 samples/sec Loss 0.3487 Epoch: 22 Global Step: 227715 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:33:25,009-Speed 129.83 samples/sec Loss 0.3098 Epoch: 22 Global Step: 227720 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:33:44,859-Speed 128.97 samples/sec Loss 0.3296 Epoch: 22 Global Step: 227725 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:34:05,094-Speed 126.52 samples/sec Loss 0.2841 Epoch: 22 Global Step: 227730 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:34:25,369-Speed 126.27 samples/sec Loss 0.3176 Epoch: 22 Global Step: 227735 Fp16 Grad Scale: 4096 Required: 26 hours Training: 2021-06-18 03:34:45,110-Speed 129.69 samples/sec Loss 0.3961 Epoch: 22 Global Step: 227740 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:35:04,950-Speed 129.03 samples/sec Loss 0.2957 Epoch: 22 Global Step: 227745 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:35:24,730-Speed 129.44 samples/sec Loss 0.3550 Epoch: 22 Global Step: 227750 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:35:44,454-Speed 129.80 samples/sec Loss 0.4658 Epoch: 22 Global Step: 227755 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:36:04,261-Speed 129.26 samples/sec Loss 0.3949 Epoch: 22 Global Step: 227760 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:36:23,976-Speed 129.85 samples/sec Loss 0.3117 Epoch: 22 Global Step: 227765 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:36:43,948-Speed 128.19 samples/sec Loss 0.2764 Epoch: 22 Global Step: 227770 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:37:03,542-Speed 130.65 samples/sec Loss 0.3789 Epoch: 22 Global Step: 227775 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:37:23,806-Speed 126.34 samples/sec Loss 0.3130 Epoch: 22 Global Step: 227780 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:37:43,551-Speed 129.65 samples/sec Loss 0.3507 Epoch: 22 Global Step: 227785 Fp16 Grad Scale: 8192 Required: 26 hours Training: 2021-06-18 03:38:03,619-Speed 127.57 samples/sec Loss 0.3407 Epoch: 22 Global Step: 227790 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:38:23,330-Speed 129.89 samples/sec Loss 0.2826 Epoch: 22 Global Step: 227795 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:38:43,028-Speed 129.97 samples/sec Loss 0.3158 Epoch: 22 Global Step: 227800 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:39:02,398-Speed 132.17 samples/sec Loss 0.3127 Epoch: 22 Global Step: 227805 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:39:22,129-Speed 129.75 samples/sec Loss 0.3392 Epoch: 22 Global Step: 227810 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:39:42,064-Speed 128.43 samples/sec Loss 0.3572 Epoch: 22 Global Step: 227815 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:40:01,779-Speed 129.86 samples/sec Loss 0.4363 Epoch: 22 Global Step: 227820 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:40:21,988-Speed 126.68 samples/sec Loss 0.3701 Epoch: 22 Global Step: 227825 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:40:41,702-Speed 129.86 samples/sec Loss 0.2670 Epoch: 22 Global Step: 227830 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:41:01,645-Speed 128.37 samples/sec Loss 0.3369 Epoch: 22 Global Step: 227835 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:41:21,353-Speed 129.90 samples/sec Loss 0.3740 Epoch: 22 Global Step: 227840 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:41:40,970-Speed 130.51 samples/sec Loss 0.3873 Epoch: 22 Global Step: 227845 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:42:00,863-Speed 128.69 samples/sec Loss 0.3286 Epoch: 22 Global Step: 227850 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:42:20,574-Speed 129.88 samples/sec Loss 0.3294 Epoch: 22 Global Step: 227855 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:42:40,592-Speed 127.89 samples/sec Loss 0.3524 Epoch: 22 Global Step: 227860 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:43:00,294-Speed 129.94 samples/sec Loss 0.2843 Epoch: 22 Global Step: 227865 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:43:20,543-Speed 126.43 samples/sec Loss 0.3263 Epoch: 22 Global Step: 227870 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:43:40,375-Speed 129.09 samples/sec Loss 0.3901 Epoch: 22 Global Step: 227875 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:44:00,350-Speed 128.17 samples/sec Loss 0.3314 Epoch: 22 Global Step: 227880 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:44:20,065-Speed 129.85 samples/sec Loss 0.3952 Epoch: 22 Global Step: 227885 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:44:39,722-Speed 130.25 samples/sec Loss 0.3673 Epoch: 22 Global Step: 227890 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:44:59,583-Speed 128.90 samples/sec Loss 0.3339 Epoch: 22 Global Step: 227895 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:45:19,235-Speed 130.27 samples/sec Loss 0.3385 Epoch: 22 Global Step: 227900 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:45:39,211-Speed 128.18 samples/sec Loss 0.3685 Epoch: 22 Global Step: 227905 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:45:58,911-Speed 129.95 samples/sec Loss 0.2788 Epoch: 22 Global Step: 227910 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:46:19,188-Speed 126.26 samples/sec Loss 0.3449 Epoch: 22 Global Step: 227915 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:46:38,896-Speed 129.90 samples/sec Loss 0.3088 Epoch: 22 Global Step: 227920 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:46:58,832-Speed 128.41 samples/sec Loss 0.3177 Epoch: 22 Global Step: 227925 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:47:18,567-Speed 129.73 samples/sec Loss 0.3575 Epoch: 22 Global Step: 227930 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:47:38,241-Speed 130.13 samples/sec Loss 0.2995 Epoch: 22 Global Step: 227935 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:47:58,105-Speed 128.89 samples/sec Loss 0.3715 Epoch: 22 Global Step: 227940 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:48:17,797-Speed 130.01 samples/sec Loss 0.3200 Epoch: 22 Global Step: 227945 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:48:37,750-Speed 128.31 samples/sec Loss 0.3295 Epoch: 22 Global Step: 227950 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:48:57,481-Speed 129.75 samples/sec Loss 0.3610 Epoch: 22 Global Step: 227955 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 03:49:17,390-Speed 128.59 samples/sec Loss 0.3071 Epoch: 22 Global Step: 227960 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:49:37,590-Speed 126.74 samples/sec Loss 0.3103 Epoch: 22 Global Step: 227965 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:49:57,684-Speed 127.40 samples/sec Loss 0.2741 Epoch: 22 Global Step: 227970 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:50:17,396-Speed 129.88 samples/sec Loss 0.3457 Epoch: 22 Global Step: 227975 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:50:37,125-Speed 129.76 samples/sec Loss 0.3888 Epoch: 22 Global Step: 227980 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:50:56,934-Speed 129.24 samples/sec Loss 0.2577 Epoch: 22 Global Step: 227985 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:51:16,653-Speed 129.83 samples/sec Loss 0.4260 Epoch: 22 Global Step: 227990 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:51:36,643-Speed 128.07 samples/sec Loss 0.2831 Epoch: 22 Global Step: 227995 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:51:56,358-Speed 129.86 samples/sec Loss 0.3002 Epoch: 22 Global Step: 228000 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:52:16,630-Speed 126.29 samples/sec Loss 0.2975 Epoch: 22 Global Step: 228005 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:52:36,362-Speed 129.74 samples/sec Loss 0.3119 Epoch: 22 Global Step: 228010 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:52:56,290-Speed 128.47 samples/sec Loss 0.3638 Epoch: 22 Global Step: 228015 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:53:15,995-Speed 129.93 samples/sec Loss 0.3779 Epoch: 22 Global Step: 228020 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:53:35,798-Speed 129.28 samples/sec Loss 0.3607 Epoch: 22 Global Step: 228025 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:53:55,516-Speed 129.83 samples/sec Loss 0.3280 Epoch: 22 Global Step: 228030 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:54:15,247-Speed 129.75 samples/sec Loss 0.2954 Epoch: 22 Global Step: 228035 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:54:35,161-Speed 128.56 samples/sec Loss 0.3010 Epoch: 22 Global Step: 228040 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:54:54,876-Speed 129.86 samples/sec Loss 0.3496 Epoch: 22 Global Step: 228045 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:55:15,167-Speed 126.17 samples/sec Loss 0.3284 Epoch: 22 Global Step: 228050 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:55:34,914-Speed 129.64 samples/sec Loss 0.2581 Epoch: 22 Global Step: 228055 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:55:54,399-Speed 131.39 samples/sec Loss 0.3343 Epoch: 22 Global Step: 228060 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:56:14,683-Speed 126.21 samples/sec Loss 0.3579 Epoch: 22 Global Step: 228065 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:56:34,472-Speed 129.37 samples/sec Loss 0.3426 Epoch: 22 Global Step: 228070 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:56:54,223-Speed 129.62 samples/sec Loss 0.3057 Epoch: 22 Global Step: 228075 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:57:14,235-Speed 127.93 samples/sec Loss 0.2760 Epoch: 22 Global Step: 228080 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:57:33,840-Speed 130.59 samples/sec Loss 0.3923 Epoch: 22 Global Step: 228085 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:57:53,539-Speed 129.96 samples/sec Loss 0.3040 Epoch: 22 Global Step: 228090 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:58:13,396-Speed 128.93 samples/sec Loss 0.3544 Epoch: 22 Global Step: 228095 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:58:33,175-Speed 129.44 samples/sec Loss 0.3560 Epoch: 22 Global Step: 228100 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:58:53,595-Speed 125.37 samples/sec Loss 0.3659 Epoch: 22 Global Step: 228105 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:59:13,241-Speed 130.32 samples/sec Loss 0.3694 Epoch: 22 Global Step: 228110 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:59:32,968-Speed 129.78 samples/sec Loss 0.3615 Epoch: 22 Global Step: 228115 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 03:59:53,246-Speed 126.25 samples/sec Loss 0.3616 Epoch: 22 Global Step: 228120 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:00:13,228-Speed 128.11 samples/sec Loss 0.3468 Epoch: 22 Global Step: 228125 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:00:32,834-Speed 130.58 samples/sec Loss 0.3581 Epoch: 22 Global Step: 228130 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:00:52,760-Speed 128.48 samples/sec Loss 0.3180 Epoch: 22 Global Step: 228135 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:01:12,361-Speed 130.61 samples/sec Loss 0.2959 Epoch: 22 Global Step: 228140 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:01:32,137-Speed 129.46 samples/sec Loss 0.3134 Epoch: 22 Global Step: 228145 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:01:52,061-Speed 128.49 samples/sec Loss 0.3361 Epoch: 22 Global Step: 228150 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:02:11,693-Speed 130.41 samples/sec Loss 0.3527 Epoch: 22 Global Step: 228155 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:02:31,571-Speed 128.79 samples/sec Loss 0.3401 Epoch: 22 Global Step: 228160 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:02:51,712-Speed 127.11 samples/sec Loss 0.3074 Epoch: 22 Global Step: 228165 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:03:11,228-Speed 131.18 samples/sec Loss 0.3038 Epoch: 22 Global Step: 228170 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:03:30,822-Speed 130.66 samples/sec Loss 0.3305 Epoch: 22 Global Step: 228175 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:03:50,730-Speed 128.60 samples/sec Loss 0.3584 Epoch: 22 Global Step: 228180 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:04:10,333-Speed 130.60 samples/sec Loss 0.3496 Epoch: 22 Global Step: 228185 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:04:30,067-Speed 129.73 samples/sec Loss 0.3660 Epoch: 22 Global Step: 228190 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:04:50,033-Speed 128.23 samples/sec Loss 0.3485 Epoch: 22 Global Step: 228195 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:05:09,664-Speed 130.41 samples/sec Loss 0.3166 Epoch: 22 Global Step: 228200 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:05:29,413-Speed 129.63 samples/sec Loss 0.3455 Epoch: 22 Global Step: 228205 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:05:49,657-Speed 126.46 samples/sec Loss 0.3164 Epoch: 22 Global Step: 228210 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:06:09,601-Speed 128.37 samples/sec Loss 0.2956 Epoch: 22 Global Step: 228215 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:06:29,303-Speed 129.94 samples/sec Loss 0.4394 Epoch: 22 Global Step: 228220 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:06:49,204-Speed 128.64 samples/sec Loss 0.3261 Epoch: 22 Global Step: 228225 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:07:08,826-Speed 130.47 samples/sec Loss 0.3428 Epoch: 22 Global Step: 228230 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:07:28,670-Speed 129.01 samples/sec Loss 0.2751 Epoch: 22 Global Step: 228235 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:07:48,560-Speed 128.72 samples/sec Loss 0.3575 Epoch: 22 Global Step: 228240 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:08:08,270-Speed 129.89 samples/sec Loss 0.3438 Epoch: 22 Global Step: 228245 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:08:28,068-Speed 129.32 samples/sec Loss 0.3699 Epoch: 22 Global Step: 228250 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:08:48,502-Speed 125.29 samples/sec Loss 0.2957 Epoch: 22 Global Step: 228255 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:09:08,592-Speed 127.43 samples/sec Loss 0.3469 Epoch: 22 Global Step: 228260 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:09:28,291-Speed 129.96 samples/sec Loss 0.3241 Epoch: 22 Global Step: 228265 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:09:48,179-Speed 128.72 samples/sec Loss 0.2645 Epoch: 22 Global Step: 228270 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:10:07,827-Speed 130.30 samples/sec Loss 0.3175 Epoch: 22 Global Step: 228275 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:10:27,607-Speed 129.43 samples/sec Loss 0.2570 Epoch: 22 Global Step: 228280 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:10:47,372-Speed 129.53 samples/sec Loss 0.3759 Epoch: 22 Global Step: 228285 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:11:07,168-Speed 129.33 samples/sec Loss 0.2763 Epoch: 22 Global Step: 228290 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:11:27,048-Speed 128.78 samples/sec Loss 0.3435 Epoch: 22 Global Step: 228295 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:11:47,274-Speed 126.57 samples/sec Loss 0.3832 Epoch: 22 Global Step: 228300 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:12:07,055-Speed 129.42 samples/sec Loss 0.2899 Epoch: 22 Global Step: 228305 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:12:26,833-Speed 129.44 samples/sec Loss 0.3376 Epoch: 22 Global Step: 228310 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:12:46,775-Speed 128.38 samples/sec Loss 0.3544 Epoch: 22 Global Step: 228315 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:13:06,445-Speed 130.16 samples/sec Loss 0.3327 Epoch: 22 Global Step: 228320 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:13:26,216-Speed 129.49 samples/sec Loss 0.3464 Epoch: 22 Global Step: 228325 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:13:45,985-Speed 129.50 samples/sec Loss 0.3461 Epoch: 22 Global Step: 228330 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:14:05,773-Speed 129.38 samples/sec Loss 0.3892 Epoch: 22 Global Step: 228335 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:14:25,638-Speed 128.87 samples/sec Loss 0.3267 Epoch: 22 Global Step: 228340 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:14:45,868-Speed 126.55 samples/sec Loss 0.3214 Epoch: 22 Global Step: 228345 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:15:05,679-Speed 129.23 samples/sec Loss 0.2367 Epoch: 22 Global Step: 228350 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:15:25,462-Speed 129.41 samples/sec Loss 0.2656 Epoch: 22 Global Step: 228355 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:15:45,353-Speed 128.71 samples/sec Loss 0.2916 Epoch: 22 Global Step: 228360 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:16:05,052-Speed 129.96 samples/sec Loss 0.4182 Epoch: 22 Global Step: 228365 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:16:24,799-Speed 129.65 samples/sec Loss 0.3722 Epoch: 22 Global Step: 228370 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:16:44,586-Speed 129.39 samples/sec Loss 0.2581 Epoch: 22 Global Step: 228375 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:17:04,386-Speed 129.30 samples/sec Loss 0.3539 Epoch: 22 Global Step: 228380 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:17:24,185-Speed 129.31 samples/sec Loss 0.3612 Epoch: 22 Global Step: 228385 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:17:44,297-Speed 127.30 samples/sec Loss 0.3000 Epoch: 22 Global Step: 228390 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:18:04,172-Speed 128.81 samples/sec Loss 0.3321 Epoch: 22 Global Step: 228395 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:18:23,962-Speed 129.36 samples/sec Loss 0.3582 Epoch: 22 Global Step: 228400 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:18:43,753-Speed 129.36 samples/sec Loss 0.2966 Epoch: 22 Global Step: 228405 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:19:03,466-Speed 129.87 samples/sec Loss 0.3024 Epoch: 22 Global Step: 228410 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:19:23,155-Speed 130.03 samples/sec Loss 0.3723 Epoch: 22 Global Step: 228415 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:19:42,982-Speed 129.12 samples/sec Loss 0.2849 Epoch: 22 Global Step: 228420 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:20:02,776-Speed 129.34 samples/sec Loss 0.4234 Epoch: 22 Global Step: 228425 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:20:22,646-Speed 128.84 samples/sec Loss 0.2928 Epoch: 22 Global Step: 228430 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:20:42,814-Speed 126.94 samples/sec Loss 0.3044 Epoch: 22 Global Step: 228435 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:21:02,648-Speed 129.08 samples/sec Loss 0.3297 Epoch: 22 Global Step: 228440 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:21:22,454-Speed 129.26 samples/sec Loss 0.3709 Epoch: 22 Global Step: 228445 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:21:42,276-Speed 129.16 samples/sec Loss 0.4203 Epoch: 22 Global Step: 228450 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:22:02,004-Speed 129.77 samples/sec Loss 0.3548 Epoch: 22 Global Step: 228455 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:22:21,830-Speed 129.13 samples/sec Loss 0.3509 Epoch: 22 Global Step: 228460 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:22:41,558-Speed 129.77 samples/sec Loss 0.3017 Epoch: 22 Global Step: 228465 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:23:01,321-Speed 129.54 samples/sec Loss 0.3834 Epoch: 22 Global Step: 228470 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:23:21,200-Speed 128.79 samples/sec Loss 0.3976 Epoch: 22 Global Step: 228475 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:23:41,333-Speed 127.16 samples/sec Loss 0.3434 Epoch: 22 Global Step: 228480 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:24:01,165-Speed 129.09 samples/sec Loss 0.2586 Epoch: 22 Global Step: 228485 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:24:20,781-Speed 130.51 samples/sec Loss 0.3703 Epoch: 22 Global Step: 228490 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:24:40,759-Speed 128.15 samples/sec Loss 0.3695 Epoch: 22 Global Step: 228495 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:24:59,941-Speed 133.47 samples/sec Loss 0.3495 Epoch: 22 Global Step: 228500 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:25:19,637-Speed 129.99 samples/sec Loss 0.3075 Epoch: 22 Global Step: 228505 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:25:39,515-Speed 128.79 samples/sec Loss 0.3241 Epoch: 22 Global Step: 228510 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:25:59,169-Speed 130.27 samples/sec Loss 0.3172 Epoch: 22 Global Step: 228515 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:26:19,114-Speed 128.36 samples/sec Loss 0.3721 Epoch: 22 Global Step: 228520 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:26:39,425-Speed 126.05 samples/sec Loss 0.3068 Epoch: 22 Global Step: 228525 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:26:59,141-Speed 129.84 samples/sec Loss 0.3353 Epoch: 22 Global Step: 228530 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:27:18,792-Speed 130.28 samples/sec Loss 0.3321 Epoch: 22 Global Step: 228535 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:27:38,725-Speed 128.44 samples/sec Loss 0.3313 Epoch: 22 Global Step: 228540 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:27:58,455-Speed 129.75 samples/sec Loss 0.3636 Epoch: 22 Global Step: 228545 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:28:18,351-Speed 128.68 samples/sec Loss 0.3721 Epoch: 22 Global Step: 228550 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:28:38,063-Speed 129.88 samples/sec Loss 0.2693 Epoch: 22 Global Step: 228555 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:28:57,925-Speed 128.89 samples/sec Loss 0.2911 Epoch: 22 Global Step: 228560 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:29:17,697-Speed 129.48 samples/sec Loss 0.3731 Epoch: 22 Global Step: 228565 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:29:38,116-Speed 125.38 samples/sec Loss 0.3290 Epoch: 22 Global Step: 228570 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:29:57,791-Speed 130.12 samples/sec Loss 0.2698 Epoch: 22 Global Step: 228575 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:30:17,627-Speed 129.06 samples/sec Loss 0.3107 Epoch: 22 Global Step: 228580 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:30:37,390-Speed 129.54 samples/sec Loss 0.3374 Epoch: 22 Global Step: 228585 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:30:57,116-Speed 129.79 samples/sec Loss 0.3230 Epoch: 22 Global Step: 228590 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:31:17,025-Speed 128.59 samples/sec Loss 0.3229 Epoch: 22 Global Step: 228595 Fp16 Grad Scale: 4096 Required: 25 hours Training: 2021-06-18 04:31:36,664-Speed 130.36 samples/sec Loss 0.3929 Epoch: 22 Global Step: 228600 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:31:56,539-Speed 128.81 samples/sec Loss 0.3218 Epoch: 22 Global Step: 228605 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:32:16,237-Speed 129.97 samples/sec Loss 0.3871 Epoch: 22 Global Step: 228610 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:32:36,615-Speed 125.63 samples/sec Loss 0.3828 Epoch: 22 Global Step: 228615 Fp16 Grad Scale: 8192 Required: 25 hours Training: 2021-06-18 04:32:56,300-Speed 130.06 samples/sec Loss 0.3456 Epoch: 22 Global Step: 228620 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:33:16,151-Speed 128.97 samples/sec Loss 0.2686 Epoch: 22 Global Step: 228625 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:33:35,889-Speed 129.71 samples/sec Loss 0.3294 Epoch: 22 Global Step: 228630 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:33:55,627-Speed 129.70 samples/sec Loss 0.3263 Epoch: 22 Global Step: 228635 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:34:15,541-Speed 128.56 samples/sec Loss 0.3840 Epoch: 22 Global Step: 228640 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:34:35,188-Speed 130.31 samples/sec Loss 0.3156 Epoch: 22 Global Step: 228645 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:34:55,074-Speed 128.74 samples/sec Loss 0.2921 Epoch: 22 Global Step: 228650 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:35:14,765-Speed 130.02 samples/sec Loss 0.3240 Epoch: 22 Global Step: 228655 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:35:35,185-Speed 125.37 samples/sec Loss 0.3666 Epoch: 22 Global Step: 228660 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:35:54,909-Speed 129.79 samples/sec Loss 0.3224 Epoch: 22 Global Step: 228665 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:36:14,773-Speed 128.88 samples/sec Loss 0.3152 Epoch: 22 Global Step: 228670 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:36:34,555-Speed 129.42 samples/sec Loss 0.3212 Epoch: 22 Global Step: 228675 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:36:54,211-Speed 130.24 samples/sec Loss 0.3761 Epoch: 22 Global Step: 228680 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:37:14,108-Speed 128.67 samples/sec Loss 0.3525 Epoch: 22 Global Step: 228685 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:37:33,806-Speed 129.97 samples/sec Loss 0.3204 Epoch: 22 Global Step: 228690 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:37:53,773-Speed 128.21 samples/sec Loss 0.3238 Epoch: 22 Global Step: 228695 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:38:13,445-Speed 130.14 samples/sec Loss 0.3873 Epoch: 22 Global Step: 228700 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:38:33,740-Speed 126.14 samples/sec Loss 0.3285 Epoch: 22 Global Step: 228705 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:38:53,646-Speed 128.61 samples/sec Loss 0.3149 Epoch: 22 Global Step: 228710 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:39:13,621-Speed 128.16 samples/sec Loss 0.3458 Epoch: 22 Global Step: 228715 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:39:33,231-Speed 130.55 samples/sec Loss 0.3233 Epoch: 22 Global Step: 228720 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:39:52,941-Speed 129.89 samples/sec Loss 0.3255 Epoch: 22 Global Step: 228725 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:40:13,528-Speed 124.36 samples/sec Loss 0.2948 Epoch: 22 Global Step: 228730 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:40:33,268-Speed 129.69 samples/sec Loss 0.3152 Epoch: 22 Global Step: 228735 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:40:53,237-Speed 128.21 samples/sec Loss 0.3250 Epoch: 22 Global Step: 228740 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:41:12,855-Speed 130.49 samples/sec Loss 0.3373 Epoch: 22 Global Step: 228745 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:41:33,226-Speed 125.68 samples/sec Loss 0.3216 Epoch: 22 Global Step: 228750 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:41:52,953-Speed 129.78 samples/sec Loss 0.2863 Epoch: 22 Global Step: 228755 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:42:12,955-Speed 127.99 samples/sec Loss 0.3156 Epoch: 22 Global Step: 228760 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:42:32,561-Speed 130.58 samples/sec Loss 0.3098 Epoch: 22 Global Step: 228765 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:42:52,296-Speed 129.73 samples/sec Loss 0.3346 Epoch: 22 Global Step: 228770 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:43:12,129-Speed 129.09 samples/sec Loss 0.4045 Epoch: 22 Global Step: 228775 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:43:31,873-Speed 129.67 samples/sec Loss 0.3201 Epoch: 22 Global Step: 228780 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:43:51,773-Speed 128.65 samples/sec Loss 0.3346 Epoch: 22 Global Step: 228785 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:44:11,381-Speed 130.56 samples/sec Loss 0.2588 Epoch: 22 Global Step: 228790 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:44:31,745-Speed 125.72 samples/sec Loss 0.3448 Epoch: 22 Global Step: 228795 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:44:50,957-Speed 133.26 samples/sec Loss 0.3463 Epoch: 22 Global Step: 228800 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:45:11,075-Speed 127.26 samples/sec Loss 0.3268 Epoch: 22 Global Step: 228805 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:45:30,711-Speed 130.38 samples/sec Loss 0.4206 Epoch: 22 Global Step: 228810 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:45:50,606-Speed 128.68 samples/sec Loss 0.3541 Epoch: 22 Global Step: 228815 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:46:10,316-Speed 129.89 samples/sec Loss 0.2866 Epoch: 22 Global Step: 228820 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:46:29,844-Speed 131.10 samples/sec Loss 0.3508 Epoch: 22 Global Step: 228825 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:46:49,685-Speed 129.03 samples/sec Loss 0.4572 Epoch: 22 Global Step: 228830 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:47:09,294-Speed 130.56 samples/sec Loss 0.3737 Epoch: 22 Global Step: 228835 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:47:28,916-Speed 130.48 samples/sec Loss 0.3523 Epoch: 22 Global Step: 228840 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:47:48,617-Speed 129.95 samples/sec Loss 0.3834 Epoch: 22 Global Step: 228845 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:48:08,506-Speed 128.72 samples/sec Loss 0.3342 Epoch: 22 Global Step: 228850 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:48:28,268-Speed 129.55 samples/sec Loss 0.3443 Epoch: 22 Global Step: 228855 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:48:47,799-Speed 131.08 samples/sec Loss 0.3295 Epoch: 22 Global Step: 228860 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:49:07,623-Speed 129.14 samples/sec Loss 0.2999 Epoch: 22 Global Step: 228865 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:49:27,270-Speed 130.31 samples/sec Loss 0.3573 Epoch: 22 Global Step: 228870 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:49:46,875-Speed 130.58 samples/sec Loss 0.3217 Epoch: 22 Global Step: 228875 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:50:06,590-Speed 129.86 samples/sec Loss 0.3447 Epoch: 22 Global Step: 228880 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:50:26,216-Speed 130.44 samples/sec Loss 0.3122 Epoch: 22 Global Step: 228885 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:50:45,963-Speed 129.65 samples/sec Loss 0.3300 Epoch: 22 Global Step: 228890 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:51:05,745-Speed 129.42 samples/sec Loss 0.3168 Epoch: 22 Global Step: 228895 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 04:51:25,550-Speed 129.26 samples/sec Loss 0.3907 Epoch: 22 Global Step: 228900 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:51:45,270-Speed 129.82 samples/sec Loss 0.3945 Epoch: 22 Global Step: 228905 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:52:05,043-Speed 129.47 samples/sec Loss 0.3953 Epoch: 22 Global Step: 228910 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:52:24,815-Speed 129.49 samples/sec Loss 0.3204 Epoch: 22 Global Step: 228915 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:52:44,401-Speed 130.71 samples/sec Loss 0.3705 Epoch: 22 Global Step: 228920 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:53:04,124-Speed 129.80 samples/sec Loss 0.3248 Epoch: 22 Global Step: 228925 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:53:23,844-Speed 129.83 samples/sec Loss 0.3877 Epoch: 22 Global Step: 228930 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:53:43,464-Speed 130.49 samples/sec Loss 0.3259 Epoch: 22 Global Step: 228935 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:54:03,327-Speed 128.89 samples/sec Loss 0.4155 Epoch: 22 Global Step: 228940 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:54:23,123-Speed 129.32 samples/sec Loss 0.3771 Epoch: 22 Global Step: 228945 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:54:42,963-Speed 129.04 samples/sec Loss 0.3574 Epoch: 22 Global Step: 228950 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:55:02,715-Speed 129.61 samples/sec Loss 0.3847 Epoch: 22 Global Step: 228955 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:55:22,466-Speed 129.62 samples/sec Loss 0.2600 Epoch: 22 Global Step: 228960 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:55:42,054-Speed 130.69 samples/sec Loss 0.3621 Epoch: 22 Global Step: 228965 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:56:01,822-Speed 129.51 samples/sec Loss 0.3725 Epoch: 22 Global Step: 228970 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:56:21,414-Speed 130.68 samples/sec Loss 0.3611 Epoch: 22 Global Step: 228975 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:56:41,083-Speed 130.16 samples/sec Loss 0.3117 Epoch: 22 Global Step: 228980 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:57:00,885-Speed 129.29 samples/sec Loss 0.3268 Epoch: 22 Global Step: 228985 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:57:20,657-Speed 129.48 samples/sec Loss 0.3734 Epoch: 22 Global Step: 228990 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:57:40,308-Speed 130.28 samples/sec Loss 0.3869 Epoch: 22 Global Step: 228995 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:58:00,073-Speed 129.53 samples/sec Loss 0.3857 Epoch: 22 Global Step: 229000 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:58:20,033-Speed 128.26 samples/sec Loss 0.4139 Epoch: 22 Global Step: 229005 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:58:39,555-Speed 131.14 samples/sec Loss 0.3503 Epoch: 22 Global Step: 229010 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:58:59,337-Speed 129.42 samples/sec Loss 0.3428 Epoch: 22 Global Step: 229015 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:59:18,918-Speed 130.74 samples/sec Loss 0.2808 Epoch: 22 Global Step: 229020 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:59:38,852-Speed 128.43 samples/sec Loss 0.2724 Epoch: 22 Global Step: 229025 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 04:59:58,738-Speed 128.74 samples/sec Loss 0.2815 Epoch: 22 Global Step: 229030 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:00:18,545-Speed 129.25 samples/sec Loss 0.3633 Epoch: 22 Global Step: 229035 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:00:38,063-Speed 131.17 samples/sec Loss 0.3136 Epoch: 22 Global Step: 229040 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:00:57,910-Speed 128.99 samples/sec Loss 0.3843 Epoch: 22 Global Step: 229045 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:01:17,865-Speed 128.30 samples/sec Loss 0.3847 Epoch: 22 Global Step: 229050 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:01:37,379-Speed 131.19 samples/sec Loss 0.3345 Epoch: 22 Global Step: 229055 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:01:57,209-Speed 129.11 samples/sec Loss 0.3584 Epoch: 22 Global Step: 229060 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:02:16,747-Speed 131.04 samples/sec Loss 0.2859 Epoch: 22 Global Step: 229065 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:02:36,480-Speed 129.74 samples/sec Loss 0.3776 Epoch: 22 Global Step: 229070 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:02:56,270-Speed 129.36 samples/sec Loss 0.3570 Epoch: 22 Global Step: 229075 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:03:16,212-Speed 128.38 samples/sec Loss 0.3255 Epoch: 22 Global Step: 229080 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:03:35,627-Speed 131.87 samples/sec Loss 0.2964 Epoch: 22 Global Step: 229085 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:03:55,368-Speed 129.68 samples/sec Loss 0.2884 Epoch: 22 Global Step: 229090 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:04:15,337-Speed 128.21 samples/sec Loss 0.3683 Epoch: 22 Global Step: 229095 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:04:34,872-Speed 131.06 samples/sec Loss 0.4116 Epoch: 22 Global Step: 229100 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:04:54,815-Speed 128.37 samples/sec Loss 0.3491 Epoch: 22 Global Step: 229105 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:05:14,242-Speed 131.79 samples/sec Loss 0.4067 Epoch: 22 Global Step: 229110 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:05:33,972-Speed 129.76 samples/sec Loss 0.3131 Epoch: 22 Global Step: 229115 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:05:53,820-Speed 128.99 samples/sec Loss 0.3619 Epoch: 22 Global Step: 229120 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:06:13,621-Speed 129.29 samples/sec Loss 0.2510 Epoch: 22 Global Step: 229125 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:06:33,884-Speed 126.34 samples/sec Loss 0.4083 Epoch: 22 Global Step: 229130 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:06:53,610-Speed 129.78 samples/sec Loss 0.3673 Epoch: 22 Global Step: 229135 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:07:13,607-Speed 128.03 samples/sec Loss 0.3300 Epoch: 22 Global Step: 229140 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:07:33,112-Speed 131.25 samples/sec Loss 0.3242 Epoch: 22 Global Step: 229145 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:07:52,981-Speed 128.85 samples/sec Loss 0.3240 Epoch: 22 Global Step: 229150 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:08:12,407-Speed 131.79 samples/sec Loss 0.3562 Epoch: 22 Global Step: 229155 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:08:32,127-Speed 129.82 samples/sec Loss 0.2953 Epoch: 22 Global Step: 229160 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:08:51,907-Speed 129.43 samples/sec Loss 0.3252 Epoch: 22 Global Step: 229165 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:09:11,687-Speed 129.43 samples/sec Loss 0.3714 Epoch: 22 Global Step: 229170 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:09:54,104-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-18 05:10:13,223-Speed 41.60 samples/sec Loss 0.3542 Epoch: 23 Global Step: 229175 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:10:33,617-Speed 125.53 samples/sec Loss 0.3176 Epoch: 23 Global Step: 229180 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:10:53,243-Speed 130.44 samples/sec Loss 0.3052 Epoch: 23 Global Step: 229185 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:11:13,109-Speed 128.87 samples/sec Loss 0.3792 Epoch: 23 Global Step: 229190 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:11:33,094-Speed 128.10 samples/sec Loss 0.3314 Epoch: 23 Global Step: 229195 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:11:53,024-Speed 128.46 samples/sec Loss 0.3483 Epoch: 23 Global Step: 229200 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:12:12,936-Speed 128.57 samples/sec Loss 0.3105 Epoch: 23 Global Step: 229205 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:12:32,742-Speed 129.26 samples/sec Loss 0.4017 Epoch: 23 Global Step: 229210 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:12:52,697-Speed 128.29 samples/sec Loss 0.3166 Epoch: 23 Global Step: 229215 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:13:12,560-Speed 128.89 samples/sec Loss 0.2519 Epoch: 23 Global Step: 229220 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:13:32,734-Speed 126.90 samples/sec Loss 0.2731 Epoch: 23 Global Step: 229225 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:13:52,718-Speed 128.12 samples/sec Loss 0.2775 Epoch: 23 Global Step: 229230 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:14:12,855-Speed 127.14 samples/sec Loss 0.2885 Epoch: 23 Global Step: 229235 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:14:32,720-Speed 128.88 samples/sec Loss 0.3038 Epoch: 23 Global Step: 229240 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:14:52,663-Speed 128.37 samples/sec Loss 0.3621 Epoch: 23 Global Step: 229245 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:15:12,586-Speed 128.51 samples/sec Loss 0.3380 Epoch: 23 Global Step: 229250 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:15:32,419-Speed 129.08 samples/sec Loss 0.2750 Epoch: 23 Global Step: 229255 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:15:52,338-Speed 128.53 samples/sec Loss 0.3060 Epoch: 23 Global Step: 229260 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:16:12,221-Speed 128.76 samples/sec Loss 0.2744 Epoch: 23 Global Step: 229265 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:16:32,573-Speed 125.79 samples/sec Loss 0.2602 Epoch: 23 Global Step: 229270 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:16:52,375-Speed 129.29 samples/sec Loss 0.3813 Epoch: 23 Global Step: 229275 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:17:12,520-Speed 127.08 samples/sec Loss 0.2984 Epoch: 23 Global Step: 229280 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:17:31,929-Speed 131.91 samples/sec Loss 0.3298 Epoch: 23 Global Step: 229285 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:17:52,169-Speed 126.49 samples/sec Loss 0.2561 Epoch: 23 Global Step: 229290 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:18:11,930-Speed 129.56 samples/sec Loss 0.3472 Epoch: 23 Global Step: 229295 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:18:31,803-Speed 128.82 samples/sec Loss 0.2960 Epoch: 23 Global Step: 229300 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:18:51,655-Speed 128.96 samples/sec Loss 0.3280 Epoch: 23 Global Step: 229305 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:19:11,538-Speed 128.76 samples/sec Loss 0.3694 Epoch: 23 Global Step: 229310 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:19:31,859-Speed 125.99 samples/sec Loss 0.3024 Epoch: 23 Global Step: 229315 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:19:51,647-Speed 129.37 samples/sec Loss 0.2728 Epoch: 23 Global Step: 229320 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:20:11,794-Speed 127.08 samples/sec Loss 0.2089 Epoch: 23 Global Step: 229325 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:20:31,691-Speed 128.67 samples/sec Loss 0.3085 Epoch: 23 Global Step: 229330 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:20:51,724-Speed 127.79 samples/sec Loss 0.3026 Epoch: 23 Global Step: 229335 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:21:11,493-Speed 129.50 samples/sec Loss 0.2885 Epoch: 23 Global Step: 229340 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:21:31,304-Speed 129.23 samples/sec Loss 0.3204 Epoch: 23 Global Step: 229345 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:21:51,262-Speed 128.28 samples/sec Loss 0.3559 Epoch: 23 Global Step: 229350 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:22:11,061-Speed 129.31 samples/sec Loss 0.2753 Epoch: 23 Global Step: 229355 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:22:31,327-Speed 126.33 samples/sec Loss 0.3378 Epoch: 23 Global Step: 229360 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:22:51,047-Speed 129.83 samples/sec Loss 0.3635 Epoch: 23 Global Step: 229365 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:23:11,112-Speed 127.59 samples/sec Loss 0.2617 Epoch: 23 Global Step: 229370 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:23:31,023-Speed 128.58 samples/sec Loss 0.2897 Epoch: 23 Global Step: 229375 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:23:50,976-Speed 128.31 samples/sec Loss 0.3217 Epoch: 23 Global Step: 229380 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:24:10,778-Speed 129.28 samples/sec Loss 0.3293 Epoch: 23 Global Step: 229385 Fp16 Grad Scale: 8192 Required: 24 hours Training: 2021-06-18 05:24:30,333-Speed 130.92 samples/sec Loss 0.3567 Epoch: 23 Global Step: 229390 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:24:50,300-Speed 128.22 samples/sec Loss 0.2970 Epoch: 23 Global Step: 229395 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:25:10,122-Speed 129.16 samples/sec Loss 0.2775 Epoch: 23 Global Step: 229400 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:25:30,354-Speed 126.54 samples/sec Loss 0.2521 Epoch: 23 Global Step: 229405 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:25:50,140-Speed 129.39 samples/sec Loss 0.2846 Epoch: 23 Global Step: 229410 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:26:10,234-Speed 127.40 samples/sec Loss 0.3858 Epoch: 23 Global Step: 229415 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:26:30,036-Speed 129.29 samples/sec Loss 0.2739 Epoch: 23 Global Step: 229420 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:26:49,984-Speed 128.34 samples/sec Loss 0.3180 Epoch: 23 Global Step: 229425 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:27:09,751-Speed 129.52 samples/sec Loss 0.3306 Epoch: 23 Global Step: 229430 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:27:29,631-Speed 128.78 samples/sec Loss 0.3588 Epoch: 23 Global Step: 229435 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:27:49,405-Speed 129.47 samples/sec Loss 0.3100 Epoch: 23 Global Step: 229440 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:28:09,224-Speed 129.18 samples/sec Loss 0.3776 Epoch: 23 Global Step: 229445 Fp16 Grad Scale: 4096 Required: 24 hours Training: 2021-06-18 05:28:29,459-Speed 126.52 samples/sec Loss 0.3282 Epoch: 23 Global Step: 229450 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:28:49,124-Speed 130.19 samples/sec Loss 0.2575 Epoch: 23 Global Step: 229455 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:29:09,264-Speed 127.12 samples/sec Loss 0.3671 Epoch: 23 Global Step: 229460 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:29:29,131-Speed 128.86 samples/sec Loss 0.3068 Epoch: 23 Global Step: 229465 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:29:49,070-Speed 128.40 samples/sec Loss 0.2521 Epoch: 23 Global Step: 229470 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:30:08,823-Speed 129.60 samples/sec Loss 0.3432 Epoch: 23 Global Step: 229475 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:30:28,661-Speed 129.06 samples/sec Loss 0.3090 Epoch: 23 Global Step: 229480 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:30:48,355-Speed 130.00 samples/sec Loss 0.2894 Epoch: 23 Global Step: 229485 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:31:08,140-Speed 129.40 samples/sec Loss 0.3430 Epoch: 23 Global Step: 229490 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:31:28,359-Speed 126.61 samples/sec Loss 0.2980 Epoch: 23 Global Step: 229495 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:31:47,975-Speed 130.52 samples/sec Loss 0.2975 Epoch: 23 Global Step: 229500 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:32:08,053-Speed 127.51 samples/sec Loss 0.2868 Epoch: 23 Global Step: 229505 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:32:27,955-Speed 128.63 samples/sec Loss 0.3494 Epoch: 23 Global Step: 229510 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:32:47,635-Speed 130.09 samples/sec Loss 0.2958 Epoch: 23 Global Step: 229515 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:33:06,944-Speed 132.59 samples/sec Loss 0.3164 Epoch: 23 Global Step: 229520 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:33:26,705-Speed 129.56 samples/sec Loss 0.3129 Epoch: 23 Global Step: 229525 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:33:46,413-Speed 129.90 samples/sec Loss 0.2994 Epoch: 23 Global Step: 229530 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:34:06,392-Speed 128.14 samples/sec Loss 0.2713 Epoch: 23 Global Step: 229535 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:34:26,099-Speed 129.91 samples/sec Loss 0.4263 Epoch: 23 Global Step: 229540 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:34:45,852-Speed 129.61 samples/sec Loss 0.3763 Epoch: 23 Global Step: 229545 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:35:05,621-Speed 129.50 samples/sec Loss 0.3216 Epoch: 23 Global Step: 229550 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:35:25,302-Speed 130.08 samples/sec Loss 0.3094 Epoch: 23 Global Step: 229555 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:35:45,406-Speed 127.34 samples/sec Loss 0.2816 Epoch: 23 Global Step: 229560 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:36:05,924-Speed 124.78 samples/sec Loss 0.3059 Epoch: 23 Global Step: 229565 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:36:25,417-Speed 131.34 samples/sec Loss 0.3701 Epoch: 23 Global Step: 229570 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:36:45,148-Speed 129.75 samples/sec Loss 0.2885 Epoch: 23 Global Step: 229575 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:37:04,793-Speed 130.31 samples/sec Loss 0.3424 Epoch: 23 Global Step: 229580 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:37:24,593-Speed 129.30 samples/sec Loss 0.2817 Epoch: 23 Global Step: 229585 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:37:44,101-Speed 131.24 samples/sec Loss 0.3950 Epoch: 23 Global Step: 229590 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:38:03,867-Speed 129.52 samples/sec Loss 0.2858 Epoch: 23 Global Step: 229595 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:38:23,418-Speed 130.95 samples/sec Loss 0.3514 Epoch: 23 Global Step: 229600 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:38:43,504-Speed 127.46 samples/sec Loss 0.3187 Epoch: 23 Global Step: 229605 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:39:03,320-Speed 129.19 samples/sec Loss 0.2750 Epoch: 23 Global Step: 229610 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:39:22,891-Speed 130.81 samples/sec Loss 0.2003 Epoch: 23 Global Step: 229615 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:39:42,680-Speed 129.38 samples/sec Loss 0.2931 Epoch: 23 Global Step: 229620 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:40:02,244-Speed 130.86 samples/sec Loss 0.3039 Epoch: 23 Global Step: 229625 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:40:21,900-Speed 130.25 samples/sec Loss 0.2723 Epoch: 23 Global Step: 229630 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:40:41,522-Speed 130.47 samples/sec Loss 0.2893 Epoch: 23 Global Step: 229635 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:41:01,294-Speed 129.48 samples/sec Loss 0.3454 Epoch: 23 Global Step: 229640 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:41:20,872-Speed 130.77 samples/sec Loss 0.3010 Epoch: 23 Global Step: 229645 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:41:40,920-Speed 127.70 samples/sec Loss 0.2877 Epoch: 23 Global Step: 229650 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:42:01,022-Speed 127.36 samples/sec Loss 0.2519 Epoch: 23 Global Step: 229655 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:42:20,453-Speed 131.75 samples/sec Loss 0.3364 Epoch: 23 Global Step: 229660 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:42:40,212-Speed 129.57 samples/sec Loss 0.3705 Epoch: 23 Global Step: 229665 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:42:59,780-Speed 130.84 samples/sec Loss 0.3355 Epoch: 23 Global Step: 229670 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:43:19,512-Speed 129.75 samples/sec Loss 0.3471 Epoch: 23 Global Step: 229675 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:43:39,116-Speed 130.59 samples/sec Loss 0.3530 Epoch: 23 Global Step: 229680 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:43:58,367-Speed 132.99 samples/sec Loss 0.2843 Epoch: 23 Global Step: 229685 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:44:17,911-Speed 130.99 samples/sec Loss 0.3077 Epoch: 23 Global Step: 229690 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:44:37,964-Speed 127.67 samples/sec Loss 0.3456 Epoch: 23 Global Step: 229695 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:44:57,999-Speed 127.79 samples/sec Loss 0.2915 Epoch: 23 Global Step: 229700 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:45:17,612-Speed 130.53 samples/sec Loss 0.3731 Epoch: 23 Global Step: 229705 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:45:37,402-Speed 129.37 samples/sec Loss 0.2916 Epoch: 23 Global Step: 229710 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:45:56,946-Speed 130.99 samples/sec Loss 0.2682 Epoch: 23 Global Step: 229715 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:46:16,687-Speed 129.69 samples/sec Loss 0.3089 Epoch: 23 Global Step: 229720 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:46:36,321-Speed 130.39 samples/sec Loss 0.3493 Epoch: 23 Global Step: 229725 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:46:56,046-Speed 129.79 samples/sec Loss 0.3383 Epoch: 23 Global Step: 229730 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:47:15,644-Speed 130.63 samples/sec Loss 0.3288 Epoch: 23 Global Step: 229735 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:47:35,696-Speed 127.67 samples/sec Loss 0.2794 Epoch: 23 Global Step: 229740 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:47:55,767-Speed 127.56 samples/sec Loss 0.2531 Epoch: 23 Global Step: 229745 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:48:15,290-Speed 131.13 samples/sec Loss 0.2918 Epoch: 23 Global Step: 229750 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:48:35,094-Speed 129.27 samples/sec Loss 0.3714 Epoch: 23 Global Step: 229755 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:48:54,619-Speed 131.12 samples/sec Loss 0.3244 Epoch: 23 Global Step: 229760 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:49:14,294-Speed 130.12 samples/sec Loss 0.2807 Epoch: 23 Global Step: 229765 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:49:33,895-Speed 130.62 samples/sec Loss 0.2757 Epoch: 23 Global Step: 229770 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:49:53,607-Speed 129.88 samples/sec Loss 0.3134 Epoch: 23 Global Step: 229775 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:50:13,093-Speed 131.38 samples/sec Loss 0.3139 Epoch: 23 Global Step: 229780 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:50:33,125-Speed 127.81 samples/sec Loss 0.2860 Epoch: 23 Global Step: 229785 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:50:53,099-Speed 128.17 samples/sec Loss 0.3679 Epoch: 23 Global Step: 229790 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:51:12,620-Speed 131.14 samples/sec Loss 0.2707 Epoch: 23 Global Step: 229795 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:51:32,280-Speed 130.22 samples/sec Loss 0.3218 Epoch: 23 Global Step: 229800 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:51:51,845-Speed 130.85 samples/sec Loss 0.2667 Epoch: 23 Global Step: 229805 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:52:11,489-Speed 130.32 samples/sec Loss 0.3875 Epoch: 23 Global Step: 229810 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:52:31,094-Speed 130.59 samples/sec Loss 0.3303 Epoch: 23 Global Step: 229815 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:52:50,687-Speed 130.66 samples/sec Loss 0.3345 Epoch: 23 Global Step: 229820 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:53:10,206-Speed 131.17 samples/sec Loss 0.3039 Epoch: 23 Global Step: 229825 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:53:30,247-Speed 127.74 samples/sec Loss 0.2744 Epoch: 23 Global Step: 229830 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:53:50,315-Speed 127.57 samples/sec Loss 0.2490 Epoch: 23 Global Step: 229835 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:54:09,796-Speed 131.42 samples/sec Loss 0.2986 Epoch: 23 Global Step: 229840 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:54:29,513-Speed 129.84 samples/sec Loss 0.3502 Epoch: 23 Global Step: 229845 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:54:49,130-Speed 130.51 samples/sec Loss 0.3385 Epoch: 23 Global Step: 229850 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:55:08,760-Speed 130.42 samples/sec Loss 0.3282 Epoch: 23 Global Step: 229855 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:55:28,303-Speed 131.00 samples/sec Loss 0.3628 Epoch: 23 Global Step: 229860 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:55:48,160-Speed 128.93 samples/sec Loss 0.3216 Epoch: 23 Global Step: 229865 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:56:08,003-Speed 129.02 samples/sec Loss 0.3017 Epoch: 23 Global Step: 229870 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:56:28,088-Speed 127.46 samples/sec Loss 0.3428 Epoch: 23 Global Step: 229875 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:56:48,064-Speed 128.16 samples/sec Loss 0.3250 Epoch: 23 Global Step: 229880 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:57:07,602-Speed 131.03 samples/sec Loss 0.2672 Epoch: 23 Global Step: 229885 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:57:27,342-Speed 129.70 samples/sec Loss 0.3578 Epoch: 23 Global Step: 229890 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 05:57:46,481-Speed 133.77 samples/sec Loss 0.3433 Epoch: 23 Global Step: 229895 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:58:06,122-Speed 130.34 samples/sec Loss 0.3131 Epoch: 23 Global Step: 229900 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:58:25,736-Speed 130.53 samples/sec Loss 0.3444 Epoch: 23 Global Step: 229905 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:58:45,534-Speed 129.31 samples/sec Loss 0.3074 Epoch: 23 Global Step: 229910 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:59:05,070-Speed 131.05 samples/sec Loss 0.2782 Epoch: 23 Global Step: 229915 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:59:25,111-Speed 127.74 samples/sec Loss 0.3069 Epoch: 23 Global Step: 229920 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 05:59:45,029-Speed 128.53 samples/sec Loss 0.3178 Epoch: 23 Global Step: 229925 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:00:04,578-Speed 130.96 samples/sec Loss 0.3160 Epoch: 23 Global Step: 229930 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:00:24,372-Speed 129.34 samples/sec Loss 0.3119 Epoch: 23 Global Step: 229935 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:00:43,896-Speed 131.13 samples/sec Loss 0.3368 Epoch: 23 Global Step: 229940 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:01:03,595-Speed 129.96 samples/sec Loss 0.2926 Epoch: 23 Global Step: 229945 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:01:23,179-Speed 130.73 samples/sec Loss 0.2414 Epoch: 23 Global Step: 229950 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:01:42,862-Speed 130.07 samples/sec Loss 0.3043 Epoch: 23 Global Step: 229955 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:02:02,447-Speed 130.72 samples/sec Loss 0.2507 Epoch: 23 Global Step: 229960 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:02:22,490-Speed 127.73 samples/sec Loss 0.3443 Epoch: 23 Global Step: 229965 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:02:42,530-Speed 127.75 samples/sec Loss 0.2757 Epoch: 23 Global Step: 229970 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:03:02,119-Speed 130.69 samples/sec Loss 0.3088 Epoch: 23 Global Step: 229975 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:03:21,861-Speed 129.68 samples/sec Loss 0.2922 Epoch: 23 Global Step: 229980 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:03:41,441-Speed 130.76 samples/sec Loss 0.3824 Epoch: 23 Global Step: 229985 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:04:01,170-Speed 129.76 samples/sec Loss 0.3563 Epoch: 23 Global Step: 229990 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:04:20,778-Speed 130.57 samples/sec Loss 0.2984 Epoch: 23 Global Step: 229995 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 06:04:40,567-Speed 129.37 samples/sec Loss 0.3288 Epoch: 23 Global Step: 230000 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 06:19:27,637-[lfw][230000]XNorm: 21.579447 Training: 2021-06-18 06:19:27,638-[lfw][230000]Accuracy-Flip: 0.99850+-0.00203 Training: 2021-06-18 06:19:27,638-[lfw][230000]Accuracy-Highest: 0.99867 Training: 2021-06-18 06:36:45,152-[cfp_fp][230000]XNorm: 23.333086 Training: 2021-06-18 06:36:45,152-[cfp_fp][230000]Accuracy-Flip: 0.99186+-0.00279 Training: 2021-06-18 06:36:45,152-[cfp_fp][230000]Accuracy-Highest: 0.99229 Training: 2021-06-18 06:51:36,719-[agedb_30][230000]XNorm: 22.590505 Training: 2021-06-18 06:51:36,719-[agedb_30][230000]Accuracy-Flip: 0.98650+-0.00560 Training: 2021-06-18 06:51:36,720-[agedb_30][230000]Accuracy-Highest: 0.98667 Training: 2021-06-18 06:51:56,376-Speed 0.90 samples/sec Loss 0.2903 Epoch: 23 Global Step: 230005 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 06:52:15,989-Speed 130.53 samples/sec Loss 0.3379 Epoch: 23 Global Step: 230010 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 06:52:36,143-Speed 127.03 samples/sec Loss 0.3418 Epoch: 23 Global Step: 230015 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 06:52:55,624-Speed 131.42 samples/sec Loss 0.3017 Epoch: 23 Global Step: 230020 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 06:53:15,702-Speed 127.51 samples/sec Loss 0.2979 Epoch: 23 Global Step: 230025 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 06:53:35,326-Speed 130.46 samples/sec Loss 0.2945 Epoch: 23 Global Step: 230030 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 06:53:55,330-Speed 127.98 samples/sec Loss 0.3945 Epoch: 23 Global Step: 230035 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 06:54:14,995-Speed 130.19 samples/sec Loss 0.2692 Epoch: 23 Global Step: 230040 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 06:54:34,314-Speed 132.52 samples/sec Loss 0.3135 Epoch: 23 Global Step: 230045 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:54:54,070-Speed 129.59 samples/sec Loss 0.3161 Epoch: 23 Global Step: 230050 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:55:13,725-Speed 130.25 samples/sec Loss 0.3708 Epoch: 23 Global Step: 230055 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:55:33,551-Speed 129.13 samples/sec Loss 0.3361 Epoch: 23 Global Step: 230060 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:55:52,975-Speed 131.80 samples/sec Loss 0.3064 Epoch: 23 Global Step: 230065 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:56:12,954-Speed 128.14 samples/sec Loss 0.2872 Epoch: 23 Global Step: 230070 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:56:32,387-Speed 131.74 samples/sec Loss 0.2989 Epoch: 23 Global Step: 230075 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:56:52,074-Speed 130.05 samples/sec Loss 0.3077 Epoch: 23 Global Step: 230080 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:57:11,923-Speed 128.98 samples/sec Loss 0.2963 Epoch: 23 Global Step: 230085 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:57:31,669-Speed 129.66 samples/sec Loss 0.2714 Epoch: 23 Global Step: 230090 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:57:51,298-Speed 130.42 samples/sec Loss 0.3027 Epoch: 23 Global Step: 230095 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:58:11,017-Speed 129.83 samples/sec Loss 0.3200 Epoch: 23 Global Step: 230100 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:58:30,655-Speed 130.37 samples/sec Loss 0.3017 Epoch: 23 Global Step: 230105 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:58:50,293-Speed 130.37 samples/sec Loss 0.2480 Epoch: 23 Global Step: 230110 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:59:10,161-Speed 128.86 samples/sec Loss 0.3053 Epoch: 23 Global Step: 230115 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:59:29,847-Speed 130.04 samples/sec Loss 0.3091 Epoch: 23 Global Step: 230120 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 06:59:49,584-Speed 129.72 samples/sec Loss 0.2796 Epoch: 23 Global Step: 230125 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:00:09,223-Speed 130.35 samples/sec Loss 0.3023 Epoch: 23 Global Step: 230130 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:00:29,008-Speed 129.40 samples/sec Loss 0.3299 Epoch: 23 Global Step: 230135 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:00:48,618-Speed 130.56 samples/sec Loss 0.3244 Epoch: 23 Global Step: 230140 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:01:07,871-Speed 132.98 samples/sec Loss 0.2620 Epoch: 23 Global Step: 230145 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:01:27,469-Speed 130.63 samples/sec Loss 0.2780 Epoch: 23 Global Step: 230150 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:01:47,182-Speed 129.88 samples/sec Loss 0.3500 Epoch: 23 Global Step: 230155 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:02:06,969-Speed 129.38 samples/sec Loss 0.3144 Epoch: 23 Global Step: 230160 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:02:26,742-Speed 129.48 samples/sec Loss 0.3337 Epoch: 23 Global Step: 230165 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:02:46,484-Speed 129.68 samples/sec Loss 0.3247 Epoch: 23 Global Step: 230170 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:03:06,155-Speed 130.14 samples/sec Loss 0.3935 Epoch: 23 Global Step: 230175 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:03:25,913-Speed 129.58 samples/sec Loss 0.3604 Epoch: 23 Global Step: 230180 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:03:45,483-Speed 130.82 samples/sec Loss 0.3355 Epoch: 23 Global Step: 230185 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:04:05,373-Speed 128.71 samples/sec Loss 0.3636 Epoch: 23 Global Step: 230190 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:04:24,978-Speed 130.59 samples/sec Loss 0.3106 Epoch: 23 Global Step: 230195 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:04:44,640-Speed 130.21 samples/sec Loss 0.2673 Epoch: 23 Global Step: 230200 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:05:04,452-Speed 129.22 samples/sec Loss 0.2868 Epoch: 23 Global Step: 230205 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:05:24,232-Speed 129.42 samples/sec Loss 0.2807 Epoch: 23 Global Step: 230210 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:05:43,953-Speed 129.82 samples/sec Loss 0.2624 Epoch: 23 Global Step: 230215 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:06:03,697-Speed 129.66 samples/sec Loss 0.3129 Epoch: 23 Global Step: 230220 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:06:23,681-Speed 128.11 samples/sec Loss 0.3817 Epoch: 23 Global Step: 230225 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:06:43,234-Speed 130.94 samples/sec Loss 0.3116 Epoch: 23 Global Step: 230230 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:07:03,071-Speed 129.06 samples/sec Loss 0.3658 Epoch: 23 Global Step: 230235 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:07:22,634-Speed 130.87 samples/sec Loss 0.3017 Epoch: 23 Global Step: 230240 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:07:42,259-Speed 130.45 samples/sec Loss 0.2871 Epoch: 23 Global Step: 230245 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 07:08:02,036-Speed 129.45 samples/sec Loss 0.3202 Epoch: 23 Global Step: 230250 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 07:08:21,869-Speed 129.08 samples/sec Loss 0.2954 Epoch: 23 Global Step: 230255 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 07:08:41,317-Speed 131.64 samples/sec Loss 0.3679 Epoch: 23 Global Step: 230260 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 07:09:01,218-Speed 128.64 samples/sec Loss 0.3418 Epoch: 23 Global Step: 230265 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 07:09:21,114-Speed 128.68 samples/sec Loss 0.3296 Epoch: 23 Global Step: 230270 Fp16 Grad Scale: 8192 Required: 23 hours Training: 2021-06-18 07:09:40,191-Speed 134.20 samples/sec Loss 0.2977 Epoch: 23 Global Step: 230275 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:09:59,788-Speed 130.64 samples/sec Loss 0.2988 Epoch: 23 Global Step: 230280 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:10:19,508-Speed 129.83 samples/sec Loss 0.2754 Epoch: 23 Global Step: 230285 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:10:39,117-Speed 130.56 samples/sec Loss 0.2890 Epoch: 23 Global Step: 230290 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:10:58,840-Speed 129.81 samples/sec Loss 0.3268 Epoch: 23 Global Step: 230295 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:11:18,616-Speed 129.46 samples/sec Loss 0.3226 Epoch: 23 Global Step: 230300 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:11:38,420-Speed 129.27 samples/sec Loss 0.2824 Epoch: 23 Global Step: 230305 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:11:58,330-Speed 128.58 samples/sec Loss 0.3160 Epoch: 23 Global Step: 230310 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:12:18,180-Speed 128.97 samples/sec Loss 0.3303 Epoch: 23 Global Step: 230315 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:12:38,112-Speed 128.44 samples/sec Loss 0.2937 Epoch: 23 Global Step: 230320 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:12:58,020-Speed 128.60 samples/sec Loss 0.3295 Epoch: 23 Global Step: 230325 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:13:17,692-Speed 130.14 samples/sec Loss 0.2661 Epoch: 23 Global Step: 230330 Fp16 Grad Scale: 4096 Required: 23 hours Training: 2021-06-18 07:13:37,513-Speed 129.17 samples/sec Loss 0.3229 Epoch: 23 Global Step: 230335 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:13:57,143-Speed 130.42 samples/sec Loss 0.2984 Epoch: 23 Global Step: 230340 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:14:17,052-Speed 128.60 samples/sec Loss 0.3353 Epoch: 23 Global Step: 230345 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:14:36,784-Speed 129.74 samples/sec Loss 0.3156 Epoch: 23 Global Step: 230350 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:14:56,787-Speed 127.98 samples/sec Loss 0.3138 Epoch: 23 Global Step: 230355 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:15:16,571-Speed 129.41 samples/sec Loss 0.3065 Epoch: 23 Global Step: 230360 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:15:37,026-Speed 125.15 samples/sec Loss 0.3457 Epoch: 23 Global Step: 230365 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:15:56,763-Speed 129.71 samples/sec Loss 0.3088 Epoch: 23 Global Step: 230370 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:16:16,533-Speed 129.50 samples/sec Loss 0.2716 Epoch: 23 Global Step: 230375 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:16:36,396-Speed 128.89 samples/sec Loss 0.2931 Epoch: 23 Global Step: 230380 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:16:56,007-Speed 130.55 samples/sec Loss 0.3394 Epoch: 23 Global Step: 230385 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:17:15,871-Speed 128.88 samples/sec Loss 0.3379 Epoch: 23 Global Step: 230390 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:17:35,594-Speed 129.80 samples/sec Loss 0.2952 Epoch: 23 Global Step: 230395 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:17:55,607-Speed 127.92 samples/sec Loss 0.2900 Epoch: 23 Global Step: 230400 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:18:15,347-Speed 129.69 samples/sec Loss 0.3265 Epoch: 23 Global Step: 230405 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:18:35,424-Speed 127.52 samples/sec Loss 0.2567 Epoch: 23 Global Step: 230410 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:18:55,145-Speed 129.82 samples/sec Loss 0.2769 Epoch: 23 Global Step: 230415 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:19:14,970-Speed 129.14 samples/sec Loss 0.3082 Epoch: 23 Global Step: 230420 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:19:34,787-Speed 129.19 samples/sec Loss 0.3370 Epoch: 23 Global Step: 230425 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:19:54,425-Speed 130.36 samples/sec Loss 0.2727 Epoch: 23 Global Step: 230430 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:20:14,317-Speed 128.71 samples/sec Loss 0.2926 Epoch: 23 Global Step: 230435 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:20:34,053-Speed 129.71 samples/sec Loss 0.2581 Epoch: 23 Global Step: 230440 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:20:54,192-Speed 127.13 samples/sec Loss 0.2621 Epoch: 23 Global Step: 230445 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:21:13,818-Speed 130.44 samples/sec Loss 0.2633 Epoch: 23 Global Step: 230450 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:21:33,887-Speed 127.57 samples/sec Loss 0.3429 Epoch: 23 Global Step: 230455 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:21:53,606-Speed 129.83 samples/sec Loss 0.3871 Epoch: 23 Global Step: 230460 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:22:13,585-Speed 128.14 samples/sec Loss 0.2816 Epoch: 23 Global Step: 230465 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:22:33,299-Speed 129.87 samples/sec Loss 0.2964 Epoch: 23 Global Step: 230470 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:22:52,950-Speed 130.28 samples/sec Loss 0.3002 Epoch: 23 Global Step: 230475 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:23:12,813-Speed 128.89 samples/sec Loss 0.2586 Epoch: 23 Global Step: 230480 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:23:32,502-Speed 130.03 samples/sec Loss 0.2967 Epoch: 23 Global Step: 230485 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:23:52,768-Speed 126.33 samples/sec Loss 0.2638 Epoch: 23 Global Step: 230490 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:24:12,391-Speed 130.47 samples/sec Loss 0.3177 Epoch: 23 Global Step: 230495 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:24:32,479-Speed 127.44 samples/sec Loss 0.3237 Epoch: 23 Global Step: 230500 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:24:52,228-Speed 129.63 samples/sec Loss 0.3022 Epoch: 23 Global Step: 230505 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:25:12,162-Speed 128.43 samples/sec Loss 0.2459 Epoch: 23 Global Step: 230510 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:25:31,860-Speed 129.97 samples/sec Loss 0.2736 Epoch: 23 Global Step: 230515 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:25:51,645-Speed 129.40 samples/sec Loss 0.3769 Epoch: 23 Global Step: 230520 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:26:11,365-Speed 129.82 samples/sec Loss 0.2150 Epoch: 23 Global Step: 230525 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:26:31,142-Speed 129.45 samples/sec Loss 0.2212 Epoch: 23 Global Step: 230530 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:26:51,367-Speed 126.59 samples/sec Loss 0.3303 Epoch: 23 Global Step: 230535 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:27:10,981-Speed 130.52 samples/sec Loss 0.3410 Epoch: 23 Global Step: 230540 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:27:31,043-Speed 127.61 samples/sec Loss 0.2877 Epoch: 23 Global Step: 230545 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:27:50,804-Speed 129.55 samples/sec Loss 0.3240 Epoch: 23 Global Step: 230550 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:28:10,686-Speed 128.76 samples/sec Loss 0.3203 Epoch: 23 Global Step: 230555 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:28:30,527-Speed 129.04 samples/sec Loss 0.3412 Epoch: 23 Global Step: 230560 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:28:50,322-Speed 129.34 samples/sec Loss 0.3971 Epoch: 23 Global Step: 230565 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:29:10,031-Speed 129.89 samples/sec Loss 0.3324 Epoch: 23 Global Step: 230570 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:29:29,802-Speed 129.49 samples/sec Loss 0.3257 Epoch: 23 Global Step: 230575 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:29:50,028-Speed 126.58 samples/sec Loss 0.3448 Epoch: 23 Global Step: 230580 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:30:09,654-Speed 130.45 samples/sec Loss 0.3685 Epoch: 23 Global Step: 230585 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:30:29,672-Speed 127.89 samples/sec Loss 0.2574 Epoch: 23 Global Step: 230590 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:30:49,506-Speed 129.08 samples/sec Loss 0.2878 Epoch: 23 Global Step: 230595 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:31:09,356-Speed 128.97 samples/sec Loss 0.3203 Epoch: 23 Global Step: 230600 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:31:29,128-Speed 129.49 samples/sec Loss 0.2851 Epoch: 23 Global Step: 230605 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:31:48,998-Speed 128.84 samples/sec Loss 0.2841 Epoch: 23 Global Step: 230610 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:32:08,676-Speed 130.10 samples/sec Loss 0.2631 Epoch: 23 Global Step: 230615 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:32:28,590-Speed 128.55 samples/sec Loss 0.2735 Epoch: 23 Global Step: 230620 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:32:48,680-Speed 127.44 samples/sec Loss 0.3008 Epoch: 23 Global Step: 230625 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:33:08,352-Speed 130.14 samples/sec Loss 0.3453 Epoch: 23 Global Step: 230630 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:33:28,781-Speed 125.32 samples/sec Loss 0.2977 Epoch: 23 Global Step: 230635 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:33:48,684-Speed 128.63 samples/sec Loss 0.3054 Epoch: 23 Global Step: 230640 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:34:08,317-Speed 130.39 samples/sec Loss 0.2895 Epoch: 23 Global Step: 230645 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:34:28,124-Speed 129.25 samples/sec Loss 0.3063 Epoch: 23 Global Step: 230650 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:34:47,880-Speed 129.59 samples/sec Loss 0.3138 Epoch: 23 Global Step: 230655 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:35:07,600-Speed 129.83 samples/sec Loss 0.3528 Epoch: 23 Global Step: 230660 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:35:27,505-Speed 128.62 samples/sec Loss 0.3077 Epoch: 23 Global Step: 230665 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:35:47,571-Speed 127.58 samples/sec Loss 0.3585 Epoch: 23 Global Step: 230670 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:36:07,242-Speed 130.15 samples/sec Loss 0.2539 Epoch: 23 Global Step: 230675 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:36:26,777-Speed 131.06 samples/sec Loss 0.3268 Epoch: 23 Global Step: 230680 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:36:46,699-Speed 128.50 samples/sec Loss 0.2777 Epoch: 23 Global Step: 230685 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:37:06,420-Speed 129.82 samples/sec Loss 0.3361 Epoch: 23 Global Step: 230690 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:37:26,184-Speed 129.54 samples/sec Loss 0.2882 Epoch: 23 Global Step: 230695 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:37:45,750-Speed 130.84 samples/sec Loss 0.3522 Epoch: 23 Global Step: 230700 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:38:05,474-Speed 129.80 samples/sec Loss 0.3073 Epoch: 23 Global Step: 230705 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:38:25,038-Speed 130.86 samples/sec Loss 0.4091 Epoch: 23 Global Step: 230710 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:38:44,613-Speed 130.79 samples/sec Loss 0.2459 Epoch: 23 Global Step: 230715 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:39:04,296-Speed 130.06 samples/sec Loss 0.3048 Epoch: 23 Global Step: 230720 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:39:24,119-Speed 129.15 samples/sec Loss 0.3154 Epoch: 23 Global Step: 230725 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:39:44,006-Speed 128.73 samples/sec Loss 0.3160 Epoch: 23 Global Step: 230730 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:40:03,565-Speed 130.90 samples/sec Loss 0.3467 Epoch: 23 Global Step: 230735 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:40:23,336-Speed 129.48 samples/sec Loss 0.3603 Epoch: 23 Global Step: 230740 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:40:43,290-Speed 128.30 samples/sec Loss 0.3484 Epoch: 23 Global Step: 230745 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:41:02,731-Speed 131.69 samples/sec Loss 0.3069 Epoch: 23 Global Step: 230750 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:41:22,454-Speed 129.81 samples/sec Loss 0.3089 Epoch: 23 Global Step: 230755 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:41:42,151-Speed 129.98 samples/sec Loss 0.2802 Epoch: 23 Global Step: 230760 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:42:01,910-Speed 129.57 samples/sec Loss 0.2842 Epoch: 23 Global Step: 230765 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:42:21,691-Speed 129.43 samples/sec Loss 0.3584 Epoch: 23 Global Step: 230770 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:42:41,583-Speed 128.70 samples/sec Loss 0.2852 Epoch: 23 Global Step: 230775 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:43:01,036-Speed 131.60 samples/sec Loss 0.3133 Epoch: 23 Global Step: 230780 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:43:20,945-Speed 128.59 samples/sec Loss 0.3484 Epoch: 23 Global Step: 230785 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:43:40,691-Speed 129.65 samples/sec Loss 0.3874 Epoch: 23 Global Step: 230790 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:44:00,253-Speed 130.87 samples/sec Loss 0.3304 Epoch: 23 Global Step: 230795 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:44:20,037-Speed 129.40 samples/sec Loss 0.3480 Epoch: 23 Global Step: 230800 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:44:39,572-Speed 131.06 samples/sec Loss 0.2926 Epoch: 23 Global Step: 230805 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:44:59,449-Speed 128.80 samples/sec Loss 0.3184 Epoch: 23 Global Step: 230810 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:45:19,205-Speed 129.59 samples/sec Loss 0.2884 Epoch: 23 Global Step: 230815 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:45:39,132-Speed 128.48 samples/sec Loss 0.2824 Epoch: 23 Global Step: 230820 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:45:58,548-Speed 131.85 samples/sec Loss 0.2792 Epoch: 23 Global Step: 230825 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:46:19,350-Speed 123.07 samples/sec Loss 0.2761 Epoch: 23 Global Step: 230830 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:46:39,221-Speed 128.84 samples/sec Loss 0.2600 Epoch: 23 Global Step: 230835 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:46:59,042-Speed 129.16 samples/sec Loss 0.2996 Epoch: 23 Global Step: 230840 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:47:18,833-Speed 129.36 samples/sec Loss 0.2809 Epoch: 23 Global Step: 230845 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:47:38,383-Speed 130.96 samples/sec Loss 0.2790 Epoch: 23 Global Step: 230850 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:47:58,196-Speed 129.22 samples/sec Loss 0.3091 Epoch: 23 Global Step: 230855 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:48:17,963-Speed 129.51 samples/sec Loss 0.2775 Epoch: 23 Global Step: 230860 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:48:37,896-Speed 128.44 samples/sec Loss 0.2710 Epoch: 23 Global Step: 230865 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:48:57,311-Speed 131.86 samples/sec Loss 0.3098 Epoch: 23 Global Step: 230870 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:49:17,157-Speed 129.00 samples/sec Loss 0.2365 Epoch: 23 Global Step: 230875 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:49:37,018-Speed 128.90 samples/sec Loss 0.2777 Epoch: 23 Global Step: 230880 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:49:56,603-Speed 130.72 samples/sec Loss 0.3956 Epoch: 23 Global Step: 230885 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:50:16,407-Speed 129.27 samples/sec Loss 0.3866 Epoch: 23 Global Step: 230890 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:50:35,925-Speed 131.16 samples/sec Loss 0.4060 Epoch: 23 Global Step: 230895 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:50:55,695-Speed 129.50 samples/sec Loss 0.2815 Epoch: 23 Global Step: 230900 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:51:15,469-Speed 129.47 samples/sec Loss 0.3448 Epoch: 23 Global Step: 230905 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:51:35,386-Speed 128.54 samples/sec Loss 0.3011 Epoch: 23 Global Step: 230910 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:51:54,807-Speed 131.82 samples/sec Loss 0.2826 Epoch: 23 Global Step: 230915 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:52:14,617-Speed 129.24 samples/sec Loss 0.2584 Epoch: 23 Global Step: 230920 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:52:34,447-Speed 129.11 samples/sec Loss 0.3163 Epoch: 23 Global Step: 230925 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:52:53,956-Speed 131.23 samples/sec Loss 0.3782 Epoch: 23 Global Step: 230930 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:53:13,876-Speed 128.52 samples/sec Loss 0.3091 Epoch: 23 Global Step: 230935 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:53:33,343-Speed 131.51 samples/sec Loss 0.2944 Epoch: 23 Global Step: 230940 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 07:53:52,692-Speed 132.32 samples/sec Loss 0.3917 Epoch: 23 Global Step: 230945 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:54:12,446-Speed 129.60 samples/sec Loss 0.3010 Epoch: 23 Global Step: 230950 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:54:32,420-Speed 128.18 samples/sec Loss 0.2988 Epoch: 23 Global Step: 230955 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:54:51,915-Speed 131.32 samples/sec Loss 0.3013 Epoch: 23 Global Step: 230960 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:55:11,691-Speed 129.46 samples/sec Loss 0.2422 Epoch: 23 Global Step: 230965 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:55:31,590-Speed 128.66 samples/sec Loss 0.2706 Epoch: 23 Global Step: 230970 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:55:51,109-Speed 131.16 samples/sec Loss 0.3274 Epoch: 23 Global Step: 230975 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:56:11,017-Speed 128.60 samples/sec Loss 0.2715 Epoch: 23 Global Step: 230980 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:56:30,537-Speed 131.15 samples/sec Loss 0.3247 Epoch: 23 Global Step: 230985 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:56:50,366-Speed 129.11 samples/sec Loss 0.2821 Epoch: 23 Global Step: 230990 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:57:10,097-Speed 129.75 samples/sec Loss 0.2504 Epoch: 23 Global Step: 230995 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:57:30,029-Speed 128.44 samples/sec Loss 0.2370 Epoch: 23 Global Step: 231000 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:57:49,440-Speed 131.89 samples/sec Loss 0.3289 Epoch: 23 Global Step: 231005 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:58:09,335-Speed 128.68 samples/sec Loss 0.3176 Epoch: 23 Global Step: 231010 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:58:28,978-Speed 130.34 samples/sec Loss 0.2822 Epoch: 23 Global Step: 231015 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:58:48,571-Speed 130.66 samples/sec Loss 0.3232 Epoch: 23 Global Step: 231020 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:59:08,528-Speed 128.28 samples/sec Loss 0.2811 Epoch: 23 Global Step: 231025 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:59:28,074-Speed 130.98 samples/sec Loss 0.3075 Epoch: 23 Global Step: 231030 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 07:59:47,977-Speed 128.63 samples/sec Loss 0.3084 Epoch: 23 Global Step: 231035 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:00:07,741-Speed 129.54 samples/sec Loss 0.3961 Epoch: 23 Global Step: 231040 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:00:27,800-Speed 127.62 samples/sec Loss 0.3404 Epoch: 23 Global Step: 231045 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 08:00:47,236-Speed 131.72 samples/sec Loss 0.3095 Epoch: 23 Global Step: 231050 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 08:01:07,165-Speed 128.46 samples/sec Loss 0.3068 Epoch: 23 Global Step: 231055 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 08:01:26,906-Speed 129.68 samples/sec Loss 0.2785 Epoch: 23 Global Step: 231060 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 08:01:46,383-Speed 131.44 samples/sec Loss 0.3399 Epoch: 23 Global Step: 231065 Fp16 Grad Scale: 8192 Required: 22 hours Training: 2021-06-18 08:02:05,906-Speed 131.14 samples/sec Loss 0.2907 Epoch: 23 Global Step: 231070 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:02:25,630-Speed 129.80 samples/sec Loss 0.3131 Epoch: 23 Global Step: 231075 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:02:45,229-Speed 130.62 samples/sec Loss 0.3385 Epoch: 23 Global Step: 231080 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:03:04,937-Speed 129.91 samples/sec Loss 0.3559 Epoch: 23 Global Step: 231085 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:03:24,788-Speed 128.97 samples/sec Loss 0.2055 Epoch: 23 Global Step: 231090 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:03:44,501-Speed 129.87 samples/sec Loss 0.3056 Epoch: 23 Global Step: 231095 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:04:04,520-Speed 127.88 samples/sec Loss 0.3165 Epoch: 23 Global Step: 231100 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:04:24,388-Speed 128.86 samples/sec Loss 0.2866 Epoch: 23 Global Step: 231105 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:04:44,395-Speed 127.96 samples/sec Loss 0.2792 Epoch: 23 Global Step: 231110 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:05:04,135-Speed 129.69 samples/sec Loss 0.4001 Epoch: 23 Global Step: 231115 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:05:23,931-Speed 129.33 samples/sec Loss 0.3655 Epoch: 23 Global Step: 231120 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:05:43,806-Speed 128.81 samples/sec Loss 0.3404 Epoch: 23 Global Step: 231125 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:06:03,523-Speed 129.84 samples/sec Loss 0.2845 Epoch: 23 Global Step: 231130 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:06:23,350-Speed 129.12 samples/sec Loss 0.3170 Epoch: 23 Global Step: 231135 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:06:43,100-Speed 129.62 samples/sec Loss 0.3392 Epoch: 23 Global Step: 231140 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:07:03,226-Speed 127.20 samples/sec Loss 0.3061 Epoch: 23 Global Step: 231145 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:07:22,960-Speed 129.73 samples/sec Loss 0.3274 Epoch: 23 Global Step: 231150 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:07:42,934-Speed 128.17 samples/sec Loss 0.3430 Epoch: 23 Global Step: 231155 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:08:02,720-Speed 129.39 samples/sec Loss 0.2866 Epoch: 23 Global Step: 231160 Fp16 Grad Scale: 4096 Required: 22 hours Training: 2021-06-18 08:08:22,643-Speed 128.50 samples/sec Loss 0.3215 Epoch: 23 Global Step: 231165 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:08:42,362-Speed 129.83 samples/sec Loss 0.2893 Epoch: 23 Global Step: 231170 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:09:01,685-Speed 132.49 samples/sec Loss 0.3098 Epoch: 23 Global Step: 231175 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:09:21,510-Speed 129.14 samples/sec Loss 0.3516 Epoch: 23 Global Step: 231180 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:09:41,219-Speed 129.89 samples/sec Loss 0.4159 Epoch: 23 Global Step: 231185 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:10:01,090-Speed 128.84 samples/sec Loss 0.3429 Epoch: 23 Global Step: 231190 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:10:20,797-Speed 129.90 samples/sec Loss 0.3377 Epoch: 23 Global Step: 231195 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:10:41,312-Speed 124.80 samples/sec Loss 0.3205 Epoch: 23 Global Step: 231200 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:11:00,993-Speed 130.08 samples/sec Loss 0.3119 Epoch: 23 Global Step: 231205 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:11:20,776-Speed 129.41 samples/sec Loss 0.3224 Epoch: 23 Global Step: 231210 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:11:40,490-Speed 129.87 samples/sec Loss 0.3369 Epoch: 23 Global Step: 231215 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:12:00,303-Speed 129.21 samples/sec Loss 0.3329 Epoch: 23 Global Step: 231220 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:12:20,130-Speed 129.13 samples/sec Loss 0.3607 Epoch: 23 Global Step: 231225 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:12:39,862-Speed 129.75 samples/sec Loss 0.2476 Epoch: 23 Global Step: 231230 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:12:59,673-Speed 129.23 samples/sec Loss 0.3051 Epoch: 23 Global Step: 231235 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:13:19,327-Speed 130.26 samples/sec Loss 0.2768 Epoch: 23 Global Step: 231240 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:13:39,537-Speed 126.67 samples/sec Loss 0.3692 Epoch: 23 Global Step: 231245 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:13:59,340-Speed 129.28 samples/sec Loss 0.3477 Epoch: 23 Global Step: 231250 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:14:19,131-Speed 129.36 samples/sec Loss 0.2833 Epoch: 23 Global Step: 231255 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:14:38,798-Speed 130.18 samples/sec Loss 0.3369 Epoch: 23 Global Step: 231260 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:14:58,636-Speed 129.05 samples/sec Loss 0.2460 Epoch: 23 Global Step: 231265 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:15:18,302-Speed 130.18 samples/sec Loss 0.3259 Epoch: 23 Global Step: 231270 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:15:38,121-Speed 129.18 samples/sec Loss 0.2935 Epoch: 23 Global Step: 231275 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:15:57,990-Speed 128.85 samples/sec Loss 0.2786 Epoch: 23 Global Step: 231280 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:16:17,608-Speed 130.50 samples/sec Loss 0.3178 Epoch: 23 Global Step: 231285 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:16:37,966-Speed 125.75 samples/sec Loss 0.2782 Epoch: 23 Global Step: 231290 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:16:57,776-Speed 129.23 samples/sec Loss 0.3100 Epoch: 23 Global Step: 231295 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:17:17,620-Speed 129.02 samples/sec Loss 0.3466 Epoch: 23 Global Step: 231300 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:17:37,220-Speed 130.62 samples/sec Loss 0.3134 Epoch: 23 Global Step: 231305 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:17:57,057-Speed 129.06 samples/sec Loss 0.3822 Epoch: 23 Global Step: 231310 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:18:16,808-Speed 129.62 samples/sec Loss 0.2732 Epoch: 23 Global Step: 231315 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:18:36,609-Speed 129.29 samples/sec Loss 0.2954 Epoch: 23 Global Step: 231320 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:18:56,467-Speed 128.92 samples/sec Loss 0.3724 Epoch: 23 Global Step: 231325 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:19:16,088-Speed 130.48 samples/sec Loss 0.2883 Epoch: 23 Global Step: 231330 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:19:36,435-Speed 125.82 samples/sec Loss 0.2640 Epoch: 23 Global Step: 231335 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:19:56,410-Speed 128.17 samples/sec Loss 0.2787 Epoch: 23 Global Step: 231340 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:20:16,119-Speed 129.89 samples/sec Loss 0.3155 Epoch: 23 Global Step: 231345 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:20:35,402-Speed 132.76 samples/sec Loss 0.3065 Epoch: 23 Global Step: 231350 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:20:55,192-Speed 129.36 samples/sec Loss 0.3029 Epoch: 23 Global Step: 231355 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:21:14,928-Speed 129.72 samples/sec Loss 0.2860 Epoch: 23 Global Step: 231360 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:21:34,894-Speed 128.23 samples/sec Loss 0.3267 Epoch: 23 Global Step: 231365 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:21:54,578-Speed 130.06 samples/sec Loss 0.3690 Epoch: 23 Global Step: 231370 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:22:14,285-Speed 129.91 samples/sec Loss 0.3257 Epoch: 23 Global Step: 231375 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:22:34,596-Speed 126.05 samples/sec Loss 0.3301 Epoch: 23 Global Step: 231380 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:22:54,573-Speed 128.15 samples/sec Loss 0.3176 Epoch: 23 Global Step: 231385 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:23:14,367-Speed 129.34 samples/sec Loss 0.2850 Epoch: 23 Global Step: 231390 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:23:34,259-Speed 128.70 samples/sec Loss 0.3155 Epoch: 23 Global Step: 231395 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:23:53,933-Speed 130.13 samples/sec Loss 0.2842 Epoch: 23 Global Step: 231400 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:24:13,694-Speed 129.56 samples/sec Loss 0.3879 Epoch: 23 Global Step: 231405 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:24:33,548-Speed 128.95 samples/sec Loss 0.2933 Epoch: 23 Global Step: 231410 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:24:53,265-Speed 129.85 samples/sec Loss 0.3115 Epoch: 23 Global Step: 231415 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:25:13,127-Speed 128.89 samples/sec Loss 0.3508 Epoch: 23 Global Step: 231420 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:25:33,311-Speed 126.84 samples/sec Loss 0.3505 Epoch: 23 Global Step: 231425 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:25:53,182-Speed 128.83 samples/sec Loss 0.2970 Epoch: 23 Global Step: 231430 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:26:12,821-Speed 130.36 samples/sec Loss 0.2385 Epoch: 23 Global Step: 231435 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:26:32,727-Speed 128.61 samples/sec Loss 0.3009 Epoch: 23 Global Step: 231440 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:26:52,383-Speed 130.25 samples/sec Loss 0.2576 Epoch: 23 Global Step: 231445 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:27:12,374-Speed 128.07 samples/sec Loss 0.2907 Epoch: 23 Global Step: 231450 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:27:32,067-Speed 130.00 samples/sec Loss 0.2653 Epoch: 23 Global Step: 231455 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:27:51,740-Speed 130.14 samples/sec Loss 0.2824 Epoch: 23 Global Step: 231460 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:28:11,570-Speed 129.10 samples/sec Loss 0.3417 Epoch: 23 Global Step: 231465 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:28:32,029-Speed 125.13 samples/sec Loss 0.3440 Epoch: 23 Global Step: 231470 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:28:51,731-Speed 129.94 samples/sec Loss 0.2561 Epoch: 23 Global Step: 231475 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:29:11,394-Speed 130.20 samples/sec Loss 0.2531 Epoch: 23 Global Step: 231480 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:29:31,214-Speed 129.17 samples/sec Loss 0.3087 Epoch: 23 Global Step: 231485 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:29:50,947-Speed 129.74 samples/sec Loss 0.2900 Epoch: 23 Global Step: 231490 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:30:10,880-Speed 128.44 samples/sec Loss 0.3237 Epoch: 23 Global Step: 231495 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:30:30,613-Speed 129.74 samples/sec Loss 0.2775 Epoch: 23 Global Step: 231500 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:30:50,262-Speed 130.29 samples/sec Loss 0.2706 Epoch: 23 Global Step: 231505 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:31:10,124-Speed 128.90 samples/sec Loss 0.3449 Epoch: 23 Global Step: 231510 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:31:30,582-Speed 125.14 samples/sec Loss 0.3484 Epoch: 23 Global Step: 231515 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:31:50,291-Speed 129.89 samples/sec Loss 0.2414 Epoch: 23 Global Step: 231520 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:32:09,996-Speed 129.92 samples/sec Loss 0.2977 Epoch: 23 Global Step: 231525 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:32:29,802-Speed 129.27 samples/sec Loss 0.2625 Epoch: 23 Global Step: 231530 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:32:49,515-Speed 129.87 samples/sec Loss 0.3845 Epoch: 23 Global Step: 231535 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:33:09,400-Speed 128.75 samples/sec Loss 0.3317 Epoch: 23 Global Step: 231540 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:33:29,107-Speed 129.91 samples/sec Loss 0.3256 Epoch: 23 Global Step: 231545 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:33:48,899-Speed 129.35 samples/sec Loss 0.2971 Epoch: 23 Global Step: 231550 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:34:08,629-Speed 129.76 samples/sec Loss 0.2461 Epoch: 23 Global Step: 231555 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:34:29,076-Speed 125.20 samples/sec Loss 0.2628 Epoch: 23 Global Step: 231560 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:34:48,790-Speed 129.87 samples/sec Loss 0.3150 Epoch: 23 Global Step: 231565 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:35:08,606-Speed 129.19 samples/sec Loss 0.3031 Epoch: 23 Global Step: 231570 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:35:28,436-Speed 129.10 samples/sec Loss 0.2956 Epoch: 23 Global Step: 231575 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:35:48,413-Speed 128.15 samples/sec Loss 0.3032 Epoch: 23 Global Step: 231580 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:36:08,326-Speed 128.57 samples/sec Loss 0.2871 Epoch: 23 Global Step: 231585 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:36:28,070-Speed 129.66 samples/sec Loss 0.3514 Epoch: 23 Global Step: 231590 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:36:47,968-Speed 128.67 samples/sec Loss 0.3389 Epoch: 23 Global Step: 231595 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:37:07,627-Speed 130.22 samples/sec Loss 0.3025 Epoch: 23 Global Step: 231600 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:37:28,062-Speed 125.28 samples/sec Loss 0.3095 Epoch: 23 Global Step: 231605 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:37:47,762-Speed 129.96 samples/sec Loss 0.2634 Epoch: 23 Global Step: 231610 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:38:08,116-Speed 125.78 samples/sec Loss 0.3429 Epoch: 23 Global Step: 231615 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:38:27,840-Speed 129.80 samples/sec Loss 0.2864 Epoch: 23 Global Step: 231620 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:38:47,562-Speed 129.81 samples/sec Loss 0.3344 Epoch: 23 Global Step: 231625 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:39:07,478-Speed 128.55 samples/sec Loss 0.3041 Epoch: 23 Global Step: 231630 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:39:27,203-Speed 129.79 samples/sec Loss 0.2832 Epoch: 23 Global Step: 231635 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:39:47,024-Speed 129.16 samples/sec Loss 0.3455 Epoch: 23 Global Step: 231640 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:40:06,756-Speed 129.74 samples/sec Loss 0.2660 Epoch: 23 Global Step: 231645 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:40:27,075-Speed 126.00 samples/sec Loss 0.2711 Epoch: 23 Global Step: 231650 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:40:46,891-Speed 129.19 samples/sec Loss 0.2492 Epoch: 23 Global Step: 231655 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:41:06,660-Speed 129.50 samples/sec Loss 0.2893 Epoch: 23 Global Step: 231660 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:41:26,515-Speed 128.94 samples/sec Loss 0.2692 Epoch: 23 Global Step: 231665 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:41:46,428-Speed 128.56 samples/sec Loss 0.2934 Epoch: 23 Global Step: 231670 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:42:06,148-Speed 129.82 samples/sec Loss 0.3100 Epoch: 23 Global Step: 231675 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:42:25,905-Speed 129.59 samples/sec Loss 0.3262 Epoch: 23 Global Step: 231680 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:42:45,674-Speed 129.50 samples/sec Loss 0.3270 Epoch: 23 Global Step: 231685 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:43:05,392-Speed 129.83 samples/sec Loss 0.3965 Epoch: 23 Global Step: 231690 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:43:25,931-Speed 124.65 samples/sec Loss 0.3634 Epoch: 23 Global Step: 231695 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:43:45,762-Speed 129.09 samples/sec Loss 0.3304 Epoch: 23 Global Step: 231700 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:44:05,546-Speed 129.40 samples/sec Loss 0.2813 Epoch: 23 Global Step: 231705 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:44:25,268-Speed 129.81 samples/sec Loss 0.2833 Epoch: 23 Global Step: 231710 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:44:45,142-Speed 128.82 samples/sec Loss 0.3100 Epoch: 23 Global Step: 231715 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:45:04,860-Speed 129.83 samples/sec Loss 0.3084 Epoch: 23 Global Step: 231720 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:45:24,232-Speed 132.16 samples/sec Loss 0.2780 Epoch: 23 Global Step: 231725 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:45:44,005-Speed 129.47 samples/sec Loss 0.2551 Epoch: 23 Global Step: 231730 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:46:03,735-Speed 129.76 samples/sec Loss 0.2645 Epoch: 23 Global Step: 231735 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:46:23,946-Speed 126.67 samples/sec Loss 0.3473 Epoch: 23 Global Step: 231740 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:46:43,736-Speed 129.36 samples/sec Loss 0.3087 Epoch: 23 Global Step: 231745 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:47:03,495-Speed 129.57 samples/sec Loss 0.3002 Epoch: 23 Global Step: 231750 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:47:23,505-Speed 127.94 samples/sec Loss 0.3343 Epoch: 23 Global Step: 231755 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:47:43,285-Speed 129.44 samples/sec Loss 0.3448 Epoch: 23 Global Step: 231760 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:48:03,348-Speed 127.60 samples/sec Loss 0.2777 Epoch: 23 Global Step: 231765 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:48:23,320-Speed 128.19 samples/sec Loss 0.2787 Epoch: 23 Global Step: 231770 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:48:43,650-Speed 125.93 samples/sec Loss 0.2834 Epoch: 23 Global Step: 231775 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:49:03,544-Speed 128.69 samples/sec Loss 0.2602 Epoch: 23 Global Step: 231780 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:49:23,677-Speed 127.16 samples/sec Loss 0.2935 Epoch: 23 Global Step: 231785 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:49:43,287-Speed 130.55 samples/sec Loss 0.2953 Epoch: 23 Global Step: 231790 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:50:03,250-Speed 128.24 samples/sec Loss 0.2812 Epoch: 23 Global Step: 231795 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:50:22,964-Speed 129.87 samples/sec Loss 0.3373 Epoch: 23 Global Step: 231800 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:50:42,670-Speed 129.92 samples/sec Loss 0.2717 Epoch: 23 Global Step: 231805 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:51:02,702-Speed 127.80 samples/sec Loss 0.2844 Epoch: 23 Global Step: 231810 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:51:22,491-Speed 129.37 samples/sec Loss 0.3150 Epoch: 23 Global Step: 231815 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:51:42,269-Speed 129.44 samples/sec Loss 0.2672 Epoch: 23 Global Step: 231820 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:52:02,045-Speed 129.45 samples/sec Loss 0.2779 Epoch: 23 Global Step: 231825 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:52:22,219-Speed 126.90 samples/sec Loss 0.3602 Epoch: 23 Global Step: 231830 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:52:41,928-Speed 129.90 samples/sec Loss 0.3340 Epoch: 23 Global Step: 231835 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:53:01,901-Speed 128.18 samples/sec Loss 0.3463 Epoch: 23 Global Step: 231840 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:53:21,602-Speed 129.95 samples/sec Loss 0.3746 Epoch: 23 Global Step: 231845 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:53:41,384-Speed 129.41 samples/sec Loss 0.3398 Epoch: 23 Global Step: 231850 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 08:54:00,984-Speed 130.62 samples/sec Loss 0.2998 Epoch: 23 Global Step: 231855 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:54:20,984-Speed 128.00 samples/sec Loss 0.2851 Epoch: 23 Global Step: 231860 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:54:40,953-Speed 128.20 samples/sec Loss 0.2777 Epoch: 23 Global Step: 231865 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:55:00,705-Speed 129.62 samples/sec Loss 0.2958 Epoch: 23 Global Step: 231870 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:55:20,601-Speed 128.68 samples/sec Loss 0.3543 Epoch: 23 Global Step: 231875 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:55:40,326-Speed 129.79 samples/sec Loss 0.2812 Epoch: 23 Global Step: 231880 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:56:00,045-Speed 129.83 samples/sec Loss 0.3107 Epoch: 23 Global Step: 231885 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:56:19,872-Speed 129.12 samples/sec Loss 0.3167 Epoch: 23 Global Step: 231890 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:56:39,565-Speed 130.00 samples/sec Loss 0.3310 Epoch: 23 Global Step: 231895 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:56:59,834-Speed 126.30 samples/sec Loss 0.3121 Epoch: 23 Global Step: 231900 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:57:19,727-Speed 128.69 samples/sec Loss 0.3817 Epoch: 23 Global Step: 231905 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:57:39,522-Speed 129.33 samples/sec Loss 0.3308 Epoch: 23 Global Step: 231910 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:57:59,252-Speed 129.76 samples/sec Loss 0.3110 Epoch: 23 Global Step: 231915 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:58:19,119-Speed 128.86 samples/sec Loss 0.3196 Epoch: 23 Global Step: 231920 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:58:38,805-Speed 130.05 samples/sec Loss 0.2940 Epoch: 23 Global Step: 231925 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:58:58,593-Speed 129.38 samples/sec Loss 0.3343 Epoch: 23 Global Step: 231930 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:59:18,394-Speed 129.30 samples/sec Loss 0.3259 Epoch: 23 Global Step: 231935 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:59:38,119-Speed 129.79 samples/sec Loss 0.2616 Epoch: 23 Global Step: 231940 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 08:59:58,350-Speed 126.54 samples/sec Loss 0.4375 Epoch: 23 Global Step: 231945 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 09:00:18,166-Speed 129.19 samples/sec Loss 0.2876 Epoch: 23 Global Step: 231950 Fp16 Grad Scale: 4096 Required: 21 hours Training: 2021-06-18 09:00:37,964-Speed 129.31 samples/sec Loss 0.3485 Epoch: 23 Global Step: 231955 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 09:00:57,858-Speed 128.69 samples/sec Loss 0.3351 Epoch: 23 Global Step: 231960 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 09:01:17,594-Speed 129.72 samples/sec Loss 0.2935 Epoch: 23 Global Step: 231965 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 09:01:37,239-Speed 130.32 samples/sec Loss 0.3498 Epoch: 23 Global Step: 231970 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 09:01:56,984-Speed 129.66 samples/sec Loss 0.3221 Epoch: 23 Global Step: 231975 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 09:02:16,769-Speed 129.39 samples/sec Loss 0.3436 Epoch: 23 Global Step: 231980 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 09:02:36,585-Speed 129.20 samples/sec Loss 0.2835 Epoch: 23 Global Step: 231985 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 09:02:56,734-Speed 127.06 samples/sec Loss 0.2532 Epoch: 23 Global Step: 231990 Fp16 Grad Scale: 8192 Required: 21 hours Training: 2021-06-18 09:03:16,546-Speed 129.22 samples/sec Loss 0.3176 Epoch: 23 Global Step: 231995 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:03:36,679-Speed 127.16 samples/sec Loss 0.2874 Epoch: 23 Global Step: 232000 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:03:56,549-Speed 128.84 samples/sec Loss 0.3326 Epoch: 23 Global Step: 232005 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:04:16,320-Speed 129.49 samples/sec Loss 0.3257 Epoch: 23 Global Step: 232010 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:04:35,996-Speed 130.11 samples/sec Loss 0.2906 Epoch: 23 Global Step: 232015 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:04:55,801-Speed 129.27 samples/sec Loss 0.2394 Epoch: 23 Global Step: 232020 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:05:15,580-Speed 129.44 samples/sec Loss 0.2913 Epoch: 23 Global Step: 232025 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:05:35,478-Speed 128.66 samples/sec Loss 0.4589 Epoch: 23 Global Step: 232030 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:05:55,619-Speed 127.11 samples/sec Loss 0.2702 Epoch: 23 Global Step: 232035 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:06:15,385-Speed 129.52 samples/sec Loss 0.3349 Epoch: 23 Global Step: 232040 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:06:35,029-Speed 130.33 samples/sec Loss 0.3428 Epoch: 23 Global Step: 232045 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:06:55,013-Speed 128.11 samples/sec Loss 0.3315 Epoch: 23 Global Step: 232050 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:07:14,712-Speed 129.96 samples/sec Loss 0.3143 Epoch: 23 Global Step: 232055 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:07:34,525-Speed 129.22 samples/sec Loss 0.3165 Epoch: 23 Global Step: 232060 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:07:54,158-Speed 130.40 samples/sec Loss 0.2534 Epoch: 23 Global Step: 232065 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:08:13,841-Speed 130.07 samples/sec Loss 0.3167 Epoch: 23 Global Step: 232070 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:08:33,725-Speed 128.75 samples/sec Loss 0.3381 Epoch: 23 Global Step: 232075 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:08:54,198-Speed 125.04 samples/sec Loss 0.3227 Epoch: 23 Global Step: 232080 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:09:13,930-Speed 129.75 samples/sec Loss 0.2603 Epoch: 23 Global Step: 232085 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:09:33,604-Speed 130.12 samples/sec Loss 0.3006 Epoch: 23 Global Step: 232090 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:09:53,560-Speed 128.29 samples/sec Loss 0.3159 Epoch: 23 Global Step: 232095 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:10:13,296-Speed 129.72 samples/sec Loss 0.3088 Epoch: 23 Global Step: 232100 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:10:33,119-Speed 129.15 samples/sec Loss 0.3109 Epoch: 23 Global Step: 232105 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:10:52,832-Speed 129.87 samples/sec Loss 0.3384 Epoch: 23 Global Step: 232110 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:11:12,632-Speed 129.29 samples/sec Loss 0.3725 Epoch: 23 Global Step: 232115 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:11:32,410-Speed 129.45 samples/sec Loss 0.2971 Epoch: 23 Global Step: 232120 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:11:52,752-Speed 125.85 samples/sec Loss 0.2501 Epoch: 23 Global Step: 232125 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:12:12,465-Speed 129.87 samples/sec Loss 0.2869 Epoch: 23 Global Step: 232130 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:12:32,261-Speed 129.32 samples/sec Loss 0.3114 Epoch: 23 Global Step: 232135 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:12:52,054-Speed 129.34 samples/sec Loss 0.2681 Epoch: 23 Global Step: 232140 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:13:11,809-Speed 129.59 samples/sec Loss 0.2881 Epoch: 23 Global Step: 232145 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:13:31,628-Speed 129.18 samples/sec Loss 0.2684 Epoch: 23 Global Step: 232150 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:13:51,362-Speed 129.73 samples/sec Loss 0.2571 Epoch: 23 Global Step: 232155 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:14:11,155-Speed 129.34 samples/sec Loss 0.2797 Epoch: 23 Global Step: 232160 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:14:30,929-Speed 129.47 samples/sec Loss 0.2775 Epoch: 23 Global Step: 232165 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:14:51,308-Speed 125.63 samples/sec Loss 0.3163 Epoch: 23 Global Step: 232170 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:15:11,036-Speed 129.77 samples/sec Loss 0.2721 Epoch: 23 Global Step: 232175 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:15:30,830-Speed 129.33 samples/sec Loss 0.2850 Epoch: 23 Global Step: 232180 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:15:50,618-Speed 129.38 samples/sec Loss 0.3040 Epoch: 23 Global Step: 232185 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:16:10,398-Speed 129.43 samples/sec Loss 0.3461 Epoch: 23 Global Step: 232190 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:16:30,231-Speed 129.08 samples/sec Loss 0.3841 Epoch: 23 Global Step: 232195 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:16:49,847-Speed 130.52 samples/sec Loss 0.3292 Epoch: 23 Global Step: 232200 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:17:09,645-Speed 129.31 samples/sec Loss 0.2787 Epoch: 23 Global Step: 232205 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:17:29,421-Speed 129.46 samples/sec Loss 0.2374 Epoch: 23 Global Step: 232210 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:17:49,789-Speed 125.69 samples/sec Loss 0.3244 Epoch: 23 Global Step: 232215 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:18:09,459-Speed 130.15 samples/sec Loss 0.3974 Epoch: 23 Global Step: 232220 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:18:29,236-Speed 129.45 samples/sec Loss 0.3519 Epoch: 23 Global Step: 232225 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:18:49,052-Speed 129.20 samples/sec Loss 0.3150 Epoch: 23 Global Step: 232230 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:19:08,817-Speed 129.53 samples/sec Loss 0.3294 Epoch: 23 Global Step: 232235 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:19:28,261-Speed 131.67 samples/sec Loss 0.3032 Epoch: 23 Global Step: 232240 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:19:47,890-Speed 130.43 samples/sec Loss 0.2856 Epoch: 23 Global Step: 232245 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:20:07,813-Speed 128.50 samples/sec Loss 0.3239 Epoch: 23 Global Step: 232250 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:20:27,597-Speed 129.40 samples/sec Loss 0.2948 Epoch: 23 Global Step: 232255 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:20:47,979-Speed 125.61 samples/sec Loss 0.2912 Epoch: 23 Global Step: 232260 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:21:07,647-Speed 130.16 samples/sec Loss 0.3213 Epoch: 23 Global Step: 232265 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:21:27,452-Speed 129.27 samples/sec Loss 0.3362 Epoch: 23 Global Step: 232270 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:21:47,230-Speed 129.45 samples/sec Loss 0.3770 Epoch: 23 Global Step: 232275 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:22:06,985-Speed 129.59 samples/sec Loss 0.3008 Epoch: 23 Global Step: 232280 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:22:26,865-Speed 128.78 samples/sec Loss 0.3061 Epoch: 23 Global Step: 232285 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:22:46,497-Speed 130.40 samples/sec Loss 0.3331 Epoch: 23 Global Step: 232290 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:23:06,305-Speed 129.25 samples/sec Loss 0.3757 Epoch: 23 Global Step: 232295 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:23:26,088-Speed 129.41 samples/sec Loss 0.2901 Epoch: 23 Global Step: 232300 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:23:46,404-Speed 126.01 samples/sec Loss 0.3418 Epoch: 23 Global Step: 232305 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:24:06,115-Speed 129.89 samples/sec Loss 0.2836 Epoch: 23 Global Step: 232310 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:24:25,873-Speed 129.57 samples/sec Loss 0.3960 Epoch: 23 Global Step: 232315 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:24:45,658-Speed 129.40 samples/sec Loss 0.3110 Epoch: 23 Global Step: 232320 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:25:05,583-Speed 128.49 samples/sec Loss 0.2571 Epoch: 23 Global Step: 232325 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:25:25,217-Speed 130.39 samples/sec Loss 0.3144 Epoch: 23 Global Step: 232330 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:25:44,898-Speed 130.08 samples/sec Loss 0.3010 Epoch: 23 Global Step: 232335 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:26:04,663-Speed 129.53 samples/sec Loss 0.3095 Epoch: 23 Global Step: 232340 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:26:24,375-Speed 129.87 samples/sec Loss 0.3383 Epoch: 23 Global Step: 232345 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:26:44,768-Speed 125.54 samples/sec Loss 0.3083 Epoch: 23 Global Step: 232350 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:27:03,836-Speed 134.26 samples/sec Loss 0.2986 Epoch: 23 Global Step: 232355 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:27:23,811-Speed 128.17 samples/sec Loss 0.2715 Epoch: 23 Global Step: 232360 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:27:43,598-Speed 129.38 samples/sec Loss 0.3049 Epoch: 23 Global Step: 232365 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:28:03,466-Speed 128.86 samples/sec Loss 0.2970 Epoch: 23 Global Step: 232370 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:28:23,168-Speed 129.94 samples/sec Loss 0.2857 Epoch: 23 Global Step: 232375 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:28:43,035-Speed 128.87 samples/sec Loss 0.3431 Epoch: 23 Global Step: 232380 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:29:02,635-Speed 130.62 samples/sec Loss 0.2913 Epoch: 23 Global Step: 232385 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:29:22,546-Speed 128.58 samples/sec Loss 0.3058 Epoch: 23 Global Step: 232390 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:29:42,587-Speed 127.74 samples/sec Loss 0.3073 Epoch: 23 Global Step: 232395 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:30:02,393-Speed 129.26 samples/sec Loss 0.2958 Epoch: 23 Global Step: 232400 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:30:22,146-Speed 129.61 samples/sec Loss 0.3583 Epoch: 23 Global Step: 232405 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:30:41,874-Speed 129.77 samples/sec Loss 0.3676 Epoch: 23 Global Step: 232410 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:31:01,845-Speed 128.19 samples/sec Loss 0.3031 Epoch: 23 Global Step: 232415 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:31:21,799-Speed 128.30 samples/sec Loss 0.3671 Epoch: 23 Global Step: 232420 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:31:41,701-Speed 128.64 samples/sec Loss 0.2905 Epoch: 23 Global Step: 232425 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:32:01,300-Speed 130.62 samples/sec Loss 0.2708 Epoch: 23 Global Step: 232430 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:32:21,308-Speed 127.96 samples/sec Loss 0.2789 Epoch: 23 Global Step: 232435 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:32:41,325-Speed 127.90 samples/sec Loss 0.2837 Epoch: 23 Global Step: 232440 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:33:01,108-Speed 129.41 samples/sec Loss 0.2967 Epoch: 23 Global Step: 232445 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:33:20,868-Speed 129.57 samples/sec Loss 0.3207 Epoch: 23 Global Step: 232450 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:33:40,580-Speed 129.88 samples/sec Loss 0.3471 Epoch: 23 Global Step: 232455 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:34:00,401-Speed 129.16 samples/sec Loss 0.3192 Epoch: 23 Global Step: 232460 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:34:20,457-Speed 127.65 samples/sec Loss 0.3412 Epoch: 23 Global Step: 232465 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:34:40,338-Speed 128.77 samples/sec Loss 0.3732 Epoch: 23 Global Step: 232470 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:34:59,951-Speed 130.53 samples/sec Loss 0.2684 Epoch: 23 Global Step: 232475 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:35:19,843-Speed 128.70 samples/sec Loss 0.3378 Epoch: 23 Global Step: 232480 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:35:39,831-Speed 128.08 samples/sec Loss 0.3647 Epoch: 23 Global Step: 232485 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:35:59,671-Speed 129.04 samples/sec Loss 0.2253 Epoch: 23 Global Step: 232490 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:36:19,388-Speed 129.85 samples/sec Loss 0.3125 Epoch: 23 Global Step: 232495 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:36:39,113-Speed 129.79 samples/sec Loss 0.2809 Epoch: 23 Global Step: 232500 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:36:59,079-Speed 128.22 samples/sec Loss 0.2808 Epoch: 23 Global Step: 232505 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:37:18,666-Speed 130.70 samples/sec Loss 0.3343 Epoch: 23 Global Step: 232510 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:37:38,462-Speed 129.33 samples/sec Loss 0.2736 Epoch: 23 Global Step: 232515 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:37:58,074-Speed 130.54 samples/sec Loss 0.3086 Epoch: 23 Global Step: 232520 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:38:17,994-Speed 128.52 samples/sec Loss 0.3440 Epoch: 23 Global Step: 232525 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:38:37,776-Speed 129.42 samples/sec Loss 0.2781 Epoch: 23 Global Step: 232530 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:38:57,668-Speed 128.70 samples/sec Loss 0.3446 Epoch: 23 Global Step: 232535 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:39:17,406-Speed 129.71 samples/sec Loss 0.2978 Epoch: 23 Global Step: 232540 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:39:37,085-Speed 130.10 samples/sec Loss 0.3359 Epoch: 23 Global Step: 232545 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:39:56,968-Speed 128.76 samples/sec Loss 0.2792 Epoch: 23 Global Step: 232550 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:40:17,253-Speed 126.21 samples/sec Loss 0.3002 Epoch: 23 Global Step: 232555 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:40:37,118-Speed 128.87 samples/sec Loss 0.3960 Epoch: 23 Global Step: 232560 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:40:56,833-Speed 129.86 samples/sec Loss 0.3145 Epoch: 23 Global Step: 232565 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:41:16,722-Speed 128.72 samples/sec Loss 0.3982 Epoch: 23 Global Step: 232570 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:41:36,512-Speed 129.37 samples/sec Loss 0.3161 Epoch: 23 Global Step: 232575 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:41:56,312-Speed 129.30 samples/sec Loss 0.3097 Epoch: 23 Global Step: 232580 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:42:16,149-Speed 129.06 samples/sec Loss 0.3658 Epoch: 23 Global Step: 232585 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:42:35,877-Speed 129.77 samples/sec Loss 0.2521 Epoch: 23 Global Step: 232590 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:42:55,714-Speed 129.06 samples/sec Loss 0.3568 Epoch: 23 Global Step: 232595 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:43:16,067-Speed 125.78 samples/sec Loss 0.2919 Epoch: 23 Global Step: 232600 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:43:35,946-Speed 128.79 samples/sec Loss 0.3769 Epoch: 23 Global Step: 232605 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:43:55,672-Speed 129.78 samples/sec Loss 0.2639 Epoch: 23 Global Step: 232610 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:44:15,568-Speed 128.68 samples/sec Loss 0.3616 Epoch: 23 Global Step: 232615 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:44:35,372-Speed 129.27 samples/sec Loss 0.2492 Epoch: 23 Global Step: 232620 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:44:55,209-Speed 129.06 samples/sec Loss 0.2587 Epoch: 23 Global Step: 232625 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:45:15,014-Speed 129.27 samples/sec Loss 0.3145 Epoch: 23 Global Step: 232630 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:45:34,778-Speed 129.53 samples/sec Loss 0.2648 Epoch: 23 Global Step: 232635 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:45:54,626-Speed 128.99 samples/sec Loss 0.2456 Epoch: 23 Global Step: 232640 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:46:15,072-Speed 125.21 samples/sec Loss 0.3074 Epoch: 23 Global Step: 232645 Fp16 Grad Scale: 8192 Required: 20 hours Training: 2021-06-18 09:46:34,357-Speed 132.75 samples/sec Loss 0.2549 Epoch: 23 Global Step: 232650 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:46:54,009-Speed 130.27 samples/sec Loss 0.3131 Epoch: 23 Global Step: 232655 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:47:13,871-Speed 128.90 samples/sec Loss 0.3025 Epoch: 23 Global Step: 232660 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:47:33,662-Speed 129.36 samples/sec Loss 0.3227 Epoch: 23 Global Step: 232665 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:47:53,582-Speed 128.53 samples/sec Loss 0.2591 Epoch: 23 Global Step: 232670 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:48:13,288-Speed 129.91 samples/sec Loss 0.3166 Epoch: 23 Global Step: 232675 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:48:33,219-Speed 128.46 samples/sec Loss 0.3342 Epoch: 23 Global Step: 232680 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:48:53,067-Speed 128.99 samples/sec Loss 0.2999 Epoch: 23 Global Step: 232685 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:49:13,503-Speed 125.27 samples/sec Loss 0.3909 Epoch: 23 Global Step: 232690 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:49:33,274-Speed 129.49 samples/sec Loss 0.2777 Epoch: 23 Global Step: 232695 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:49:52,945-Speed 130.15 samples/sec Loss 0.3401 Epoch: 23 Global Step: 232700 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:50:12,803-Speed 128.92 samples/sec Loss 0.3244 Epoch: 23 Global Step: 232705 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:50:32,599-Speed 129.33 samples/sec Loss 0.2922 Epoch: 23 Global Step: 232710 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:50:52,394-Speed 129.33 samples/sec Loss 0.3448 Epoch: 23 Global Step: 232715 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:51:12,107-Speed 129.87 samples/sec Loss 0.3401 Epoch: 23 Global Step: 232720 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:51:31,833-Speed 129.79 samples/sec Loss 0.2700 Epoch: 23 Global Step: 232725 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:51:51,688-Speed 128.94 samples/sec Loss 0.2709 Epoch: 23 Global Step: 232730 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:52:12,094-Speed 125.45 samples/sec Loss 0.3220 Epoch: 23 Global Step: 232735 Fp16 Grad Scale: 4096 Required: 20 hours Training: 2021-06-18 09:52:31,232-Speed 133.77 samples/sec Loss 0.3514 Epoch: 23 Global Step: 232740 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:52:51,105-Speed 128.83 samples/sec Loss 0.2933 Epoch: 23 Global Step: 232745 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:53:10,849-Speed 129.66 samples/sec Loss 0.3330 Epoch: 23 Global Step: 232750 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:53:30,659-Speed 129.24 samples/sec Loss 0.3061 Epoch: 23 Global Step: 232755 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:53:50,558-Speed 128.65 samples/sec Loss 0.2884 Epoch: 23 Global Step: 232760 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:54:10,268-Speed 129.89 samples/sec Loss 0.3212 Epoch: 23 Global Step: 232765 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:54:30,092-Speed 129.14 samples/sec Loss 0.2903 Epoch: 23 Global Step: 232770 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:54:49,924-Speed 129.09 samples/sec Loss 0.3070 Epoch: 23 Global Step: 232775 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:55:10,003-Speed 127.50 samples/sec Loss 0.3174 Epoch: 23 Global Step: 232780 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:55:29,899-Speed 128.68 samples/sec Loss 0.3252 Epoch: 23 Global Step: 232785 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:55:49,923-Speed 127.86 samples/sec Loss 0.3009 Epoch: 23 Global Step: 232790 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:56:09,614-Speed 130.02 samples/sec Loss 0.2753 Epoch: 23 Global Step: 232795 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:56:29,412-Speed 129.31 samples/sec Loss 0.3961 Epoch: 23 Global Step: 232800 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:56:49,304-Speed 128.70 samples/sec Loss 0.2939 Epoch: 23 Global Step: 232805 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:57:09,021-Speed 129.85 samples/sec Loss 0.3073 Epoch: 23 Global Step: 232810 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:57:28,947-Speed 128.48 samples/sec Loss 0.3759 Epoch: 23 Global Step: 232815 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:57:48,572-Speed 130.45 samples/sec Loss 0.3479 Epoch: 23 Global Step: 232820 Fp16 Grad Scale: 2048 Required: 20 hours Training: 2021-06-18 09:58:08,572-Speed 128.00 samples/sec Loss 0.3003 Epoch: 23 Global Step: 232825 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 09:58:28,491-Speed 128.53 samples/sec Loss 0.3073 Epoch: 23 Global Step: 232830 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 09:58:48,417-Speed 128.48 samples/sec Loss 0.2205 Epoch: 23 Global Step: 232835 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 09:59:08,157-Speed 129.69 samples/sec Loss 0.2755 Epoch: 23 Global Step: 232840 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 09:59:27,940-Speed 129.41 samples/sec Loss 0.2912 Epoch: 23 Global Step: 232845 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 09:59:47,796-Speed 128.94 samples/sec Loss 0.2939 Epoch: 23 Global Step: 232850 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:00:07,530-Speed 129.73 samples/sec Loss 0.3392 Epoch: 23 Global Step: 232855 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:00:27,451-Speed 128.51 samples/sec Loss 0.3170 Epoch: 23 Global Step: 232860 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:00:47,059-Speed 130.56 samples/sec Loss 0.2502 Epoch: 23 Global Step: 232865 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:01:07,100-Speed 127.74 samples/sec Loss 0.2815 Epoch: 23 Global Step: 232870 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:01:26,944-Speed 129.02 samples/sec Loss 0.2928 Epoch: 23 Global Step: 232875 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:01:47,024-Speed 127.49 samples/sec Loss 0.3405 Epoch: 23 Global Step: 232880 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:02:06,657-Speed 130.40 samples/sec Loss 0.2488 Epoch: 23 Global Step: 232885 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:02:26,541-Speed 128.75 samples/sec Loss 0.2946 Epoch: 23 Global Step: 232890 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:02:46,304-Speed 129.55 samples/sec Loss 0.2836 Epoch: 23 Global Step: 232895 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:03:06,048-Speed 129.67 samples/sec Loss 0.2900 Epoch: 23 Global Step: 232900 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:03:25,964-Speed 128.54 samples/sec Loss 0.2958 Epoch: 23 Global Step: 232905 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:03:45,590-Speed 130.44 samples/sec Loss 0.3032 Epoch: 23 Global Step: 232910 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:04:05,908-Speed 126.01 samples/sec Loss 0.3082 Epoch: 23 Global Step: 232915 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:04:25,634-Speed 129.78 samples/sec Loss 0.3897 Epoch: 23 Global Step: 232920 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:04:45,738-Speed 127.35 samples/sec Loss 0.3080 Epoch: 23 Global Step: 232925 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:05:05,523-Speed 129.39 samples/sec Loss 0.3131 Epoch: 23 Global Step: 232930 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:05:25,419-Speed 128.68 samples/sec Loss 0.2796 Epoch: 23 Global Step: 232935 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:05:45,140-Speed 129.81 samples/sec Loss 0.2473 Epoch: 23 Global Step: 232940 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:06:04,787-Speed 130.31 samples/sec Loss 0.3535 Epoch: 23 Global Step: 232945 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:06:24,637-Speed 128.98 samples/sec Loss 0.3084 Epoch: 23 Global Step: 232950 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:06:44,266-Speed 130.42 samples/sec Loss 0.2852 Epoch: 23 Global Step: 232955 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:07:04,482-Speed 126.64 samples/sec Loss 0.3161 Epoch: 23 Global Step: 232960 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:07:23,722-Speed 133.06 samples/sec Loss 0.3567 Epoch: 23 Global Step: 232965 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:07:43,809-Speed 127.45 samples/sec Loss 0.3534 Epoch: 23 Global Step: 232970 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:08:03,414-Speed 130.58 samples/sec Loss 0.3026 Epoch: 23 Global Step: 232975 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:08:23,293-Speed 128.79 samples/sec Loss 0.3039 Epoch: 23 Global Step: 232980 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:08:43,069-Speed 129.46 samples/sec Loss 0.3090 Epoch: 23 Global Step: 232985 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:09:02,747-Speed 130.10 samples/sec Loss 0.2668 Epoch: 23 Global Step: 232990 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:09:22,633-Speed 128.74 samples/sec Loss 0.3153 Epoch: 23 Global Step: 232995 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:09:42,241-Speed 130.56 samples/sec Loss 0.2531 Epoch: 23 Global Step: 233000 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:10:02,116-Speed 128.81 samples/sec Loss 0.3152 Epoch: 23 Global Step: 233005 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:10:21,888-Speed 129.49 samples/sec Loss 0.2527 Epoch: 23 Global Step: 233010 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:10:42,264-Speed 125.64 samples/sec Loss 0.3081 Epoch: 23 Global Step: 233015 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:11:01,885-Speed 130.47 samples/sec Loss 0.3532 Epoch: 23 Global Step: 233020 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:11:21,773-Speed 128.73 samples/sec Loss 0.2940 Epoch: 23 Global Step: 233025 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:11:41,550-Speed 129.45 samples/sec Loss 0.3011 Epoch: 23 Global Step: 233030 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:12:01,223-Speed 130.14 samples/sec Loss 0.3523 Epoch: 23 Global Step: 233035 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:12:21,111-Speed 128.73 samples/sec Loss 0.2946 Epoch: 23 Global Step: 233040 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:12:40,727-Speed 130.52 samples/sec Loss 0.3334 Epoch: 23 Global Step: 233045 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:13:00,517-Speed 129.37 samples/sec Loss 0.3598 Epoch: 23 Global Step: 233050 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:13:20,276-Speed 129.56 samples/sec Loss 0.3313 Epoch: 23 Global Step: 233055 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:13:40,504-Speed 126.56 samples/sec Loss 0.2757 Epoch: 23 Global Step: 233060 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:14:00,240-Speed 129.72 samples/sec Loss 0.2747 Epoch: 23 Global Step: 233065 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:14:20,059-Speed 129.18 samples/sec Loss 0.3170 Epoch: 23 Global Step: 233070 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:14:39,846-Speed 129.38 samples/sec Loss 0.2771 Epoch: 23 Global Step: 233075 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:14:59,498-Speed 130.28 samples/sec Loss 0.3865 Epoch: 23 Global Step: 233080 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:15:19,314-Speed 129.20 samples/sec Loss 0.3762 Epoch: 23 Global Step: 233085 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:15:38,992-Speed 130.10 samples/sec Loss 0.2978 Epoch: 23 Global Step: 233090 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:15:58,797-Speed 129.27 samples/sec Loss 0.3187 Epoch: 23 Global Step: 233095 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:16:18,570-Speed 129.47 samples/sec Loss 0.3163 Epoch: 23 Global Step: 233100 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:16:38,114-Speed 130.99 samples/sec Loss 0.3925 Epoch: 23 Global Step: 233105 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:16:57,785-Speed 130.15 samples/sec Loss 0.2720 Epoch: 23 Global Step: 233110 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:17:17,623-Speed 129.05 samples/sec Loss 0.3654 Epoch: 23 Global Step: 233115 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:17:37,403-Speed 129.44 samples/sec Loss 0.3689 Epoch: 23 Global Step: 233120 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:17:57,085-Speed 130.07 samples/sec Loss 0.2904 Epoch: 23 Global Step: 233125 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:18:16,495-Speed 131.90 samples/sec Loss 0.3241 Epoch: 23 Global Step: 233130 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:18:36,192-Speed 129.98 samples/sec Loss 0.3358 Epoch: 23 Global Step: 233135 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:18:55,982-Speed 129.36 samples/sec Loss 0.3182 Epoch: 23 Global Step: 233140 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:19:15,710-Speed 129.77 samples/sec Loss 0.3298 Epoch: 23 Global Step: 233145 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:19:36,026-Speed 126.02 samples/sec Loss 0.3148 Epoch: 23 Global Step: 233150 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:19:55,753-Speed 129.78 samples/sec Loss 0.2517 Epoch: 23 Global Step: 233155 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:20:15,810-Speed 127.64 samples/sec Loss 0.2965 Epoch: 23 Global Step: 233160 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:20:35,636-Speed 129.13 samples/sec Loss 0.3407 Epoch: 23 Global Step: 233165 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:20:55,499-Speed 128.88 samples/sec Loss 0.3575 Epoch: 23 Global Step: 233170 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:21:15,226-Speed 129.78 samples/sec Loss 0.3228 Epoch: 23 Global Step: 233175 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:21:35,088-Speed 128.89 samples/sec Loss 0.2925 Epoch: 23 Global Step: 233180 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:21:54,806-Speed 129.84 samples/sec Loss 0.3201 Epoch: 23 Global Step: 233185 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:22:14,480-Speed 130.12 samples/sec Loss 0.3021 Epoch: 23 Global Step: 233190 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:22:34,690-Speed 126.68 samples/sec Loss 0.3029 Epoch: 23 Global Step: 233195 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:22:54,435-Speed 129.66 samples/sec Loss 0.3903 Epoch: 23 Global Step: 233200 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:23:14,505-Speed 127.56 samples/sec Loss 0.3105 Epoch: 23 Global Step: 233205 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:23:34,177-Speed 130.14 samples/sec Loss 0.3273 Epoch: 23 Global Step: 233210 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:23:54,127-Speed 128.32 samples/sec Loss 0.3200 Epoch: 23 Global Step: 233215 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:24:13,845-Speed 129.84 samples/sec Loss 0.2533 Epoch: 23 Global Step: 233220 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:24:33,759-Speed 128.56 samples/sec Loss 0.2536 Epoch: 23 Global Step: 233225 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:24:53,459-Speed 129.95 samples/sec Loss 0.3021 Epoch: 23 Global Step: 233230 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:25:13,199-Speed 129.69 samples/sec Loss 0.4390 Epoch: 23 Global Step: 233235 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:25:33,377-Speed 126.88 samples/sec Loss 0.4309 Epoch: 23 Global Step: 233240 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:25:53,050-Speed 130.13 samples/sec Loss 0.3176 Epoch: 23 Global Step: 233245 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:26:13,156-Speed 127.33 samples/sec Loss 0.3184 Epoch: 23 Global Step: 233250 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:26:32,884-Speed 129.77 samples/sec Loss 0.3127 Epoch: 23 Global Step: 233255 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:26:52,665-Speed 129.43 samples/sec Loss 0.3113 Epoch: 23 Global Step: 233260 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:27:11,946-Speed 132.78 samples/sec Loss 0.3387 Epoch: 23 Global Step: 233265 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:27:31,835-Speed 128.72 samples/sec Loss 0.2726 Epoch: 23 Global Step: 233270 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:27:51,544-Speed 129.90 samples/sec Loss 0.3143 Epoch: 23 Global Step: 233275 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:28:11,198-Speed 130.26 samples/sec Loss 0.4235 Epoch: 23 Global Step: 233280 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:28:31,069-Speed 128.84 samples/sec Loss 0.3696 Epoch: 23 Global Step: 233285 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:28:50,793-Speed 129.79 samples/sec Loss 0.3296 Epoch: 23 Global Step: 233290 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:29:11,157-Speed 125.72 samples/sec Loss 0.2958 Epoch: 23 Global Step: 233295 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:29:30,940-Speed 129.41 samples/sec Loss 0.2502 Epoch: 23 Global Step: 233300 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:29:50,669-Speed 129.76 samples/sec Loss 0.3089 Epoch: 23 Global Step: 233305 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:30:10,402-Speed 129.74 samples/sec Loss 0.3347 Epoch: 23 Global Step: 233310 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:30:30,295-Speed 128.70 samples/sec Loss 0.3626 Epoch: 23 Global Step: 233315 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:30:50,014-Speed 129.83 samples/sec Loss 0.2796 Epoch: 23 Global Step: 233320 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:31:09,790-Speed 129.46 samples/sec Loss 0.2813 Epoch: 23 Global Step: 233325 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:31:29,591-Speed 129.29 samples/sec Loss 0.3068 Epoch: 23 Global Step: 233330 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:31:49,263-Speed 130.14 samples/sec Loss 0.3750 Epoch: 23 Global Step: 233335 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:32:09,571-Speed 126.07 samples/sec Loss 0.3138 Epoch: 23 Global Step: 233340 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:32:29,357-Speed 129.39 samples/sec Loss 0.3385 Epoch: 23 Global Step: 233345 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:32:49,066-Speed 129.90 samples/sec Loss 0.3429 Epoch: 23 Global Step: 233350 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:33:08,795-Speed 129.76 samples/sec Loss 0.2926 Epoch: 23 Global Step: 233355 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:33:28,660-Speed 128.88 samples/sec Loss 0.3271 Epoch: 23 Global Step: 233360 Fp16 Grad Scale: 2048 Required: 19 hours Training: 2021-06-18 10:33:48,363-Speed 129.94 samples/sec Loss 0.3307 Epoch: 23 Global Step: 233365 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:34:08,166-Speed 129.28 samples/sec Loss 0.2637 Epoch: 23 Global Step: 233370 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:34:27,948-Speed 129.42 samples/sec Loss 0.2942 Epoch: 23 Global Step: 233375 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:34:47,688-Speed 129.69 samples/sec Loss 0.3012 Epoch: 23 Global Step: 233380 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:35:08,019-Speed 125.92 samples/sec Loss 0.2803 Epoch: 23 Global Step: 233385 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:35:27,836-Speed 129.19 samples/sec Loss 0.2767 Epoch: 23 Global Step: 233390 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:35:47,482-Speed 130.31 samples/sec Loss 0.3472 Epoch: 23 Global Step: 233395 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:36:07,255-Speed 129.48 samples/sec Loss 0.3821 Epoch: 23 Global Step: 233400 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:36:27,048-Speed 129.35 samples/sec Loss 0.3871 Epoch: 23 Global Step: 233405 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:36:46,756-Speed 129.90 samples/sec Loss 0.2632 Epoch: 23 Global Step: 233410 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:37:06,547-Speed 129.36 samples/sec Loss 0.3009 Epoch: 23 Global Step: 233415 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:37:26,343-Speed 129.32 samples/sec Loss 0.2876 Epoch: 23 Global Step: 233420 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:37:46,227-Speed 128.76 samples/sec Loss 0.2860 Epoch: 23 Global Step: 233425 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:38:06,461-Speed 126.52 samples/sec Loss 0.3530 Epoch: 23 Global Step: 233430 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:38:26,260-Speed 129.30 samples/sec Loss 0.3719 Epoch: 23 Global Step: 233435 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:38:46,043-Speed 129.42 samples/sec Loss 0.3699 Epoch: 23 Global Step: 233440 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:39:05,941-Speed 128.66 samples/sec Loss 0.2811 Epoch: 23 Global Step: 233445 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:39:25,653-Speed 129.88 samples/sec Loss 0.3332 Epoch: 23 Global Step: 233450 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:39:45,395-Speed 129.68 samples/sec Loss 0.3509 Epoch: 23 Global Step: 233455 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:40:05,143-Speed 129.64 samples/sec Loss 0.2817 Epoch: 23 Global Step: 233460 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:40:24,938-Speed 129.33 samples/sec Loss 0.2546 Epoch: 23 Global Step: 233465 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:40:44,769-Speed 129.10 samples/sec Loss 0.3128 Epoch: 23 Global Step: 233470 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:41:04,983-Speed 126.65 samples/sec Loss 0.3085 Epoch: 23 Global Step: 233475 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:41:24,727-Speed 129.67 samples/sec Loss 0.3984 Epoch: 23 Global Step: 233480 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:41:44,503-Speed 129.46 samples/sec Loss 0.2732 Epoch: 23 Global Step: 233485 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:42:04,393-Speed 128.71 samples/sec Loss 0.2931 Epoch: 23 Global Step: 233490 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:42:24,138-Speed 129.66 samples/sec Loss 0.2947 Epoch: 23 Global Step: 233495 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:42:43,956-Speed 129.18 samples/sec Loss 0.3044 Epoch: 23 Global Step: 233500 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:43:03,636-Speed 130.09 samples/sec Loss 0.2868 Epoch: 23 Global Step: 233505 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:43:23,442-Speed 129.26 samples/sec Loss 0.2925 Epoch: 23 Global Step: 233510 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:43:43,343-Speed 128.64 samples/sec Loss 0.3574 Epoch: 23 Global Step: 233515 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:44:03,487-Speed 127.09 samples/sec Loss 0.2648 Epoch: 23 Global Step: 233520 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:44:23,277-Speed 129.36 samples/sec Loss 0.3380 Epoch: 23 Global Step: 233525 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:44:42,989-Speed 129.88 samples/sec Loss 0.2645 Epoch: 23 Global Step: 233530 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:45:02,905-Speed 128.55 samples/sec Loss 0.2492 Epoch: 23 Global Step: 233535 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:45:22,620-Speed 129.85 samples/sec Loss 0.2675 Epoch: 23 Global Step: 233540 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:45:42,327-Speed 129.92 samples/sec Loss 0.3216 Epoch: 23 Global Step: 233545 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:46:02,093-Speed 129.52 samples/sec Loss 0.3689 Epoch: 23 Global Step: 233550 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:46:21,883-Speed 129.37 samples/sec Loss 0.3685 Epoch: 23 Global Step: 233555 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:46:41,769-Speed 128.74 samples/sec Loss 0.2988 Epoch: 23 Global Step: 233560 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:47:02,041-Speed 126.29 samples/sec Loss 0.3398 Epoch: 23 Global Step: 233565 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:47:21,756-Speed 129.85 samples/sec Loss 0.3017 Epoch: 23 Global Step: 233570 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:47:41,510-Speed 129.60 samples/sec Loss 0.3332 Epoch: 23 Global Step: 233575 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:48:01,666-Speed 127.02 samples/sec Loss 0.3276 Epoch: 23 Global Step: 233580 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:48:21,396-Speed 129.75 samples/sec Loss 0.3149 Epoch: 23 Global Step: 233585 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:48:41,314-Speed 128.54 samples/sec Loss 0.2742 Epoch: 23 Global Step: 233590 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:49:00,960-Speed 130.31 samples/sec Loss 0.2563 Epoch: 23 Global Step: 233595 Fp16 Grad Scale: 8192 Required: 19 hours Training: 2021-06-18 10:49:20,423-Speed 131.54 samples/sec Loss 0.3738 Epoch: 23 Global Step: 233600 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:49:40,248-Speed 129.14 samples/sec Loss 0.3619 Epoch: 23 Global Step: 233605 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:50:00,627-Speed 125.63 samples/sec Loss 0.3090 Epoch: 23 Global Step: 233610 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:50:20,217-Speed 130.68 samples/sec Loss 0.2968 Epoch: 23 Global Step: 233615 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:50:39,902-Speed 130.06 samples/sec Loss 0.3489 Epoch: 23 Global Step: 233620 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:50:59,779-Speed 128.80 samples/sec Loss 0.3426 Epoch: 23 Global Step: 233625 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:51:19,518-Speed 129.70 samples/sec Loss 0.2868 Epoch: 23 Global Step: 233630 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:51:39,414-Speed 128.68 samples/sec Loss 0.3172 Epoch: 23 Global Step: 233635 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:51:59,174-Speed 129.56 samples/sec Loss 0.3179 Epoch: 23 Global Step: 233640 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:52:19,130-Speed 128.29 samples/sec Loss 0.3002 Epoch: 23 Global Step: 233645 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:52:38,842-Speed 129.87 samples/sec Loss 0.3169 Epoch: 23 Global Step: 233650 Fp16 Grad Scale: 4096 Required: 19 hours Training: 2021-06-18 10:52:59,174-Speed 125.91 samples/sec Loss 0.2879 Epoch: 23 Global Step: 233655 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 10:53:18,785-Speed 130.54 samples/sec Loss 0.2908 Epoch: 23 Global Step: 233660 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 10:53:38,641-Speed 128.93 samples/sec Loss 0.2762 Epoch: 23 Global Step: 233665 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 10:53:58,468-Speed 129.13 samples/sec Loss 0.2693 Epoch: 23 Global Step: 233670 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 10:54:18,201-Speed 129.74 samples/sec Loss 0.3505 Epoch: 23 Global Step: 233675 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 10:54:38,085-Speed 128.75 samples/sec Loss 0.2664 Epoch: 23 Global Step: 233680 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 10:54:57,754-Speed 130.16 samples/sec Loss 0.3444 Epoch: 23 Global Step: 233685 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 10:55:17,600-Speed 129.00 samples/sec Loss 0.3214 Epoch: 23 Global Step: 233690 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 10:55:37,354-Speed 129.60 samples/sec Loss 0.2991 Epoch: 23 Global Step: 233695 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 10:55:57,744-Speed 125.55 samples/sec Loss 0.2769 Epoch: 23 Global Step: 233700 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:56:17,370-Speed 130.45 samples/sec Loss 0.3272 Epoch: 23 Global Step: 233705 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:56:37,194-Speed 129.14 samples/sec Loss 0.3314 Epoch: 23 Global Step: 233710 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:56:56,998-Speed 129.28 samples/sec Loss 0.3164 Epoch: 23 Global Step: 233715 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:57:16,752-Speed 129.60 samples/sec Loss 0.3497 Epoch: 23 Global Step: 233720 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:57:36,612-Speed 128.91 samples/sec Loss 0.3412 Epoch: 23 Global Step: 233725 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:57:56,221-Speed 130.56 samples/sec Loss 0.3413 Epoch: 23 Global Step: 233730 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:58:16,119-Speed 128.66 samples/sec Loss 0.3418 Epoch: 23 Global Step: 233735 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:58:35,913-Speed 129.34 samples/sec Loss 0.3028 Epoch: 23 Global Step: 233740 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:58:56,293-Speed 125.62 samples/sec Loss 0.3432 Epoch: 23 Global Step: 233745 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:59:15,902-Speed 130.55 samples/sec Loss 0.2998 Epoch: 23 Global Step: 233750 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:59:35,809-Speed 128.60 samples/sec Loss 0.2849 Epoch: 23 Global Step: 233755 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 10:59:55,117-Speed 132.60 samples/sec Loss 0.3132 Epoch: 23 Global Step: 233760 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:00:14,936-Speed 129.17 samples/sec Loss 0.2989 Epoch: 23 Global Step: 233765 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:00:34,733-Speed 129.33 samples/sec Loss 0.3428 Epoch: 23 Global Step: 233770 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:00:54,350-Speed 130.50 samples/sec Loss 0.2659 Epoch: 23 Global Step: 233775 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:01:14,254-Speed 128.63 samples/sec Loss 0.3067 Epoch: 23 Global Step: 233780 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:01:34,046-Speed 129.35 samples/sec Loss 0.3875 Epoch: 23 Global Step: 233785 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:01:54,192-Speed 127.08 samples/sec Loss 0.3109 Epoch: 23 Global Step: 233790 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:02:13,808-Speed 130.51 samples/sec Loss 0.3049 Epoch: 23 Global Step: 233795 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:02:33,903-Speed 127.40 samples/sec Loss 0.2343 Epoch: 23 Global Step: 233800 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:02:53,604-Speed 129.95 samples/sec Loss 0.3498 Epoch: 23 Global Step: 233805 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:03:13,508-Speed 128.62 samples/sec Loss 0.3676 Epoch: 23 Global Step: 233810 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:03:33,240-Speed 129.75 samples/sec Loss 0.2626 Epoch: 23 Global Step: 233815 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:03:52,879-Speed 130.36 samples/sec Loss 0.2411 Epoch: 23 Global Step: 233820 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:04:12,796-Speed 128.54 samples/sec Loss 0.3486 Epoch: 23 Global Step: 233825 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:04:32,469-Speed 130.13 samples/sec Loss 0.3135 Epoch: 23 Global Step: 233830 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:04:52,730-Speed 126.36 samples/sec Loss 0.2581 Epoch: 23 Global Step: 233835 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:05:11,896-Speed 133.58 samples/sec Loss 0.3147 Epoch: 23 Global Step: 233840 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:05:31,982-Speed 127.46 samples/sec Loss 0.3662 Epoch: 23 Global Step: 233845 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:05:51,753-Speed 129.49 samples/sec Loss 0.2696 Epoch: 23 Global Step: 233850 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:06:11,651-Speed 128.66 samples/sec Loss 0.2767 Epoch: 23 Global Step: 233855 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:06:31,389-Speed 129.70 samples/sec Loss 0.3193 Epoch: 23 Global Step: 233860 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:06:51,045-Speed 130.25 samples/sec Loss 0.2833 Epoch: 23 Global Step: 233865 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:07:10,888-Speed 129.02 samples/sec Loss 0.3259 Epoch: 23 Global Step: 233870 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:07:30,676-Speed 129.38 samples/sec Loss 0.2909 Epoch: 23 Global Step: 233875 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:07:50,568-Speed 128.70 samples/sec Loss 0.3162 Epoch: 23 Global Step: 233880 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:08:10,306-Speed 129.71 samples/sec Loss 0.2839 Epoch: 23 Global Step: 233885 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:08:30,578-Speed 126.29 samples/sec Loss 0.3399 Epoch: 23 Global Step: 233890 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:08:50,362-Speed 129.41 samples/sec Loss 0.3041 Epoch: 23 Global Step: 233895 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:09:10,193-Speed 129.10 samples/sec Loss 0.2753 Epoch: 23 Global Step: 233900 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:09:29,915-Speed 129.81 samples/sec Loss 0.2660 Epoch: 23 Global Step: 233905 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:09:49,757-Speed 129.03 samples/sec Loss 0.3652 Epoch: 23 Global Step: 233910 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:10:09,524-Speed 129.52 samples/sec Loss 0.2786 Epoch: 23 Global Step: 233915 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:10:29,378-Speed 128.95 samples/sec Loss 0.2854 Epoch: 23 Global Step: 233920 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:10:49,245-Speed 128.87 samples/sec Loss 0.3095 Epoch: 23 Global Step: 233925 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:11:09,014-Speed 129.50 samples/sec Loss 0.3643 Epoch: 23 Global Step: 233930 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:11:29,300-Speed 126.20 samples/sec Loss 0.2955 Epoch: 23 Global Step: 233935 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:11:49,108-Speed 129.25 samples/sec Loss 0.3399 Epoch: 23 Global Step: 233940 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:12:08,856-Speed 129.63 samples/sec Loss 0.3067 Epoch: 23 Global Step: 233945 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:12:28,583-Speed 129.78 samples/sec Loss 0.3204 Epoch: 23 Global Step: 233950 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:12:48,378-Speed 129.34 samples/sec Loss 0.3454 Epoch: 23 Global Step: 233955 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:13:08,071-Speed 130.00 samples/sec Loss 0.3131 Epoch: 23 Global Step: 233960 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:13:27,963-Speed 128.70 samples/sec Loss 0.3632 Epoch: 23 Global Step: 233965 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:13:47,882-Speed 128.53 samples/sec Loss 0.2871 Epoch: 23 Global Step: 233970 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:14:07,620-Speed 129.71 samples/sec Loss 0.2941 Epoch: 23 Global Step: 233975 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:14:27,983-Speed 125.73 samples/sec Loss 0.2802 Epoch: 23 Global Step: 233980 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:14:47,949-Speed 128.22 samples/sec Loss 0.3171 Epoch: 23 Global Step: 233985 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:15:07,575-Speed 130.44 samples/sec Loss 0.3196 Epoch: 23 Global Step: 233990 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:15:27,357-Speed 129.42 samples/sec Loss 0.3949 Epoch: 23 Global Step: 233995 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:15:47,134-Speed 129.45 samples/sec Loss 0.3358 Epoch: 23 Global Step: 234000 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:16:06,754-Speed 130.49 samples/sec Loss 0.3461 Epoch: 23 Global Step: 234005 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:16:26,533-Speed 129.44 samples/sec Loss 0.3561 Epoch: 23 Global Step: 234010 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:16:46,322-Speed 129.37 samples/sec Loss 0.3702 Epoch: 23 Global Step: 234015 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:17:06,098-Speed 129.46 samples/sec Loss 0.3106 Epoch: 23 Global Step: 234020 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:17:26,302-Speed 126.71 samples/sec Loss 0.2896 Epoch: 23 Global Step: 234025 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:17:46,118-Speed 129.20 samples/sec Loss 0.2702 Epoch: 23 Global Step: 234030 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:18:05,907-Speed 129.37 samples/sec Loss 0.3113 Epoch: 23 Global Step: 234035 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:18:25,674-Speed 129.51 samples/sec Loss 0.3737 Epoch: 23 Global Step: 234040 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:18:45,463-Speed 129.37 samples/sec Loss 0.2500 Epoch: 23 Global Step: 234045 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:19:05,167-Speed 129.93 samples/sec Loss 0.3486 Epoch: 23 Global Step: 234050 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:19:24,957-Speed 129.36 samples/sec Loss 0.2724 Epoch: 23 Global Step: 234055 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:19:44,756-Speed 129.31 samples/sec Loss 0.3511 Epoch: 23 Global Step: 234060 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:20:04,504-Speed 129.64 samples/sec Loss 0.2861 Epoch: 23 Global Step: 234065 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:20:24,705-Speed 126.73 samples/sec Loss 0.3477 Epoch: 23 Global Step: 234070 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:20:45,194-Speed 124.96 samples/sec Loss 0.3003 Epoch: 23 Global Step: 234075 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:21:05,008-Speed 129.21 samples/sec Loss 0.2717 Epoch: 23 Global Step: 234080 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:21:24,950-Speed 128.38 samples/sec Loss 0.3314 Epoch: 23 Global Step: 234085 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:21:44,719-Speed 129.50 samples/sec Loss 0.3262 Epoch: 23 Global Step: 234090 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:22:04,384-Speed 130.18 samples/sec Loss 0.2762 Epoch: 23 Global Step: 234095 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:22:24,130-Speed 129.66 samples/sec Loss 0.3101 Epoch: 23 Global Step: 234100 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:22:43,939-Speed 129.24 samples/sec Loss 0.3870 Epoch: 23 Global Step: 234105 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:23:03,836-Speed 128.67 samples/sec Loss 0.3675 Epoch: 23 Global Step: 234110 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:23:23,943-Speed 127.32 samples/sec Loss 0.2966 Epoch: 23 Global Step: 234115 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:23:43,719-Speed 129.45 samples/sec Loss 0.2844 Epoch: 23 Global Step: 234120 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:24:03,494-Speed 129.46 samples/sec Loss 0.3432 Epoch: 23 Global Step: 234125 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:24:23,424-Speed 128.46 samples/sec Loss 0.2540 Epoch: 23 Global Step: 234130 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:24:43,226-Speed 129.29 samples/sec Loss 0.2850 Epoch: 23 Global Step: 234135 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:25:02,913-Speed 130.04 samples/sec Loss 0.3113 Epoch: 23 Global Step: 234140 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:25:22,623-Speed 129.89 samples/sec Loss 0.3614 Epoch: 23 Global Step: 234145 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:25:42,346-Speed 129.81 samples/sec Loss 0.2843 Epoch: 23 Global Step: 234150 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:26:01,828-Speed 131.41 samples/sec Loss 0.3058 Epoch: 23 Global Step: 234155 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:26:21,960-Speed 127.17 samples/sec Loss 0.3439 Epoch: 23 Global Step: 234160 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:26:41,700-Speed 129.69 samples/sec Loss 0.3137 Epoch: 23 Global Step: 234165 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:27:01,391-Speed 130.01 samples/sec Loss 0.2572 Epoch: 23 Global Step: 234170 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:27:21,313-Speed 128.51 samples/sec Loss 0.3621 Epoch: 23 Global Step: 234175 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:27:41,012-Speed 129.96 samples/sec Loss 0.3542 Epoch: 23 Global Step: 234180 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:28:00,813-Speed 129.30 samples/sec Loss 0.3237 Epoch: 23 Global Step: 234185 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:28:20,401-Speed 130.70 samples/sec Loss 0.3280 Epoch: 23 Global Step: 234190 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:28:40,128-Speed 129.78 samples/sec Loss 0.2566 Epoch: 23 Global Step: 234195 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:29:00,051-Speed 128.50 samples/sec Loss 0.2743 Epoch: 23 Global Step: 234200 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:29:20,319-Speed 126.31 samples/sec Loss 0.3025 Epoch: 23 Global Step: 234205 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:29:39,935-Speed 130.52 samples/sec Loss 0.3078 Epoch: 23 Global Step: 234210 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:29:59,623-Speed 130.03 samples/sec Loss 0.3349 Epoch: 23 Global Step: 234215 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:30:19,527-Speed 128.62 samples/sec Loss 0.2839 Epoch: 23 Global Step: 234220 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:30:39,229-Speed 129.95 samples/sec Loss 0.3139 Epoch: 23 Global Step: 234225 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:30:59,029-Speed 129.30 samples/sec Loss 0.3424 Epoch: 23 Global Step: 234230 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:31:18,708-Speed 130.09 samples/sec Loss 0.3280 Epoch: 23 Global Step: 234235 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:31:38,442-Speed 129.73 samples/sec Loss 0.3101 Epoch: 23 Global Step: 234240 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:31:58,382-Speed 128.39 samples/sec Loss 0.3388 Epoch: 23 Global Step: 234245 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:32:18,656-Speed 126.27 samples/sec Loss 0.3763 Epoch: 23 Global Step: 234250 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:32:38,294-Speed 130.37 samples/sec Loss 0.2807 Epoch: 23 Global Step: 234255 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:32:58,048-Speed 129.60 samples/sec Loss 0.2380 Epoch: 23 Global Step: 234260 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:33:17,995-Speed 128.35 samples/sec Loss 0.3053 Epoch: 23 Global Step: 234265 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:33:37,691-Speed 129.98 samples/sec Loss 0.3479 Epoch: 23 Global Step: 234270 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:33:57,512-Speed 129.16 samples/sec Loss 0.2948 Epoch: 23 Global Step: 234275 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:34:17,137-Speed 130.45 samples/sec Loss 0.3091 Epoch: 23 Global Step: 234280 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:34:37,106-Speed 128.21 samples/sec Loss 0.2869 Epoch: 23 Global Step: 234285 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:34:56,907-Speed 129.29 samples/sec Loss 0.3254 Epoch: 23 Global Step: 234290 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:35:17,202-Speed 126.15 samples/sec Loss 0.2766 Epoch: 23 Global Step: 234295 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:35:36,833-Speed 130.41 samples/sec Loss 0.3103 Epoch: 23 Global Step: 234300 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:35:56,491-Speed 130.23 samples/sec Loss 0.2751 Epoch: 23 Global Step: 234305 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:36:16,419-Speed 128.47 samples/sec Loss 0.2537 Epoch: 23 Global Step: 234310 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:36:36,138-Speed 129.83 samples/sec Loss 0.3021 Epoch: 23 Global Step: 234315 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:36:55,938-Speed 129.30 samples/sec Loss 0.2775 Epoch: 23 Global Step: 234320 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:37:15,656-Speed 129.83 samples/sec Loss 0.2670 Epoch: 23 Global Step: 234325 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:37:35,444-Speed 129.38 samples/sec Loss 0.3127 Epoch: 23 Global Step: 234330 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:37:55,426-Speed 128.12 samples/sec Loss 0.3053 Epoch: 23 Global Step: 234335 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:38:15,723-Speed 126.13 samples/sec Loss 0.3202 Epoch: 23 Global Step: 234340 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:38:35,389-Speed 130.18 samples/sec Loss 0.2910 Epoch: 23 Global Step: 234345 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:38:55,041-Speed 130.28 samples/sec Loss 0.2876 Epoch: 23 Global Step: 234350 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:39:14,972-Speed 128.45 samples/sec Loss 0.3805 Epoch: 23 Global Step: 234355 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:39:34,804-Speed 129.09 samples/sec Loss 0.3809 Epoch: 23 Global Step: 234360 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:39:54,592-Speed 129.38 samples/sec Loss 0.3187 Epoch: 23 Global Step: 234365 Fp16 Grad Scale: 8192 Required: 18 hours Training: 2021-06-18 11:40:13,877-Speed 132.75 samples/sec Loss 0.2420 Epoch: 23 Global Step: 234370 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:40:33,670-Speed 129.35 samples/sec Loss 0.2859 Epoch: 23 Global Step: 234375 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:40:53,441-Speed 129.49 samples/sec Loss 0.3119 Epoch: 23 Global Step: 234380 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:41:13,476-Speed 127.78 samples/sec Loss 0.2598 Epoch: 23 Global Step: 234385 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:41:33,339-Speed 128.90 samples/sec Loss 0.3053 Epoch: 23 Global Step: 234390 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:41:52,948-Speed 130.56 samples/sec Loss 0.3923 Epoch: 23 Global Step: 234395 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:42:12,895-Speed 128.35 samples/sec Loss 0.3265 Epoch: 23 Global Step: 234400 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:42:32,631-Speed 129.71 samples/sec Loss 0.3302 Epoch: 23 Global Step: 234405 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:42:52,668-Speed 127.77 samples/sec Loss 0.2750 Epoch: 23 Global Step: 234410 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:43:12,341-Speed 130.14 samples/sec Loss 0.2582 Epoch: 23 Global Step: 234415 Fp16 Grad Scale: 4096 Required: 18 hours Training: 2021-06-18 11:43:31,694-Speed 132.29 samples/sec Loss 0.3462 Epoch: 23 Global Step: 234420 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:43:51,506-Speed 129.22 samples/sec Loss 0.2742 Epoch: 23 Global Step: 234425 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:44:11,367-Speed 128.91 samples/sec Loss 0.3166 Epoch: 23 Global Step: 234430 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:44:31,171-Speed 129.27 samples/sec Loss 0.3605 Epoch: 23 Global Step: 234435 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:44:50,794-Speed 130.46 samples/sec Loss 0.2941 Epoch: 23 Global Step: 234440 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:45:10,990-Speed 126.77 samples/sec Loss 0.3506 Epoch: 23 Global Step: 234445 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:45:30,685-Speed 129.99 samples/sec Loss 0.3234 Epoch: 23 Global Step: 234450 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:45:50,665-Speed 128.14 samples/sec Loss 0.3368 Epoch: 23 Global Step: 234455 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:46:10,260-Speed 130.65 samples/sec Loss 0.3916 Epoch: 23 Global Step: 234460 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:46:30,142-Speed 128.77 samples/sec Loss 0.2999 Epoch: 23 Global Step: 234465 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:46:49,847-Speed 129.93 samples/sec Loss 0.3463 Epoch: 23 Global Step: 234470 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:47:09,680-Speed 129.09 samples/sec Loss 0.4072 Epoch: 23 Global Step: 234475 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:47:29,391-Speed 129.88 samples/sec Loss 0.3713 Epoch: 23 Global Step: 234480 Fp16 Grad Scale: 2048 Required: 18 hours Training: 2021-06-18 11:47:49,286-Speed 128.69 samples/sec Loss 0.2326 Epoch: 23 Global Step: 234485 Fp16 Grad Scale: 2048 Required: 17 hours Training: 2021-06-18 11:48:09,176-Speed 128.71 samples/sec Loss 0.2993 Epoch: 23 Global Step: 234490 Fp16 Grad Scale: 2048 Required: 17 hours Training: 2021-06-18 11:48:28,937-Speed 129.55 samples/sec Loss 0.3353 Epoch: 23 Global Step: 234495 Fp16 Grad Scale: 2048 Required: 17 hours Training: 2021-06-18 11:48:49,212-Speed 126.27 samples/sec Loss 0.3554 Epoch: 23 Global Step: 234500 Fp16 Grad Scale: 2048 Required: 17 hours Training: 2021-06-18 11:49:08,867-Speed 130.25 samples/sec Loss 0.3873 Epoch: 23 Global Step: 234505 Fp16 Grad Scale: 2048 Required: 17 hours Training: 2021-06-18 11:49:28,605-Speed 129.70 samples/sec Loss 0.2885 Epoch: 23 Global Step: 234510 Fp16 Grad Scale: 2048 Required: 17 hours Training: 2021-06-18 11:49:49,222-Speed 124.17 samples/sec Loss 0.2958 Epoch: 23 Global Step: 234515 Fp16 Grad Scale: 2048 Required: 17 hours Training: 2021-06-18 11:50:09,131-Speed 128.59 samples/sec Loss 0.3087 Epoch: 23 Global Step: 234520 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:50:28,827-Speed 129.98 samples/sec Loss 0.2690 Epoch: 23 Global Step: 234525 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:50:48,488-Speed 130.22 samples/sec Loss 0.3427 Epoch: 23 Global Step: 234530 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:51:08,330-Speed 129.03 samples/sec Loss 0.3328 Epoch: 23 Global Step: 234535 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:51:28,086-Speed 129.59 samples/sec Loss 0.3470 Epoch: 23 Global Step: 234540 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:51:48,429-Speed 125.85 samples/sec Loss 0.3219 Epoch: 23 Global Step: 234545 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:52:08,161-Speed 129.74 samples/sec Loss 0.3715 Epoch: 23 Global Step: 234550 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:52:27,942-Speed 129.42 samples/sec Loss 0.2954 Epoch: 23 Global Step: 234555 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:52:47,721-Speed 129.44 samples/sec Loss 0.3059 Epoch: 23 Global Step: 234560 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:53:07,858-Speed 127.14 samples/sec Loss 0.2816 Epoch: 23 Global Step: 234565 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:53:27,581-Speed 129.80 samples/sec Loss 0.2738 Epoch: 23 Global Step: 234570 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:53:47,306-Speed 129.80 samples/sec Loss 0.2879 Epoch: 23 Global Step: 234575 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:54:07,087-Speed 129.42 samples/sec Loss 0.3281 Epoch: 23 Global Step: 234580 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:54:26,877-Speed 129.37 samples/sec Loss 0.2387 Epoch: 23 Global Step: 234585 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:54:47,106-Speed 126.55 samples/sec Loss 0.3879 Epoch: 23 Global Step: 234590 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:55:06,913-Speed 129.25 samples/sec Loss 0.3359 Epoch: 23 Global Step: 234595 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:55:26,708-Speed 129.33 samples/sec Loss 0.3211 Epoch: 23 Global Step: 234600 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:55:46,551-Speed 129.03 samples/sec Loss 0.2738 Epoch: 23 Global Step: 234605 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:56:06,399-Speed 128.98 samples/sec Loss 0.2534 Epoch: 23 Global Step: 234610 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:56:26,064-Speed 130.19 samples/sec Loss 0.3045 Epoch: 23 Global Step: 234615 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:56:45,870-Speed 129.26 samples/sec Loss 0.3251 Epoch: 23 Global Step: 234620 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 11:57:05,589-Speed 129.83 samples/sec Loss 0.2711 Epoch: 23 Global Step: 234625 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 11:57:25,412-Speed 129.15 samples/sec Loss 0.3177 Epoch: 23 Global Step: 234630 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 11:57:45,756-Speed 125.84 samples/sec Loss 0.3157 Epoch: 23 Global Step: 234635 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 11:58:05,597-Speed 129.03 samples/sec Loss 0.3073 Epoch: 23 Global Step: 234640 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 11:58:25,291-Speed 130.00 samples/sec Loss 0.3211 Epoch: 23 Global Step: 234645 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 11:58:45,141-Speed 128.97 samples/sec Loss 0.3766 Epoch: 23 Global Step: 234650 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 11:59:04,552-Speed 131.89 samples/sec Loss 0.2942 Epoch: 23 Global Step: 234655 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:59:24,190-Speed 130.36 samples/sec Loss 0.3313 Epoch: 23 Global Step: 234660 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 11:59:44,019-Speed 129.11 samples/sec Loss 0.3274 Epoch: 23 Global Step: 234665 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:00:03,730-Speed 129.88 samples/sec Loss 0.3060 Epoch: 23 Global Step: 234670 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:00:23,487-Speed 129.58 samples/sec Loss 0.3720 Epoch: 23 Global Step: 234675 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:00:43,786-Speed 126.12 samples/sec Loss 0.2890 Epoch: 23 Global Step: 234680 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:01:03,591-Speed 129.27 samples/sec Loss 0.3252 Epoch: 23 Global Step: 234685 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:01:23,296-Speed 129.92 samples/sec Loss 0.3313 Epoch: 23 Global Step: 234690 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:01:43,198-Speed 128.64 samples/sec Loss 0.3266 Epoch: 23 Global Step: 234695 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:02:02,982-Speed 129.41 samples/sec Loss 0.3522 Epoch: 23 Global Step: 234700 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:02:22,589-Speed 130.57 samples/sec Loss 0.3377 Epoch: 23 Global Step: 234705 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:02:42,397-Speed 129.24 samples/sec Loss 0.2659 Epoch: 23 Global Step: 234710 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:03:02,114-Speed 129.84 samples/sec Loss 0.3604 Epoch: 23 Global Step: 234715 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:03:21,884-Speed 129.50 samples/sec Loss 0.2998 Epoch: 23 Global Step: 234720 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:03:42,188-Speed 126.09 samples/sec Loss 0.3635 Epoch: 23 Global Step: 234725 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:04:02,013-Speed 129.13 samples/sec Loss 0.3105 Epoch: 23 Global Step: 234730 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:04:21,708-Speed 129.99 samples/sec Loss 0.3296 Epoch: 23 Global Step: 234735 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:04:41,637-Speed 128.46 samples/sec Loss 0.3309 Epoch: 23 Global Step: 234740 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:05:01,371-Speed 129.73 samples/sec Loss 0.2863 Epoch: 23 Global Step: 234745 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:05:21,119-Speed 129.64 samples/sec Loss 0.3199 Epoch: 23 Global Step: 234750 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:05:40,884-Speed 129.53 samples/sec Loss 0.2845 Epoch: 23 Global Step: 234755 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:06:00,607-Speed 129.81 samples/sec Loss 0.3311 Epoch: 23 Global Step: 234760 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:06:20,496-Speed 128.72 samples/sec Loss 0.2718 Epoch: 23 Global Step: 234765 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:06:40,710-Speed 126.65 samples/sec Loss 0.2625 Epoch: 23 Global Step: 234770 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:07:00,534-Speed 129.14 samples/sec Loss 0.3173 Epoch: 23 Global Step: 234775 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:07:20,125-Speed 130.68 samples/sec Loss 0.3009 Epoch: 23 Global Step: 234780 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:07:40,012-Speed 128.73 samples/sec Loss 0.3342 Epoch: 23 Global Step: 234785 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:07:59,730-Speed 129.84 samples/sec Loss 0.3030 Epoch: 23 Global Step: 234790 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:08:19,490-Speed 129.56 samples/sec Loss 0.2656 Epoch: 23 Global Step: 234795 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:08:39,257-Speed 129.52 samples/sec Loss 0.3126 Epoch: 23 Global Step: 234800 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:08:58,901-Speed 130.33 samples/sec Loss 0.2525 Epoch: 23 Global Step: 234805 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:09:18,775-Speed 128.81 samples/sec Loss 0.3371 Epoch: 23 Global Step: 234810 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:09:39,272-Speed 124.90 samples/sec Loss 0.3814 Epoch: 23 Global Step: 234815 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:09:59,029-Speed 129.58 samples/sec Loss 0.2896 Epoch: 23 Global Step: 234820 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:10:18,245-Speed 133.22 samples/sec Loss 0.3737 Epoch: 23 Global Step: 234825 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:10:38,151-Speed 128.61 samples/sec Loss 0.3801 Epoch: 23 Global Step: 234830 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:10:57,821-Speed 130.15 samples/sec Loss 0.3472 Epoch: 23 Global Step: 234835 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:11:17,654-Speed 129.09 samples/sec Loss 0.3625 Epoch: 23 Global Step: 234840 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:11:37,407-Speed 129.61 samples/sec Loss 0.2864 Epoch: 23 Global Step: 234845 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:11:57,453-Speed 127.71 samples/sec Loss 0.2906 Epoch: 23 Global Step: 234850 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:12:17,404-Speed 128.32 samples/sec Loss 0.3245 Epoch: 23 Global Step: 234855 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:12:37,406-Speed 127.99 samples/sec Loss 0.3389 Epoch: 23 Global Step: 234860 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:12:57,292-Speed 128.74 samples/sec Loss 0.2865 Epoch: 23 Global Step: 234865 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:13:16,963-Speed 130.15 samples/sec Loss 0.3477 Epoch: 23 Global Step: 234870 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:13:36,708-Speed 129.66 samples/sec Loss 0.3042 Epoch: 23 Global Step: 234875 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:13:56,490-Speed 129.42 samples/sec Loss 0.2978 Epoch: 23 Global Step: 234880 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:14:16,612-Speed 127.23 samples/sec Loss 0.3424 Epoch: 23 Global Step: 234885 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:14:36,306-Speed 130.00 samples/sec Loss 0.3491 Epoch: 23 Global Step: 234890 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:14:56,079-Speed 129.48 samples/sec Loss 0.2821 Epoch: 23 Global Step: 234895 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:15:15,825-Speed 129.65 samples/sec Loss 0.3078 Epoch: 23 Global Step: 234900 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:15:35,932-Speed 127.32 samples/sec Loss 0.3157 Epoch: 23 Global Step: 234905 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:15:55,716-Speed 129.40 samples/sec Loss 0.3501 Epoch: 23 Global Step: 234910 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:16:15,437-Speed 129.82 samples/sec Loss 0.2872 Epoch: 23 Global Step: 234915 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:16:35,086-Speed 130.30 samples/sec Loss 0.3256 Epoch: 23 Global Step: 234920 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:16:54,951-Speed 128.88 samples/sec Loss 0.3010 Epoch: 23 Global Step: 234925 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:17:15,127-Speed 126.89 samples/sec Loss 0.3775 Epoch: 23 Global Step: 234930 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:17:34,796-Speed 130.16 samples/sec Loss 0.2646 Epoch: 23 Global Step: 234935 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:17:54,484-Speed 130.04 samples/sec Loss 0.2841 Epoch: 23 Global Step: 234940 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:18:14,287-Speed 129.28 samples/sec Loss 0.3416 Epoch: 23 Global Step: 234945 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:18:34,385-Speed 127.38 samples/sec Loss 0.3768 Epoch: 23 Global Step: 234950 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:18:54,264-Speed 128.79 samples/sec Loss 0.3123 Epoch: 23 Global Step: 234955 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:19:13,869-Speed 130.58 samples/sec Loss 0.2947 Epoch: 23 Global Step: 234960 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:19:33,075-Speed 133.30 samples/sec Loss 0.3006 Epoch: 23 Global Step: 234965 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:19:52,933-Speed 128.93 samples/sec Loss 0.3561 Epoch: 23 Global Step: 234970 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:20:13,100-Speed 126.94 samples/sec Loss 0.3368 Epoch: 23 Global Step: 234975 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:20:32,825-Speed 129.79 samples/sec Loss 0.3821 Epoch: 23 Global Step: 234980 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:20:52,526-Speed 129.95 samples/sec Loss 0.2964 Epoch: 23 Global Step: 234985 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:21:12,233-Speed 129.91 samples/sec Loss 0.3491 Epoch: 23 Global Step: 234990 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:21:32,003-Speed 129.49 samples/sec Loss 0.4006 Epoch: 23 Global Step: 234995 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:21:51,857-Speed 128.95 samples/sec Loss 0.3278 Epoch: 23 Global Step: 235000 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:22:11,549-Speed 130.01 samples/sec Loss 0.3103 Epoch: 23 Global Step: 235005 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:22:31,581-Speed 127.81 samples/sec Loss 0.3065 Epoch: 23 Global Step: 235010 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:22:51,185-Speed 130.59 samples/sec Loss 0.3153 Epoch: 23 Global Step: 235015 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:23:11,605-Speed 125.37 samples/sec Loss 0.3173 Epoch: 23 Global Step: 235020 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:23:31,259-Speed 130.26 samples/sec Loss 0.3287 Epoch: 23 Global Step: 235025 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:23:51,036-Speed 129.45 samples/sec Loss 0.2889 Epoch: 23 Global Step: 235030 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:24:10,699-Speed 130.20 samples/sec Loss 0.2810 Epoch: 23 Global Step: 235035 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:24:30,596-Speed 128.67 samples/sec Loss 0.2495 Epoch: 23 Global Step: 235040 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:24:50,522-Speed 128.48 samples/sec Loss 0.3468 Epoch: 23 Global Step: 235045 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:25:10,139-Speed 130.51 samples/sec Loss 0.3286 Epoch: 23 Global Step: 235050 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:25:29,936-Speed 129.33 samples/sec Loss 0.3267 Epoch: 23 Global Step: 235055 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:25:49,542-Speed 130.57 samples/sec Loss 0.3279 Epoch: 23 Global Step: 235060 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:26:09,922-Speed 125.62 samples/sec Loss 0.3828 Epoch: 23 Global Step: 235065 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:26:29,658-Speed 129.72 samples/sec Loss 0.3003 Epoch: 23 Global Step: 235070 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:26:49,437-Speed 129.43 samples/sec Loss 0.4145 Epoch: 23 Global Step: 235075 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:27:08,598-Speed 133.62 samples/sec Loss 0.2823 Epoch: 23 Global Step: 235080 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:27:28,640-Speed 127.74 samples/sec Loss 0.2507 Epoch: 23 Global Step: 235085 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:27:48,365-Speed 129.79 samples/sec Loss 0.3007 Epoch: 23 Global Step: 235090 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:28:08,060-Speed 129.99 samples/sec Loss 0.2526 Epoch: 23 Global Step: 235095 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:28:27,855-Speed 129.33 samples/sec Loss 0.3010 Epoch: 23 Global Step: 235100 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:28:47,493-Speed 130.37 samples/sec Loss 0.3000 Epoch: 23 Global Step: 235105 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:29:07,891-Speed 125.51 samples/sec Loss 0.3639 Epoch: 23 Global Step: 235110 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:29:27,637-Speed 129.65 samples/sec Loss 0.3097 Epoch: 23 Global Step: 235115 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:29:47,392-Speed 129.59 samples/sec Loss 0.3582 Epoch: 23 Global Step: 235120 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:30:07,134-Speed 129.68 samples/sec Loss 0.3065 Epoch: 23 Global Step: 235125 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:30:27,019-Speed 128.74 samples/sec Loss 0.3345 Epoch: 23 Global Step: 235130 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:30:46,725-Speed 129.92 samples/sec Loss 0.3273 Epoch: 23 Global Step: 235135 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:31:06,560-Speed 129.07 samples/sec Loss 0.3215 Epoch: 23 Global Step: 235140 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:31:26,316-Speed 129.59 samples/sec Loss 0.3083 Epoch: 23 Global Step: 235145 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:31:46,078-Speed 129.54 samples/sec Loss 0.3094 Epoch: 23 Global Step: 235150 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:32:06,439-Speed 125.74 samples/sec Loss 0.3125 Epoch: 23 Global Step: 235155 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:32:26,344-Speed 128.62 samples/sec Loss 0.3404 Epoch: 23 Global Step: 235160 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:32:46,032-Speed 130.04 samples/sec Loss 0.2447 Epoch: 23 Global Step: 235165 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:33:05,724-Speed 130.01 samples/sec Loss 0.2792 Epoch: 23 Global Step: 235170 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:33:25,613-Speed 128.72 samples/sec Loss 0.2621 Epoch: 23 Global Step: 235175 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:33:45,366-Speed 129.61 samples/sec Loss 0.3130 Epoch: 23 Global Step: 235180 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:34:05,157-Speed 129.36 samples/sec Loss 0.3715 Epoch: 23 Global Step: 235185 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:34:24,921-Speed 129.54 samples/sec Loss 0.2876 Epoch: 23 Global Step: 235190 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:34:45,530-Speed 124.22 samples/sec Loss 0.3339 Epoch: 23 Global Step: 235195 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:35:05,912-Speed 125.61 samples/sec Loss 0.3258 Epoch: 23 Global Step: 235200 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:35:25,741-Speed 129.11 samples/sec Loss 0.3195 Epoch: 23 Global Step: 235205 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:35:45,510-Speed 129.50 samples/sec Loss 0.3157 Epoch: 23 Global Step: 235210 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:36:05,119-Speed 130.56 samples/sec Loss 0.3128 Epoch: 23 Global Step: 235215 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:36:25,015-Speed 128.68 samples/sec Loss 0.2572 Epoch: 23 Global Step: 235220 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:36:44,745-Speed 129.76 samples/sec Loss 0.3570 Epoch: 23 Global Step: 235225 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:37:04,477-Speed 129.75 samples/sec Loss 0.2812 Epoch: 23 Global Step: 235230 Fp16 Grad Scale: 8192 Required: 17 hours Training: 2021-06-18 12:37:23,783-Speed 132.61 samples/sec Loss 0.3226 Epoch: 23 Global Step: 235235 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:37:43,410-Speed 130.44 samples/sec Loss 0.3602 Epoch: 23 Global Step: 235240 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:38:03,599-Speed 126.81 samples/sec Loss 0.3228 Epoch: 23 Global Step: 235245 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:38:23,400-Speed 129.29 samples/sec Loss 0.3774 Epoch: 23 Global Step: 235250 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:38:43,173-Speed 129.48 samples/sec Loss 0.2615 Epoch: 23 Global Step: 235255 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:39:03,014-Speed 129.03 samples/sec Loss 0.3473 Epoch: 23 Global Step: 235260 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:39:22,561-Speed 130.97 samples/sec Loss 0.3276 Epoch: 23 Global Step: 235265 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:39:42,509-Speed 128.34 samples/sec Loss 0.2807 Epoch: 23 Global Step: 235270 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:40:02,458-Speed 128.33 samples/sec Loss 0.3471 Epoch: 23 Global Step: 235275 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:40:22,073-Speed 130.52 samples/sec Loss 0.3508 Epoch: 23 Global Step: 235280 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:40:41,860-Speed 129.38 samples/sec Loss 0.3268 Epoch: 23 Global Step: 235285 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:41:01,899-Speed 127.76 samples/sec Loss 0.2593 Epoch: 23 Global Step: 235290 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:41:21,726-Speed 129.12 samples/sec Loss 0.2915 Epoch: 23 Global Step: 235295 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:41:41,502-Speed 129.45 samples/sec Loss 0.3210 Epoch: 23 Global Step: 235300 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:42:01,213-Speed 129.89 samples/sec Loss 0.3230 Epoch: 23 Global Step: 235305 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:42:21,093-Speed 128.78 samples/sec Loss 0.3001 Epoch: 23 Global Step: 235310 Fp16 Grad Scale: 4096 Required: 17 hours Training: 2021-06-18 12:42:41,039-Speed 128.35 samples/sec Loss 0.3170 Epoch: 23 Global Step: 235315 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:43:00,959-Speed 128.52 samples/sec Loss 0.3257 Epoch: 23 Global Step: 235320 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:43:20,576-Speed 130.50 samples/sec Loss 0.3117 Epoch: 23 Global Step: 235325 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:43:40,437-Speed 128.90 samples/sec Loss 0.2738 Epoch: 23 Global Step: 235330 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:44:00,371-Speed 128.43 samples/sec Loss 0.3673 Epoch: 23 Global Step: 235335 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:44:20,131-Speed 129.56 samples/sec Loss 0.2564 Epoch: 23 Global Step: 235340 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:44:39,968-Speed 129.06 samples/sec Loss 0.2950 Epoch: 23 Global Step: 235345 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:44:59,686-Speed 129.84 samples/sec Loss 0.3458 Epoch: 23 Global Step: 235350 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:45:19,504-Speed 129.18 samples/sec Loss 0.3776 Epoch: 23 Global Step: 235355 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:45:39,449-Speed 128.36 samples/sec Loss 0.3253 Epoch: 23 Global Step: 235360 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:45:59,309-Speed 128.91 samples/sec Loss 0.3460 Epoch: 23 Global Step: 235365 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:46:18,992-Speed 130.07 samples/sec Loss 0.2790 Epoch: 23 Global Step: 235370 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:46:38,898-Speed 128.61 samples/sec Loss 0.3361 Epoch: 23 Global Step: 235375 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:46:58,889-Speed 128.06 samples/sec Loss 0.2824 Epoch: 23 Global Step: 235380 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:47:18,806-Speed 128.54 samples/sec Loss 0.3029 Epoch: 23 Global Step: 235385 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:47:38,544-Speed 129.71 samples/sec Loss 0.4094 Epoch: 23 Global Step: 235390 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:47:58,471-Speed 128.48 samples/sec Loss 0.3378 Epoch: 23 Global Step: 235395 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:48:18,327-Speed 128.94 samples/sec Loss 0.3374 Epoch: 23 Global Step: 235400 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:48:38,421-Speed 127.40 samples/sec Loss 0.3382 Epoch: 23 Global Step: 235405 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:48:58,153-Speed 129.75 samples/sec Loss 0.3578 Epoch: 23 Global Step: 235410 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:49:17,998-Speed 129.01 samples/sec Loss 0.3096 Epoch: 23 Global Step: 235415 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:49:37,863-Speed 128.87 samples/sec Loss 0.3333 Epoch: 23 Global Step: 235420 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:49:57,872-Speed 127.95 samples/sec Loss 0.3369 Epoch: 23 Global Step: 235425 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:50:17,834-Speed 128.25 samples/sec Loss 0.2789 Epoch: 23 Global Step: 235430 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:50:37,630-Speed 129.33 samples/sec Loss 0.3381 Epoch: 23 Global Step: 235435 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:50:57,525-Speed 128.68 samples/sec Loss 0.3513 Epoch: 23 Global Step: 235440 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:51:17,228-Speed 129.94 samples/sec Loss 0.3172 Epoch: 23 Global Step: 235445 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:51:37,317-Speed 127.44 samples/sec Loss 0.3221 Epoch: 23 Global Step: 235450 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:51:57,075-Speed 129.57 samples/sec Loss 0.3347 Epoch: 23 Global Step: 235455 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:52:16,914-Speed 129.05 samples/sec Loss 0.3678 Epoch: 23 Global Step: 235460 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:52:36,757-Speed 129.01 samples/sec Loss 0.3393 Epoch: 23 Global Step: 235465 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:52:56,736-Speed 128.14 samples/sec Loss 0.2528 Epoch: 23 Global Step: 235470 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:53:16,652-Speed 128.54 samples/sec Loss 0.2700 Epoch: 23 Global Step: 235475 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:53:36,344-Speed 130.01 samples/sec Loss 0.2652 Epoch: 23 Global Step: 235480 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 12:53:55,756-Speed 131.88 samples/sec Loss 0.2790 Epoch: 23 Global Step: 235485 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:54:15,468-Speed 129.88 samples/sec Loss 0.3075 Epoch: 23 Global Step: 235490 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:54:35,543-Speed 127.53 samples/sec Loss 0.2993 Epoch: 23 Global Step: 235495 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:54:55,348-Speed 129.27 samples/sec Loss 0.3039 Epoch: 23 Global Step: 235500 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:55:15,325-Speed 128.15 samples/sec Loss 0.3439 Epoch: 23 Global Step: 235505 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:55:34,932-Speed 130.57 samples/sec Loss 0.2786 Epoch: 23 Global Step: 235510 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:55:54,702-Speed 129.50 samples/sec Loss 0.3436 Epoch: 23 Global Step: 235515 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:56:14,491-Speed 129.37 samples/sec Loss 0.3574 Epoch: 23 Global Step: 235520 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:56:34,560-Speed 127.57 samples/sec Loss 0.3172 Epoch: 23 Global Step: 235525 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:56:54,352-Speed 129.35 samples/sec Loss 0.3566 Epoch: 23 Global Step: 235530 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:57:14,053-Speed 129.95 samples/sec Loss 0.2979 Epoch: 23 Global Step: 235535 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:57:34,071-Speed 127.89 samples/sec Loss 0.2745 Epoch: 23 Global Step: 235540 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:57:53,850-Speed 129.44 samples/sec Loss 0.3312 Epoch: 23 Global Step: 235545 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:58:13,762-Speed 128.58 samples/sec Loss 0.2696 Epoch: 23 Global Step: 235550 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:58:33,393-Speed 130.41 samples/sec Loss 0.3324 Epoch: 23 Global Step: 235555 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:58:53,181-Speed 129.38 samples/sec Loss 0.3170 Epoch: 23 Global Step: 235560 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:59:12,986-Speed 129.27 samples/sec Loss 0.3176 Epoch: 23 Global Step: 235565 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:59:33,102-Speed 127.27 samples/sec Loss 0.2818 Epoch: 23 Global Step: 235570 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 12:59:52,899-Speed 129.32 samples/sec Loss 0.4006 Epoch: 23 Global Step: 235575 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:00:12,622-Speed 129.80 samples/sec Loss 0.3384 Epoch: 23 Global Step: 235580 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:00:32,681-Speed 127.63 samples/sec Loss 0.3511 Epoch: 23 Global Step: 235585 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:00:52,431-Speed 129.63 samples/sec Loss 0.3233 Epoch: 23 Global Step: 235590 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:01:12,296-Speed 128.88 samples/sec Loss 0.3170 Epoch: 23 Global Step: 235595 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:01:31,906-Speed 130.55 samples/sec Loss 0.3033 Epoch: 23 Global Step: 235600 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:01:51,797-Speed 128.71 samples/sec Loss 0.2322 Epoch: 23 Global Step: 235605 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:02:11,544-Speed 129.65 samples/sec Loss 0.3320 Epoch: 23 Global Step: 235610 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:02:31,544-Speed 128.01 samples/sec Loss 0.3411 Epoch: 23 Global Step: 235615 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:02:51,308-Speed 129.54 samples/sec Loss 0.3855 Epoch: 23 Global Step: 235620 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:03:11,034-Speed 129.78 samples/sec Loss 0.3326 Epoch: 23 Global Step: 235625 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:03:31,101-Speed 127.58 samples/sec Loss 0.3260 Epoch: 23 Global Step: 235630 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:03:50,982-Speed 128.77 samples/sec Loss 0.3357 Epoch: 23 Global Step: 235635 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:04:10,728-Speed 129.65 samples/sec Loss 0.3374 Epoch: 23 Global Step: 235640 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:04:30,361-Speed 130.40 samples/sec Loss 0.2623 Epoch: 23 Global Step: 235645 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:04:50,275-Speed 128.56 samples/sec Loss 0.2595 Epoch: 23 Global Step: 235650 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:05:09,584-Speed 132.58 samples/sec Loss 0.2635 Epoch: 23 Global Step: 235655 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:05:29,786-Speed 126.73 samples/sec Loss 0.3356 Epoch: 23 Global Step: 235660 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:05:49,451-Speed 130.19 samples/sec Loss 0.3138 Epoch: 23 Global Step: 235665 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:06:09,183-Speed 129.75 samples/sec Loss 0.3143 Epoch: 23 Global Step: 235670 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:06:29,285-Speed 127.35 samples/sec Loss 0.2464 Epoch: 23 Global Step: 235675 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:06:49,186-Speed 128.64 samples/sec Loss 0.3212 Epoch: 23 Global Step: 235680 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:07:08,970-Speed 129.40 samples/sec Loss 0.3829 Epoch: 23 Global Step: 235685 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:07:28,624-Speed 130.26 samples/sec Loss 0.3160 Epoch: 23 Global Step: 235690 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:07:48,472-Speed 128.99 samples/sec Loss 0.3103 Epoch: 23 Global Step: 235695 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:08:08,190-Speed 129.84 samples/sec Loss 0.3145 Epoch: 23 Global Step: 235700 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:08:28,366-Speed 126.89 samples/sec Loss 0.3135 Epoch: 23 Global Step: 235705 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:08:48,047-Speed 130.08 samples/sec Loss 0.3878 Epoch: 23 Global Step: 235710 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:09:07,761-Speed 129.87 samples/sec Loss 0.3087 Epoch: 23 Global Step: 235715 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:09:27,857-Speed 127.40 samples/sec Loss 0.3818 Epoch: 23 Global Step: 235720 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:09:47,574-Speed 129.84 samples/sec Loss 0.2703 Epoch: 23 Global Step: 235725 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:10:07,377-Speed 129.28 samples/sec Loss 0.2812 Epoch: 23 Global Step: 235730 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:10:27,093-Speed 129.85 samples/sec Loss 0.2894 Epoch: 23 Global Step: 235735 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:10:46,944-Speed 128.97 samples/sec Loss 0.2445 Epoch: 23 Global Step: 235740 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:11:06,682-Speed 129.70 samples/sec Loss 0.3689 Epoch: 23 Global Step: 235745 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:11:26,778-Speed 127.39 samples/sec Loss 0.2845 Epoch: 23 Global Step: 235750 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:11:46,468-Speed 130.03 samples/sec Loss 0.3169 Epoch: 23 Global Step: 235755 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:12:06,208-Speed 129.69 samples/sec Loss 0.2876 Epoch: 23 Global Step: 235760 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:12:26,262-Speed 127.66 samples/sec Loss 0.3038 Epoch: 23 Global Step: 235765 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:12:46,067-Speed 129.26 samples/sec Loss 0.3199 Epoch: 23 Global Step: 235770 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:13:05,795-Speed 129.77 samples/sec Loss 0.4340 Epoch: 23 Global Step: 235775 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:13:25,769-Speed 128.18 samples/sec Loss 0.2846 Epoch: 23 Global Step: 235780 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:13:45,498-Speed 129.76 samples/sec Loss 0.3267 Epoch: 23 Global Step: 235785 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:14:05,315-Speed 129.19 samples/sec Loss 0.3078 Epoch: 23 Global Step: 235790 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:14:25,373-Speed 127.63 samples/sec Loss 0.3072 Epoch: 23 Global Step: 235795 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:14:45,165-Speed 129.36 samples/sec Loss 0.3755 Epoch: 23 Global Step: 235800 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:15:05,003-Speed 129.05 samples/sec Loss 0.3367 Epoch: 23 Global Step: 235805 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:15:24,970-Speed 128.22 samples/sec Loss 0.3559 Epoch: 23 Global Step: 235810 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:15:44,717-Speed 129.64 samples/sec Loss 0.3072 Epoch: 23 Global Step: 235815 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:16:03,981-Speed 132.89 samples/sec Loss 0.3306 Epoch: 23 Global Step: 235820 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:16:23,932-Speed 128.32 samples/sec Loss 0.3777 Epoch: 23 Global Step: 235825 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:16:43,660-Speed 129.77 samples/sec Loss 0.3356 Epoch: 23 Global Step: 235830 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:17:03,543-Speed 128.76 samples/sec Loss 0.2995 Epoch: 23 Global Step: 235835 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:17:23,288-Speed 129.66 samples/sec Loss 0.2660 Epoch: 23 Global Step: 235840 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:17:43,142-Speed 128.95 samples/sec Loss 0.2991 Epoch: 23 Global Step: 235845 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:18:02,700-Speed 130.90 samples/sec Loss 0.3328 Epoch: 23 Global Step: 235850 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:18:22,249-Speed 130.96 samples/sec Loss 0.3142 Epoch: 23 Global Step: 235855 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:18:42,028-Speed 129.43 samples/sec Loss 0.2557 Epoch: 23 Global Step: 235860 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:19:01,599-Speed 130.81 samples/sec Loss 0.2992 Epoch: 23 Global Step: 235865 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:19:21,443-Speed 129.02 samples/sec Loss 0.3213 Epoch: 23 Global Step: 235870 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:19:41,098-Speed 130.25 samples/sec Loss 0.3271 Epoch: 23 Global Step: 235875 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:20:01,173-Speed 127.53 samples/sec Loss 0.2970 Epoch: 23 Global Step: 235880 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:20:20,860-Speed 130.04 samples/sec Loss 0.2461 Epoch: 23 Global Step: 235885 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:20:40,472-Speed 130.54 samples/sec Loss 0.2849 Epoch: 23 Global Step: 235890 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:21:00,238-Speed 129.52 samples/sec Loss 0.3194 Epoch: 23 Global Step: 235895 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:21:19,771-Speed 131.06 samples/sec Loss 0.3404 Epoch: 23 Global Step: 235900 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:21:39,646-Speed 128.81 samples/sec Loss 0.2826 Epoch: 23 Global Step: 235905 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:21:59,135-Speed 131.37 samples/sec Loss 0.2677 Epoch: 23 Global Step: 235910 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:22:18,996-Speed 128.90 samples/sec Loss 0.3634 Epoch: 23 Global Step: 235915 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:22:38,532-Speed 131.05 samples/sec Loss 0.3871 Epoch: 23 Global Step: 235920 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:22:58,722-Speed 126.80 samples/sec Loss 0.3221 Epoch: 23 Global Step: 235925 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:23:18,326-Speed 130.59 samples/sec Loss 0.3053 Epoch: 23 Global Step: 235930 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:23:37,926-Speed 130.62 samples/sec Loss 0.3894 Epoch: 23 Global Step: 235935 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:23:57,646-Speed 129.82 samples/sec Loss 0.3534 Epoch: 23 Global Step: 235940 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:24:17,260-Speed 130.53 samples/sec Loss 0.3624 Epoch: 23 Global Step: 235945 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:24:37,155-Speed 128.68 samples/sec Loss 0.2422 Epoch: 23 Global Step: 235950 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:24:56,698-Speed 131.00 samples/sec Loss 0.2944 Epoch: 23 Global Step: 235955 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:25:16,512-Speed 129.21 samples/sec Loss 0.3814 Epoch: 23 Global Step: 235960 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:25:36,053-Speed 131.02 samples/sec Loss 0.2982 Epoch: 23 Global Step: 235965 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:25:56,108-Speed 127.65 samples/sec Loss 0.3316 Epoch: 23 Global Step: 235970 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:26:15,837-Speed 129.77 samples/sec Loss 0.3247 Epoch: 23 Global Step: 235975 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:26:35,358-Speed 131.15 samples/sec Loss 0.3030 Epoch: 23 Global Step: 235980 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:26:55,270-Speed 128.57 samples/sec Loss 0.3034 Epoch: 23 Global Step: 235985 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:27:14,760-Speed 131.36 samples/sec Loss 0.3200 Epoch: 23 Global Step: 235990 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:27:34,722-Speed 128.25 samples/sec Loss 0.2718 Epoch: 23 Global Step: 235995 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:27:54,269-Speed 130.98 samples/sec Loss 0.2965 Epoch: 23 Global Step: 236000 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:28:13,948-Speed 130.09 samples/sec Loss 0.4146 Epoch: 23 Global Step: 236005 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:28:33,525-Speed 130.77 samples/sec Loss 0.3236 Epoch: 23 Global Step: 236010 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:28:53,550-Speed 127.84 samples/sec Loss 0.3156 Epoch: 23 Global Step: 236015 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:29:13,314-Speed 129.54 samples/sec Loss 0.3588 Epoch: 23 Global Step: 236020 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:29:32,858-Speed 130.99 samples/sec Loss 0.2820 Epoch: 23 Global Step: 236025 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:29:52,696-Speed 129.05 samples/sec Loss 0.3516 Epoch: 23 Global Step: 236030 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:30:12,120-Speed 131.80 samples/sec Loss 0.2405 Epoch: 23 Global Step: 236035 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:30:31,938-Speed 129.18 samples/sec Loss 0.3766 Epoch: 23 Global Step: 236040 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:30:51,526-Speed 130.70 samples/sec Loss 0.3053 Epoch: 23 Global Step: 236045 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:31:11,296-Speed 129.50 samples/sec Loss 0.3652 Epoch: 23 Global Step: 236050 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:31:30,885-Speed 130.69 samples/sec Loss 0.3175 Epoch: 23 Global Step: 236055 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:31:50,877-Speed 128.06 samples/sec Loss 0.2647 Epoch: 23 Global Step: 236060 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:32:10,653-Speed 129.45 samples/sec Loss 0.3047 Epoch: 23 Global Step: 236065 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:32:30,497-Speed 129.01 samples/sec Loss 0.2934 Epoch: 23 Global Step: 236070 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:32:50,255-Speed 129.58 samples/sec Loss 0.2336 Epoch: 23 Global Step: 236075 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:33:09,782-Speed 131.11 samples/sec Loss 0.3171 Epoch: 23 Global Step: 236080 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:33:29,671-Speed 128.72 samples/sec Loss 0.3141 Epoch: 23 Global Step: 236085 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:33:49,187-Speed 131.19 samples/sec Loss 0.3304 Epoch: 23 Global Step: 236090 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:34:08,997-Speed 129.23 samples/sec Loss 0.2743 Epoch: 23 Global Step: 236095 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:34:28,542-Speed 130.99 samples/sec Loss 0.3702 Epoch: 23 Global Step: 236100 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:34:49,181-Speed 124.04 samples/sec Loss 0.3355 Epoch: 23 Global Step: 236105 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:35:08,801-Speed 130.49 samples/sec Loss 0.3050 Epoch: 23 Global Step: 236110 Fp16 Grad Scale: 8192 Required: 16 hours Training: 2021-06-18 13:35:28,296-Speed 131.32 samples/sec Loss 0.3436 Epoch: 23 Global Step: 236115 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:35:47,786-Speed 131.36 samples/sec Loss 0.2855 Epoch: 23 Global Step: 236120 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:36:07,417-Speed 130.42 samples/sec Loss 0.3162 Epoch: 23 Global Step: 236125 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:36:27,018-Speed 130.61 samples/sec Loss 0.3316 Epoch: 23 Global Step: 236130 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:36:46,646-Speed 130.43 samples/sec Loss 0.3185 Epoch: 23 Global Step: 236135 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:37:06,261-Speed 130.52 samples/sec Loss 0.3317 Epoch: 23 Global Step: 236140 Fp16 Grad Scale: 4096 Required: 16 hours Training: 2021-06-18 13:37:25,884-Speed 130.47 samples/sec Loss 0.3133 Epoch: 23 Global Step: 236145 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:37:45,752-Speed 128.86 samples/sec Loss 0.3114 Epoch: 23 Global Step: 236150 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:38:05,462-Speed 129.88 samples/sec Loss 0.3463 Epoch: 23 Global Step: 236155 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:38:25,018-Speed 130.91 samples/sec Loss 0.2907 Epoch: 23 Global Step: 236160 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:38:44,890-Speed 128.83 samples/sec Loss 0.2739 Epoch: 23 Global Step: 236165 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:39:04,559-Speed 130.16 samples/sec Loss 0.3122 Epoch: 23 Global Step: 236170 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:39:24,165-Speed 130.58 samples/sec Loss 0.3633 Epoch: 23 Global Step: 236175 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:39:43,874-Speed 129.90 samples/sec Loss 0.2671 Epoch: 23 Global Step: 236180 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:40:03,421-Speed 130.97 samples/sec Loss 0.2938 Epoch: 23 Global Step: 236185 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:40:23,344-Speed 128.51 samples/sec Loss 0.3297 Epoch: 23 Global Step: 236190 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:40:43,480-Speed 127.14 samples/sec Loss 0.3321 Epoch: 23 Global Step: 236195 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:41:03,089-Speed 130.55 samples/sec Loss 0.2901 Epoch: 23 Global Step: 236200 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:41:22,795-Speed 129.92 samples/sec Loss 0.2982 Epoch: 23 Global Step: 236205 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:41:42,641-Speed 129.00 samples/sec Loss 0.3544 Epoch: 23 Global Step: 236210 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:42:02,585-Speed 128.36 samples/sec Loss 0.3595 Epoch: 23 Global Step: 236215 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:42:22,151-Speed 130.85 samples/sec Loss 0.2854 Epoch: 23 Global Step: 236220 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:42:41,880-Speed 129.76 samples/sec Loss 0.3005 Epoch: 23 Global Step: 236225 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:43:01,040-Speed 133.62 samples/sec Loss 0.3350 Epoch: 23 Global Step: 236230 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:43:20,836-Speed 129.33 samples/sec Loss 0.3684 Epoch: 23 Global Step: 236235 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:43:40,628-Speed 129.35 samples/sec Loss 0.3209 Epoch: 23 Global Step: 236240 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:44:00,258-Speed 130.42 samples/sec Loss 0.3792 Epoch: 23 Global Step: 236245 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:44:19,885-Speed 130.44 samples/sec Loss 0.2758 Epoch: 23 Global Step: 236250 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:44:39,797-Speed 128.57 samples/sec Loss 0.4300 Epoch: 23 Global Step: 236255 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:44:59,612-Speed 129.20 samples/sec Loss 0.3031 Epoch: 23 Global Step: 236260 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:45:19,104-Speed 131.35 samples/sec Loss 0.3058 Epoch: 23 Global Step: 236265 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:45:38,799-Speed 129.99 samples/sec Loss 0.2680 Epoch: 23 Global Step: 236270 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:45:58,824-Speed 127.85 samples/sec Loss 0.3016 Epoch: 23 Global Step: 236275 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:46:18,569-Speed 129.66 samples/sec Loss 0.2618 Epoch: 23 Global Step: 236280 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:46:38,366-Speed 129.32 samples/sec Loss 0.2694 Epoch: 23 Global Step: 236285 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:46:58,024-Speed 130.23 samples/sec Loss 0.2966 Epoch: 23 Global Step: 236290 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:47:17,672-Speed 130.30 samples/sec Loss 0.3287 Epoch: 23 Global Step: 236295 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:47:37,557-Speed 128.75 samples/sec Loss 0.3534 Epoch: 23 Global Step: 236300 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:47:57,256-Speed 129.96 samples/sec Loss 0.2866 Epoch: 23 Global Step: 236305 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:48:16,882-Speed 130.45 samples/sec Loss 0.3602 Epoch: 23 Global Step: 236310 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:48:36,655-Speed 129.48 samples/sec Loss 0.3591 Epoch: 23 Global Step: 236315 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:48:56,213-Speed 130.90 samples/sec Loss 0.3426 Epoch: 23 Global Step: 236320 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:49:15,943-Speed 129.75 samples/sec Loss 0.3045 Epoch: 23 Global Step: 236325 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:49:35,727-Speed 129.41 samples/sec Loss 0.3990 Epoch: 23 Global Step: 236330 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:49:55,368-Speed 130.34 samples/sec Loss 0.3398 Epoch: 23 Global Step: 236335 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:50:14,964-Speed 130.64 samples/sec Loss 0.2765 Epoch: 23 Global Step: 236340 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:50:34,723-Speed 129.57 samples/sec Loss 0.3416 Epoch: 23 Global Step: 236345 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:50:54,593-Speed 128.84 samples/sec Loss 0.3094 Epoch: 23 Global Step: 236350 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:51:14,146-Speed 130.93 samples/sec Loss 0.3387 Epoch: 23 Global Step: 236355 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:51:33,894-Speed 129.64 samples/sec Loss 0.3158 Epoch: 23 Global Step: 236360 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:51:53,470-Speed 130.78 samples/sec Loss 0.2664 Epoch: 23 Global Step: 236365 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:52:13,191-Speed 129.82 samples/sec Loss 0.2560 Epoch: 23 Global Step: 236370 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:52:33,033-Speed 129.03 samples/sec Loss 0.2488 Epoch: 23 Global Step: 236375 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:52:52,743-Speed 129.89 samples/sec Loss 0.2909 Epoch: 23 Global Step: 236380 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:53:12,406-Speed 130.20 samples/sec Loss 0.3012 Epoch: 23 Global Step: 236385 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:53:32,227-Speed 129.16 samples/sec Loss 0.3904 Epoch: 23 Global Step: 236390 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:53:52,155-Speed 128.47 samples/sec Loss 0.2717 Epoch: 23 Global Step: 236395 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:54:11,684-Speed 131.10 samples/sec Loss 0.3656 Epoch: 23 Global Step: 236400 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:54:31,389-Speed 129.92 samples/sec Loss 0.3137 Epoch: 23 Global Step: 236405 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 13:54:50,536-Speed 133.71 samples/sec Loss 0.2600 Epoch: 23 Global Step: 236410 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:55:10,147-Speed 130.55 samples/sec Loss 0.3287 Epoch: 23 Global Step: 236415 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:55:29,946-Speed 129.30 samples/sec Loss 0.2579 Epoch: 23 Global Step: 236420 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:55:49,486-Speed 131.02 samples/sec Loss 0.2954 Epoch: 23 Global Step: 236425 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:56:09,196-Speed 129.88 samples/sec Loss 0.3544 Epoch: 23 Global Step: 236430 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:56:29,584-Speed 125.58 samples/sec Loss 0.2696 Epoch: 23 Global Step: 236435 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:56:49,100-Speed 131.18 samples/sec Loss 0.3319 Epoch: 23 Global Step: 236440 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:57:09,131-Speed 127.81 samples/sec Loss 0.3077 Epoch: 23 Global Step: 236445 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:57:28,890-Speed 129.57 samples/sec Loss 0.2846 Epoch: 23 Global Step: 236450 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:57:48,751-Speed 128.90 samples/sec Loss 0.3452 Epoch: 23 Global Step: 236455 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:58:08,529-Speed 129.44 samples/sec Loss 0.3311 Epoch: 23 Global Step: 236460 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:58:28,557-Speed 127.83 samples/sec Loss 0.3155 Epoch: 23 Global Step: 236465 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:58:48,272-Speed 129.86 samples/sec Loss 0.3458 Epoch: 23 Global Step: 236470 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:59:07,940-Speed 130.17 samples/sec Loss 0.3650 Epoch: 23 Global Step: 236475 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:59:28,169-Speed 126.55 samples/sec Loss 0.2792 Epoch: 23 Global Step: 236480 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 13:59:47,802-Speed 130.40 samples/sec Loss 0.2839 Epoch: 23 Global Step: 236485 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:00:07,884-Speed 127.49 samples/sec Loss 0.2631 Epoch: 23 Global Step: 236490 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:00:27,538-Speed 130.26 samples/sec Loss 0.3700 Epoch: 23 Global Step: 236495 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:00:47,393-Speed 128.94 samples/sec Loss 0.3419 Epoch: 23 Global Step: 236500 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:01:07,109-Speed 129.85 samples/sec Loss 0.3027 Epoch: 23 Global Step: 236505 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:01:27,041-Speed 128.44 samples/sec Loss 0.3006 Epoch: 23 Global Step: 236510 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:01:46,765-Speed 129.80 samples/sec Loss 0.3427 Epoch: 23 Global Step: 236515 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:02:06,421-Speed 130.25 samples/sec Loss 0.3311 Epoch: 23 Global Step: 236520 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:02:26,666-Speed 126.46 samples/sec Loss 0.2854 Epoch: 23 Global Step: 236525 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:02:46,394-Speed 129.77 samples/sec Loss 0.3171 Epoch: 23 Global Step: 236530 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:03:06,475-Speed 127.49 samples/sec Loss 0.2642 Epoch: 23 Global Step: 236535 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:03:26,140-Speed 130.19 samples/sec Loss 0.3266 Epoch: 23 Global Step: 236540 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:03:45,991-Speed 128.96 samples/sec Loss 0.3026 Epoch: 23 Global Step: 236545 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:04:05,327-Speed 132.41 samples/sec Loss 0.3046 Epoch: 23 Global Step: 236550 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:04:25,171-Speed 129.01 samples/sec Loss 0.3409 Epoch: 23 Global Step: 236555 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:04:44,883-Speed 129.87 samples/sec Loss 0.3666 Epoch: 23 Global Step: 236560 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:05:04,543-Speed 130.23 samples/sec Loss 0.3356 Epoch: 23 Global Step: 236565 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:05:24,389-Speed 129.00 samples/sec Loss 0.2831 Epoch: 23 Global Step: 236570 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:05:45,335-Speed 122.23 samples/sec Loss 0.2581 Epoch: 23 Global Step: 236575 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:06:05,409-Speed 127.53 samples/sec Loss 0.3905 Epoch: 23 Global Step: 236580 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:06:24,924-Speed 131.19 samples/sec Loss 0.2860 Epoch: 23 Global Step: 236585 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:06:44,446-Speed 131.14 samples/sec Loss 0.3526 Epoch: 23 Global Step: 236590 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:07:04,359-Speed 128.56 samples/sec Loss 0.3014 Epoch: 23 Global Step: 236595 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:07:24,268-Speed 128.59 samples/sec Loss 0.2553 Epoch: 23 Global Step: 236600 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:07:43,688-Speed 131.83 samples/sec Loss 0.2667 Epoch: 23 Global Step: 236605 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:08:03,350-Speed 130.21 samples/sec Loss 0.3288 Epoch: 23 Global Step: 236610 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:08:22,933-Speed 130.73 samples/sec Loss 0.2861 Epoch: 23 Global Step: 236615 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:08:42,855-Speed 128.50 samples/sec Loss 0.2606 Epoch: 23 Global Step: 236620 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:09:02,647-Speed 129.36 samples/sec Loss 0.2767 Epoch: 23 Global Step: 236625 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:09:22,303-Speed 130.25 samples/sec Loss 0.3301 Epoch: 23 Global Step: 236630 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:09:41,880-Speed 130.77 samples/sec Loss 0.3254 Epoch: 23 Global Step: 236635 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:10:01,662-Speed 129.41 samples/sec Loss 0.3466 Epoch: 23 Global Step: 236640 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:10:21,575-Speed 128.57 samples/sec Loss 0.3010 Epoch: 23 Global Step: 236645 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:10:41,088-Speed 131.20 samples/sec Loss 0.3478 Epoch: 23 Global Step: 236650 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:11:00,790-Speed 129.94 samples/sec Loss 0.3496 Epoch: 23 Global Step: 236655 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:11:20,374-Speed 130.73 samples/sec Loss 0.3334 Epoch: 23 Global Step: 236660 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:11:40,186-Speed 129.22 samples/sec Loss 0.3122 Epoch: 23 Global Step: 236665 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:11:59,993-Speed 129.26 samples/sec Loss 0.3212 Epoch: 23 Global Step: 236670 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:12:19,685-Speed 130.01 samples/sec Loss 0.3250 Epoch: 23 Global Step: 236675 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:12:39,265-Speed 130.75 samples/sec Loss 0.3658 Epoch: 23 Global Step: 236680 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:12:59,053-Speed 129.38 samples/sec Loss 0.3141 Epoch: 23 Global Step: 236685 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:13:18,946-Speed 128.70 samples/sec Loss 0.3015 Epoch: 23 Global Step: 236690 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:13:38,468-Speed 131.14 samples/sec Loss 0.3030 Epoch: 23 Global Step: 236695 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:13:58,276-Speed 129.25 samples/sec Loss 0.2892 Epoch: 23 Global Step: 236700 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:14:17,790-Speed 131.20 samples/sec Loss 0.2776 Epoch: 23 Global Step: 236705 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:14:37,628-Speed 129.05 samples/sec Loss 0.2837 Epoch: 23 Global Step: 236710 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:14:57,580-Speed 128.31 samples/sec Loss 0.3641 Epoch: 23 Global Step: 236715 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:15:17,531-Speed 128.32 samples/sec Loss 0.2928 Epoch: 23 Global Step: 236720 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:15:36,967-Speed 131.72 samples/sec Loss 0.2953 Epoch: 23 Global Step: 236725 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:15:56,751-Speed 129.40 samples/sec Loss 0.3169 Epoch: 23 Global Step: 236730 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:16:16,461-Speed 129.89 samples/sec Loss 0.3320 Epoch: 23 Global Step: 236735 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:16:35,998-Speed 131.04 samples/sec Loss 0.3832 Epoch: 23 Global Step: 236740 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:16:55,804-Speed 129.26 samples/sec Loss 0.3204 Epoch: 23 Global Step: 236745 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:17:15,242-Speed 131.71 samples/sec Loss 0.3130 Epoch: 23 Global Step: 236750 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:17:35,219-Speed 128.16 samples/sec Loss 0.3414 Epoch: 23 Global Step: 236755 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:17:54,996-Speed 129.45 samples/sec Loss 0.2849 Epoch: 23 Global Step: 236760 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:18:14,821-Speed 129.14 samples/sec Loss 0.3181 Epoch: 23 Global Step: 236765 Fp16 Grad Scale: 8192 Required: 15 hours Training: 2021-06-18 14:18:33,816-Speed 134.77 samples/sec Loss 0.3131 Epoch: 23 Global Step: 236770 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:18:53,595-Speed 129.44 samples/sec Loss 0.2421 Epoch: 23 Global Step: 236775 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:19:13,342-Speed 129.65 samples/sec Loss 0.2733 Epoch: 23 Global Step: 236780 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:19:32,960-Speed 130.50 samples/sec Loss 0.3398 Epoch: 23 Global Step: 236785 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:19:52,677-Speed 129.84 samples/sec Loss 0.3543 Epoch: 23 Global Step: 236790 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:20:12,170-Speed 131.33 samples/sec Loss 0.3098 Epoch: 23 Global Step: 236795 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:20:32,065-Speed 128.69 samples/sec Loss 0.2769 Epoch: 23 Global Step: 236800 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:20:51,942-Speed 128.80 samples/sec Loss 0.3237 Epoch: 23 Global Step: 236805 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:21:11,726-Speed 129.40 samples/sec Loss 0.3749 Epoch: 23 Global Step: 236810 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:21:31,155-Speed 131.77 samples/sec Loss 0.3066 Epoch: 23 Global Step: 236815 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:21:50,944-Speed 129.37 samples/sec Loss 0.4053 Epoch: 23 Global Step: 236820 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:22:10,721-Speed 129.45 samples/sec Loss 0.2959 Epoch: 23 Global Step: 236825 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:22:30,316-Speed 130.65 samples/sec Loss 0.3532 Epoch: 23 Global Step: 236830 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:22:49,953-Speed 130.37 samples/sec Loss 0.2449 Epoch: 23 Global Step: 236835 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:23:09,431-Speed 131.44 samples/sec Loss 0.2866 Epoch: 23 Global Step: 236840 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:23:29,296-Speed 128.88 samples/sec Loss 0.2905 Epoch: 23 Global Step: 236845 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:23:49,283-Speed 128.09 samples/sec Loss 0.3104 Epoch: 23 Global Step: 236850 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:24:08,374-Speed 134.10 samples/sec Loss 0.4113 Epoch: 23 Global Step: 236855 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:24:28,372-Speed 128.02 samples/sec Loss 0.3317 Epoch: 23 Global Step: 236860 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:24:48,033-Speed 130.21 samples/sec Loss 0.3261 Epoch: 23 Global Step: 236865 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:25:07,780-Speed 129.64 samples/sec Loss 0.3625 Epoch: 23 Global Step: 236870 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:25:27,792-Speed 127.93 samples/sec Loss 0.3473 Epoch: 23 Global Step: 236875 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:25:47,444-Speed 130.27 samples/sec Loss 0.3026 Epoch: 23 Global Step: 236880 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:26:07,236-Speed 129.35 samples/sec Loss 0.2622 Epoch: 23 Global Step: 236885 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:26:26,862-Speed 130.45 samples/sec Loss 0.2989 Epoch: 23 Global Step: 236890 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:26:46,830-Speed 128.22 samples/sec Loss 0.3498 Epoch: 23 Global Step: 236895 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:27:06,577-Speed 129.65 samples/sec Loss 0.3414 Epoch: 23 Global Step: 236900 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:27:26,347-Speed 129.49 samples/sec Loss 0.3650 Epoch: 23 Global Step: 236905 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:27:46,321-Speed 128.18 samples/sec Loss 0.3006 Epoch: 23 Global Step: 236910 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:28:06,063-Speed 129.68 samples/sec Loss 0.2318 Epoch: 23 Global Step: 236915 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:28:26,164-Speed 127.37 samples/sec Loss 0.2694 Epoch: 23 Global Step: 236920 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:28:45,989-Speed 129.13 samples/sec Loss 0.3712 Epoch: 23 Global Step: 236925 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:29:05,678-Speed 130.03 samples/sec Loss 0.4243 Epoch: 23 Global Step: 236930 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:29:25,416-Speed 129.71 samples/sec Loss 0.2835 Epoch: 23 Global Step: 236935 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:29:45,227-Speed 129.22 samples/sec Loss 0.2978 Epoch: 23 Global Step: 236940 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:30:05,023-Speed 129.33 samples/sec Loss 0.3302 Epoch: 23 Global Step: 236945 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:30:24,798-Speed 129.46 samples/sec Loss 0.2753 Epoch: 23 Global Step: 236950 Fp16 Grad Scale: 2048 Required: 15 hours Training: 2021-06-18 14:30:44,830-Speed 127.80 samples/sec Loss 0.3029 Epoch: 23 Global Step: 236955 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:31:04,453-Speed 130.47 samples/sec Loss 0.3938 Epoch: 23 Global Step: 236960 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:31:24,605-Speed 127.04 samples/sec Loss 0.2853 Epoch: 23 Global Step: 236965 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:31:44,495-Speed 128.72 samples/sec Loss 0.3681 Epoch: 23 Global Step: 236970 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:32:04,114-Speed 130.49 samples/sec Loss 0.4141 Epoch: 23 Global Step: 236975 Fp16 Grad Scale: 4096 Required: 15 hours Training: 2021-06-18 14:32:23,877-Speed 129.54 samples/sec Loss 0.3437 Epoch: 23 Global Step: 236980 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:32:43,680-Speed 129.28 samples/sec Loss 0.3267 Epoch: 23 Global Step: 236985 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:33:03,470-Speed 129.37 samples/sec Loss 0.3145 Epoch: 23 Global Step: 236990 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:33:23,375-Speed 128.62 samples/sec Loss 0.3396 Epoch: 23 Global Step: 236995 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:33:43,287-Speed 128.57 samples/sec Loss 0.3673 Epoch: 23 Global Step: 237000 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:34:02,989-Speed 129.94 samples/sec Loss 0.3448 Epoch: 23 Global Step: 237005 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:34:23,106-Speed 127.26 samples/sec Loss 0.2949 Epoch: 23 Global Step: 237010 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:34:43,028-Speed 128.51 samples/sec Loss 0.3187 Epoch: 23 Global Step: 237015 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:35:02,633-Speed 130.58 samples/sec Loss 0.2913 Epoch: 23 Global Step: 237020 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:35:22,373-Speed 129.69 samples/sec Loss 0.2729 Epoch: 23 Global Step: 237025 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:35:42,146-Speed 129.48 samples/sec Loss 0.3233 Epoch: 23 Global Step: 237030 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:36:01,943-Speed 129.32 samples/sec Loss 0.3299 Epoch: 23 Global Step: 237035 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:36:21,758-Speed 129.20 samples/sec Loss 0.3352 Epoch: 23 Global Step: 237040 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:36:41,622-Speed 128.88 samples/sec Loss 0.3208 Epoch: 23 Global Step: 237045 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:37:01,312-Speed 130.02 samples/sec Loss 0.3189 Epoch: 23 Global Step: 237050 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:37:21,417-Speed 127.34 samples/sec Loss 0.3322 Epoch: 23 Global Step: 237055 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:37:41,377-Speed 128.26 samples/sec Loss 0.2570 Epoch: 23 Global Step: 237060 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:38:01,094-Speed 129.84 samples/sec Loss 0.3453 Epoch: 23 Global Step: 237065 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:38:20,882-Speed 129.37 samples/sec Loss 0.2175 Epoch: 23 Global Step: 237070 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:38:40,573-Speed 130.02 samples/sec Loss 0.3301 Epoch: 23 Global Step: 237075 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:38:59,869-Speed 132.67 samples/sec Loss 0.3557 Epoch: 23 Global Step: 237080 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:39:19,794-Speed 128.49 samples/sec Loss 0.2820 Epoch: 23 Global Step: 237085 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:39:39,521-Speed 129.78 samples/sec Loss 0.3189 Epoch: 23 Global Step: 237090 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:39:59,151-Speed 130.42 samples/sec Loss 0.3019 Epoch: 23 Global Step: 237095 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:40:19,203-Speed 127.67 samples/sec Loss 0.3761 Epoch: 23 Global Step: 237100 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:40:39,236-Speed 127.80 samples/sec Loss 0.2549 Epoch: 23 Global Step: 237105 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:40:58,672-Speed 131.72 samples/sec Loss 0.2992 Epoch: 23 Global Step: 237110 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:41:18,272-Speed 130.62 samples/sec Loss 0.3839 Epoch: 23 Global Step: 237115 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:41:38,004-Speed 129.74 samples/sec Loss 0.3328 Epoch: 23 Global Step: 237120 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:41:57,536-Speed 131.07 samples/sec Loss 0.2625 Epoch: 23 Global Step: 237125 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:42:17,442-Speed 128.61 samples/sec Loss 0.2862 Epoch: 23 Global Step: 237130 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:42:36,864-Speed 131.82 samples/sec Loss 0.3260 Epoch: 23 Global Step: 237135 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:42:56,522-Speed 130.24 samples/sec Loss 0.3349 Epoch: 23 Global Step: 237140 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:43:16,262-Speed 129.70 samples/sec Loss 0.2959 Epoch: 23 Global Step: 237145 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:43:36,135-Speed 128.82 samples/sec Loss 0.3102 Epoch: 23 Global Step: 237150 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:43:55,565-Speed 131.76 samples/sec Loss 0.3221 Epoch: 23 Global Step: 237155 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:44:15,257-Speed 130.01 samples/sec Loss 0.3130 Epoch: 23 Global Step: 237160 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:44:35,146-Speed 128.72 samples/sec Loss 0.3200 Epoch: 23 Global Step: 237165 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:44:54,605-Speed 131.56 samples/sec Loss 0.2800 Epoch: 23 Global Step: 237170 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:45:14,572-Speed 128.22 samples/sec Loss 0.2613 Epoch: 23 Global Step: 237175 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:45:34,106-Speed 131.06 samples/sec Loss 0.2705 Epoch: 23 Global Step: 237180 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:45:53,849-Speed 129.68 samples/sec Loss 0.2902 Epoch: 23 Global Step: 237185 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:46:13,606-Speed 129.58 samples/sec Loss 0.3804 Epoch: 23 Global Step: 237190 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:46:33,438-Speed 129.09 samples/sec Loss 0.3020 Epoch: 23 Global Step: 237195 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:46:52,857-Speed 131.84 samples/sec Loss 0.3336 Epoch: 23 Global Step: 237200 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:47:12,743-Speed 128.74 samples/sec Loss 0.3127 Epoch: 23 Global Step: 237205 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:47:32,458-Speed 129.85 samples/sec Loss 0.4009 Epoch: 23 Global Step: 237210 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:47:51,951-Speed 131.34 samples/sec Loss 0.3208 Epoch: 23 Global Step: 237215 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:48:11,919-Speed 128.21 samples/sec Loss 0.3078 Epoch: 23 Global Step: 237220 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:48:31,525-Speed 130.58 samples/sec Loss 0.3333 Epoch: 23 Global Step: 237225 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:48:51,246-Speed 129.82 samples/sec Loss 0.2643 Epoch: 23 Global Step: 237230 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:49:10,967-Speed 129.82 samples/sec Loss 0.3479 Epoch: 23 Global Step: 237235 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:49:30,885-Speed 128.53 samples/sec Loss 0.3208 Epoch: 23 Global Step: 237240 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:49:50,420-Speed 131.05 samples/sec Loss 0.3073 Epoch: 23 Global Step: 237245 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:50:10,243-Speed 129.15 samples/sec Loss 0.2753 Epoch: 23 Global Step: 237250 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:50:30,212-Speed 128.21 samples/sec Loss 0.3363 Epoch: 23 Global Step: 237255 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:50:49,788-Speed 130.78 samples/sec Loss 0.3179 Epoch: 23 Global Step: 237260 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:51:09,665-Speed 128.80 samples/sec Loss 0.2954 Epoch: 23 Global Step: 237265 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:51:29,230-Speed 130.85 samples/sec Loss 0.2890 Epoch: 23 Global Step: 237270 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:51:49,000-Speed 129.50 samples/sec Loss 0.2820 Epoch: 23 Global Step: 237275 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:52:08,699-Speed 129.97 samples/sec Loss 0.3047 Epoch: 23 Global Step: 237280 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:52:28,649-Speed 128.32 samples/sec Loss 0.3442 Epoch: 23 Global Step: 237285 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:52:48,076-Speed 131.78 samples/sec Loss 0.3051 Epoch: 23 Global Step: 237290 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:53:07,913-Speed 129.05 samples/sec Loss 0.4042 Epoch: 23 Global Step: 237295 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:53:27,784-Speed 128.84 samples/sec Loss 0.3190 Epoch: 23 Global Step: 237300 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:53:47,402-Speed 130.50 samples/sec Loss 0.3179 Epoch: 23 Global Step: 237305 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:54:07,191-Speed 129.38 samples/sec Loss 0.2595 Epoch: 23 Global Step: 237310 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 14:54:26,176-Speed 134.85 samples/sec Loss 0.3238 Epoch: 23 Global Step: 237315 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:54:45,907-Speed 129.75 samples/sec Loss 0.3070 Epoch: 23 Global Step: 237320 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:55:05,697-Speed 129.37 samples/sec Loss 0.3114 Epoch: 23 Global Step: 237325 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:55:25,679-Speed 128.12 samples/sec Loss 0.2759 Epoch: 23 Global Step: 237330 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:55:45,136-Speed 131.57 samples/sec Loss 0.3413 Epoch: 23 Global Step: 237335 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:56:04,950-Speed 129.21 samples/sec Loss 0.2981 Epoch: 23 Global Step: 237340 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:56:24,851-Speed 128.65 samples/sec Loss 0.3367 Epoch: 23 Global Step: 237345 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:56:44,319-Speed 131.51 samples/sec Loss 0.3825 Epoch: 23 Global Step: 237350 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:57:04,207-Speed 128.73 samples/sec Loss 0.2762 Epoch: 23 Global Step: 237355 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:57:23,721-Speed 131.19 samples/sec Loss 0.3545 Epoch: 23 Global Step: 237360 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:57:43,449-Speed 129.77 samples/sec Loss 0.2570 Epoch: 23 Global Step: 237365 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:58:03,250-Speed 129.30 samples/sec Loss 0.3238 Epoch: 23 Global Step: 237370 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:58:23,139-Speed 128.72 samples/sec Loss 0.3672 Epoch: 23 Global Step: 237375 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:58:42,578-Speed 131.70 samples/sec Loss 0.3016 Epoch: 23 Global Step: 237380 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:59:02,407-Speed 129.11 samples/sec Loss 0.3575 Epoch: 23 Global Step: 237385 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:59:22,279-Speed 128.83 samples/sec Loss 0.2748 Epoch: 23 Global Step: 237390 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 14:59:41,735-Speed 131.59 samples/sec Loss 0.3335 Epoch: 23 Global Step: 237395 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:00:01,719-Speed 128.11 samples/sec Loss 0.2459 Epoch: 23 Global Step: 237400 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:00:21,144-Speed 131.79 samples/sec Loss 0.2862 Epoch: 23 Global Step: 237405 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:00:40,900-Speed 129.59 samples/sec Loss 0.2591 Epoch: 23 Global Step: 237410 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:01:00,697-Speed 129.32 samples/sec Loss 0.3250 Epoch: 23 Global Step: 237415 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:01:20,528-Speed 129.09 samples/sec Loss 0.3014 Epoch: 23 Global Step: 237420 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:01:39,958-Speed 131.76 samples/sec Loss 0.2350 Epoch: 23 Global Step: 237425 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:01:59,227-Speed 132.86 samples/sec Loss 0.3492 Epoch: 23 Global Step: 237430 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:02:18,808-Speed 130.75 samples/sec Loss 0.3560 Epoch: 23 Global Step: 237435 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:02:38,546-Speed 129.70 samples/sec Loss 0.3348 Epoch: 23 Global Step: 237440 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:02:58,161-Speed 130.52 samples/sec Loss 0.3148 Epoch: 23 Global Step: 237445 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:03:17,762-Speed 130.61 samples/sec Loss 0.2713 Epoch: 23 Global Step: 237450 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:03:37,303-Speed 131.02 samples/sec Loss 0.3118 Epoch: 23 Global Step: 237455 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:03:57,149-Speed 129.00 samples/sec Loss 0.2601 Epoch: 23 Global Step: 237460 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:04:16,718-Speed 130.83 samples/sec Loss 0.2959 Epoch: 23 Global Step: 237465 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:04:36,888-Speed 126.93 samples/sec Loss 0.3448 Epoch: 23 Global Step: 237470 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:04:56,724-Speed 129.07 samples/sec Loss 0.3470 Epoch: 23 Global Step: 237475 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:05:16,273-Speed 130.96 samples/sec Loss 0.2415 Epoch: 23 Global Step: 237480 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:05:36,019-Speed 129.65 samples/sec Loss 0.2638 Epoch: 23 Global Step: 237485 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:05:55,634-Speed 130.52 samples/sec Loss 0.3036 Epoch: 23 Global Step: 237490 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:06:15,319-Speed 130.06 samples/sec Loss 0.3152 Epoch: 23 Global Step: 237495 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:06:34,926-Speed 130.57 samples/sec Loss 0.3609 Epoch: 23 Global Step: 237500 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:06:54,650-Speed 129.80 samples/sec Loss 0.3268 Epoch: 23 Global Step: 237505 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:07:14,244-Speed 130.66 samples/sec Loss 0.3514 Epoch: 23 Global Step: 237510 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:07:34,164-Speed 128.52 samples/sec Loss 0.3679 Epoch: 23 Global Step: 237515 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:07:54,231-Speed 127.57 samples/sec Loss 0.2862 Epoch: 23 Global Step: 237520 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:08:13,792-Speed 130.88 samples/sec Loss 0.3672 Epoch: 23 Global Step: 237525 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:08:33,406-Speed 130.52 samples/sec Loss 0.3363 Epoch: 23 Global Step: 237530 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:08:53,010-Speed 130.59 samples/sec Loss 0.2991 Epoch: 23 Global Step: 237535 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:09:12,803-Speed 129.35 samples/sec Loss 0.3729 Epoch: 23 Global Step: 237540 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:09:32,433-Speed 130.42 samples/sec Loss 0.2685 Epoch: 23 Global Step: 237545 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:09:52,176-Speed 129.68 samples/sec Loss 0.3672 Epoch: 23 Global Step: 237550 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:10:11,634-Speed 131.57 samples/sec Loss 0.3204 Epoch: 23 Global Step: 237555 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:10:31,610-Speed 128.16 samples/sec Loss 0.2875 Epoch: 23 Global Step: 237560 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:10:51,836-Speed 126.58 samples/sec Loss 0.3224 Epoch: 23 Global Step: 237565 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:11:11,396-Speed 130.88 samples/sec Loss 0.3033 Epoch: 23 Global Step: 237570 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:11:31,111-Speed 129.86 samples/sec Loss 0.3426 Epoch: 23 Global Step: 237575 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:11:50,679-Speed 130.83 samples/sec Loss 0.3816 Epoch: 23 Global Step: 237580 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:12:10,292-Speed 130.54 samples/sec Loss 0.3684 Epoch: 23 Global Step: 237585 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:12:29,907-Speed 130.52 samples/sec Loss 0.3655 Epoch: 23 Global Step: 237590 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:12:49,642-Speed 129.73 samples/sec Loss 0.3683 Epoch: 23 Global Step: 237595 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:13:09,218-Speed 130.78 samples/sec Loss 0.2772 Epoch: 23 Global Step: 237600 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:13:29,183-Speed 128.23 samples/sec Loss 0.4148 Epoch: 23 Global Step: 237605 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:13:49,080-Speed 128.67 samples/sec Loss 0.3078 Epoch: 23 Global Step: 237610 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:14:08,669-Speed 130.69 samples/sec Loss 0.2806 Epoch: 23 Global Step: 237615 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:14:28,422-Speed 129.61 samples/sec Loss 0.3313 Epoch: 23 Global Step: 237620 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:14:47,995-Speed 130.80 samples/sec Loss 0.3317 Epoch: 23 Global Step: 237625 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:15:07,656-Speed 130.21 samples/sec Loss 0.3220 Epoch: 23 Global Step: 237630 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:15:27,402-Speed 129.65 samples/sec Loss 0.3443 Epoch: 23 Global Step: 237635 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:15:47,128-Speed 129.78 samples/sec Loss 0.2836 Epoch: 23 Global Step: 237640 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:16:06,614-Speed 131.38 samples/sec Loss 0.2970 Epoch: 23 Global Step: 237645 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:16:26,629-Speed 127.91 samples/sec Loss 0.3224 Epoch: 23 Global Step: 237650 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:16:46,578-Speed 128.34 samples/sec Loss 0.3187 Epoch: 23 Global Step: 237655 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:17:06,118-Speed 131.02 samples/sec Loss 0.3264 Epoch: 23 Global Step: 237660 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:17:25,914-Speed 129.32 samples/sec Loss 0.3399 Epoch: 23 Global Step: 237665 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:17:45,514-Speed 130.62 samples/sec Loss 0.2659 Epoch: 23 Global Step: 237670 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:18:04,687-Speed 133.53 samples/sec Loss 0.3268 Epoch: 23 Global Step: 237675 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:18:24,661-Speed 128.17 samples/sec Loss 0.2986 Epoch: 23 Global Step: 237680 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:18:44,765-Speed 127.35 samples/sec Loss 0.3577 Epoch: 23 Global Step: 237685 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:19:04,449-Speed 130.06 samples/sec Loss 0.3196 Epoch: 23 Global Step: 237690 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:19:24,592-Speed 127.10 samples/sec Loss 0.3054 Epoch: 23 Global Step: 237695 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:19:44,593-Speed 127.99 samples/sec Loss 0.3800 Epoch: 23 Global Step: 237700 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:20:04,345-Speed 129.61 samples/sec Loss 0.3013 Epoch: 23 Global Step: 237705 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:20:24,161-Speed 129.19 samples/sec Loss 0.3445 Epoch: 23 Global Step: 237710 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:20:43,895-Speed 129.74 samples/sec Loss 0.3760 Epoch: 23 Global Step: 237715 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:21:03,603-Speed 129.90 samples/sec Loss 0.3619 Epoch: 23 Global Step: 237720 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:21:23,836-Speed 126.53 samples/sec Loss 0.2895 Epoch: 23 Global Step: 237725 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:21:43,580-Speed 129.66 samples/sec Loss 0.2664 Epoch: 23 Global Step: 237730 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:22:03,242-Speed 130.21 samples/sec Loss 0.3040 Epoch: 23 Global Step: 237735 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:22:23,325-Speed 127.48 samples/sec Loss 0.3051 Epoch: 23 Global Step: 237740 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:22:43,299-Speed 128.17 samples/sec Loss 0.3305 Epoch: 23 Global Step: 237745 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:23:03,073-Speed 129.47 samples/sec Loss 0.3708 Epoch: 23 Global Step: 237750 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:23:22,989-Speed 128.54 samples/sec Loss 0.3032 Epoch: 23 Global Step: 237755 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:23:42,720-Speed 129.75 samples/sec Loss 0.3240 Epoch: 23 Global Step: 237760 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:24:02,363-Speed 130.33 samples/sec Loss 0.3380 Epoch: 23 Global Step: 237765 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:24:22,570-Speed 126.70 samples/sec Loss 0.3613 Epoch: 23 Global Step: 237770 Fp16 Grad Scale: 4096 Required: 14 hours Training: 2021-06-18 15:24:42,291-Speed 129.81 samples/sec Loss 0.2953 Epoch: 23 Global Step: 237775 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:25:02,094-Speed 129.28 samples/sec Loss 0.2546 Epoch: 23 Global Step: 237780 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:25:22,864-Speed 123.27 samples/sec Loss 0.3616 Epoch: 23 Global Step: 237785 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:25:42,717-Speed 128.95 samples/sec Loss 0.3616 Epoch: 23 Global Step: 237790 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:26:02,516-Speed 129.30 samples/sec Loss 0.2915 Epoch: 23 Global Step: 237795 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:26:22,412-Speed 128.68 samples/sec Loss 0.2527 Epoch: 23 Global Step: 237800 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:26:42,029-Speed 130.50 samples/sec Loss 0.2924 Epoch: 23 Global Step: 237805 Fp16 Grad Scale: 8192 Required: 14 hours Training: 2021-06-18 15:27:01,848-Speed 129.18 samples/sec Loss 0.2739 Epoch: 23 Global Step: 237810 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:27:21,843-Speed 128.04 samples/sec Loss 0.3422 Epoch: 23 Global Step: 237815 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:27:41,651-Speed 129.25 samples/sec Loss 0.3231 Epoch: 23 Global Step: 237820 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:28:01,451-Speed 129.30 samples/sec Loss 0.3738 Epoch: 23 Global Step: 237825 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:28:21,567-Speed 127.27 samples/sec Loss 0.3029 Epoch: 23 Global Step: 237830 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:28:41,339-Speed 129.49 samples/sec Loss 0.3365 Epoch: 23 Global Step: 237835 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:29:01,264-Speed 128.49 samples/sec Loss 0.3661 Epoch: 23 Global Step: 237840 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:29:21,163-Speed 128.66 samples/sec Loss 0.2997 Epoch: 23 Global Step: 237845 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:29:40,820-Speed 130.24 samples/sec Loss 0.3209 Epoch: 23 Global Step: 237850 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:30:00,619-Speed 129.31 samples/sec Loss 0.2270 Epoch: 23 Global Step: 237855 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:30:20,630-Speed 127.94 samples/sec Loss 0.3653 Epoch: 23 Global Step: 237860 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:30:40,388-Speed 129.58 samples/sec Loss 0.3540 Epoch: 23 Global Step: 237865 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:31:00,152-Speed 129.54 samples/sec Loss 0.3808 Epoch: 23 Global Step: 237870 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:31:20,220-Speed 127.57 samples/sec Loss 0.3004 Epoch: 23 Global Step: 237875 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:31:39,949-Speed 129.77 samples/sec Loss 0.3529 Epoch: 23 Global Step: 237880 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:31:59,761-Speed 129.22 samples/sec Loss 0.3131 Epoch: 23 Global Step: 237885 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:32:19,696-Speed 128.42 samples/sec Loss 0.2907 Epoch: 23 Global Step: 237890 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:32:39,363-Speed 130.18 samples/sec Loss 0.3127 Epoch: 23 Global Step: 237895 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:32:59,251-Speed 128.73 samples/sec Loss 0.3341 Epoch: 23 Global Step: 237900 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:33:19,292-Speed 127.74 samples/sec Loss 0.3010 Epoch: 23 Global Step: 237905 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:33:39,263-Speed 128.19 samples/sec Loss 0.3544 Epoch: 23 Global Step: 237910 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:33:58,890-Speed 130.44 samples/sec Loss 0.3891 Epoch: 23 Global Step: 237915 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:34:18,977-Speed 127.45 samples/sec Loss 0.2802 Epoch: 23 Global Step: 237920 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:34:38,581-Speed 130.60 samples/sec Loss 0.2945 Epoch: 23 Global Step: 237925 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:34:58,425-Speed 129.01 samples/sec Loss 0.2714 Epoch: 23 Global Step: 237930 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:35:18,407-Speed 128.12 samples/sec Loss 0.3084 Epoch: 23 Global Step: 237935 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:35:38,147-Speed 129.69 samples/sec Loss 0.3276 Epoch: 23 Global Step: 237940 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:35:58,028-Speed 128.77 samples/sec Loss 0.3260 Epoch: 23 Global Step: 237945 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:36:17,941-Speed 128.57 samples/sec Loss 0.3120 Epoch: 23 Global Step: 237950 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:36:37,854-Speed 128.56 samples/sec Loss 0.4293 Epoch: 23 Global Step: 237955 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:36:57,204-Speed 132.31 samples/sec Loss 0.3224 Epoch: 23 Global Step: 237960 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:37:17,296-Speed 127.42 samples/sec Loss 0.2936 Epoch: 23 Global Step: 237965 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:37:36,915-Speed 130.50 samples/sec Loss 0.3764 Epoch: 23 Global Step: 237970 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:37:56,836-Speed 128.51 samples/sec Loss 0.3708 Epoch: 23 Global Step: 237975 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:38:16,557-Speed 129.82 samples/sec Loss 0.3071 Epoch: 23 Global Step: 237980 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:38:36,398-Speed 129.03 samples/sec Loss 0.3018 Epoch: 23 Global Step: 237985 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:38:56,281-Speed 128.76 samples/sec Loss 0.3550 Epoch: 23 Global Step: 237990 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:39:16,841-Speed 124.52 samples/sec Loss 0.2977 Epoch: 23 Global Step: 237995 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:39:36,733-Speed 128.70 samples/sec Loss 0.2949 Epoch: 23 Global Step: 238000 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:39:56,550-Speed 129.19 samples/sec Loss 0.2705 Epoch: 23 Global Step: 238005 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:40:16,622-Speed 127.55 samples/sec Loss 0.3323 Epoch: 23 Global Step: 238010 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:40:36,230-Speed 130.57 samples/sec Loss 0.3758 Epoch: 23 Global Step: 238015 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:40:56,130-Speed 128.64 samples/sec Loss 0.3401 Epoch: 23 Global Step: 238020 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:41:15,853-Speed 129.81 samples/sec Loss 0.3665 Epoch: 23 Global Step: 238025 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:41:35,677-Speed 129.15 samples/sec Loss 0.2693 Epoch: 23 Global Step: 238030 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:41:55,501-Speed 129.14 samples/sec Loss 0.3300 Epoch: 23 Global Step: 238035 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:42:15,420-Speed 128.52 samples/sec Loss 0.3149 Epoch: 23 Global Step: 238040 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:42:35,284-Speed 128.89 samples/sec Loss 0.2949 Epoch: 23 Global Step: 238045 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:42:55,093-Speed 129.24 samples/sec Loss 0.3124 Epoch: 23 Global Step: 238050 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:43:15,108-Speed 127.91 samples/sec Loss 0.3196 Epoch: 23 Global Step: 238055 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:43:34,716-Speed 130.57 samples/sec Loss 0.3500 Epoch: 23 Global Step: 238060 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:43:54,668-Speed 128.32 samples/sec Loss 0.2627 Epoch: 23 Global Step: 238065 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:44:14,379-Speed 129.88 samples/sec Loss 0.3233 Epoch: 23 Global Step: 238070 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:44:34,138-Speed 129.56 samples/sec Loss 0.2989 Epoch: 23 Global Step: 238075 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:44:53,975-Speed 129.06 samples/sec Loss 0.3337 Epoch: 23 Global Step: 238080 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:45:13,895-Speed 128.52 samples/sec Loss 0.3331 Epoch: 23 Global Step: 238085 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:45:33,804-Speed 128.59 samples/sec Loss 0.2921 Epoch: 23 Global Step: 238090 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:45:53,522-Speed 129.84 samples/sec Loss 0.3016 Epoch: 23 Global Step: 238095 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:46:13,688-Speed 126.95 samples/sec Loss 0.2552 Epoch: 23 Global Step: 238100 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:46:33,299-Speed 130.55 samples/sec Loss 0.2927 Epoch: 23 Global Step: 238105 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:46:53,214-Speed 128.55 samples/sec Loss 0.3125 Epoch: 23 Global Step: 238110 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:47:12,898-Speed 130.06 samples/sec Loss 0.3539 Epoch: 23 Global Step: 238115 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:47:32,650-Speed 129.62 samples/sec Loss 0.2843 Epoch: 23 Global Step: 238120 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:47:52,585-Speed 128.42 samples/sec Loss 0.3454 Epoch: 23 Global Step: 238125 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:48:12,638-Speed 127.66 samples/sec Loss 0.3691 Epoch: 23 Global Step: 238130 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:48:32,523-Speed 128.75 samples/sec Loss 0.2984 Epoch: 23 Global Step: 238135 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:48:52,152-Speed 130.42 samples/sec Loss 0.3110 Epoch: 23 Global Step: 238140 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:49:12,314-Speed 126.98 samples/sec Loss 0.2992 Epoch: 23 Global Step: 238145 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:49:31,930-Speed 130.51 samples/sec Loss 0.3054 Epoch: 23 Global Step: 238150 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:49:51,869-Speed 128.40 samples/sec Loss 0.3328 Epoch: 23 Global Step: 238155 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:50:11,738-Speed 128.85 samples/sec Loss 0.3277 Epoch: 23 Global Step: 238160 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:50:31,095-Speed 132.26 samples/sec Loss 0.3312 Epoch: 23 Global Step: 238165 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:50:50,931-Speed 129.06 samples/sec Loss 0.3748 Epoch: 23 Global Step: 238170 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:51:10,896-Speed 128.23 samples/sec Loss 0.3166 Epoch: 23 Global Step: 238175 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:51:30,755-Speed 128.92 samples/sec Loss 0.2714 Epoch: 23 Global Step: 238180 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:51:50,177-Speed 131.82 samples/sec Loss 0.2608 Epoch: 23 Global Step: 238185 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:52:10,084-Speed 128.60 samples/sec Loss 0.3408 Epoch: 23 Global Step: 238190 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:52:29,606-Speed 131.14 samples/sec Loss 0.2951 Epoch: 23 Global Step: 238195 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:52:49,231-Speed 130.46 samples/sec Loss 0.3594 Epoch: 23 Global Step: 238200 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:53:08,879-Speed 130.30 samples/sec Loss 0.2932 Epoch: 23 Global Step: 238205 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:53:28,364-Speed 131.39 samples/sec Loss 0.3097 Epoch: 23 Global Step: 238210 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:53:48,417-Speed 127.67 samples/sec Loss 0.2861 Epoch: 23 Global Step: 238215 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:54:08,493-Speed 127.52 samples/sec Loss 0.2789 Epoch: 23 Global Step: 238220 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:54:27,919-Speed 131.79 samples/sec Loss 0.3197 Epoch: 23 Global Step: 238225 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:54:47,556-Speed 130.37 samples/sec Loss 0.3073 Epoch: 23 Global Step: 238230 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:55:07,192-Speed 130.38 samples/sec Loss 0.2585 Epoch: 23 Global Step: 238235 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:55:26,806-Speed 130.53 samples/sec Loss 0.2650 Epoch: 23 Global Step: 238240 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:55:46,427-Speed 130.48 samples/sec Loss 0.3042 Epoch: 23 Global Step: 238245 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:56:06,092-Speed 130.19 samples/sec Loss 0.3514 Epoch: 23 Global Step: 238250 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:56:25,565-Speed 131.47 samples/sec Loss 0.2673 Epoch: 23 Global Step: 238255 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:56:45,768-Speed 126.72 samples/sec Loss 0.3147 Epoch: 23 Global Step: 238260 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 15:57:05,719-Speed 128.32 samples/sec Loss 0.3516 Epoch: 23 Global Step: 238265 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:57:25,624-Speed 128.62 samples/sec Loss 0.3163 Epoch: 23 Global Step: 238270 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:57:45,271-Speed 130.31 samples/sec Loss 0.3513 Epoch: 23 Global Step: 238275 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:58:04,848-Speed 130.77 samples/sec Loss 0.3085 Epoch: 23 Global Step: 238280 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:58:24,541-Speed 130.00 samples/sec Loss 0.3071 Epoch: 23 Global Step: 238285 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:58:44,294-Speed 129.61 samples/sec Loss 0.2865 Epoch: 23 Global Step: 238290 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:59:04,127-Speed 129.09 samples/sec Loss 0.3192 Epoch: 23 Global Step: 238295 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:59:23,730-Speed 130.60 samples/sec Loss 0.3289 Epoch: 23 Global Step: 238300 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 15:59:43,847-Speed 127.26 samples/sec Loss 0.3739 Epoch: 23 Global Step: 238305 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:00:03,847-Speed 128.01 samples/sec Loss 0.2822 Epoch: 23 Global Step: 238310 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:00:23,412-Speed 130.85 samples/sec Loss 0.3502 Epoch: 23 Global Step: 238315 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:00:43,121-Speed 129.89 samples/sec Loss 0.3954 Epoch: 23 Global Step: 238320 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:01:02,713-Speed 130.68 samples/sec Loss 0.2980 Epoch: 23 Global Step: 238325 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:01:22,504-Speed 129.36 samples/sec Loss 0.3102 Epoch: 23 Global Step: 238330 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:01:42,105-Speed 130.61 samples/sec Loss 0.4415 Epoch: 23 Global Step: 238335 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:02:01,784-Speed 130.10 samples/sec Loss 0.2817 Epoch: 23 Global Step: 238340 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:02:21,329-Speed 130.98 samples/sec Loss 0.3076 Epoch: 23 Global Step: 238345 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:02:41,333-Speed 127.98 samples/sec Loss 0.2817 Epoch: 23 Global Step: 238350 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:03:01,342-Speed 127.95 samples/sec Loss 0.2911 Epoch: 23 Global Step: 238355 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:03:20,869-Speed 131.11 samples/sec Loss 0.3186 Epoch: 23 Global Step: 238360 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:03:40,555-Speed 130.05 samples/sec Loss 0.2589 Epoch: 23 Global Step: 238365 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:04:00,335-Speed 129.43 samples/sec Loss 0.3281 Epoch: 23 Global Step: 238370 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:04:20,076-Speed 129.69 samples/sec Loss 0.3153 Epoch: 23 Global Step: 238375 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:04:39,698-Speed 130.47 samples/sec Loss 0.2847 Epoch: 23 Global Step: 238380 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:04:59,397-Speed 129.96 samples/sec Loss 0.3236 Epoch: 23 Global Step: 238385 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:05:18,962-Speed 130.85 samples/sec Loss 0.3394 Epoch: 23 Global Step: 238390 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:05:38,968-Speed 127.97 samples/sec Loss 0.2714 Epoch: 23 Global Step: 238395 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:05:58,948-Speed 128.14 samples/sec Loss 0.3513 Epoch: 23 Global Step: 238400 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:06:18,494-Speed 130.97 samples/sec Loss 0.3132 Epoch: 23 Global Step: 238405 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:06:38,245-Speed 129.63 samples/sec Loss 0.2854 Epoch: 23 Global Step: 238410 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:06:57,835-Speed 130.69 samples/sec Loss 0.2376 Epoch: 23 Global Step: 238415 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:07:17,526-Speed 130.01 samples/sec Loss 0.3254 Epoch: 23 Global Step: 238420 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:07:37,261-Speed 129.73 samples/sec Loss 0.3168 Epoch: 23 Global Step: 238425 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:07:56,993-Speed 129.75 samples/sec Loss 0.3680 Epoch: 23 Global Step: 238430 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:08:16,551-Speed 130.90 samples/sec Loss 0.3001 Epoch: 23 Global Step: 238435 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:08:36,544-Speed 128.05 samples/sec Loss 0.3380 Epoch: 23 Global Step: 238440 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:08:56,582-Speed 127.76 samples/sec Loss 0.3567 Epoch: 23 Global Step: 238445 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:09:15,679-Speed 134.06 samples/sec Loss 0.3766 Epoch: 23 Global Step: 238450 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:09:35,660-Speed 128.13 samples/sec Loss 0.3030 Epoch: 23 Global Step: 238455 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:09:55,659-Speed 128.02 samples/sec Loss 0.3040 Epoch: 23 Global Step: 238460 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:10:15,203-Speed 130.99 samples/sec Loss 0.3697 Epoch: 23 Global Step: 238465 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:10:35,058-Speed 128.95 samples/sec Loss 0.2710 Epoch: 23 Global Step: 238470 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:10:54,569-Speed 131.21 samples/sec Loss 0.3112 Epoch: 23 Global Step: 238475 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:11:14,368-Speed 129.31 samples/sec Loss 0.3359 Epoch: 23 Global Step: 238480 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:11:34,246-Speed 128.79 samples/sec Loss 0.2952 Epoch: 23 Global Step: 238485 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:11:54,189-Speed 128.37 samples/sec Loss 0.3051 Epoch: 23 Global Step: 238490 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:12:13,602-Speed 131.88 samples/sec Loss 0.2932 Epoch: 23 Global Step: 238495 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:12:33,546-Speed 128.36 samples/sec Loss 0.3094 Epoch: 23 Global Step: 238500 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:12:53,159-Speed 130.54 samples/sec Loss 0.3492 Epoch: 23 Global Step: 238505 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:13:12,679-Speed 131.15 samples/sec Loss 0.3617 Epoch: 23 Global Step: 238510 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:13:32,565-Speed 128.74 samples/sec Loss 0.2948 Epoch: 23 Global Step: 238515 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:13:52,118-Speed 130.93 samples/sec Loss 0.3051 Epoch: 23 Global Step: 238520 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:14:11,897-Speed 129.44 samples/sec Loss 0.3466 Epoch: 23 Global Step: 238525 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:14:31,541-Speed 130.32 samples/sec Loss 0.3237 Epoch: 23 Global Step: 238530 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:14:51,442-Speed 128.65 samples/sec Loss 0.3369 Epoch: 23 Global Step: 238535 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:15:10,889-Speed 131.64 samples/sec Loss 0.3097 Epoch: 23 Global Step: 238540 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:15:30,805-Speed 128.54 samples/sec Loss 0.2684 Epoch: 23 Global Step: 238545 Fp16 Grad Scale: 4096 Required: 13 hours Training: 2021-06-18 16:15:50,429-Speed 130.46 samples/sec Loss 0.3743 Epoch: 23 Global Step: 238550 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:16:09,986-Speed 130.91 samples/sec Loss 0.3582 Epoch: 23 Global Step: 238555 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:16:30,044-Speed 127.63 samples/sec Loss 0.3819 Epoch: 23 Global Step: 238560 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:16:49,592-Speed 130.97 samples/sec Loss 0.3276 Epoch: 23 Global Step: 238565 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:17:09,486-Speed 128.69 samples/sec Loss 0.2845 Epoch: 23 Global Step: 238570 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:17:29,201-Speed 129.86 samples/sec Loss 0.2705 Epoch: 23 Global Step: 238575 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:17:49,172-Speed 128.19 samples/sec Loss 0.3430 Epoch: 23 Global Step: 238580 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:18:08,635-Speed 131.54 samples/sec Loss 0.2995 Epoch: 23 Global Step: 238585 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:18:28,441-Speed 129.26 samples/sec Loss 0.3649 Epoch: 23 Global Step: 238590 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:18:48,053-Speed 130.55 samples/sec Loss 0.2553 Epoch: 23 Global Step: 238595 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:19:07,636-Speed 130.73 samples/sec Loss 0.3037 Epoch: 23 Global Step: 238600 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:19:27,577-Speed 128.39 samples/sec Loss 0.3019 Epoch: 23 Global Step: 238605 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:19:47,003-Speed 131.79 samples/sec Loss 0.2289 Epoch: 23 Global Step: 238610 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:20:06,894-Speed 128.71 samples/sec Loss 0.2935 Epoch: 23 Global Step: 238615 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:20:26,561-Speed 130.17 samples/sec Loss 0.2728 Epoch: 23 Global Step: 238620 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:20:46,444-Speed 128.76 samples/sec Loss 0.2823 Epoch: 23 Global Step: 238625 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:21:05,873-Speed 131.77 samples/sec Loss 0.2746 Epoch: 23 Global Step: 238630 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:21:25,699-Speed 129.13 samples/sec Loss 0.3481 Epoch: 23 Global Step: 238635 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:21:45,311-Speed 130.54 samples/sec Loss 0.3386 Epoch: 23 Global Step: 238640 Fp16 Grad Scale: 8192 Required: 13 hours Training: 2021-06-18 16:22:04,939-Speed 130.43 samples/sec Loss 0.3117 Epoch: 23 Global Step: 238645 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:22:24,726-Speed 129.39 samples/sec Loss 0.3488 Epoch: 23 Global Step: 238650 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:22:44,269-Speed 131.00 samples/sec Loss 0.2747 Epoch: 23 Global Step: 238655 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:23:04,159-Speed 128.72 samples/sec Loss 0.3485 Epoch: 23 Global Step: 238660 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:23:23,951-Speed 129.36 samples/sec Loss 0.2555 Epoch: 23 Global Step: 238665 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:23:43,722-Speed 129.49 samples/sec Loss 0.3444 Epoch: 23 Global Step: 238670 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:24:03,354-Speed 130.40 samples/sec Loss 0.2880 Epoch: 23 Global Step: 238675 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:24:23,203-Speed 128.98 samples/sec Loss 0.2909 Epoch: 23 Global Step: 238680 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:24:42,813-Speed 130.56 samples/sec Loss 0.3004 Epoch: 23 Global Step: 238685 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:25:02,438-Speed 130.45 samples/sec Loss 0.3119 Epoch: 23 Global Step: 238690 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:25:22,169-Speed 129.76 samples/sec Loss 0.3725 Epoch: 23 Global Step: 238695 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:25:41,774-Speed 130.59 samples/sec Loss 0.3227 Epoch: 23 Global Step: 238700 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:26:01,640-Speed 128.87 samples/sec Loss 0.2982 Epoch: 23 Global Step: 238705 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:26:21,626-Speed 128.10 samples/sec Loss 0.2317 Epoch: 23 Global Step: 238710 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:26:40,909-Speed 132.77 samples/sec Loss 0.2903 Epoch: 23 Global Step: 238715 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:27:00,493-Speed 130.72 samples/sec Loss 0.3040 Epoch: 23 Global Step: 238720 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:27:20,434-Speed 128.38 samples/sec Loss 0.3132 Epoch: 23 Global Step: 238725 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:27:40,067-Speed 130.40 samples/sec Loss 0.3196 Epoch: 23 Global Step: 238730 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:27:59,676-Speed 130.56 samples/sec Loss 0.2755 Epoch: 23 Global Step: 238735 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:28:19,412-Speed 129.72 samples/sec Loss 0.2622 Epoch: 23 Global Step: 238740 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:28:39,014-Speed 130.61 samples/sec Loss 0.2781 Epoch: 23 Global Step: 238745 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:28:58,883-Speed 128.86 samples/sec Loss 0.3222 Epoch: 23 Global Step: 238750 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:29:18,681-Speed 129.31 samples/sec Loss 0.4133 Epoch: 23 Global Step: 238755 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:29:38,388-Speed 129.91 samples/sec Loss 0.2976 Epoch: 23 Global Step: 238760 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:29:57,965-Speed 130.77 samples/sec Loss 0.3306 Epoch: 23 Global Step: 238765 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:30:17,898-Speed 128.44 samples/sec Loss 0.3766 Epoch: 23 Global Step: 238770 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:30:37,619-Speed 129.81 samples/sec Loss 0.3244 Epoch: 23 Global Step: 238775 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:30:57,220-Speed 130.62 samples/sec Loss 0.2859 Epoch: 23 Global Step: 238780 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:31:16,998-Speed 129.44 samples/sec Loss 0.2744 Epoch: 23 Global Step: 238785 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:31:36,528-Speed 131.09 samples/sec Loss 0.3102 Epoch: 23 Global Step: 238790 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:31:56,400-Speed 128.83 samples/sec Loss 0.2377 Epoch: 23 Global Step: 238795 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:32:16,575-Speed 126.90 samples/sec Loss 0.2920 Epoch: 23 Global Step: 238800 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:32:36,292-Speed 129.85 samples/sec Loss 0.3422 Epoch: 23 Global Step: 238805 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:32:55,869-Speed 130.77 samples/sec Loss 0.3427 Epoch: 23 Global Step: 238810 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:33:15,740-Speed 128.84 samples/sec Loss 0.3589 Epoch: 23 Global Step: 238815 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:33:35,462-Speed 129.81 samples/sec Loss 0.3042 Epoch: 23 Global Step: 238820 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:33:55,049-Speed 130.70 samples/sec Loss 0.2877 Epoch: 23 Global Step: 238825 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:34:14,754-Speed 129.93 samples/sec Loss 0.3529 Epoch: 23 Global Step: 238830 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:34:34,362-Speed 130.56 samples/sec Loss 0.3984 Epoch: 23 Global Step: 238835 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:34:53,967-Speed 130.58 samples/sec Loss 0.2936 Epoch: 23 Global Step: 238840 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:35:13,851-Speed 128.76 samples/sec Loss 0.3110 Epoch: 23 Global Step: 238845 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:35:33,605-Speed 129.59 samples/sec Loss 0.3168 Epoch: 23 Global Step: 238850 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:35:53,933-Speed 125.94 samples/sec Loss 0.2398 Epoch: 23 Global Step: 238855 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:36:13,733-Speed 129.30 samples/sec Loss 0.3042 Epoch: 23 Global Step: 238860 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:36:33,789-Speed 127.65 samples/sec Loss 0.2792 Epoch: 23 Global Step: 238865 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:36:53,350-Speed 130.88 samples/sec Loss 0.3405 Epoch: 23 Global Step: 238870 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:37:13,058-Speed 129.90 samples/sec Loss 0.3023 Epoch: 23 Global Step: 238875 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:37:32,744-Speed 130.05 samples/sec Loss 0.4096 Epoch: 23 Global Step: 238880 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:37:52,373-Speed 130.43 samples/sec Loss 0.3413 Epoch: 23 Global Step: 238885 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:38:12,553-Speed 126.87 samples/sec Loss 0.2761 Epoch: 23 Global Step: 238890 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:38:32,257-Speed 129.92 samples/sec Loss 0.2866 Epoch: 23 Global Step: 238895 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:38:52,030-Speed 129.48 samples/sec Loss 0.3161 Epoch: 23 Global Step: 238900 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:39:11,814-Speed 129.40 samples/sec Loss 0.2799 Epoch: 23 Global Step: 238905 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:39:31,628-Speed 129.21 samples/sec Loss 0.3223 Epoch: 23 Global Step: 238910 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:39:51,161-Speed 131.07 samples/sec Loss 0.3706 Epoch: 23 Global Step: 238915 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:40:10,858-Speed 129.97 samples/sec Loss 0.3601 Epoch: 23 Global Step: 238920 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:40:30,471-Speed 130.54 samples/sec Loss 0.3214 Epoch: 23 Global Step: 238925 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:40:50,158-Speed 130.04 samples/sec Loss 0.3325 Epoch: 23 Global Step: 238930 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:41:10,055-Speed 128.67 samples/sec Loss 0.2821 Epoch: 23 Global Step: 238935 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:41:29,807-Speed 129.61 samples/sec Loss 0.2515 Epoch: 23 Global Step: 238940 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:41:49,519-Speed 129.88 samples/sec Loss 0.2987 Epoch: 23 Global Step: 238945 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:42:09,419-Speed 128.65 samples/sec Loss 0.3196 Epoch: 23 Global Step: 238950 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:42:28,588-Speed 133.55 samples/sec Loss 0.2828 Epoch: 23 Global Step: 238955 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:42:48,223-Speed 130.39 samples/sec Loss 0.3153 Epoch: 23 Global Step: 238960 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:43:07,759-Speed 131.05 samples/sec Loss 0.2742 Epoch: 23 Global Step: 238965 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:43:27,654-Speed 128.68 samples/sec Loss 0.3335 Epoch: 23 Global Step: 238970 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:43:47,396-Speed 129.69 samples/sec Loss 0.3298 Epoch: 23 Global Step: 238975 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:44:07,298-Speed 128.64 samples/sec Loss 0.3036 Epoch: 23 Global Step: 238980 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:44:26,966-Speed 130.16 samples/sec Loss 0.3430 Epoch: 23 Global Step: 238985 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:44:46,748-Speed 129.42 samples/sec Loss 0.2857 Epoch: 23 Global Step: 238990 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:45:06,992-Speed 126.47 samples/sec Loss 0.3240 Epoch: 23 Global Step: 238995 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:45:26,987-Speed 128.03 samples/sec Loss 0.3171 Epoch: 23 Global Step: 239000 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:45:46,973-Speed 128.10 samples/sec Loss 0.3394 Epoch: 23 Global Step: 239005 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:46:06,756-Speed 129.41 samples/sec Loss 0.3582 Epoch: 23 Global Step: 239010 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:46:26,542-Speed 129.39 samples/sec Loss 0.3441 Epoch: 23 Global Step: 239015 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:46:46,377-Speed 129.07 samples/sec Loss 0.3234 Epoch: 23 Global Step: 239020 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:47:06,238-Speed 128.90 samples/sec Loss 0.2684 Epoch: 23 Global Step: 239025 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:47:25,852-Speed 130.53 samples/sec Loss 0.2700 Epoch: 23 Global Step: 239030 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:47:45,742-Speed 128.72 samples/sec Loss 0.3424 Epoch: 23 Global Step: 239035 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:48:05,829-Speed 127.45 samples/sec Loss 0.3062 Epoch: 23 Global Step: 239040 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:48:25,607-Speed 129.44 samples/sec Loss 0.3437 Epoch: 23 Global Step: 239045 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:48:45,542-Speed 128.44 samples/sec Loss 0.3314 Epoch: 23 Global Step: 239050 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 16:49:05,432-Speed 128.71 samples/sec Loss 0.3681 Epoch: 23 Global Step: 239055 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:49:25,154-Speed 129.81 samples/sec Loss 0.3625 Epoch: 23 Global Step: 239060 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:49:44,984-Speed 129.10 samples/sec Loss 0.2602 Epoch: 23 Global Step: 239065 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:50:04,827-Speed 129.02 samples/sec Loss 0.3413 Epoch: 23 Global Step: 239070 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:50:24,471-Speed 130.33 samples/sec Loss 0.3434 Epoch: 23 Global Step: 239075 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:50:44,379-Speed 128.60 samples/sec Loss 0.2762 Epoch: 23 Global Step: 239080 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:51:04,467-Speed 127.45 samples/sec Loss 0.3470 Epoch: 23 Global Step: 239085 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:51:24,266-Speed 129.30 samples/sec Loss 0.3157 Epoch: 23 Global Step: 239090 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:51:44,419-Speed 127.04 samples/sec Loss 0.3096 Epoch: 23 Global Step: 239095 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:52:04,886-Speed 125.09 samples/sec Loss 0.3279 Epoch: 23 Global Step: 239100 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:52:24,653-Speed 129.51 samples/sec Loss 0.2783 Epoch: 23 Global Step: 239105 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:52:44,654-Speed 128.00 samples/sec Loss 0.2849 Epoch: 23 Global Step: 239110 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:53:04,392-Speed 129.70 samples/sec Loss 0.3211 Epoch: 23 Global Step: 239115 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:53:24,046-Speed 130.26 samples/sec Loss 0.3687 Epoch: 23 Global Step: 239120 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:53:43,931-Speed 128.75 samples/sec Loss 0.2495 Epoch: 23 Global Step: 239125 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:54:03,864-Speed 128.43 samples/sec Loss 0.2862 Epoch: 23 Global Step: 239130 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:54:23,494-Speed 130.42 samples/sec Loss 0.3239 Epoch: 23 Global Step: 239135 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:55:02,574-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth' Training: 2021-06-18 16:55:25,492-Speed 41.29 samples/sec Loss 0.3543 Epoch: 24 Global Step: 239140 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:55:45,647-Speed 127.02 samples/sec Loss 0.2616 Epoch: 24 Global Step: 239145 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:56:05,739-Speed 127.42 samples/sec Loss 0.2801 Epoch: 24 Global Step: 239150 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:56:25,666-Speed 128.47 samples/sec Loss 0.2589 Epoch: 24 Global Step: 239155 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:56:45,800-Speed 127.16 samples/sec Loss 0.3056 Epoch: 24 Global Step: 239160 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:57:05,821-Speed 127.87 samples/sec Loss 0.2775 Epoch: 24 Global Step: 239165 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:57:25,570-Speed 129.63 samples/sec Loss 0.2835 Epoch: 24 Global Step: 239170 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:57:45,622-Speed 127.67 samples/sec Loss 0.3035 Epoch: 24 Global Step: 239175 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:58:05,517-Speed 128.69 samples/sec Loss 0.2810 Epoch: 24 Global Step: 239180 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:58:25,247-Speed 129.76 samples/sec Loss 0.2174 Epoch: 24 Global Step: 239185 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:58:45,164-Speed 128.60 samples/sec Loss 0.3139 Epoch: 24 Global Step: 239190 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:59:05,130-Speed 128.22 samples/sec Loss 0.2590 Epoch: 24 Global Step: 239195 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:59:25,048-Speed 128.54 samples/sec Loss 0.2930 Epoch: 24 Global Step: 239200 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 16:59:44,790-Speed 129.68 samples/sec Loss 0.2804 Epoch: 24 Global Step: 239205 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:00:04,609-Speed 129.18 samples/sec Loss 0.2707 Epoch: 24 Global Step: 239210 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:00:24,401-Speed 129.35 samples/sec Loss 0.3053 Epoch: 24 Global Step: 239215 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:00:43,723-Speed 132.50 samples/sec Loss 0.3028 Epoch: 24 Global Step: 239220 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:01:03,613-Speed 128.71 samples/sec Loss 0.2935 Epoch: 24 Global Step: 239225 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:01:23,594-Speed 128.13 samples/sec Loss 0.3631 Epoch: 24 Global Step: 239230 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:01:43,576-Speed 128.12 samples/sec Loss 0.3247 Epoch: 24 Global Step: 239235 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:02:03,509-Speed 128.44 samples/sec Loss 0.2591 Epoch: 24 Global Step: 239240 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:02:23,369-Speed 128.91 samples/sec Loss 0.2604 Epoch: 24 Global Step: 239245 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:02:43,003-Speed 130.39 samples/sec Loss 0.2645 Epoch: 24 Global Step: 239250 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:03:02,893-Speed 128.72 samples/sec Loss 0.2329 Epoch: 24 Global Step: 239255 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:03:22,413-Speed 131.15 samples/sec Loss 0.2301 Epoch: 24 Global Step: 239260 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:03:42,247-Speed 129.08 samples/sec Loss 0.2390 Epoch: 24 Global Step: 239265 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:04:01,828-Speed 130.75 samples/sec Loss 0.2917 Epoch: 24 Global Step: 239270 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:04:21,559-Speed 129.75 samples/sec Loss 0.2687 Epoch: 24 Global Step: 239275 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:04:41,178-Speed 130.49 samples/sec Loss 0.2364 Epoch: 24 Global Step: 239280 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:05:01,015-Speed 129.06 samples/sec Loss 0.3210 Epoch: 24 Global Step: 239285 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:05:20,944-Speed 128.46 samples/sec Loss 0.3157 Epoch: 24 Global Step: 239290 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:05:40,375-Speed 131.76 samples/sec Loss 0.2895 Epoch: 24 Global Step: 239295 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:06:00,362-Speed 128.09 samples/sec Loss 0.2658 Epoch: 24 Global Step: 239300 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:06:19,826-Speed 131.53 samples/sec Loss 0.3179 Epoch: 24 Global Step: 239305 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:06:39,740-Speed 128.56 samples/sec Loss 0.3539 Epoch: 24 Global Step: 239310 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:06:59,290-Speed 130.95 samples/sec Loss 0.2729 Epoch: 24 Global Step: 239315 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:07:18,933-Speed 130.33 samples/sec Loss 0.2449 Epoch: 24 Global Step: 239320 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:07:38,572-Speed 130.36 samples/sec Loss 0.3882 Epoch: 24 Global Step: 239325 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:07:58,791-Speed 126.62 samples/sec Loss 0.2877 Epoch: 24 Global Step: 239330 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:08:18,608-Speed 129.19 samples/sec Loss 0.3149 Epoch: 24 Global Step: 239335 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:08:38,131-Speed 131.14 samples/sec Loss 0.2840 Epoch: 24 Global Step: 239340 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:08:57,942-Speed 129.23 samples/sec Loss 0.2423 Epoch: 24 Global Step: 239345 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:09:16,936-Speed 134.79 samples/sec Loss 0.3351 Epoch: 24 Global Step: 239350 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:09:36,819-Speed 128.76 samples/sec Loss 0.2539 Epoch: 24 Global Step: 239355 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:09:56,371-Speed 130.94 samples/sec Loss 0.2890 Epoch: 24 Global Step: 239360 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:10:16,188-Speed 129.19 samples/sec Loss 0.2928 Epoch: 24 Global Step: 239365 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:10:35,763-Speed 130.78 samples/sec Loss 0.3278 Epoch: 24 Global Step: 239370 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:10:55,767-Speed 127.98 samples/sec Loss 0.2694 Epoch: 24 Global Step: 239375 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:11:15,519-Speed 129.62 samples/sec Loss 0.3119 Epoch: 24 Global Step: 239380 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:11:35,043-Speed 131.12 samples/sec Loss 0.3455 Epoch: 24 Global Step: 239385 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:11:54,943-Speed 128.65 samples/sec Loss 0.2163 Epoch: 24 Global Step: 239390 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:12:14,389-Speed 131.66 samples/sec Loss 0.2431 Epoch: 24 Global Step: 239395 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:12:34,233-Speed 129.01 samples/sec Loss 0.2313 Epoch: 24 Global Step: 239400 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:12:53,770-Speed 131.04 samples/sec Loss 0.3484 Epoch: 24 Global Step: 239405 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:13:13,519-Speed 129.64 samples/sec Loss 0.2564 Epoch: 24 Global Step: 239410 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:13:33,123-Speed 130.60 samples/sec Loss 0.3125 Epoch: 24 Global Step: 239415 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:13:53,110-Speed 128.09 samples/sec Loss 0.3661 Epoch: 24 Global Step: 239420 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:14:12,849-Speed 129.70 samples/sec Loss 0.2956 Epoch: 24 Global Step: 239425 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:14:32,462-Speed 130.53 samples/sec Loss 0.2780 Epoch: 24 Global Step: 239430 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:14:52,379-Speed 128.54 samples/sec Loss 0.3076 Epoch: 24 Global Step: 239435 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:15:11,857-Speed 131.43 samples/sec Loss 0.2566 Epoch: 24 Global Step: 239440 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:15:31,754-Speed 128.68 samples/sec Loss 0.2754 Epoch: 24 Global Step: 239445 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-06-18 17:15:51,290-Speed 131.05 samples/sec Loss 0.2940 Epoch: 24 Global Step: 239450 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:16:11,193-Speed 128.63 samples/sec Loss 0.3058 Epoch: 24 Global Step: 239455 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:16:30,700-Speed 131.24 samples/sec Loss 0.2825 Epoch: 24 Global Step: 239460 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:16:50,790-Speed 127.43 samples/sec Loss 0.2753 Epoch: 24 Global Step: 239465 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:17:10,433-Speed 130.33 samples/sec Loss 0.2529 Epoch: 24 Global Step: 239470 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:17:29,980-Speed 130.97 samples/sec Loss 0.2732 Epoch: 24 Global Step: 239475 Fp16 Grad Scale: 8192 Required: 12 hours Training: 2021-06-18 17:17:49,764-Speed 129.41 samples/sec Loss 0.2894 Epoch: 24 Global Step: 239480 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:18:09,350-Speed 130.71 samples/sec Loss 0.3224 Epoch: 24 Global Step: 239485 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:18:29,242-Speed 128.70 samples/sec Loss 0.2838 Epoch: 24 Global Step: 239490 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:18:48,709-Speed 131.51 samples/sec Loss 0.3132 Epoch: 24 Global Step: 239495 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:19:08,438-Speed 129.76 samples/sec Loss 0.2612 Epoch: 24 Global Step: 239500 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:19:28,053-Speed 130.52 samples/sec Loss 0.2553 Epoch: 24 Global Step: 239505 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:19:48,187-Speed 127.16 samples/sec Loss 0.2782 Epoch: 24 Global Step: 239510 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:20:07,920-Speed 129.74 samples/sec Loss 0.2793 Epoch: 24 Global Step: 239515 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:20:27,477-Speed 130.90 samples/sec Loss 0.3150 Epoch: 24 Global Step: 239520 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:20:47,305-Speed 129.12 samples/sec Loss 0.3510 Epoch: 24 Global Step: 239525 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:21:06,911-Speed 130.58 samples/sec Loss 0.2951 Epoch: 24 Global Step: 239530 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:21:26,792-Speed 128.77 samples/sec Loss 0.2850 Epoch: 24 Global Step: 239535 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:21:46,319-Speed 131.10 samples/sec Loss 0.3729 Epoch: 24 Global Step: 239540 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:22:06,086-Speed 129.52 samples/sec Loss 0.3124 Epoch: 24 Global Step: 239545 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:22:25,428-Speed 132.36 samples/sec Loss 0.2966 Epoch: 24 Global Step: 239550 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:22:45,144-Speed 129.84 samples/sec Loss 0.3116 Epoch: 24 Global Step: 239555 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:23:05,246-Speed 127.36 samples/sec Loss 0.2662 Epoch: 24 Global Step: 239560 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:23:24,971-Speed 129.79 samples/sec Loss 0.2622 Epoch: 24 Global Step: 239565 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:23:45,197-Speed 126.58 samples/sec Loss 0.2271 Epoch: 24 Global Step: 239570 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:24:04,785-Speed 130.70 samples/sec Loss 0.2758 Epoch: 24 Global Step: 239575 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:24:24,754-Speed 128.21 samples/sec Loss 0.2295 Epoch: 24 Global Step: 239580 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:24:44,638-Speed 128.75 samples/sec Loss 0.3113 Epoch: 24 Global Step: 239585 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:25:04,438-Speed 129.30 samples/sec Loss 0.3111 Epoch: 24 Global Step: 239590 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:25:24,062-Speed 130.46 samples/sec Loss 0.3135 Epoch: 24 Global Step: 239595 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:25:43,804-Speed 129.67 samples/sec Loss 0.3147 Epoch: 24 Global Step: 239600 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:26:04,236-Speed 125.30 samples/sec Loss 0.2628 Epoch: 24 Global Step: 239605 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:26:23,911-Speed 130.12 samples/sec Loss 0.3387 Epoch: 24 Global Step: 239610 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:26:43,739-Speed 129.12 samples/sec Loss 0.3131 Epoch: 24 Global Step: 239615 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:27:03,566-Speed 129.12 samples/sec Loss 0.2571 Epoch: 24 Global Step: 239620 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:27:23,563-Speed 128.02 samples/sec Loss 0.3197 Epoch: 24 Global Step: 239625 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:27:43,280-Speed 129.84 samples/sec Loss 0.2492 Epoch: 24 Global Step: 239630 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:28:03,122-Speed 129.03 samples/sec Loss 0.3071 Epoch: 24 Global Step: 239635 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:28:22,867-Speed 129.66 samples/sec Loss 0.2739 Epoch: 24 Global Step: 239640 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:28:42,607-Speed 129.69 samples/sec Loss 0.3024 Epoch: 24 Global Step: 239645 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:29:02,766-Speed 127.00 samples/sec Loss 0.3561 Epoch: 24 Global Step: 239650 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:29:22,380-Speed 130.53 samples/sec Loss 0.2852 Epoch: 24 Global Step: 239655 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:29:42,291-Speed 128.58 samples/sec Loss 0.2501 Epoch: 24 Global Step: 239660 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:30:02,182-Speed 128.71 samples/sec Loss 0.2771 Epoch: 24 Global Step: 239665 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:30:22,419-Speed 126.51 samples/sec Loss 0.3295 Epoch: 24 Global Step: 239670 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:30:42,219-Speed 129.30 samples/sec Loss 0.3741 Epoch: 24 Global Step: 239675 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:31:02,002-Speed 129.41 samples/sec Loss 0.2789 Epoch: 24 Global Step: 239680 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:31:21,726-Speed 129.80 samples/sec Loss 0.2739 Epoch: 24 Global Step: 239685 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:31:41,735-Speed 127.95 samples/sec Loss 0.2755 Epoch: 24 Global Step: 239690 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:32:01,048-Speed 132.56 samples/sec Loss 0.2677 Epoch: 24 Global Step: 239695 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:32:20,803-Speed 129.59 samples/sec Loss 0.3169 Epoch: 24 Global Step: 239700 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:32:40,805-Speed 127.99 samples/sec Loss 0.2228 Epoch: 24 Global Step: 239705 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:33:00,591-Speed 129.39 samples/sec Loss 0.2814 Epoch: 24 Global Step: 239710 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:33:20,648-Speed 127.64 samples/sec Loss 0.2660 Epoch: 24 Global Step: 239715 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:33:40,317-Speed 130.16 samples/sec Loss 0.2519 Epoch: 24 Global Step: 239720 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:34:00,184-Speed 128.86 samples/sec Loss 0.2442 Epoch: 24 Global Step: 239725 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:34:19,896-Speed 129.88 samples/sec Loss 0.3115 Epoch: 24 Global Step: 239730 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:34:39,427-Speed 131.08 samples/sec Loss 0.2855 Epoch: 24 Global Step: 239735 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:34:59,290-Speed 128.90 samples/sec Loss 0.3067 Epoch: 24 Global Step: 239740 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:35:18,806-Speed 131.18 samples/sec Loss 0.2590 Epoch: 24 Global Step: 239745 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:35:38,714-Speed 128.60 samples/sec Loss 0.2583 Epoch: 24 Global Step: 239750 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:35:58,346-Speed 130.41 samples/sec Loss 0.2917 Epoch: 24 Global Step: 239755 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:36:18,350-Speed 127.98 samples/sec Loss 0.2492 Epoch: 24 Global Step: 239760 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:36:38,035-Speed 130.05 samples/sec Loss 0.2657 Epoch: 24 Global Step: 239765 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:36:57,636-Speed 130.61 samples/sec Loss 0.2870 Epoch: 24 Global Step: 239770 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:37:17,408-Speed 129.49 samples/sec Loss 0.2389 Epoch: 24 Global Step: 239775 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:37:36,944-Speed 131.05 samples/sec Loss 0.2763 Epoch: 24 Global Step: 239780 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:37:56,789-Speed 129.00 samples/sec Loss 0.2838 Epoch: 24 Global Step: 239785 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:38:16,313-Speed 131.13 samples/sec Loss 0.2886 Epoch: 24 Global Step: 239790 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:38:36,202-Speed 128.72 samples/sec Loss 0.2911 Epoch: 24 Global Step: 239795 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:38:55,809-Speed 130.57 samples/sec Loss 0.3402 Epoch: 24 Global Step: 239800 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:39:15,849-Speed 127.75 samples/sec Loss 0.2774 Epoch: 24 Global Step: 239805 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:39:35,518-Speed 130.16 samples/sec Loss 0.2593 Epoch: 24 Global Step: 239810 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:39:54,694-Speed 133.51 samples/sec Loss 0.2813 Epoch: 24 Global Step: 239815 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:40:14,339-Speed 130.32 samples/sec Loss 0.3159 Epoch: 24 Global Step: 239820 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:40:33,910-Speed 130.81 samples/sec Loss 0.3011 Epoch: 24 Global Step: 239825 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:40:53,726-Speed 129.20 samples/sec Loss 0.2653 Epoch: 24 Global Step: 239830 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:41:13,185-Speed 131.57 samples/sec Loss 0.3322 Epoch: 24 Global Step: 239835 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:41:33,044-Speed 128.92 samples/sec Loss 0.3086 Epoch: 24 Global Step: 239840 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:41:52,499-Speed 131.59 samples/sec Loss 0.3294 Epoch: 24 Global Step: 239845 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:42:12,646-Speed 127.07 samples/sec Loss 0.3258 Epoch: 24 Global Step: 239850 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:42:32,283-Speed 130.37 samples/sec Loss 0.2811 Epoch: 24 Global Step: 239855 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:42:51,905-Speed 130.47 samples/sec Loss 0.2583 Epoch: 24 Global Step: 239860 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:43:11,592-Speed 130.04 samples/sec Loss 0.2968 Epoch: 24 Global Step: 239865 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:43:31,153-Speed 130.88 samples/sec Loss 0.3645 Epoch: 24 Global Step: 239870 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:43:51,129-Speed 128.16 samples/sec Loss 0.3627 Epoch: 24 Global Step: 239875 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:44:10,575-Speed 131.66 samples/sec Loss 0.3619 Epoch: 24 Global Step: 239880 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:44:30,530-Speed 128.29 samples/sec Loss 0.2593 Epoch: 24 Global Step: 239885 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:44:50,122-Speed 130.67 samples/sec Loss 0.3373 Epoch: 24 Global Step: 239890 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:45:10,228-Speed 127.34 samples/sec Loss 0.2865 Epoch: 24 Global Step: 239895 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:45:29,986-Speed 129.57 samples/sec Loss 0.3208 Epoch: 24 Global Step: 239900 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:45:49,570-Speed 130.73 samples/sec Loss 0.2599 Epoch: 24 Global Step: 239905 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:46:09,206-Speed 130.38 samples/sec Loss 0.2721 Epoch: 24 Global Step: 239910 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:46:28,813-Speed 130.57 samples/sec Loss 0.3592 Epoch: 24 Global Step: 239915 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:46:48,651-Speed 129.05 samples/sec Loss 0.3474 Epoch: 24 Global Step: 239920 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:47:08,218-Speed 130.84 samples/sec Loss 0.2631 Epoch: 24 Global Step: 239925 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:47:28,112-Speed 128.69 samples/sec Loss 0.3363 Epoch: 24 Global Step: 239930 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:47:47,679-Speed 130.84 samples/sec Loss 0.3434 Epoch: 24 Global Step: 239935 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:48:07,844-Speed 126.96 samples/sec Loss 0.3438 Epoch: 24 Global Step: 239940 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:48:27,565-Speed 129.82 samples/sec Loss 0.3164 Epoch: 24 Global Step: 239945 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:48:47,197-Speed 130.41 samples/sec Loss 0.2711 Epoch: 24 Global Step: 239950 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:49:06,925-Speed 129.77 samples/sec Loss 0.3127 Epoch: 24 Global Step: 239955 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:49:26,536-Speed 130.55 samples/sec Loss 0.2669 Epoch: 24 Global Step: 239960 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:49:46,393-Speed 128.93 samples/sec Loss 0.2856 Epoch: 24 Global Step: 239965 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:50:05,951-Speed 130.90 samples/sec Loss 0.3440 Epoch: 24 Global Step: 239970 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:50:25,880-Speed 128.46 samples/sec Loss 0.2665 Epoch: 24 Global Step: 239975 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 17:50:45,332-Speed 131.62 samples/sec Loss 0.2826 Epoch: 24 Global Step: 239980 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:51:05,302-Speed 128.19 samples/sec Loss 0.3232 Epoch: 24 Global Step: 239985 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:51:25,295-Speed 128.05 samples/sec Loss 0.3329 Epoch: 24 Global Step: 239990 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:51:45,060-Speed 129.53 samples/sec Loss 0.3042 Epoch: 24 Global Step: 239995 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 17:52:05,112-Speed 127.67 samples/sec Loss 0.3160 Epoch: 24 Global Step: 240000 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:06:55,767-[lfw][240000]XNorm: 22.079786 Training: 2021-06-18 18:06:55,767-[lfw][240000]Accuracy-Flip: 0.99817+-0.00203 Training: 2021-06-18 18:06:55,767-[lfw][240000]Accuracy-Highest: 0.99867 Training: 2021-06-18 18:24:12,347-[cfp_fp][240000]XNorm: 23.795517 Training: 2021-06-18 18:24:12,347-[cfp_fp][240000]Accuracy-Flip: 0.99243+-0.00286 Training: 2021-06-18 18:24:12,348-[cfp_fp][240000]Accuracy-Highest: 0.99243 Training: 2021-06-18 18:39:01,424-[agedb_30][240000]XNorm: 23.137918 Training: 2021-06-18 18:39:01,424-[agedb_30][240000]Accuracy-Flip: 0.98600+-0.00554 Training: 2021-06-18 18:39:01,424-[agedb_30][240000]Accuracy-Highest: 0.98667 Training: 2021-06-18 18:39:21,440-Speed 0.90 samples/sec Loss 0.2507 Epoch: 24 Global Step: 240005 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:39:41,248-Speed 129.24 samples/sec Loss 0.2747 Epoch: 24 Global Step: 240010 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:40:00,936-Speed 130.04 samples/sec Loss 0.3670 Epoch: 24 Global Step: 240015 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:40:20,836-Speed 128.65 samples/sec Loss 0.2243 Epoch: 24 Global Step: 240020 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:40:40,548-Speed 129.88 samples/sec Loss 0.2676 Epoch: 24 Global Step: 240025 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:41:00,617-Speed 127.56 samples/sec Loss 0.2636 Epoch: 24 Global Step: 240030 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:41:20,481-Speed 128.89 samples/sec Loss 0.3347 Epoch: 24 Global Step: 240035 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:41:40,104-Speed 130.46 samples/sec Loss 0.3113 Epoch: 24 Global Step: 240040 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:42:00,012-Speed 128.60 samples/sec Loss 0.2994 Epoch: 24 Global Step: 240045 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:42:19,738-Speed 129.79 samples/sec Loss 0.3293 Epoch: 24 Global Step: 240050 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:42:39,365-Speed 130.44 samples/sec Loss 0.3379 Epoch: 24 Global Step: 240055 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:42:59,364-Speed 128.01 samples/sec Loss 0.3158 Epoch: 24 Global Step: 240060 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:43:19,185-Speed 129.16 samples/sec Loss 0.2650 Epoch: 24 Global Step: 240065 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:43:38,993-Speed 129.25 samples/sec Loss 0.3595 Epoch: 24 Global Step: 240070 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:43:58,926-Speed 128.44 samples/sec Loss 0.2604 Epoch: 24 Global Step: 240075 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:44:18,896-Speed 128.19 samples/sec Loss 0.2660 Epoch: 24 Global Step: 240080 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:44:38,627-Speed 129.75 samples/sec Loss 0.3513 Epoch: 24 Global Step: 240085 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:44:58,046-Speed 131.83 samples/sec Loss 0.2971 Epoch: 24 Global Step: 240090 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:45:17,781-Speed 129.73 samples/sec Loss 0.2464 Epoch: 24 Global Step: 240095 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:45:37,390-Speed 130.56 samples/sec Loss 0.2651 Epoch: 24 Global Step: 240100 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:45:57,420-Speed 127.82 samples/sec Loss 0.3317 Epoch: 24 Global Step: 240105 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:46:17,307-Speed 128.73 samples/sec Loss 0.2853 Epoch: 24 Global Step: 240110 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:46:37,053-Speed 129.65 samples/sec Loss 0.2942 Epoch: 24 Global Step: 240115 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:46:56,766-Speed 129.87 samples/sec Loss 0.2621 Epoch: 24 Global Step: 240120 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:47:16,525-Speed 129.57 samples/sec Loss 0.2544 Epoch: 24 Global Step: 240125 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:47:36,126-Speed 130.61 samples/sec Loss 0.2965 Epoch: 24 Global Step: 240130 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:47:55,906-Speed 129.43 samples/sec Loss 0.3293 Epoch: 24 Global Step: 240135 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:48:15,846-Speed 128.40 samples/sec Loss 0.2117 Epoch: 24 Global Step: 240140 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:48:35,566-Speed 129.82 samples/sec Loss 0.2397 Epoch: 24 Global Step: 240145 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:48:55,759-Speed 126.78 samples/sec Loss 0.2415 Epoch: 24 Global Step: 240150 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:49:15,600-Speed 129.03 samples/sec Loss 0.2514 Epoch: 24 Global Step: 240155 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:49:35,400-Speed 129.30 samples/sec Loss 0.2322 Epoch: 24 Global Step: 240160 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:49:55,117-Speed 129.84 samples/sec Loss 0.2933 Epoch: 24 Global Step: 240165 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:50:14,921-Speed 129.27 samples/sec Loss 0.2805 Epoch: 24 Global Step: 240170 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:50:34,541-Speed 130.49 samples/sec Loss 0.3577 Epoch: 24 Global Step: 240175 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:50:54,329-Speed 129.38 samples/sec Loss 0.2826 Epoch: 24 Global Step: 240180 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:51:14,251-Speed 128.51 samples/sec Loss 0.2753 Epoch: 24 Global Step: 240185 Fp16 Grad Scale: 4096 Required: 11 hours Training: 2021-06-18 18:51:33,847-Speed 130.64 samples/sec Loss 0.2643 Epoch: 24 Global Step: 240190 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:51:54,025-Speed 126.88 samples/sec Loss 0.2652 Epoch: 24 Global Step: 240195 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:52:13,987-Speed 128.24 samples/sec Loss 0.3103 Epoch: 24 Global Step: 240200 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:52:33,791-Speed 129.28 samples/sec Loss 0.2855 Epoch: 24 Global Step: 240205 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:52:53,446-Speed 130.25 samples/sec Loss 0.2861 Epoch: 24 Global Step: 240210 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:53:13,290-Speed 129.01 samples/sec Loss 0.2731 Epoch: 24 Global Step: 240215 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:53:32,912-Speed 130.48 samples/sec Loss 0.2744 Epoch: 24 Global Step: 240220 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:53:52,795-Speed 128.76 samples/sec Loss 0.3217 Epoch: 24 Global Step: 240225 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:54:12,619-Speed 129.16 samples/sec Loss 0.3278 Epoch: 24 Global Step: 240230 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:54:32,245-Speed 130.45 samples/sec Loss 0.2991 Epoch: 24 Global Step: 240235 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:54:52,423-Speed 126.88 samples/sec Loss 0.2608 Epoch: 24 Global Step: 240240 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:55:12,495-Speed 127.55 samples/sec Loss 0.2913 Epoch: 24 Global Step: 240245 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:55:32,318-Speed 129.15 samples/sec Loss 0.2885 Epoch: 24 Global Step: 240250 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:55:51,967-Speed 130.29 samples/sec Loss 0.2816 Epoch: 24 Global Step: 240255 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:56:11,744-Speed 129.45 samples/sec Loss 0.2730 Epoch: 24 Global Step: 240260 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:56:31,471-Speed 129.78 samples/sec Loss 0.2960 Epoch: 24 Global Step: 240265 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:56:51,299-Speed 129.12 samples/sec Loss 0.2640 Epoch: 24 Global Step: 240270 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:57:11,158-Speed 128.91 samples/sec Loss 0.2472 Epoch: 24 Global Step: 240275 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:57:30,822-Speed 130.20 samples/sec Loss 0.3243 Epoch: 24 Global Step: 240280 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:57:51,064-Speed 126.48 samples/sec Loss 0.3173 Epoch: 24 Global Step: 240285 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:58:10,976-Speed 128.57 samples/sec Loss 0.2689 Epoch: 24 Global Step: 240290 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:58:30,766-Speed 129.36 samples/sec Loss 0.3028 Epoch: 24 Global Step: 240295 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:58:50,421-Speed 130.25 samples/sec Loss 0.2382 Epoch: 24 Global Step: 240300 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:59:10,196-Speed 129.47 samples/sec Loss 0.3418 Epoch: 24 Global Step: 240305 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:59:29,939-Speed 129.67 samples/sec Loss 0.2776 Epoch: 24 Global Step: 240310 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 18:59:49,822-Speed 128.76 samples/sec Loss 0.2896 Epoch: 24 Global Step: 240315 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 19:00:09,647-Speed 129.14 samples/sec Loss 0.2522 Epoch: 24 Global Step: 240320 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 19:00:29,289-Speed 130.34 samples/sec Loss 0.2847 Epoch: 24 Global Step: 240325 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 19:00:49,541-Speed 126.41 samples/sec Loss 0.2571 Epoch: 24 Global Step: 240330 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 19:01:09,429-Speed 128.73 samples/sec Loss 0.3055 Epoch: 24 Global Step: 240335 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-06-18 19:01:29,146-Speed 129.84 samples/sec Loss 0.3181 Epoch: 24 Global Step: 240340 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:01:48,843-Speed 129.98 samples/sec Loss 0.3094 Epoch: 24 Global Step: 240345 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:02:08,564-Speed 129.81 samples/sec Loss 0.2815 Epoch: 24 Global Step: 240350 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:02:28,309-Speed 129.66 samples/sec Loss 0.2562 Epoch: 24 Global Step: 240355 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:02:48,211-Speed 128.63 samples/sec Loss 0.2493 Epoch: 24 Global Step: 240360 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:03:08,044-Speed 129.08 samples/sec Loss 0.2529 Epoch: 24 Global Step: 240365 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:03:27,693-Speed 130.29 samples/sec Loss 0.3273 Epoch: 24 Global Step: 240370 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:03:47,926-Speed 126.53 samples/sec Loss 0.2795 Epoch: 24 Global Step: 240375 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:04:07,906-Speed 128.13 samples/sec Loss 0.2660 Epoch: 24 Global Step: 240380 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:04:27,800-Speed 128.69 samples/sec Loss 0.3049 Epoch: 24 Global Step: 240385 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:04:47,603-Speed 129.28 samples/sec Loss 0.3864 Epoch: 24 Global Step: 240390 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:05:07,284-Speed 130.08 samples/sec Loss 0.2718 Epoch: 24 Global Step: 240395 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:05:27,103-Speed 129.18 samples/sec Loss 0.2955 Epoch: 24 Global Step: 240400 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:05:47,075-Speed 128.19 samples/sec Loss 0.2807 Epoch: 24 Global Step: 240405 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:06:06,806-Speed 129.75 samples/sec Loss 0.3965 Epoch: 24 Global Step: 240410 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:06:26,592-Speed 129.39 samples/sec Loss 0.3004 Epoch: 24 Global Step: 240415 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:06:46,710-Speed 127.26 samples/sec Loss 0.3149 Epoch: 24 Global Step: 240420 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:07:06,648-Speed 128.41 samples/sec Loss 0.2609 Epoch: 24 Global Step: 240425 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:07:26,367-Speed 129.83 samples/sec Loss 0.2525 Epoch: 24 Global Step: 240430 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:07:46,181-Speed 129.21 samples/sec Loss 0.2658 Epoch: 24 Global Step: 240435 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:08:05,883-Speed 129.94 samples/sec Loss 0.2233 Epoch: 24 Global Step: 240440 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:08:25,598-Speed 129.86 samples/sec Loss 0.2484 Epoch: 24 Global Step: 240445 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:08:45,496-Speed 128.67 samples/sec Loss 0.3271 Epoch: 24 Global Step: 240450 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:09:05,214-Speed 129.84 samples/sec Loss 0.2097 Epoch: 24 Global Step: 240455 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:09:25,001-Speed 129.38 samples/sec Loss 0.2284 Epoch: 24 Global Step: 240460 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:09:45,165-Speed 126.97 samples/sec Loss 0.2410 Epoch: 24 Global Step: 240465 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:10:05,203-Speed 127.76 samples/sec Loss 0.3297 Epoch: 24 Global Step: 240470 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:10:24,899-Speed 129.98 samples/sec Loss 0.2801 Epoch: 24 Global Step: 240475 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:10:44,725-Speed 129.13 samples/sec Loss 0.2614 Epoch: 24 Global Step: 240480 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:11:04,424-Speed 129.96 samples/sec Loss 0.2539 Epoch: 24 Global Step: 240485 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:11:24,172-Speed 129.64 samples/sec Loss 0.3408 Epoch: 24 Global Step: 240490 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:11:44,044-Speed 128.83 samples/sec Loss 0.3144 Epoch: 24 Global Step: 240495 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:12:03,779-Speed 129.72 samples/sec Loss 0.2914 Epoch: 24 Global Step: 240500 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:12:23,580-Speed 129.29 samples/sec Loss 0.2957 Epoch: 24 Global Step: 240505 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:12:43,795-Speed 126.64 samples/sec Loss 0.2637 Epoch: 24 Global Step: 240510 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:13:03,668-Speed 128.82 samples/sec Loss 0.3456 Epoch: 24 Global Step: 240515 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:13:23,415-Speed 129.64 samples/sec Loss 0.2905 Epoch: 24 Global Step: 240520 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:13:43,286-Speed 128.84 samples/sec Loss 0.3209 Epoch: 24 Global Step: 240525 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:14:02,989-Speed 129.94 samples/sec Loss 0.2561 Epoch: 24 Global Step: 240530 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:14:22,788-Speed 129.31 samples/sec Loss 0.3341 Epoch: 24 Global Step: 240535 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:14:42,676-Speed 128.73 samples/sec Loss 0.3560 Epoch: 24 Global Step: 240540 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:15:02,399-Speed 129.81 samples/sec Loss 0.2958 Epoch: 24 Global Step: 240545 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:15:22,292-Speed 128.69 samples/sec Loss 0.2851 Epoch: 24 Global Step: 240550 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:15:42,461-Speed 126.93 samples/sec Loss 0.3573 Epoch: 24 Global Step: 240555 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:16:02,404-Speed 128.37 samples/sec Loss 0.2909 Epoch: 24 Global Step: 240560 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:16:22,084-Speed 130.09 samples/sec Loss 0.2840 Epoch: 24 Global Step: 240565 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:16:41,965-Speed 128.77 samples/sec Loss 0.3132 Epoch: 24 Global Step: 240570 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:17:01,575-Speed 130.55 samples/sec Loss 0.2649 Epoch: 24 Global Step: 240575 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:17:21,456-Speed 128.77 samples/sec Loss 0.2935 Epoch: 24 Global Step: 240580 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:17:41,259-Speed 129.28 samples/sec Loss 0.2378 Epoch: 24 Global Step: 240585 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:18:01,136-Speed 128.80 samples/sec Loss 0.3061 Epoch: 24 Global Step: 240590 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:18:20,989-Speed 128.95 samples/sec Loss 0.3177 Epoch: 24 Global Step: 240595 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:18:41,314-Speed 125.96 samples/sec Loss 0.2826 Epoch: 24 Global Step: 240600 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:19:01,127-Speed 129.22 samples/sec Loss 0.3375 Epoch: 24 Global Step: 240605 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:19:20,851-Speed 129.79 samples/sec Loss 0.3066 Epoch: 24 Global Step: 240610 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:19:40,720-Speed 128.85 samples/sec Loss 0.2857 Epoch: 24 Global Step: 240615 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:20:00,321-Speed 130.61 samples/sec Loss 0.2837 Epoch: 24 Global Step: 240620 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:20:20,198-Speed 128.80 samples/sec Loss 0.2786 Epoch: 24 Global Step: 240625 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:20:39,963-Speed 129.53 samples/sec Loss 0.3144 Epoch: 24 Global Step: 240630 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:20:59,698-Speed 129.73 samples/sec Loss 0.3464 Epoch: 24 Global Step: 240635 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:21:19,578-Speed 128.78 samples/sec Loss 0.3454 Epoch: 24 Global Step: 240640 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:21:39,917-Speed 125.87 samples/sec Loss 0.3299 Epoch: 24 Global Step: 240645 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:21:59,679-Speed 129.55 samples/sec Loss 0.3064 Epoch: 24 Global Step: 240650 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:22:19,447-Speed 129.50 samples/sec Loss 0.2779 Epoch: 24 Global Step: 240655 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:22:39,266-Speed 129.18 samples/sec Loss 0.2503 Epoch: 24 Global Step: 240660 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:22:59,062-Speed 129.33 samples/sec Loss 0.2342 Epoch: 24 Global Step: 240665 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:23:18,933-Speed 128.83 samples/sec Loss 0.3362 Epoch: 24 Global Step: 240670 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:23:38,677-Speed 129.67 samples/sec Loss 0.3043 Epoch: 24 Global Step: 240675 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:23:58,865-Speed 126.82 samples/sec Loss 0.2648 Epoch: 24 Global Step: 240680 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:24:18,566-Speed 129.94 samples/sec Loss 0.2525 Epoch: 24 Global Step: 240685 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:24:38,908-Speed 125.85 samples/sec Loss 0.2475 Epoch: 24 Global Step: 240690 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:24:58,724-Speed 129.20 samples/sec Loss 0.2644 Epoch: 24 Global Step: 240695 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:25:18,629-Speed 128.62 samples/sec Loss 0.2832 Epoch: 24 Global Step: 240700 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:25:38,900-Speed 126.30 samples/sec Loss 0.2449 Epoch: 24 Global Step: 240705 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:25:58,671-Speed 129.49 samples/sec Loss 0.2340 Epoch: 24 Global Step: 240710 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:26:18,522-Speed 128.97 samples/sec Loss 0.3680 Epoch: 24 Global Step: 240715 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:26:37,806-Speed 132.76 samples/sec Loss 0.2944 Epoch: 24 Global Step: 240720 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:26:57,707-Speed 128.64 samples/sec Loss 0.2771 Epoch: 24 Global Step: 240725 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:27:17,438-Speed 129.75 samples/sec Loss 0.3194 Epoch: 24 Global Step: 240730 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:27:37,948-Speed 124.83 samples/sec Loss 0.3471 Epoch: 24 Global Step: 240735 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:27:57,738-Speed 129.36 samples/sec Loss 0.3320 Epoch: 24 Global Step: 240740 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:28:17,543-Speed 129.27 samples/sec Loss 0.2245 Epoch: 24 Global Step: 240745 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:28:37,266-Speed 129.81 samples/sec Loss 0.2365 Epoch: 24 Global Step: 240750 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:28:57,070-Speed 129.27 samples/sec Loss 0.2957 Epoch: 24 Global Step: 240755 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:29:16,942-Speed 128.83 samples/sec Loss 0.2808 Epoch: 24 Global Step: 240760 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:29:36,677-Speed 129.72 samples/sec Loss 0.3370 Epoch: 24 Global Step: 240765 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:29:56,594-Speed 128.54 samples/sec Loss 0.3227 Epoch: 24 Global Step: 240770 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:30:16,279-Speed 130.05 samples/sec Loss 0.2627 Epoch: 24 Global Step: 240775 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:30:36,683-Speed 125.48 samples/sec Loss 0.3866 Epoch: 24 Global Step: 240780 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:30:56,458-Speed 129.46 samples/sec Loss 0.2798 Epoch: 24 Global Step: 240785 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:31:16,322-Speed 128.88 samples/sec Loss 0.3179 Epoch: 24 Global Step: 240790 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:31:36,074-Speed 129.61 samples/sec Loss 0.2988 Epoch: 24 Global Step: 240795 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:31:55,865-Speed 129.36 samples/sec Loss 0.3629 Epoch: 24 Global Step: 240800 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:32:15,689-Speed 129.15 samples/sec Loss 0.2948 Epoch: 24 Global Step: 240805 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:32:35,466-Speed 129.45 samples/sec Loss 0.2457 Epoch: 24 Global Step: 240810 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:32:55,374-Speed 128.60 samples/sec Loss 0.2253 Epoch: 24 Global Step: 240815 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:33:14,994-Speed 130.48 samples/sec Loss 0.2480 Epoch: 24 Global Step: 240820 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:33:35,364-Speed 125.68 samples/sec Loss 0.2906 Epoch: 24 Global Step: 240825 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:33:55,140-Speed 129.46 samples/sec Loss 0.2801 Epoch: 24 Global Step: 240830 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:34:15,091-Speed 128.32 samples/sec Loss 0.2839 Epoch: 24 Global Step: 240835 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:34:34,720-Speed 130.43 samples/sec Loss 0.2438 Epoch: 24 Global Step: 240840 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:34:54,485-Speed 129.53 samples/sec Loss 0.2997 Epoch: 24 Global Step: 240845 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:35:14,310-Speed 129.13 samples/sec Loss 0.2935 Epoch: 24 Global Step: 240850 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:35:34,110-Speed 129.30 samples/sec Loss 0.2916 Epoch: 24 Global Step: 240855 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:35:54,014-Speed 128.62 samples/sec Loss 0.2509 Epoch: 24 Global Step: 240860 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:36:13,641-Speed 130.44 samples/sec Loss 0.2401 Epoch: 24 Global Step: 240865 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:36:34,020-Speed 125.63 samples/sec Loss 0.2782 Epoch: 24 Global Step: 240870 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:36:53,722-Speed 129.94 samples/sec Loss 0.3057 Epoch: 24 Global Step: 240875 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:37:13,534-Speed 129.22 samples/sec Loss 0.2915 Epoch: 24 Global Step: 240880 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:37:33,176-Speed 130.35 samples/sec Loss 0.2691 Epoch: 24 Global Step: 240885 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:37:53,050-Speed 128.82 samples/sec Loss 0.3232 Epoch: 24 Global Step: 240890 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:38:12,915-Speed 128.88 samples/sec Loss 0.2534 Epoch: 24 Global Step: 240895 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:38:32,727-Speed 129.22 samples/sec Loss 0.2937 Epoch: 24 Global Step: 240900 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:38:52,498-Speed 129.49 samples/sec Loss 0.3204 Epoch: 24 Global Step: 240905 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:39:12,127-Speed 130.43 samples/sec Loss 0.2305 Epoch: 24 Global Step: 240910 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:39:32,429-Speed 126.14 samples/sec Loss 0.2841 Epoch: 24 Global Step: 240915 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:39:52,280-Speed 128.96 samples/sec Loss 0.2831 Epoch: 24 Global Step: 240920 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:40:12,040-Speed 129.56 samples/sec Loss 0.2868 Epoch: 24 Global Step: 240925 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:40:31,652-Speed 130.54 samples/sec Loss 0.2602 Epoch: 24 Global Step: 240930 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:40:51,420-Speed 129.51 samples/sec Loss 0.3002 Epoch: 24 Global Step: 240935 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:41:11,281-Speed 128.90 samples/sec Loss 0.2630 Epoch: 24 Global Step: 240940 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:41:31,027-Speed 129.66 samples/sec Loss 0.2515 Epoch: 24 Global Step: 240945 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:41:50,821-Speed 129.34 samples/sec Loss 0.3157 Epoch: 24 Global Step: 240950 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:42:10,492-Speed 130.14 samples/sec Loss 0.2959 Epoch: 24 Global Step: 240955 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:42:30,807-Speed 126.02 samples/sec Loss 0.3269 Epoch: 24 Global Step: 240960 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:42:50,597-Speed 129.36 samples/sec Loss 0.2876 Epoch: 24 Global Step: 240965 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:43:10,391-Speed 129.34 samples/sec Loss 0.2976 Epoch: 24 Global Step: 240970 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:43:30,037-Speed 130.32 samples/sec Loss 0.3967 Epoch: 24 Global Step: 240975 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:43:49,734-Speed 129.97 samples/sec Loss 0.2993 Epoch: 24 Global Step: 240980 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:44:09,643-Speed 128.59 samples/sec Loss 0.2908 Epoch: 24 Global Step: 240985 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:44:29,384-Speed 129.70 samples/sec Loss 0.2950 Epoch: 24 Global Step: 240990 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:44:49,219-Speed 129.07 samples/sec Loss 0.3165 Epoch: 24 Global Step: 240995 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:45:08,880-Speed 130.21 samples/sec Loss 0.3227 Epoch: 24 Global Step: 241000 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:45:29,250-Speed 125.68 samples/sec Loss 0.3427 Epoch: 24 Global Step: 241005 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:45:49,039-Speed 129.37 samples/sec Loss 0.3017 Epoch: 24 Global Step: 241010 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:46:08,700-Speed 130.21 samples/sec Loss 0.2607 Epoch: 24 Global Step: 241015 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:46:28,547-Speed 128.99 samples/sec Loss 0.3008 Epoch: 24 Global Step: 241020 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:46:48,254-Speed 129.91 samples/sec Loss 0.3611 Epoch: 24 Global Step: 241025 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:47:08,141-Speed 128.73 samples/sec Loss 0.3194 Epoch: 24 Global Step: 241030 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:47:27,942-Speed 129.30 samples/sec Loss 0.3145 Epoch: 24 Global Step: 241035 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:47:47,753-Speed 129.22 samples/sec Loss 0.2372 Epoch: 24 Global Step: 241040 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:48:07,500-Speed 129.65 samples/sec Loss 0.3057 Epoch: 24 Global Step: 241045 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:48:27,157-Speed 130.24 samples/sec Loss 0.3310 Epoch: 24 Global Step: 241050 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:48:47,112-Speed 128.30 samples/sec Loss 0.3461 Epoch: 24 Global Step: 241055 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:49:07,099-Speed 128.09 samples/sec Loss 0.3279 Epoch: 24 Global Step: 241060 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:49:26,808-Speed 129.89 samples/sec Loss 0.2715 Epoch: 24 Global Step: 241065 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:49:46,627-Speed 129.17 samples/sec Loss 0.3019 Epoch: 24 Global Step: 241070 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:50:06,396-Speed 129.50 samples/sec Loss 0.3208 Epoch: 24 Global Step: 241075 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:50:26,150-Speed 129.60 samples/sec Loss 0.3285 Epoch: 24 Global Step: 241080 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:50:45,942-Speed 129.35 samples/sec Loss 0.2918 Epoch: 24 Global Step: 241085 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:51:05,686-Speed 129.67 samples/sec Loss 0.3053 Epoch: 24 Global Step: 241090 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:51:25,829-Speed 127.10 samples/sec Loss 0.2374 Epoch: 24 Global Step: 241095 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:51:45,698-Speed 128.85 samples/sec Loss 0.2911 Epoch: 24 Global Step: 241100 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:52:05,372-Speed 130.12 samples/sec Loss 0.2954 Epoch: 24 Global Step: 241105 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:52:25,264-Speed 128.70 samples/sec Loss 0.2893 Epoch: 24 Global Step: 241110 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:52:44,983-Speed 129.83 samples/sec Loss 0.2473 Epoch: 24 Global Step: 241115 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:53:05,001-Speed 127.89 samples/sec Loss 0.2888 Epoch: 24 Global Step: 241120 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:53:24,854-Speed 128.95 samples/sec Loss 0.2231 Epoch: 24 Global Step: 241125 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:53:44,590-Speed 129.72 samples/sec Loss 0.3339 Epoch: 24 Global Step: 241130 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:54:04,329-Speed 129.70 samples/sec Loss 0.3589 Epoch: 24 Global Step: 241135 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:54:24,427-Speed 127.38 samples/sec Loss 0.2761 Epoch: 24 Global Step: 241140 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:54:44,208-Speed 129.42 samples/sec Loss 0.3193 Epoch: 24 Global Step: 241145 Fp16 Grad Scale: 4096 Required: 10 hours Training: 2021-06-18 19:55:03,875-Speed 130.18 samples/sec Loss 0.2780 Epoch: 24 Global Step: 241150 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:55:23,776-Speed 128.65 samples/sec Loss 0.2924 Epoch: 24 Global Step: 241155 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:55:43,414-Speed 130.37 samples/sec Loss 0.3088 Epoch: 24 Global Step: 241160 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:56:03,473-Speed 127.63 samples/sec Loss 0.3020 Epoch: 24 Global Step: 241165 Fp16 Grad Scale: 8192 Required: 10 hours Training: 2021-06-18 19:56:23,396-Speed 128.50 samples/sec Loss 0.2743 Epoch: 24 Global Step: 241170 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 19:56:43,113-Speed 129.84 samples/sec Loss 0.3004 Epoch: 24 Global Step: 241175 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 19:57:02,877-Speed 129.54 samples/sec Loss 0.2909 Epoch: 24 Global Step: 241180 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 19:57:22,925-Speed 127.70 samples/sec Loss 0.2263 Epoch: 24 Global Step: 241185 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 19:57:42,647-Speed 129.81 samples/sec Loss 0.3826 Epoch: 24 Global Step: 241190 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 19:58:02,523-Speed 128.81 samples/sec Loss 0.3558 Epoch: 24 Global Step: 241195 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 19:58:22,272-Speed 129.63 samples/sec Loss 0.3646 Epoch: 24 Global Step: 241200 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 19:58:42,252-Speed 128.14 samples/sec Loss 0.3368 Epoch: 24 Global Step: 241205 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 19:59:02,363-Speed 127.30 samples/sec Loss 0.3335 Epoch: 24 Global Step: 241210 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 19:59:22,299-Speed 128.42 samples/sec Loss 0.3099 Epoch: 24 Global Step: 241215 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 19:59:42,874-Speed 124.43 samples/sec Loss 0.2621 Epoch: 24 Global Step: 241220 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:00:02,796-Speed 128.51 samples/sec Loss 0.2957 Epoch: 24 Global Step: 241225 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:00:22,706-Speed 128.58 samples/sec Loss 0.2951 Epoch: 24 Global Step: 241230 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:00:42,437-Speed 129.75 samples/sec Loss 0.3177 Epoch: 24 Global Step: 241235 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:01:02,337-Speed 128.65 samples/sec Loss 0.2823 Epoch: 24 Global Step: 241240 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:01:22,082-Speed 129.66 samples/sec Loss 0.2981 Epoch: 24 Global Step: 241245 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:01:41,788-Speed 129.91 samples/sec Loss 0.2352 Epoch: 24 Global Step: 241250 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:02:01,848-Speed 127.63 samples/sec Loss 0.2911 Epoch: 24 Global Step: 241255 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:02:21,742-Speed 128.69 samples/sec Loss 0.3193 Epoch: 24 Global Step: 241260 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:02:41,438-Speed 129.98 samples/sec Loss 0.2928 Epoch: 24 Global Step: 241265 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:03:01,362-Speed 128.49 samples/sec Loss 0.2910 Epoch: 24 Global Step: 241270 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:03:20,577-Speed 133.24 samples/sec Loss 0.3114 Epoch: 24 Global Step: 241275 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:03:40,297-Speed 129.82 samples/sec Loss 0.2880 Epoch: 24 Global Step: 241280 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:04:00,462-Speed 126.96 samples/sec Loss 0.2649 Epoch: 24 Global Step: 241285 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:04:20,205-Speed 129.67 samples/sec Loss 0.3142 Epoch: 24 Global Step: 241290 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:04:40,002-Speed 129.32 samples/sec Loss 0.3198 Epoch: 24 Global Step: 241295 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:04:59,891-Speed 128.72 samples/sec Loss 0.2767 Epoch: 24 Global Step: 241300 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:05:19,773-Speed 128.77 samples/sec Loss 0.3104 Epoch: 24 Global Step: 241305 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:05:39,405-Speed 130.40 samples/sec Loss 0.2713 Epoch: 24 Global Step: 241310 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:05:59,316-Speed 128.58 samples/sec Loss 0.3451 Epoch: 24 Global Step: 241315 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:06:18,932-Speed 130.51 samples/sec Loss 0.2886 Epoch: 24 Global Step: 241320 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:06:38,766-Speed 129.08 samples/sec Loss 0.2922 Epoch: 24 Global Step: 241325 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:06:58,553-Speed 129.38 samples/sec Loss 0.3171 Epoch: 24 Global Step: 241330 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:07:18,381-Speed 129.12 samples/sec Loss 0.3434 Epoch: 24 Global Step: 241335 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:07:38,225-Speed 129.01 samples/sec Loss 0.3740 Epoch: 24 Global Step: 241340 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:07:58,466-Speed 126.48 samples/sec Loss 0.2241 Epoch: 24 Global Step: 241345 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:08:18,323-Speed 128.93 samples/sec Loss 0.3642 Epoch: 24 Global Step: 241350 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:08:38,008-Speed 130.05 samples/sec Loss 0.2682 Epoch: 24 Global Step: 241355 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:08:57,885-Speed 128.80 samples/sec Loss 0.2984 Epoch: 24 Global Step: 241360 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:09:17,603-Speed 129.84 samples/sec Loss 0.2997 Epoch: 24 Global Step: 241365 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:09:37,429-Speed 129.13 samples/sec Loss 0.2335 Epoch: 24 Global Step: 241370 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:09:57,670-Speed 126.48 samples/sec Loss 0.2999 Epoch: 24 Global Step: 241375 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:10:17,372-Speed 129.94 samples/sec Loss 0.3174 Epoch: 24 Global Step: 241380 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:10:37,195-Speed 129.15 samples/sec Loss 0.3137 Epoch: 24 Global Step: 241385 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:10:57,394-Speed 126.75 samples/sec Loss 0.3018 Epoch: 24 Global Step: 241390 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:11:17,262-Speed 128.86 samples/sec Loss 0.2693 Epoch: 24 Global Step: 241395 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:11:36,981-Speed 129.83 samples/sec Loss 0.3200 Epoch: 24 Global Step: 241400 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:11:56,838-Speed 128.93 samples/sec Loss 0.3138 Epoch: 24 Global Step: 241405 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:12:16,424-Speed 130.71 samples/sec Loss 0.3074 Epoch: 24 Global Step: 241410 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:12:36,246-Speed 129.16 samples/sec Loss 0.3222 Epoch: 24 Global Step: 241415 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:12:56,113-Speed 128.87 samples/sec Loss 0.2570 Epoch: 24 Global Step: 241420 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:13:15,852-Speed 129.70 samples/sec Loss 0.2387 Epoch: 24 Global Step: 241425 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:13:35,652-Speed 129.30 samples/sec Loss 0.3047 Epoch: 24 Global Step: 241430 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:13:55,848-Speed 126.76 samples/sec Loss 0.2452 Epoch: 24 Global Step: 241435 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:14:15,762-Speed 128.56 samples/sec Loss 0.2523 Epoch: 24 Global Step: 241440 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:14:35,467-Speed 129.92 samples/sec Loss 0.2162 Epoch: 24 Global Step: 241445 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:14:55,383-Speed 128.55 samples/sec Loss 0.3189 Epoch: 24 Global Step: 241450 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:15:14,419-Speed 134.49 samples/sec Loss 0.2772 Epoch: 24 Global Step: 241455 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:15:34,476-Speed 127.64 samples/sec Loss 0.2893 Epoch: 24 Global Step: 241460 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:15:54,304-Speed 129.12 samples/sec Loss 0.2654 Epoch: 24 Global Step: 241465 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:16:14,031-Speed 129.78 samples/sec Loss 0.2775 Epoch: 24 Global Step: 241470 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:16:33,828-Speed 129.32 samples/sec Loss 0.3445 Epoch: 24 Global Step: 241475 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:16:53,957-Speed 127.18 samples/sec Loss 0.3127 Epoch: 24 Global Step: 241480 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:17:14,166-Speed 126.68 samples/sec Loss 0.2751 Epoch: 24 Global Step: 241485 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:17:33,900-Speed 129.73 samples/sec Loss 0.2768 Epoch: 24 Global Step: 241490 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:17:53,706-Speed 129.26 samples/sec Loss 0.3422 Epoch: 24 Global Step: 241495 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:18:13,317-Speed 130.55 samples/sec Loss 0.3074 Epoch: 24 Global Step: 241500 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:18:33,078-Speed 129.56 samples/sec Loss 0.2472 Epoch: 24 Global Step: 241505 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:18:53,205-Speed 127.20 samples/sec Loss 0.3526 Epoch: 24 Global Step: 241510 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:19:12,945-Speed 129.69 samples/sec Loss 0.2537 Epoch: 24 Global Step: 241515 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:19:32,672-Speed 129.78 samples/sec Loss 0.2666 Epoch: 24 Global Step: 241520 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:19:52,977-Speed 126.08 samples/sec Loss 0.2401 Epoch: 24 Global Step: 241525 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:20:12,921-Speed 128.37 samples/sec Loss 0.3218 Epoch: 24 Global Step: 241530 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:20:32,567-Speed 130.31 samples/sec Loss 0.2723 Epoch: 24 Global Step: 241535 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:20:52,416-Speed 128.98 samples/sec Loss 0.2559 Epoch: 24 Global Step: 241540 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:21:12,036-Speed 130.49 samples/sec Loss 0.3235 Epoch: 24 Global Step: 241545 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:21:31,947-Speed 128.58 samples/sec Loss 0.3483 Epoch: 24 Global Step: 241550 Fp16 Grad Scale: 4096 Required: 9 hours Training: 2021-06-18 20:21:51,669-Speed 129.80 samples/sec Loss 0.3233 Epoch: 24 Global Step: 241555 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:22:11,480-Speed 129.23 samples/sec Loss 0.2775 Epoch: 24 Global Step: 241560 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:22:31,281-Speed 129.29 samples/sec Loss 0.2621 Epoch: 24 Global Step: 241565 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:22:51,577-Speed 126.14 samples/sec Loss 0.3132 Epoch: 24 Global Step: 241570 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:23:11,382-Speed 129.27 samples/sec Loss 0.2702 Epoch: 24 Global Step: 241575 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:23:31,030-Speed 130.30 samples/sec Loss 0.3334 Epoch: 24 Global Step: 241580 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:23:50,884-Speed 128.95 samples/sec Loss 0.3859 Epoch: 24 Global Step: 241585 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:24:10,504-Speed 130.48 samples/sec Loss 0.3091 Epoch: 24 Global Step: 241590 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:24:30,406-Speed 128.64 samples/sec Loss 0.4158 Epoch: 24 Global Step: 241595 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:24:50,168-Speed 129.55 samples/sec Loss 0.3170 Epoch: 24 Global Step: 241600 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:25:09,854-Speed 130.05 samples/sec Loss 0.3420 Epoch: 24 Global Step: 241605 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:25:29,666-Speed 129.22 samples/sec Loss 0.2928 Epoch: 24 Global Step: 241610 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:25:49,959-Speed 126.15 samples/sec Loss 0.3936 Epoch: 24 Global Step: 241615 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:26:09,747-Speed 129.38 samples/sec Loss 0.3295 Epoch: 24 Global Step: 241620 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:26:29,362-Speed 130.52 samples/sec Loss 0.2898 Epoch: 24 Global Step: 241625 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:26:49,218-Speed 128.93 samples/sec Loss 0.2304 Epoch: 24 Global Step: 241630 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:27:08,838-Speed 130.48 samples/sec Loss 0.2622 Epoch: 24 Global Step: 241635 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:27:28,775-Speed 128.41 samples/sec Loss 0.2627 Epoch: 24 Global Step: 241640 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:27:48,496-Speed 129.82 samples/sec Loss 0.3065 Epoch: 24 Global Step: 241645 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:28:08,207-Speed 129.89 samples/sec Loss 0.2988 Epoch: 24 Global Step: 241650 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:28:27,990-Speed 129.41 samples/sec Loss 0.3357 Epoch: 24 Global Step: 241655 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:28:48,130-Speed 127.11 samples/sec Loss 0.3065 Epoch: 24 Global Step: 241660 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:29:08,049-Speed 128.52 samples/sec Loss 0.4206 Epoch: 24 Global Step: 241665 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:29:27,666-Speed 130.51 samples/sec Loss 0.2965 Epoch: 24 Global Step: 241670 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:29:47,587-Speed 128.51 samples/sec Loss 0.4050 Epoch: 24 Global Step: 241675 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:30:07,264-Speed 130.11 samples/sec Loss 0.3183 Epoch: 24 Global Step: 241680 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:30:27,163-Speed 128.66 samples/sec Loss 0.2907 Epoch: 24 Global Step: 241685 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:30:46,950-Speed 129.39 samples/sec Loss 0.2598 Epoch: 24 Global Step: 241690 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:31:06,589-Speed 130.36 samples/sec Loss 0.3822 Epoch: 24 Global Step: 241695 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:31:26,520-Speed 128.45 samples/sec Loss 0.2263 Epoch: 24 Global Step: 241700 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:31:46,685-Speed 126.96 samples/sec Loss 0.3035 Epoch: 24 Global Step: 241705 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:32:06,628-Speed 128.37 samples/sec Loss 0.3446 Epoch: 24 Global Step: 241710 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:32:26,233-Speed 130.58 samples/sec Loss 0.3120 Epoch: 24 Global Step: 241715 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:32:46,164-Speed 128.46 samples/sec Loss 0.2763 Epoch: 24 Global Step: 241720 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:33:05,878-Speed 129.86 samples/sec Loss 0.3065 Epoch: 24 Global Step: 241725 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:33:25,728-Speed 128.97 samples/sec Loss 0.2826 Epoch: 24 Global Step: 241730 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:33:45,529-Speed 129.30 samples/sec Loss 0.2932 Epoch: 24 Global Step: 241735 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:34:05,155-Speed 130.45 samples/sec Loss 0.2776 Epoch: 24 Global Step: 241740 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:34:24,999-Speed 129.01 samples/sec Loss 0.2703 Epoch: 24 Global Step: 241745 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:34:45,153-Speed 127.03 samples/sec Loss 0.2806 Epoch: 24 Global Step: 241750 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:35:05,137-Speed 128.10 samples/sec Loss 0.2445 Epoch: 24 Global Step: 241755 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:35:24,745-Speed 130.57 samples/sec Loss 0.3614 Epoch: 24 Global Step: 241760 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:35:44,767-Speed 127.87 samples/sec Loss 0.2413 Epoch: 24 Global Step: 241765 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:36:04,496-Speed 129.76 samples/sec Loss 0.3009 Epoch: 24 Global Step: 241770 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:36:24,386-Speed 128.72 samples/sec Loss 0.2172 Epoch: 24 Global Step: 241775 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:36:44,168-Speed 129.42 samples/sec Loss 0.2655 Epoch: 24 Global Step: 241780 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:37:03,837-Speed 130.16 samples/sec Loss 0.3322 Epoch: 24 Global Step: 241785 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:37:23,688-Speed 128.97 samples/sec Loss 0.3014 Epoch: 24 Global Step: 241790 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:37:44,067-Speed 125.63 samples/sec Loss 0.2843 Epoch: 24 Global Step: 241795 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:38:03,793-Speed 129.78 samples/sec Loss 0.2561 Epoch: 24 Global Step: 241800 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:38:23,398-Speed 130.58 samples/sec Loss 0.2372 Epoch: 24 Global Step: 241805 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:38:43,266-Speed 128.86 samples/sec Loss 0.3290 Epoch: 24 Global Step: 241810 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:39:02,993-Speed 129.77 samples/sec Loss 0.2305 Epoch: 24 Global Step: 241815 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:39:23,009-Speed 127.90 samples/sec Loss 0.3253 Epoch: 24 Global Step: 241820 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:39:42,626-Speed 130.52 samples/sec Loss 0.3260 Epoch: 24 Global Step: 241825 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:40:02,302-Speed 130.11 samples/sec Loss 0.2550 Epoch: 24 Global Step: 241830 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:40:22,178-Speed 128.80 samples/sec Loss 0.2615 Epoch: 24 Global Step: 241835 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:40:42,577-Speed 125.50 samples/sec Loss 0.3382 Epoch: 24 Global Step: 241840 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:41:02,345-Speed 129.51 samples/sec Loss 0.2704 Epoch: 24 Global Step: 241845 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:41:22,032-Speed 130.04 samples/sec Loss 0.2635 Epoch: 24 Global Step: 241850 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:41:41,903-Speed 128.84 samples/sec Loss 0.2875 Epoch: 24 Global Step: 241855 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:42:01,602-Speed 129.96 samples/sec Loss 0.2733 Epoch: 24 Global Step: 241860 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:42:21,509-Speed 128.60 samples/sec Loss 0.3050 Epoch: 24 Global Step: 241865 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:42:41,211-Speed 129.95 samples/sec Loss 0.4169 Epoch: 24 Global Step: 241870 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:43:00,883-Speed 130.14 samples/sec Loss 0.2888 Epoch: 24 Global Step: 241875 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:43:20,765-Speed 128.77 samples/sec Loss 0.2414 Epoch: 24 Global Step: 241880 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:43:41,187-Speed 125.36 samples/sec Loss 0.3273 Epoch: 24 Global Step: 241885 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:44:00,885-Speed 129.97 samples/sec Loss 0.3005 Epoch: 24 Global Step: 241890 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:44:20,523-Speed 130.37 samples/sec Loss 0.3118 Epoch: 24 Global Step: 241895 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:44:40,411-Speed 128.73 samples/sec Loss 0.2515 Epoch: 24 Global Step: 241900 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:45:00,262-Speed 128.97 samples/sec Loss 0.2761 Epoch: 24 Global Step: 241905 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:45:20,247-Speed 128.10 samples/sec Loss 0.2605 Epoch: 24 Global Step: 241910 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:45:39,865-Speed 130.50 samples/sec Loss 0.2800 Epoch: 24 Global Step: 241915 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:45:59,541-Speed 130.12 samples/sec Loss 0.2714 Epoch: 24 Global Step: 241920 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:46:19,441-Speed 128.65 samples/sec Loss 0.2464 Epoch: 24 Global Step: 241925 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:46:39,885-Speed 125.22 samples/sec Loss 0.2716 Epoch: 24 Global Step: 241930 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:46:59,637-Speed 129.62 samples/sec Loss 0.2777 Epoch: 24 Global Step: 241935 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:47:19,265-Speed 130.43 samples/sec Loss 0.3075 Epoch: 24 Global Step: 241940 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:47:39,328-Speed 127.60 samples/sec Loss 0.3313 Epoch: 24 Global Step: 241945 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:47:59,050-Speed 129.81 samples/sec Loss 0.3047 Epoch: 24 Global Step: 241950 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:48:19,031-Speed 128.13 samples/sec Loss 0.3157 Epoch: 24 Global Step: 241955 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:48:38,664-Speed 130.40 samples/sec Loss 0.2748 Epoch: 24 Global Step: 241960 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:48:58,343-Speed 130.10 samples/sec Loss 0.2918 Epoch: 24 Global Step: 241965 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:49:18,235-Speed 128.70 samples/sec Loss 0.2950 Epoch: 24 Global Step: 241970 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:49:38,701-Speed 125.09 samples/sec Loss 0.2777 Epoch: 24 Global Step: 241975 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:49:58,331-Speed 130.42 samples/sec Loss 0.2449 Epoch: 24 Global Step: 241980 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:50:17,939-Speed 130.57 samples/sec Loss 0.2792 Epoch: 24 Global Step: 241985 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:50:37,880-Speed 128.39 samples/sec Loss 0.2342 Epoch: 24 Global Step: 241990 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:50:57,603-Speed 129.81 samples/sec Loss 0.3091 Epoch: 24 Global Step: 241995 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:51:17,672-Speed 127.56 samples/sec Loss 0.2165 Epoch: 24 Global Step: 242000 Fp16 Grad Scale: 8192 Required: 9 hours Training: 2021-06-18 20:51:37,306-Speed 130.39 samples/sec Loss 0.2716 Epoch: 24 Global Step: 242005 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:51:56,974-Speed 130.17 samples/sec Loss 0.2711 Epoch: 24 Global Step: 242010 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:52:16,901-Speed 128.48 samples/sec Loss 0.3273 Epoch: 24 Global Step: 242015 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:52:37,216-Speed 126.02 samples/sec Loss 0.2722 Epoch: 24 Global Step: 242020 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:52:56,987-Speed 129.49 samples/sec Loss 0.3968 Epoch: 24 Global Step: 242025 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:53:16,607-Speed 130.48 samples/sec Loss 0.3388 Epoch: 24 Global Step: 242030 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:53:36,518-Speed 128.58 samples/sec Loss 0.2933 Epoch: 24 Global Step: 242035 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:53:56,236-Speed 129.84 samples/sec Loss 0.2551 Epoch: 24 Global Step: 242040 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:54:16,023-Speed 129.39 samples/sec Loss 0.3138 Epoch: 24 Global Step: 242045 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:54:35,808-Speed 129.40 samples/sec Loss 0.2970 Epoch: 24 Global Step: 242050 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:54:55,541-Speed 129.74 samples/sec Loss 0.2596 Epoch: 24 Global Step: 242055 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:55:15,407-Speed 128.87 samples/sec Loss 0.2547 Epoch: 24 Global Step: 242060 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:55:35,706-Speed 126.12 samples/sec Loss 0.3091 Epoch: 24 Global Step: 242065 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 20:55:54,712-Speed 134.70 samples/sec Loss 0.3324 Epoch: 24 Global Step: 242070 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:56:14,743-Speed 127.80 samples/sec Loss 0.2833 Epoch: 24 Global Step: 242075 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:56:34,687-Speed 128.37 samples/sec Loss 0.3695 Epoch: 24 Global Step: 242080 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:56:54,399-Speed 129.88 samples/sec Loss 0.2834 Epoch: 24 Global Step: 242085 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:57:14,230-Speed 129.09 samples/sec Loss 0.2963 Epoch: 24 Global Step: 242090 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:57:33,978-Speed 129.64 samples/sec Loss 0.2822 Epoch: 24 Global Step: 242095 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:57:53,713-Speed 129.73 samples/sec Loss 0.2781 Epoch: 24 Global Step: 242100 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:58:13,635-Speed 128.50 samples/sec Loss 0.3483 Epoch: 24 Global Step: 242105 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:58:33,823-Speed 126.81 samples/sec Loss 0.3534 Epoch: 24 Global Step: 242110 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:58:53,653-Speed 129.11 samples/sec Loss 0.3298 Epoch: 24 Global Step: 242115 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:59:13,346-Speed 130.00 samples/sec Loss 0.3223 Epoch: 24 Global Step: 242120 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:59:33,130-Speed 129.41 samples/sec Loss 0.2921 Epoch: 24 Global Step: 242125 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 20:59:53,002-Speed 128.83 samples/sec Loss 0.3652 Epoch: 24 Global Step: 242130 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:00:13,153-Speed 127.05 samples/sec Loss 0.3280 Epoch: 24 Global Step: 242135 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:00:32,780-Speed 130.44 samples/sec Loss 0.2622 Epoch: 24 Global Step: 242140 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:00:52,563-Speed 129.41 samples/sec Loss 0.2773 Epoch: 24 Global Step: 242145 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:01:12,421-Speed 128.92 samples/sec Loss 0.2625 Epoch: 24 Global Step: 242150 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:01:32,441-Speed 127.88 samples/sec Loss 0.2715 Epoch: 24 Global Step: 242155 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:01:52,378-Speed 128.41 samples/sec Loss 0.3139 Epoch: 24 Global Step: 242160 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:02:12,022-Speed 130.33 samples/sec Loss 0.3051 Epoch: 24 Global Step: 242165 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:02:31,819-Speed 129.32 samples/sec Loss 0.2759 Epoch: 24 Global Step: 242170 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:02:51,673-Speed 128.95 samples/sec Loss 0.2431 Epoch: 24 Global Step: 242175 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:03:11,839-Speed 126.95 samples/sec Loss 0.2881 Epoch: 24 Global Step: 242180 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:03:31,522-Speed 130.07 samples/sec Loss 0.2439 Epoch: 24 Global Step: 242185 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:03:51,275-Speed 129.60 samples/sec Loss 0.3451 Epoch: 24 Global Step: 242190 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:04:11,137-Speed 128.90 samples/sec Loss 0.2987 Epoch: 24 Global Step: 242195 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:04:31,260-Speed 127.22 samples/sec Loss 0.3206 Epoch: 24 Global Step: 242200 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:04:51,163-Speed 128.63 samples/sec Loss 0.2511 Epoch: 24 Global Step: 242205 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:05:10,892-Speed 129.76 samples/sec Loss 0.3223 Epoch: 24 Global Step: 242210 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:05:30,908-Speed 127.91 samples/sec Loss 0.3659 Epoch: 24 Global Step: 242215 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:05:50,632-Speed 129.80 samples/sec Loss 0.2587 Epoch: 24 Global Step: 242220 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:06:10,902-Speed 126.30 samples/sec Loss 0.3135 Epoch: 24 Global Step: 242225 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:06:30,621-Speed 129.83 samples/sec Loss 0.3047 Epoch: 24 Global Step: 242230 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:06:50,657-Speed 127.77 samples/sec Loss 0.2840 Epoch: 24 Global Step: 242235 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:07:10,426-Speed 129.51 samples/sec Loss 0.2655 Epoch: 24 Global Step: 242240 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:07:30,554-Speed 127.19 samples/sec Loss 0.2708 Epoch: 24 Global Step: 242245 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:07:50,420-Speed 128.87 samples/sec Loss 0.2913 Epoch: 24 Global Step: 242250 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:08:10,157-Speed 129.72 samples/sec Loss 0.2850 Epoch: 24 Global Step: 242255 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:08:30,082-Speed 128.49 samples/sec Loss 0.2622 Epoch: 24 Global Step: 242260 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:08:49,267-Speed 133.44 samples/sec Loss 0.3535 Epoch: 24 Global Step: 242265 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:09:09,461-Speed 126.78 samples/sec Loss 0.3157 Epoch: 24 Global Step: 242270 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:09:29,168-Speed 129.91 samples/sec Loss 0.3072 Epoch: 24 Global Step: 242275 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:09:48,992-Speed 129.14 samples/sec Loss 0.2563 Epoch: 24 Global Step: 242280 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:10:08,660-Speed 130.17 samples/sec Loss 0.2457 Epoch: 24 Global Step: 242285 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:10:28,504-Speed 129.01 samples/sec Loss 0.3279 Epoch: 24 Global Step: 242290 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:10:48,353-Speed 128.98 samples/sec Loss 0.2806 Epoch: 24 Global Step: 242295 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:11:08,380-Speed 127.83 samples/sec Loss 0.2643 Epoch: 24 Global Step: 242300 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:11:28,475-Speed 127.41 samples/sec Loss 0.3217 Epoch: 24 Global Step: 242305 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:11:48,232-Speed 129.58 samples/sec Loss 0.2463 Epoch: 24 Global Step: 242310 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:12:08,708-Speed 125.03 samples/sec Loss 0.2219 Epoch: 24 Global Step: 242315 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:12:28,421-Speed 129.87 samples/sec Loss 0.2541 Epoch: 24 Global Step: 242320 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:12:48,306-Speed 128.74 samples/sec Loss 0.2520 Epoch: 24 Global Step: 242325 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:13:08,010-Speed 129.93 samples/sec Loss 0.3192 Epoch: 24 Global Step: 242330 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:13:27,834-Speed 129.14 samples/sec Loss 0.3100 Epoch: 24 Global Step: 242335 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:13:47,713-Speed 128.79 samples/sec Loss 0.2709 Epoch: 24 Global Step: 242340 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:14:07,380-Speed 130.18 samples/sec Loss 0.2902 Epoch: 24 Global Step: 242345 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:14:27,423-Speed 127.73 samples/sec Loss 0.2778 Epoch: 24 Global Step: 242350 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:14:47,135-Speed 129.88 samples/sec Loss 0.3066 Epoch: 24 Global Step: 242355 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:15:07,594-Speed 125.13 samples/sec Loss 0.2632 Epoch: 24 Global Step: 242360 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:15:27,227-Speed 130.40 samples/sec Loss 0.2641 Epoch: 24 Global Step: 242365 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:15:47,162-Speed 128.42 samples/sec Loss 0.3253 Epoch: 24 Global Step: 242370 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:16:06,451-Speed 132.73 samples/sec Loss 0.2849 Epoch: 24 Global Step: 242375 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:16:26,289-Speed 129.05 samples/sec Loss 0.3759 Epoch: 24 Global Step: 242380 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:16:46,428-Speed 127.13 samples/sec Loss 0.2371 Epoch: 24 Global Step: 242385 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:17:06,069-Speed 130.35 samples/sec Loss 0.3013 Epoch: 24 Global Step: 242390 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:17:25,973-Speed 128.63 samples/sec Loss 0.2911 Epoch: 24 Global Step: 242395 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:17:45,702-Speed 129.76 samples/sec Loss 0.3239 Epoch: 24 Global Step: 242400 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:18:06,152-Speed 125.19 samples/sec Loss 0.3083 Epoch: 24 Global Step: 242405 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:18:25,771-Speed 130.49 samples/sec Loss 0.3066 Epoch: 24 Global Step: 242410 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:18:45,685-Speed 128.56 samples/sec Loss 0.2959 Epoch: 24 Global Step: 242415 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:19:05,406-Speed 129.82 samples/sec Loss 0.2624 Epoch: 24 Global Step: 242420 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:19:25,252-Speed 129.00 samples/sec Loss 0.3157 Epoch: 24 Global Step: 242425 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:19:45,107-Speed 128.94 samples/sec Loss 0.3026 Epoch: 24 Global Step: 242430 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:20:04,719-Speed 130.54 samples/sec Loss 0.3012 Epoch: 24 Global Step: 242435 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:20:24,632-Speed 128.57 samples/sec Loss 0.3022 Epoch: 24 Global Step: 242440 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:20:44,346-Speed 129.86 samples/sec Loss 0.2762 Epoch: 24 Global Step: 242445 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:21:04,786-Speed 125.25 samples/sec Loss 0.2960 Epoch: 24 Global Step: 242450 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:21:24,407-Speed 130.48 samples/sec Loss 0.2482 Epoch: 24 Global Step: 242455 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:21:44,267-Speed 128.91 samples/sec Loss 0.3220 Epoch: 24 Global Step: 242460 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:22:04,064-Speed 129.32 samples/sec Loss 0.3230 Epoch: 24 Global Step: 242465 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:22:23,826-Speed 129.55 samples/sec Loss 0.2438 Epoch: 24 Global Step: 242470 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:22:43,683-Speed 128.93 samples/sec Loss 0.3064 Epoch: 24 Global Step: 242475 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:23:03,314-Speed 130.41 samples/sec Loss 0.3187 Epoch: 24 Global Step: 242480 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:23:23,247-Speed 128.44 samples/sec Loss 0.2650 Epoch: 24 Global Step: 242485 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:23:42,988-Speed 129.68 samples/sec Loss 0.2581 Epoch: 24 Global Step: 242490 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:24:03,327-Speed 125.87 samples/sec Loss 0.2478 Epoch: 24 Global Step: 242495 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:24:22,977-Speed 130.29 samples/sec Loss 0.2458 Epoch: 24 Global Step: 242500 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:24:42,746-Speed 129.50 samples/sec Loss 0.2971 Epoch: 24 Global Step: 242505 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:25:02,698-Speed 128.32 samples/sec Loss 0.2915 Epoch: 24 Global Step: 242510 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:25:22,487-Speed 129.37 samples/sec Loss 0.3293 Epoch: 24 Global Step: 242515 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:25:42,370-Speed 128.76 samples/sec Loss 0.2971 Epoch: 24 Global Step: 242520 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:26:02,016-Speed 130.31 samples/sec Loss 0.2621 Epoch: 24 Global Step: 242525 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:26:21,932-Speed 128.55 samples/sec Loss 0.2582 Epoch: 24 Global Step: 242530 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:26:41,661-Speed 129.77 samples/sec Loss 0.3036 Epoch: 24 Global Step: 242535 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:27:02,007-Speed 125.83 samples/sec Loss 0.2831 Epoch: 24 Global Step: 242540 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:27:21,730-Speed 129.80 samples/sec Loss 0.3123 Epoch: 24 Global Step: 242545 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:27:41,478-Speed 129.64 samples/sec Loss 0.3125 Epoch: 24 Global Step: 242550 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:28:01,347-Speed 128.85 samples/sec Loss 0.3049 Epoch: 24 Global Step: 242555 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:28:21,053-Speed 129.92 samples/sec Loss 0.2552 Epoch: 24 Global Step: 242560 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:28:40,977-Speed 128.49 samples/sec Loss 0.3518 Epoch: 24 Global Step: 242565 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:29:00,739-Speed 129.55 samples/sec Loss 0.2941 Epoch: 24 Global Step: 242570 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:29:20,594-Speed 128.94 samples/sec Loss 0.2494 Epoch: 24 Global Step: 242575 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:29:40,344-Speed 129.63 samples/sec Loss 0.2261 Epoch: 24 Global Step: 242580 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:30:00,710-Speed 125.71 samples/sec Loss 0.3086 Epoch: 24 Global Step: 242585 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:30:20,433-Speed 129.80 samples/sec Loss 0.3466 Epoch: 24 Global Step: 242590 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:30:40,276-Speed 129.01 samples/sec Loss 0.3397 Epoch: 24 Global Step: 242595 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:31:00,102-Speed 129.13 samples/sec Loss 0.2850 Epoch: 24 Global Step: 242600 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:31:19,904-Speed 129.29 samples/sec Loss 0.2474 Epoch: 24 Global Step: 242605 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:31:39,789-Speed 128.74 samples/sec Loss 0.3164 Epoch: 24 Global Step: 242610 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:31:59,519-Speed 129.76 samples/sec Loss 0.3219 Epoch: 24 Global Step: 242615 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:32:19,437-Speed 128.53 samples/sec Loss 0.3202 Epoch: 24 Global Step: 242620 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:32:39,126-Speed 130.03 samples/sec Loss 0.3217 Epoch: 24 Global Step: 242625 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:32:58,644-Speed 131.16 samples/sec Loss 0.2836 Epoch: 24 Global Step: 242630 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:33:18,541-Speed 128.67 samples/sec Loss 0.2623 Epoch: 24 Global Step: 242635 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:33:38,638-Speed 127.39 samples/sec Loss 0.2248 Epoch: 24 Global Step: 242640 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:33:58,372-Speed 129.73 samples/sec Loss 0.2814 Epoch: 24 Global Step: 242645 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:34:18,207-Speed 129.07 samples/sec Loss 0.2695 Epoch: 24 Global Step: 242650 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:34:37,972-Speed 129.53 samples/sec Loss 0.2486 Epoch: 24 Global Step: 242655 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:34:57,698-Speed 129.79 samples/sec Loss 0.2735 Epoch: 24 Global Step: 242660 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:35:17,629-Speed 128.45 samples/sec Loss 0.2688 Epoch: 24 Global Step: 242665 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:35:37,346-Speed 129.84 samples/sec Loss 0.3556 Epoch: 24 Global Step: 242670 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:35:57,386-Speed 127.75 samples/sec Loss 0.3106 Epoch: 24 Global Step: 242675 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:36:17,259-Speed 128.84 samples/sec Loss 0.2399 Epoch: 24 Global Step: 242680 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:36:37,587-Speed 125.94 samples/sec Loss 0.3619 Epoch: 24 Global Step: 242685 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:36:57,316-Speed 129.76 samples/sec Loss 0.2802 Epoch: 24 Global Step: 242690 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:37:17,092-Speed 129.45 samples/sec Loss 0.3076 Epoch: 24 Global Step: 242695 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:37:36,864-Speed 129.48 samples/sec Loss 0.3581 Epoch: 24 Global Step: 242700 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:37:56,548-Speed 130.06 samples/sec Loss 0.3444 Epoch: 24 Global Step: 242705 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:38:16,490-Speed 128.38 samples/sec Loss 0.2607 Epoch: 24 Global Step: 242710 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:38:36,247-Speed 129.58 samples/sec Loss 0.3312 Epoch: 24 Global Step: 242715 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:38:57,163-Speed 122.40 samples/sec Loss 0.3473 Epoch: 24 Global Step: 242720 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:39:16,868-Speed 129.92 samples/sec Loss 0.2455 Epoch: 24 Global Step: 242725 Fp16 Grad Scale: 4096 Required: 8 hours Training: 2021-06-18 21:39:37,007-Speed 127.12 samples/sec Loss 0.3485 Epoch: 24 Global Step: 242730 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:39:56,741-Speed 129.73 samples/sec Loss 0.2907 Epoch: 24 Global Step: 242735 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:40:16,707-Speed 128.22 samples/sec Loss 0.3333 Epoch: 24 Global Step: 242740 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:40:36,823-Speed 127.27 samples/sec Loss 0.2803 Epoch: 24 Global Step: 242745 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:40:56,530-Speed 129.91 samples/sec Loss 0.3219 Epoch: 24 Global Step: 242750 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:41:16,430-Speed 128.65 samples/sec Loss 0.3150 Epoch: 24 Global Step: 242755 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:41:36,253-Speed 129.15 samples/sec Loss 0.2664 Epoch: 24 Global Step: 242760 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:41:56,431-Speed 126.87 samples/sec Loss 0.3932 Epoch: 24 Global Step: 242765 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:42:16,096-Speed 130.19 samples/sec Loss 0.3984 Epoch: 24 Global Step: 242770 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:42:36,213-Speed 127.26 samples/sec Loss 0.2604 Epoch: 24 Global Step: 242775 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:42:55,944-Speed 129.75 samples/sec Loss 0.2871 Epoch: 24 Global Step: 242780 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:43:15,956-Speed 127.93 samples/sec Loss 0.3062 Epoch: 24 Global Step: 242785 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:43:35,727-Speed 129.49 samples/sec Loss 0.2906 Epoch: 24 Global Step: 242790 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:43:55,491-Speed 129.54 samples/sec Loss 0.3120 Epoch: 24 Global Step: 242795 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:44:15,410-Speed 128.53 samples/sec Loss 0.3282 Epoch: 24 Global Step: 242800 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:44:35,120-Speed 129.89 samples/sec Loss 0.2580 Epoch: 24 Global Step: 242805 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:44:55,371-Speed 126.42 samples/sec Loss 0.2978 Epoch: 24 Global Step: 242810 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:45:15,106-Speed 129.72 samples/sec Loss 0.2627 Epoch: 24 Global Step: 242815 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:45:35,250-Speed 127.09 samples/sec Loss 0.2350 Epoch: 24 Global Step: 242820 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:45:55,002-Speed 129.61 samples/sec Loss 0.3300 Epoch: 24 Global Step: 242825 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:46:14,863-Speed 128.90 samples/sec Loss 0.3509 Epoch: 24 Global Step: 242830 Fp16 Grad Scale: 8192 Required: 8 hours Training: 2021-06-18 21:46:34,600-Speed 129.71 samples/sec Loss 0.2611 Epoch: 24 Global Step: 242835 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:46:54,466-Speed 128.87 samples/sec Loss 0.3444 Epoch: 24 Global Step: 242840 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:47:14,194-Speed 129.77 samples/sec Loss 0.2601 Epoch: 24 Global Step: 242845 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:47:33,981-Speed 129.38 samples/sec Loss 0.2580 Epoch: 24 Global Step: 242850 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:47:54,214-Speed 126.53 samples/sec Loss 0.2994 Epoch: 24 Global Step: 242855 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:48:13,948-Speed 129.73 samples/sec Loss 0.3122 Epoch: 24 Global Step: 242860 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:48:34,053-Speed 127.34 samples/sec Loss 0.2849 Epoch: 24 Global Step: 242865 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:48:53,797-Speed 129.67 samples/sec Loss 0.3314 Epoch: 24 Global Step: 242870 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:49:13,739-Speed 128.38 samples/sec Loss 0.2269 Epoch: 24 Global Step: 242875 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:49:33,461-Speed 129.81 samples/sec Loss 0.2663 Epoch: 24 Global Step: 242880 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:49:53,262-Speed 129.29 samples/sec Loss 0.3006 Epoch: 24 Global Step: 242885 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:50:13,062-Speed 129.30 samples/sec Loss 0.1966 Epoch: 24 Global Step: 242890 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:50:32,846-Speed 129.40 samples/sec Loss 0.3029 Epoch: 24 Global Step: 242895 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:50:53,084-Speed 126.50 samples/sec Loss 0.2584 Epoch: 24 Global Step: 242900 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:51:12,788-Speed 129.93 samples/sec Loss 0.2842 Epoch: 24 Global Step: 242905 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:51:32,859-Speed 127.55 samples/sec Loss 0.3183 Epoch: 24 Global Step: 242910 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:51:52,682-Speed 129.15 samples/sec Loss 0.2892 Epoch: 24 Global Step: 242915 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:52:12,536-Speed 128.95 samples/sec Loss 0.3044 Epoch: 24 Global Step: 242920 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:52:32,443-Speed 128.61 samples/sec Loss 0.3412 Epoch: 24 Global Step: 242925 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:52:52,254-Speed 129.23 samples/sec Loss 0.3540 Epoch: 24 Global Step: 242930 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:53:11,967-Speed 129.87 samples/sec Loss 0.2919 Epoch: 24 Global Step: 242935 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:53:31,640-Speed 130.13 samples/sec Loss 0.2746 Epoch: 24 Global Step: 242940 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:53:51,941-Speed 126.11 samples/sec Loss 0.3603 Epoch: 24 Global Step: 242945 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:54:11,669-Speed 129.77 samples/sec Loss 0.2486 Epoch: 24 Global Step: 242950 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:54:31,751-Speed 127.48 samples/sec Loss 0.2495 Epoch: 24 Global Step: 242955 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:54:51,563-Speed 129.22 samples/sec Loss 0.2742 Epoch: 24 Global Step: 242960 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:55:11,394-Speed 129.10 samples/sec Loss 0.2908 Epoch: 24 Global Step: 242965 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:55:31,117-Speed 129.80 samples/sec Loss 0.2545 Epoch: 24 Global Step: 242970 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:55:51,041-Speed 128.49 samples/sec Loss 0.2796 Epoch: 24 Global Step: 242975 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:56:10,793-Speed 129.62 samples/sec Loss 0.3341 Epoch: 24 Global Step: 242980 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:56:30,540-Speed 129.65 samples/sec Loss 0.2475 Epoch: 24 Global Step: 242985 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:56:50,705-Speed 126.96 samples/sec Loss 0.3422 Epoch: 24 Global Step: 242990 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:57:10,568-Speed 128.89 samples/sec Loss 0.2787 Epoch: 24 Global Step: 242995 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:57:30,661-Speed 127.41 samples/sec Loss 0.2705 Epoch: 24 Global Step: 243000 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:57:50,662-Speed 128.00 samples/sec Loss 0.3531 Epoch: 24 Global Step: 243005 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:58:10,898-Speed 126.51 samples/sec Loss 0.3228 Epoch: 24 Global Step: 243010 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:58:30,988-Speed 127.44 samples/sec Loss 0.2658 Epoch: 24 Global Step: 243015 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:58:50,751-Speed 129.54 samples/sec Loss 0.2692 Epoch: 24 Global Step: 243020 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:59:10,508-Speed 129.58 samples/sec Loss 0.2363 Epoch: 24 Global Step: 243025 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:59:30,467-Speed 128.27 samples/sec Loss 0.2728 Epoch: 24 Global Step: 243030 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 21:59:50,386-Speed 128.52 samples/sec Loss 0.2874 Epoch: 24 Global Step: 243035 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:00:10,067-Speed 130.09 samples/sec Loss 0.2596 Epoch: 24 Global Step: 243040 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:00:30,157-Speed 127.43 samples/sec Loss 0.3843 Epoch: 24 Global Step: 243045 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:00:50,041-Speed 128.75 samples/sec Loss 0.3074 Epoch: 24 Global Step: 243050 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:01:09,757-Speed 129.85 samples/sec Loss 0.3980 Epoch: 24 Global Step: 243055 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:01:29,514-Speed 129.58 samples/sec Loss 0.3574 Epoch: 24 Global Step: 243060 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:01:49,334-Speed 129.17 samples/sec Loss 0.2818 Epoch: 24 Global Step: 243065 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:02:08,627-Speed 132.70 samples/sec Loss 0.3084 Epoch: 24 Global Step: 243070 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:02:28,449-Speed 129.16 samples/sec Loss 0.2349 Epoch: 24 Global Step: 243075 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:02:48,276-Speed 129.13 samples/sec Loss 0.2871 Epoch: 24 Global Step: 243080 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:03:08,031-Speed 129.59 samples/sec Loss 0.2269 Epoch: 24 Global Step: 243085 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:03:28,042-Speed 127.93 samples/sec Loss 0.2925 Epoch: 24 Global Step: 243090 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:03:48,064-Speed 127.87 samples/sec Loss 0.3074 Epoch: 24 Global Step: 243095 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:04:07,591-Speed 131.11 samples/sec Loss 0.3110 Epoch: 24 Global Step: 243100 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:04:27,097-Speed 131.25 samples/sec Loss 0.2715 Epoch: 24 Global Step: 243105 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:04:47,100-Speed 127.99 samples/sec Loss 0.3124 Epoch: 24 Global Step: 243110 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:05:06,632-Speed 131.07 samples/sec Loss 0.3007 Epoch: 24 Global Step: 243115 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:05:26,617-Speed 128.11 samples/sec Loss 0.3074 Epoch: 24 Global Step: 243120 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:05:46,131-Speed 131.19 samples/sec Loss 0.2774 Epoch: 24 Global Step: 243125 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:06:05,977-Speed 129.00 samples/sec Loss 0.2689 Epoch: 24 Global Step: 243130 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:06:25,751-Speed 129.47 samples/sec Loss 0.2471 Epoch: 24 Global Step: 243135 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:06:45,660-Speed 128.59 samples/sec Loss 0.2864 Epoch: 24 Global Step: 243140 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:07:05,083-Speed 131.81 samples/sec Loss 0.3001 Epoch: 24 Global Step: 243145 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:07:24,981-Speed 128.66 samples/sec Loss 0.2695 Epoch: 24 Global Step: 243150 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:07:44,941-Speed 128.26 samples/sec Loss 0.2725 Epoch: 24 Global Step: 243155 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:08:04,482-Speed 131.01 samples/sec Loss 0.2584 Epoch: 24 Global Step: 243160 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:08:24,374-Speed 128.70 samples/sec Loss 0.2375 Epoch: 24 Global Step: 243165 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:08:43,885-Speed 131.21 samples/sec Loss 0.3084 Epoch: 24 Global Step: 243170 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:09:03,660-Speed 129.46 samples/sec Loss 0.3226 Epoch: 24 Global Step: 243175 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:09:22,987-Speed 132.47 samples/sec Loss 0.3484 Epoch: 24 Global Step: 243180 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:09:43,002-Speed 127.91 samples/sec Loss 0.2773 Epoch: 24 Global Step: 243185 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:10:02,578-Speed 130.78 samples/sec Loss 0.3059 Epoch: 24 Global Step: 243190 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:10:22,426-Speed 128.99 samples/sec Loss 0.2330 Epoch: 24 Global Step: 243195 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:10:42,337-Speed 128.58 samples/sec Loss 0.2719 Epoch: 24 Global Step: 243200 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:11:01,930-Speed 130.66 samples/sec Loss 0.3150 Epoch: 24 Global Step: 243205 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:11:21,832-Speed 128.64 samples/sec Loss 0.3386 Epoch: 24 Global Step: 243210 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:11:41,282-Speed 131.62 samples/sec Loss 0.3264 Epoch: 24 Global Step: 243215 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:12:01,025-Speed 129.67 samples/sec Loss 0.2641 Epoch: 24 Global Step: 243220 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:12:20,820-Speed 129.34 samples/sec Loss 0.3301 Epoch: 24 Global Step: 243225 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:12:40,817-Speed 128.02 samples/sec Loss 0.2853 Epoch: 24 Global Step: 243230 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:13:00,257-Speed 131.70 samples/sec Loss 0.2910 Epoch: 24 Global Step: 243235 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:13:20,090-Speed 129.08 samples/sec Loss 0.2964 Epoch: 24 Global Step: 243240 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:13:39,978-Speed 128.73 samples/sec Loss 0.3424 Epoch: 24 Global Step: 243245 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:13:59,580-Speed 130.61 samples/sec Loss 0.3079 Epoch: 24 Global Step: 243250 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:14:19,722-Speed 127.10 samples/sec Loss 0.3254 Epoch: 24 Global Step: 243255 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:14:39,191-Speed 131.50 samples/sec Loss 0.2889 Epoch: 24 Global Step: 243260 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:14:58,928-Speed 129.72 samples/sec Loss 0.2422 Epoch: 24 Global Step: 243265 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:15:18,714-Speed 129.39 samples/sec Loss 0.3063 Epoch: 24 Global Step: 243270 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:15:38,703-Speed 128.08 samples/sec Loss 0.2742 Epoch: 24 Global Step: 243275 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:15:58,140-Speed 131.72 samples/sec Loss 0.3395 Epoch: 24 Global Step: 243280 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:16:17,826-Speed 130.04 samples/sec Loss 0.2624 Epoch: 24 Global Step: 243285 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:16:37,809-Speed 128.12 samples/sec Loss 0.2861 Epoch: 24 Global Step: 243290 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:16:57,544-Speed 129.73 samples/sec Loss 0.2714 Epoch: 24 Global Step: 243295 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:17:17,534-Speed 128.07 samples/sec Loss 0.2783 Epoch: 24 Global Step: 243300 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:17:36,966-Speed 131.75 samples/sec Loss 0.3047 Epoch: 24 Global Step: 243305 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:17:56,745-Speed 129.44 samples/sec Loss 0.3250 Epoch: 24 Global Step: 243310 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:18:16,550-Speed 129.26 samples/sec Loss 0.2587 Epoch: 24 Global Step: 243315 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:18:36,554-Speed 127.98 samples/sec Loss 0.3115 Epoch: 24 Global Step: 243320 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:18:56,007-Speed 131.60 samples/sec Loss 0.3257 Epoch: 24 Global Step: 243325 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:19:15,840-Speed 129.09 samples/sec Loss 0.3082 Epoch: 24 Global Step: 243330 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:19:35,895-Speed 127.65 samples/sec Loss 0.2819 Epoch: 24 Global Step: 243335 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:19:55,439-Speed 131.00 samples/sec Loss 0.2580 Epoch: 24 Global Step: 243340 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:20:15,091-Speed 130.27 samples/sec Loss 0.2407 Epoch: 24 Global Step: 243345 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:20:34,662-Speed 130.81 samples/sec Loss 0.2968 Epoch: 24 Global Step: 243350 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:20:54,387-Speed 129.80 samples/sec Loss 0.2835 Epoch: 24 Global Step: 243355 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:21:14,183-Speed 129.32 samples/sec Loss 0.2267 Epoch: 24 Global Step: 243360 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:21:34,096-Speed 128.56 samples/sec Loss 0.3271 Epoch: 24 Global Step: 243365 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:21:53,174-Speed 134.20 samples/sec Loss 0.2743 Epoch: 24 Global Step: 243370 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:22:12,908-Speed 129.73 samples/sec Loss 0.3161 Epoch: 24 Global Step: 243375 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:22:32,805-Speed 128.67 samples/sec Loss 0.3475 Epoch: 24 Global Step: 243380 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:22:52,610-Speed 129.26 samples/sec Loss 0.3050 Epoch: 24 Global Step: 243385 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:23:12,412-Speed 129.29 samples/sec Loss 0.2384 Epoch: 24 Global Step: 243390 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:23:32,144-Speed 129.75 samples/sec Loss 0.2664 Epoch: 24 Global Step: 243395 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:23:51,793-Speed 130.29 samples/sec Loss 0.2935 Epoch: 24 Global Step: 243400 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:24:11,622-Speed 129.11 samples/sec Loss 0.3474 Epoch: 24 Global Step: 243405 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:24:31,360-Speed 129.71 samples/sec Loss 0.2648 Epoch: 24 Global Step: 243410 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:24:51,294-Speed 128.43 samples/sec Loss 0.2818 Epoch: 24 Global Step: 243415 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:25:11,371-Speed 127.52 samples/sec Loss 0.2596 Epoch: 24 Global Step: 243420 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:25:31,202-Speed 129.09 samples/sec Loss 0.3247 Epoch: 24 Global Step: 243425 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:25:51,267-Speed 127.60 samples/sec Loss 0.2653 Epoch: 24 Global Step: 243430 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:26:11,058-Speed 129.36 samples/sec Loss 0.2684 Epoch: 24 Global Step: 243435 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:26:30,769-Speed 129.88 samples/sec Loss 0.3067 Epoch: 24 Global Step: 243440 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:26:50,510-Speed 129.68 samples/sec Loss 0.2889 Epoch: 24 Global Step: 243445 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:27:10,388-Speed 128.80 samples/sec Loss 0.2632 Epoch: 24 Global Step: 243450 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:27:30,126-Speed 129.71 samples/sec Loss 0.3260 Epoch: 24 Global Step: 243455 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:27:49,980-Speed 128.95 samples/sec Loss 0.3014 Epoch: 24 Global Step: 243460 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:28:09,970-Speed 128.07 samples/sec Loss 0.3204 Epoch: 24 Global Step: 243465 Fp16 Grad Scale: 4096 Required: 7 hours Training: 2021-06-18 22:28:29,663-Speed 130.00 samples/sec Loss 0.3260 Epoch: 24 Global Step: 243470 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:28:50,153-Speed 124.95 samples/sec Loss 0.3059 Epoch: 24 Global Step: 243475 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:29:09,951-Speed 129.31 samples/sec Loss 0.3413 Epoch: 24 Global Step: 243480 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:29:29,657-Speed 129.91 samples/sec Loss 0.3306 Epoch: 24 Global Step: 243485 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:29:49,525-Speed 128.86 samples/sec Loss 0.3408 Epoch: 24 Global Step: 243490 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:30:09,264-Speed 129.71 samples/sec Loss 0.2372 Epoch: 24 Global Step: 243495 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:30:29,048-Speed 129.40 samples/sec Loss 0.3268 Epoch: 24 Global Step: 243500 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:30:48,845-Speed 129.32 samples/sec Loss 0.3134 Epoch: 24 Global Step: 243505 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:31:08,871-Speed 127.84 samples/sec Loss 0.3293 Epoch: 24 Global Step: 243510 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:31:28,537-Speed 130.18 samples/sec Loss 0.2806 Epoch: 24 Global Step: 243515 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:31:48,643-Speed 127.33 samples/sec Loss 0.2742 Epoch: 24 Global Step: 243520 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:32:08,581-Speed 128.40 samples/sec Loss 0.2489 Epoch: 24 Global Step: 243525 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:32:28,250-Speed 130.16 samples/sec Loss 0.2806 Epoch: 24 Global Step: 243530 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:32:48,018-Speed 129.51 samples/sec Loss 0.2600 Epoch: 24 Global Step: 243535 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:33:07,780-Speed 129.55 samples/sec Loss 0.2693 Epoch: 24 Global Step: 243540 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:33:27,537-Speed 129.58 samples/sec Loss 0.2974 Epoch: 24 Global Step: 243545 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:33:47,442-Speed 128.62 samples/sec Loss 0.3450 Epoch: 24 Global Step: 243550 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:34:07,351-Speed 128.59 samples/sec Loss 0.2984 Epoch: 24 Global Step: 243555 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:34:27,106-Speed 129.60 samples/sec Loss 0.2986 Epoch: 24 Global Step: 243560 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:34:47,241-Speed 127.15 samples/sec Loss 0.2739 Epoch: 24 Global Step: 243565 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:35:07,121-Speed 128.78 samples/sec Loss 0.2578 Epoch: 24 Global Step: 243570 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:35:26,748-Speed 130.44 samples/sec Loss 0.3268 Epoch: 24 Global Step: 243575 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:35:46,452-Speed 129.93 samples/sec Loss 0.2764 Epoch: 24 Global Step: 243580 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:36:06,340-Speed 128.73 samples/sec Loss 0.2872 Epoch: 24 Global Step: 243585 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:36:26,079-Speed 129.70 samples/sec Loss 0.2855 Epoch: 24 Global Step: 243590 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:36:45,862-Speed 129.41 samples/sec Loss 0.2746 Epoch: 24 Global Step: 243595 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:37:05,891-Speed 127.82 samples/sec Loss 0.2656 Epoch: 24 Global Step: 243600 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:37:25,624-Speed 129.74 samples/sec Loss 0.2642 Epoch: 24 Global Step: 243605 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:37:45,734-Speed 127.31 samples/sec Loss 0.3458 Epoch: 24 Global Step: 243610 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:38:05,626-Speed 128.70 samples/sec Loss 0.2918 Epoch: 24 Global Step: 243615 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:38:25,249-Speed 130.46 samples/sec Loss 0.3342 Epoch: 24 Global Step: 243620 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:38:44,950-Speed 129.95 samples/sec Loss 0.2848 Epoch: 24 Global Step: 243625 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:39:04,870-Speed 128.52 samples/sec Loss 0.2656 Epoch: 24 Global Step: 243630 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:39:24,710-Speed 129.04 samples/sec Loss 0.3205 Epoch: 24 Global Step: 243635 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:39:44,490-Speed 129.43 samples/sec Loss 0.2837 Epoch: 24 Global Step: 243640 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:40:04,560-Speed 127.56 samples/sec Loss 0.2795 Epoch: 24 Global Step: 243645 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:40:24,284-Speed 129.80 samples/sec Loss 0.2977 Epoch: 24 Global Step: 243650 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:40:44,449-Speed 126.96 samples/sec Loss 0.2558 Epoch: 24 Global Step: 243655 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:41:04,209-Speed 129.56 samples/sec Loss 0.2131 Epoch: 24 Global Step: 243660 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:41:23,983-Speed 129.47 samples/sec Loss 0.2920 Epoch: 24 Global Step: 243665 Fp16 Grad Scale: 8192 Required: 7 hours Training: 2021-06-18 22:41:43,724-Speed 129.68 samples/sec Loss 0.2492 Epoch: 24 Global Step: 243670 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:42:03,647-Speed 128.50 samples/sec Loss 0.2979 Epoch: 24 Global Step: 243675 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:42:23,383-Speed 129.72 samples/sec Loss 0.3040 Epoch: 24 Global Step: 243680 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:42:43,116-Speed 129.74 samples/sec Loss 0.2767 Epoch: 24 Global Step: 243685 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:43:03,190-Speed 127.53 samples/sec Loss 0.2577 Epoch: 24 Global Step: 243690 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:43:23,008-Speed 129.19 samples/sec Loss 0.3103 Epoch: 24 Global Step: 243695 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:43:43,122-Speed 127.28 samples/sec Loss 0.2998 Epoch: 24 Global Step: 243700 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:44:02,809-Speed 130.04 samples/sec Loss 0.3507 Epoch: 24 Global Step: 243705 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:44:22,562-Speed 129.61 samples/sec Loss 0.3355 Epoch: 24 Global Step: 243710 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:44:42,310-Speed 129.64 samples/sec Loss 0.3001 Epoch: 24 Global Step: 243715 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:45:02,099-Speed 129.37 samples/sec Loss 0.2573 Epoch: 24 Global Step: 243720 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:45:21,900-Speed 129.29 samples/sec Loss 0.2610 Epoch: 24 Global Step: 243725 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:45:41,644-Speed 129.67 samples/sec Loss 0.2527 Epoch: 24 Global Step: 243730 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:46:01,647-Speed 127.99 samples/sec Loss 0.3123 Epoch: 24 Global Step: 243735 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:46:21,350-Speed 129.93 samples/sec Loss 0.2646 Epoch: 24 Global Step: 243740 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:46:41,483-Speed 127.16 samples/sec Loss 0.3699 Epoch: 24 Global Step: 243745 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:47:01,302-Speed 129.17 samples/sec Loss 0.2734 Epoch: 24 Global Step: 243750 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:47:20,924-Speed 130.47 samples/sec Loss 0.3131 Epoch: 24 Global Step: 243755 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:47:40,690-Speed 129.52 samples/sec Loss 0.3238 Epoch: 24 Global Step: 243760 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:48:00,505-Speed 129.20 samples/sec Loss 0.2691 Epoch: 24 Global Step: 243765 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:48:20,340-Speed 129.07 samples/sec Loss 0.3130 Epoch: 24 Global Step: 243770 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 22:48:39,711-Speed 132.16 samples/sec Loss 0.2576 Epoch: 24 Global Step: 243775 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:48:59,443-Speed 129.74 samples/sec Loss 0.3121 Epoch: 24 Global Step: 243780 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:49:19,234-Speed 129.36 samples/sec Loss 0.3098 Epoch: 24 Global Step: 243785 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:49:39,689-Speed 125.16 samples/sec Loss 0.3145 Epoch: 24 Global Step: 243790 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:49:59,518-Speed 129.11 samples/sec Loss 0.3029 Epoch: 24 Global Step: 243795 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:50:19,251-Speed 129.74 samples/sec Loss 0.3233 Epoch: 24 Global Step: 243800 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:50:39,005-Speed 129.60 samples/sec Loss 0.2855 Epoch: 24 Global Step: 243805 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:50:58,848-Speed 129.02 samples/sec Loss 0.2970 Epoch: 24 Global Step: 243810 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:51:18,640-Speed 129.35 samples/sec Loss 0.2671 Epoch: 24 Global Step: 243815 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:51:38,506-Speed 128.87 samples/sec Loss 0.2564 Epoch: 24 Global Step: 243820 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:51:58,231-Speed 129.79 samples/sec Loss 0.3519 Epoch: 24 Global Step: 243825 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:52:17,965-Speed 129.73 samples/sec Loss 0.2711 Epoch: 24 Global Step: 243830 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:52:38,311-Speed 125.83 samples/sec Loss 0.3223 Epoch: 24 Global Step: 243835 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:52:57,646-Speed 132.41 samples/sec Loss 0.3277 Epoch: 24 Global Step: 243840 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:53:17,365-Speed 129.83 samples/sec Loss 0.2634 Epoch: 24 Global Step: 243845 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:53:37,106-Speed 129.68 samples/sec Loss 0.3393 Epoch: 24 Global Step: 243850 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:53:56,910-Speed 129.27 samples/sec Loss 0.2680 Epoch: 24 Global Step: 243855 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:54:16,663-Speed 129.61 samples/sec Loss 0.3840 Epoch: 24 Global Step: 243860 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:54:36,468-Speed 129.27 samples/sec Loss 0.3323 Epoch: 24 Global Step: 243865 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:54:56,205-Speed 129.71 samples/sec Loss 0.3741 Epoch: 24 Global Step: 243870 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:55:15,912-Speed 129.91 samples/sec Loss 0.2373 Epoch: 24 Global Step: 243875 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:55:36,297-Speed 125.59 samples/sec Loss 0.3559 Epoch: 24 Global Step: 243880 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:55:56,167-Speed 128.85 samples/sec Loss 0.2105 Epoch: 24 Global Step: 243885 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:56:15,867-Speed 129.95 samples/sec Loss 0.2969 Epoch: 24 Global Step: 243890 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:56:35,614-Speed 129.65 samples/sec Loss 0.3266 Epoch: 24 Global Step: 243895 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:56:55,411-Speed 129.32 samples/sec Loss 0.3116 Epoch: 24 Global Step: 243900 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:57:15,218-Speed 129.25 samples/sec Loss 0.3242 Epoch: 24 Global Step: 243905 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:57:35,137-Speed 128.53 samples/sec Loss 0.2936 Epoch: 24 Global Step: 243910 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:57:54,874-Speed 129.71 samples/sec Loss 0.3037 Epoch: 24 Global Step: 243915 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:58:14,628-Speed 129.60 samples/sec Loss 0.3048 Epoch: 24 Global Step: 243920 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:58:35,041-Speed 125.42 samples/sec Loss 0.2583 Epoch: 24 Global Step: 243925 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:58:54,932-Speed 128.70 samples/sec Loss 0.2762 Epoch: 24 Global Step: 243930 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:59:14,676-Speed 129.67 samples/sec Loss 0.3005 Epoch: 24 Global Step: 243935 Fp16 Grad Scale: 2048 Required: 6 hours Training: 2021-06-18 22:59:34,425-Speed 129.63 samples/sec Loss 0.3455 Epoch: 24 Global Step: 243940 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 22:59:54,183-Speed 129.57 samples/sec Loss 0.2911 Epoch: 24 Global Step: 243945 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:00:13,987-Speed 129.28 samples/sec Loss 0.3289 Epoch: 24 Global Step: 243950 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:00:33,897-Speed 128.58 samples/sec Loss 0.3233 Epoch: 24 Global Step: 243955 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:00:53,640-Speed 129.68 samples/sec Loss 0.3250 Epoch: 24 Global Step: 243960 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:01:13,410-Speed 129.49 samples/sec Loss 0.2527 Epoch: 24 Global Step: 243965 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:01:33,828-Speed 125.39 samples/sec Loss 0.2877 Epoch: 24 Global Step: 243970 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:01:53,780-Speed 128.31 samples/sec Loss 0.2760 Epoch: 24 Global Step: 243975 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:02:13,510-Speed 129.76 samples/sec Loss 0.2205 Epoch: 24 Global Step: 243980 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:02:33,219-Speed 129.90 samples/sec Loss 0.2532 Epoch: 24 Global Step: 243985 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:02:53,004-Speed 129.40 samples/sec Loss 0.3129 Epoch: 24 Global Step: 243990 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:03:12,784-Speed 129.43 samples/sec Loss 0.2827 Epoch: 24 Global Step: 243995 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:03:32,685-Speed 128.64 samples/sec Loss 0.3164 Epoch: 24 Global Step: 244000 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:03:52,428-Speed 129.67 samples/sec Loss 0.2613 Epoch: 24 Global Step: 244005 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:04:12,098-Speed 130.15 samples/sec Loss 0.2871 Epoch: 24 Global Step: 244010 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:04:32,468-Speed 125.68 samples/sec Loss 0.3635 Epoch: 24 Global Step: 244015 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:04:52,385-Speed 128.54 samples/sec Loss 0.3077 Epoch: 24 Global Step: 244020 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:05:12,111-Speed 129.78 samples/sec Loss 0.2981 Epoch: 24 Global Step: 244025 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:05:32,011-Speed 128.65 samples/sec Loss 0.2860 Epoch: 24 Global Step: 244030 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:05:51,673-Speed 130.21 samples/sec Loss 0.2523 Epoch: 24 Global Step: 244035 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:06:11,485-Speed 129.22 samples/sec Loss 0.2811 Epoch: 24 Global Step: 244040 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:06:31,928-Speed 125.24 samples/sec Loss 0.2783 Epoch: 24 Global Step: 244045 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:06:51,673-Speed 129.66 samples/sec Loss 0.2904 Epoch: 24 Global Step: 244050 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:07:11,479-Speed 129.26 samples/sec Loss 0.2713 Epoch: 24 Global Step: 244055 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:07:31,732-Speed 126.40 samples/sec Loss 0.3626 Epoch: 24 Global Step: 244060 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:07:51,585-Speed 128.96 samples/sec Loss 0.2920 Epoch: 24 Global Step: 244065 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:08:11,304-Speed 129.83 samples/sec Loss 0.3547 Epoch: 24 Global Step: 244070 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:08:31,118-Speed 129.21 samples/sec Loss 0.2174 Epoch: 24 Global Step: 244075 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:08:50,854-Speed 129.72 samples/sec Loss 0.2764 Epoch: 24 Global Step: 244080 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:09:10,614-Speed 129.56 samples/sec Loss 0.2428 Epoch: 24 Global Step: 244085 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:09:30,486-Speed 128.83 samples/sec Loss 0.2698 Epoch: 24 Global Step: 244090 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:09:50,817-Speed 125.92 samples/sec Loss 0.3147 Epoch: 24 Global Step: 244095 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:10:10,617-Speed 129.30 samples/sec Loss 0.2990 Epoch: 24 Global Step: 244100 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:10:31,063-Speed 125.21 samples/sec Loss 0.2656 Epoch: 24 Global Step: 244105 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:10:50,129-Speed 134.28 samples/sec Loss 0.2860 Epoch: 24 Global Step: 244110 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:11:10,190-Speed 127.62 samples/sec Loss 0.2570 Epoch: 24 Global Step: 244115 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:11:30,146-Speed 128.28 samples/sec Loss 0.3364 Epoch: 24 Global Step: 244120 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:11:49,949-Speed 129.28 samples/sec Loss 0.3311 Epoch: 24 Global Step: 244125 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:12:09,870-Speed 128.51 samples/sec Loss 0.2171 Epoch: 24 Global Step: 244130 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:12:29,579-Speed 129.90 samples/sec Loss 0.2955 Epoch: 24 Global Step: 244135 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:12:49,371-Speed 129.35 samples/sec Loss 0.3038 Epoch: 24 Global Step: 244140 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:13:09,131-Speed 129.56 samples/sec Loss 0.3295 Epoch: 24 Global Step: 244145 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:13:29,231-Speed 127.37 samples/sec Loss 0.2813 Epoch: 24 Global Step: 244150 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:13:49,132-Speed 128.65 samples/sec Loss 0.3417 Epoch: 24 Global Step: 244155 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:14:08,888-Speed 129.59 samples/sec Loss 0.2940 Epoch: 24 Global Step: 244160 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:14:28,569-Speed 130.08 samples/sec Loss 0.2668 Epoch: 24 Global Step: 244165 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:14:48,514-Speed 128.36 samples/sec Loss 0.2802 Epoch: 24 Global Step: 244170 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:15:08,611-Speed 127.39 samples/sec Loss 0.3238 Epoch: 24 Global Step: 244175 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:15:29,008-Speed 125.51 samples/sec Loss 0.3716 Epoch: 24 Global Step: 244180 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:15:48,829-Speed 129.16 samples/sec Loss 0.2932 Epoch: 24 Global Step: 244185 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:16:08,603-Speed 129.47 samples/sec Loss 0.2983 Epoch: 24 Global Step: 244190 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:16:28,704-Speed 127.36 samples/sec Loss 0.2623 Epoch: 24 Global Step: 244195 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:16:48,602-Speed 128.67 samples/sec Loss 0.4390 Epoch: 24 Global Step: 244200 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:17:08,251-Speed 130.29 samples/sec Loss 0.3247 Epoch: 24 Global Step: 244205 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:17:28,020-Speed 129.50 samples/sec Loss 0.2928 Epoch: 24 Global Step: 244210 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:17:47,960-Speed 128.40 samples/sec Loss 0.2848 Epoch: 24 Global Step: 244215 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:18:08,071-Speed 127.30 samples/sec Loss 0.3357 Epoch: 24 Global Step: 244220 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:18:27,968-Speed 128.67 samples/sec Loss 0.2716 Epoch: 24 Global Step: 244225 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:18:47,739-Speed 129.49 samples/sec Loss 0.2937 Epoch: 24 Global Step: 244230 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:19:07,579-Speed 129.04 samples/sec Loss 0.2684 Epoch: 24 Global Step: 244235 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:19:27,615-Speed 127.78 samples/sec Loss 0.3231 Epoch: 24 Global Step: 244240 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:19:47,527-Speed 128.57 samples/sec Loss 0.2510 Epoch: 24 Global Step: 244245 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:20:07,219-Speed 130.00 samples/sec Loss 0.3042 Epoch: 24 Global Step: 244250 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:20:26,971-Speed 129.62 samples/sec Loss 0.3055 Epoch: 24 Global Step: 244255 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:20:46,974-Speed 127.99 samples/sec Loss 0.2870 Epoch: 24 Global Step: 244260 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:21:07,091-Speed 127.26 samples/sec Loss 0.2247 Epoch: 24 Global Step: 244265 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:21:26,324-Speed 133.11 samples/sec Loss 0.3040 Epoch: 24 Global Step: 244270 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:21:46,105-Speed 129.42 samples/sec Loss 0.2936 Epoch: 24 Global Step: 244275 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:22:06,082-Speed 128.16 samples/sec Loss 0.2693 Epoch: 24 Global Step: 244280 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:22:25,756-Speed 130.13 samples/sec Loss 0.2905 Epoch: 24 Global Step: 244285 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:22:45,692-Speed 128.41 samples/sec Loss 0.2720 Epoch: 24 Global Step: 244290 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:23:05,347-Speed 130.25 samples/sec Loss 0.3363 Epoch: 24 Global Step: 244295 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:23:25,194-Speed 129.00 samples/sec Loss 0.3253 Epoch: 24 Global Step: 244300 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:23:45,004-Speed 129.23 samples/sec Loss 0.2794 Epoch: 24 Global Step: 244305 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:24:06,001-Speed 121.93 samples/sec Loss 0.3279 Epoch: 24 Global Step: 244310 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:24:25,700-Speed 129.96 samples/sec Loss 0.3102 Epoch: 24 Global Step: 244315 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:24:45,376-Speed 130.12 samples/sec Loss 0.2459 Epoch: 24 Global Step: 244320 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:25:05,325-Speed 128.34 samples/sec Loss 0.2642 Epoch: 24 Global Step: 244325 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:25:24,965-Speed 130.35 samples/sec Loss 0.2757 Epoch: 24 Global Step: 244330 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:25:44,870-Speed 128.62 samples/sec Loss 0.3426 Epoch: 24 Global Step: 244335 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:26:04,599-Speed 129.76 samples/sec Loss 0.3690 Epoch: 24 Global Step: 244340 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:26:24,526-Speed 128.47 samples/sec Loss 0.3517 Epoch: 24 Global Step: 244345 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:26:44,323-Speed 129.32 samples/sec Loss 0.2801 Epoch: 24 Global Step: 244350 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:27:05,215-Speed 122.54 samples/sec Loss 0.2691 Epoch: 24 Global Step: 244355 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:27:24,939-Speed 129.80 samples/sec Loss 0.3380 Epoch: 24 Global Step: 244360 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:27:44,694-Speed 129.59 samples/sec Loss 0.2788 Epoch: 24 Global Step: 244365 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:28:04,564-Speed 128.85 samples/sec Loss 0.2602 Epoch: 24 Global Step: 244370 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:28:24,192-Speed 130.43 samples/sec Loss 0.2710 Epoch: 24 Global Step: 244375 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:28:44,093-Speed 128.64 samples/sec Loss 0.2789 Epoch: 24 Global Step: 244380 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:29:03,671-Speed 130.76 samples/sec Loss 0.3663 Epoch: 24 Global Step: 244385 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:29:23,451-Speed 129.44 samples/sec Loss 0.2547 Epoch: 24 Global Step: 244390 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:29:43,299-Speed 128.98 samples/sec Loss 0.3031 Epoch: 24 Global Step: 244395 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:30:03,594-Speed 126.14 samples/sec Loss 0.2990 Epoch: 24 Global Step: 244400 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:30:23,399-Speed 129.27 samples/sec Loss 0.3322 Epoch: 24 Global Step: 244405 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:30:43,178-Speed 129.44 samples/sec Loss 0.2654 Epoch: 24 Global Step: 244410 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:31:03,029-Speed 128.97 samples/sec Loss 0.2737 Epoch: 24 Global Step: 244415 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:31:22,658-Speed 130.42 samples/sec Loss 0.3379 Epoch: 24 Global Step: 244420 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:31:42,579-Speed 128.51 samples/sec Loss 0.2872 Epoch: 24 Global Step: 244425 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:32:02,317-Speed 129.71 samples/sec Loss 0.3129 Epoch: 24 Global Step: 244430 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:32:22,301-Speed 128.11 samples/sec Loss 0.2729 Epoch: 24 Global Step: 244435 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:32:42,044-Speed 129.67 samples/sec Loss 0.3088 Epoch: 24 Global Step: 244440 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:33:02,592-Speed 124.59 samples/sec Loss 0.3429 Epoch: 24 Global Step: 244445 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:33:22,317-Speed 129.79 samples/sec Loss 0.2937 Epoch: 24 Global Step: 244450 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:33:42,216-Speed 128.66 samples/sec Loss 0.3104 Epoch: 24 Global Step: 244455 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:34:01,925-Speed 129.89 samples/sec Loss 0.2677 Epoch: 24 Global Step: 244460 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:34:21,628-Speed 129.94 samples/sec Loss 0.2385 Epoch: 24 Global Step: 244465 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:34:41,495-Speed 128.86 samples/sec Loss 0.3101 Epoch: 24 Global Step: 244470 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:35:01,265-Speed 129.50 samples/sec Loss 0.3235 Epoch: 24 Global Step: 244475 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:35:21,244-Speed 128.14 samples/sec Loss 0.2449 Epoch: 24 Global Step: 244480 Fp16 Grad Scale: 4096 Required: 6 hours Training: 2021-06-18 23:35:40,897-Speed 130.26 samples/sec Loss 0.3493 Epoch: 24 Global Step: 244485 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:36:01,259-Speed 125.73 samples/sec Loss 0.3451 Epoch: 24 Global Step: 244490 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:36:20,994-Speed 129.72 samples/sec Loss 0.3053 Epoch: 24 Global Step: 244495 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:36:40,965-Speed 128.19 samples/sec Loss 0.2812 Epoch: 24 Global Step: 244500 Fp16 Grad Scale: 8192 Required: 6 hours Training: 2021-06-18 23:37:00,099-Speed 133.80 samples/sec Loss 0.3017 Epoch: 24 Global Step: 244505 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:37:19,814-Speed 129.86 samples/sec Loss 0.3003 Epoch: 24 Global Step: 244510 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:37:39,641-Speed 129.12 samples/sec Loss 0.2912 Epoch: 24 Global Step: 244515 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:37:59,392-Speed 129.62 samples/sec Loss 0.2574 Epoch: 24 Global Step: 244520 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:38:19,356-Speed 128.24 samples/sec Loss 0.3796 Epoch: 24 Global Step: 244525 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:38:39,114-Speed 129.57 samples/sec Loss 0.3052 Epoch: 24 Global Step: 244530 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:38:59,243-Speed 127.18 samples/sec Loss 0.3404 Epoch: 24 Global Step: 244535 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:39:18,968-Speed 129.79 samples/sec Loss 0.2656 Epoch: 24 Global Step: 244540 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:39:39,066-Speed 127.38 samples/sec Loss 0.2539 Epoch: 24 Global Step: 244545 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:39:58,691-Speed 130.45 samples/sec Loss 0.2657 Epoch: 24 Global Step: 244550 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:40:18,390-Speed 129.96 samples/sec Loss 0.3555 Epoch: 24 Global Step: 244555 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:40:38,247-Speed 128.93 samples/sec Loss 0.3315 Epoch: 24 Global Step: 244560 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:40:57,939-Speed 130.01 samples/sec Loss 0.2331 Epoch: 24 Global Step: 244565 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:41:17,924-Speed 128.10 samples/sec Loss 0.2665 Epoch: 24 Global Step: 244570 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:41:37,546-Speed 130.47 samples/sec Loss 0.3077 Epoch: 24 Global Step: 244575 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:41:57,659-Speed 127.29 samples/sec Loss 0.2963 Epoch: 24 Global Step: 244580 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:42:17,477-Speed 129.18 samples/sec Loss 0.2305 Epoch: 24 Global Step: 244585 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:42:37,651-Speed 126.90 samples/sec Loss 0.3098 Epoch: 24 Global Step: 244590 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:42:57,339-Speed 130.04 samples/sec Loss 0.2827 Epoch: 24 Global Step: 244595 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:43:17,180-Speed 129.03 samples/sec Loss 0.3547 Epoch: 24 Global Step: 244600 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:43:37,014-Speed 129.08 samples/sec Loss 0.2342 Epoch: 24 Global Step: 244605 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:43:56,724-Speed 129.89 samples/sec Loss 0.2926 Epoch: 24 Global Step: 244610 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:44:16,690-Speed 128.23 samples/sec Loss 0.2833 Epoch: 24 Global Step: 244615 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:44:36,322-Speed 130.41 samples/sec Loss 0.2799 Epoch: 24 Global Step: 244620 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:44:56,493-Speed 126.92 samples/sec Loss 0.3021 Epoch: 24 Global Step: 244625 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:45:16,212-Speed 129.83 samples/sec Loss 0.2440 Epoch: 24 Global Step: 244630 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:45:36,399-Speed 126.82 samples/sec Loss 0.2969 Epoch: 24 Global Step: 244635 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:45:56,023-Speed 130.46 samples/sec Loss 0.2802 Epoch: 24 Global Step: 244640 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:46:15,946-Speed 128.50 samples/sec Loss 0.2801 Epoch: 24 Global Step: 244645 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:46:35,754-Speed 129.25 samples/sec Loss 0.3816 Epoch: 24 Global Step: 244650 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:46:55,524-Speed 129.50 samples/sec Loss 0.3831 Epoch: 24 Global Step: 244655 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:47:15,468-Speed 128.37 samples/sec Loss 0.2753 Epoch: 24 Global Step: 244660 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:47:35,099-Speed 130.41 samples/sec Loss 0.3020 Epoch: 24 Global Step: 244665 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:47:55,348-Speed 126.43 samples/sec Loss 0.2888 Epoch: 24 Global Step: 244670 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:48:15,080-Speed 129.74 samples/sec Loss 0.3323 Epoch: 24 Global Step: 244675 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:48:35,256-Speed 126.89 samples/sec Loss 0.2860 Epoch: 24 Global Step: 244680 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:48:54,874-Speed 130.50 samples/sec Loss 0.2869 Epoch: 24 Global Step: 244685 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:49:14,655-Speed 129.42 samples/sec Loss 0.3172 Epoch: 24 Global Step: 244690 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:49:34,458-Speed 129.28 samples/sec Loss 0.2798 Epoch: 24 Global Step: 244695 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:49:54,144-Speed 130.05 samples/sec Loss 0.3530 Epoch: 24 Global Step: 244700 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:50:14,136-Speed 128.06 samples/sec Loss 0.2824 Epoch: 24 Global Step: 244705 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:50:33,750-Speed 130.53 samples/sec Loss 0.2591 Epoch: 24 Global Step: 244710 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:50:53,845-Speed 127.40 samples/sec Loss 0.2166 Epoch: 24 Global Step: 244715 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:51:13,671-Speed 129.13 samples/sec Loss 0.2843 Epoch: 24 Global Step: 244720 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:51:33,794-Speed 127.23 samples/sec Loss 0.2509 Epoch: 24 Global Step: 244725 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:51:53,426-Speed 130.41 samples/sec Loss 0.3216 Epoch: 24 Global Step: 244730 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:52:13,212-Speed 129.39 samples/sec Loss 0.2638 Epoch: 24 Global Step: 244735 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:52:33,028-Speed 129.19 samples/sec Loss 0.2634 Epoch: 24 Global Step: 244740 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:52:52,999-Speed 128.19 samples/sec Loss 0.2760 Epoch: 24 Global Step: 244745 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-18 23:53:12,447-Speed 131.64 samples/sec Loss 0.2460 Epoch: 24 Global Step: 244750 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:53:32,072-Speed 130.45 samples/sec Loss 0.3097 Epoch: 24 Global Step: 244755 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:53:51,971-Speed 128.66 samples/sec Loss 0.2750 Epoch: 24 Global Step: 244760 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:54:11,678-Speed 129.91 samples/sec Loss 0.2991 Epoch: 24 Global Step: 244765 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:54:31,861-Speed 126.84 samples/sec Loss 0.2819 Epoch: 24 Global Step: 244770 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:54:51,475-Speed 130.53 samples/sec Loss 0.3246 Epoch: 24 Global Step: 244775 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:55:11,002-Speed 131.11 samples/sec Loss 0.2662 Epoch: 24 Global Step: 244780 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:55:30,561-Speed 130.90 samples/sec Loss 0.3383 Epoch: 24 Global Step: 244785 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:55:50,360-Speed 129.30 samples/sec Loss 0.3352 Epoch: 24 Global Step: 244790 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:56:09,969-Speed 130.56 samples/sec Loss 0.2665 Epoch: 24 Global Step: 244795 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:56:29,599-Speed 130.41 samples/sec Loss 0.3116 Epoch: 24 Global Step: 244800 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:56:49,219-Speed 130.49 samples/sec Loss 0.2457 Epoch: 24 Global Step: 244805 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:57:08,946-Speed 129.78 samples/sec Loss 0.2444 Epoch: 24 Global Step: 244810 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:57:28,893-Speed 128.35 samples/sec Loss 0.2480 Epoch: 24 Global Step: 244815 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:57:48,656-Speed 129.54 samples/sec Loss 0.2783 Epoch: 24 Global Step: 244820 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:58:08,326-Speed 130.16 samples/sec Loss 0.2666 Epoch: 24 Global Step: 244825 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:58:28,184-Speed 128.92 samples/sec Loss 0.3126 Epoch: 24 Global Step: 244830 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:58:47,993-Speed 129.24 samples/sec Loss 0.3270 Epoch: 24 Global Step: 244835 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:59:07,567-Speed 130.80 samples/sec Loss 0.2811 Epoch: 24 Global Step: 244840 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:59:27,343-Speed 129.46 samples/sec Loss 0.3182 Epoch: 24 Global Step: 244845 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-18 23:59:46,956-Speed 130.53 samples/sec Loss 0.3512 Epoch: 24 Global Step: 244850 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:00:06,705-Speed 129.63 samples/sec Loss 0.2980 Epoch: 24 Global Step: 244855 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:00:26,508-Speed 129.27 samples/sec Loss 0.3258 Epoch: 24 Global Step: 244860 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:00:46,361-Speed 128.96 samples/sec Loss 0.2659 Epoch: 24 Global Step: 244865 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:01:06,027-Speed 130.18 samples/sec Loss 0.3360 Epoch: 24 Global Step: 244870 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:01:25,842-Speed 129.20 samples/sec Loss 0.2758 Epoch: 24 Global Step: 244875 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:01:45,716-Speed 128.82 samples/sec Loss 0.2702 Epoch: 24 Global Step: 244880 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:02:05,309-Speed 130.66 samples/sec Loss 0.2792 Epoch: 24 Global Step: 244885 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:02:25,090-Speed 129.43 samples/sec Loss 0.2629 Epoch: 24 Global Step: 244890 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:02:44,682-Speed 130.68 samples/sec Loss 0.3043 Epoch: 24 Global Step: 244895 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:03:04,449-Speed 129.52 samples/sec Loss 0.2462 Epoch: 24 Global Step: 244900 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:03:24,232-Speed 129.41 samples/sec Loss 0.2954 Epoch: 24 Global Step: 244905 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:03:44,048-Speed 129.19 samples/sec Loss 0.3082 Epoch: 24 Global Step: 244910 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:04:03,667-Speed 130.49 samples/sec Loss 0.3443 Epoch: 24 Global Step: 244915 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:04:23,518-Speed 128.97 samples/sec Loss 0.4364 Epoch: 24 Global Step: 244920 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:04:43,411-Speed 128.70 samples/sec Loss 0.2897 Epoch: 24 Global Step: 244925 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:05:02,997-Speed 130.71 samples/sec Loss 0.3111 Epoch: 24 Global Step: 244930 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:05:22,720-Speed 129.81 samples/sec Loss 0.2526 Epoch: 24 Global Step: 244935 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:05:42,316-Speed 130.64 samples/sec Loss 0.2912 Epoch: 24 Global Step: 244940 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:06:01,615-Speed 132.66 samples/sec Loss 0.3931 Epoch: 24 Global Step: 244945 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:06:21,450-Speed 129.07 samples/sec Loss 0.3275 Epoch: 24 Global Step: 244950 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:06:41,523-Speed 127.54 samples/sec Loss 0.2826 Epoch: 24 Global Step: 244955 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:07:01,112-Speed 130.69 samples/sec Loss 0.3580 Epoch: 24 Global Step: 244960 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:07:20,884-Speed 129.48 samples/sec Loss 0.2853 Epoch: 24 Global Step: 244965 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:07:40,723-Speed 129.05 samples/sec Loss 0.3718 Epoch: 24 Global Step: 244970 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:08:00,341-Speed 130.50 samples/sec Loss 0.2439 Epoch: 24 Global Step: 244975 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:08:20,150-Speed 129.24 samples/sec Loss 0.2657 Epoch: 24 Global Step: 244980 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:08:39,715-Speed 130.86 samples/sec Loss 0.3301 Epoch: 24 Global Step: 244985 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:08:59,346-Speed 130.41 samples/sec Loss 0.2801 Epoch: 24 Global Step: 244990 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:09:19,170-Speed 129.15 samples/sec Loss 0.3766 Epoch: 24 Global Step: 244995 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:09:39,134-Speed 128.24 samples/sec Loss 0.3131 Epoch: 24 Global Step: 245000 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:09:58,656-Speed 131.14 samples/sec Loss 0.2758 Epoch: 24 Global Step: 245005 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:10:18,381-Speed 129.79 samples/sec Loss 0.2718 Epoch: 24 Global Step: 245010 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:10:38,354-Speed 128.18 samples/sec Loss 0.2797 Epoch: 24 Global Step: 245015 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:10:57,804-Speed 131.63 samples/sec Loss 0.3742 Epoch: 24 Global Step: 245020 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:11:17,543-Speed 129.70 samples/sec Loss 0.2959 Epoch: 24 Global Step: 245025 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:11:37,116-Speed 130.80 samples/sec Loss 0.2328 Epoch: 24 Global Step: 245030 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:11:56,762-Speed 130.31 samples/sec Loss 0.2942 Epoch: 24 Global Step: 245035 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:12:16,547-Speed 129.40 samples/sec Loss 0.3388 Epoch: 24 Global Step: 245040 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:12:36,288-Speed 129.69 samples/sec Loss 0.2896 Epoch: 24 Global Step: 245045 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:12:55,923-Speed 130.39 samples/sec Loss 0.2321 Epoch: 24 Global Step: 245050 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:13:15,634-Speed 129.88 samples/sec Loss 0.2912 Epoch: 24 Global Step: 245055 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:13:35,622-Speed 128.08 samples/sec Loss 0.2658 Epoch: 24 Global Step: 245060 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:13:55,203-Speed 130.75 samples/sec Loss 0.3460 Epoch: 24 Global Step: 245065 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:14:14,982-Speed 129.44 samples/sec Loss 0.3017 Epoch: 24 Global Step: 245070 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:14:34,549-Speed 130.84 samples/sec Loss 0.3493 Epoch: 24 Global Step: 245075 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:14:54,170-Speed 130.48 samples/sec Loss 0.3255 Epoch: 24 Global Step: 245080 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:15:13,966-Speed 129.32 samples/sec Loss 0.3480 Epoch: 24 Global Step: 245085 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:15:33,775-Speed 129.24 samples/sec Loss 0.3266 Epoch: 24 Global Step: 245090 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:15:53,382-Speed 130.57 samples/sec Loss 0.2696 Epoch: 24 Global Step: 245095 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:16:13,237-Speed 128.94 samples/sec Loss 0.2620 Epoch: 24 Global Step: 245100 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:16:33,117-Speed 128.78 samples/sec Loss 0.2225 Epoch: 24 Global Step: 245105 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:16:52,752-Speed 130.39 samples/sec Loss 0.2915 Epoch: 24 Global Step: 245110 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:17:12,556-Speed 129.27 samples/sec Loss 0.3277 Epoch: 24 Global Step: 245115 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:17:32,150-Speed 130.66 samples/sec Loss 0.2321 Epoch: 24 Global Step: 245120 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:17:51,863-Speed 129.87 samples/sec Loss 0.2738 Epoch: 24 Global Step: 245125 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:18:11,663-Speed 129.31 samples/sec Loss 0.3094 Epoch: 24 Global Step: 245130 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:18:31,667-Speed 127.98 samples/sec Loss 0.3566 Epoch: 24 Global Step: 245135 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:18:51,116-Speed 131.63 samples/sec Loss 0.2578 Epoch: 24 Global Step: 245140 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:19:10,967-Speed 128.96 samples/sec Loss 0.2246 Epoch: 24 Global Step: 245145 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:19:30,897-Speed 128.46 samples/sec Loss 0.2919 Epoch: 24 Global Step: 245150 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:19:50,404-Speed 131.24 samples/sec Loss 0.2922 Epoch: 24 Global Step: 245155 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:20:10,314-Speed 128.58 samples/sec Loss 0.2615 Epoch: 24 Global Step: 245160 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:20:29,775-Speed 131.55 samples/sec Loss 0.2284 Epoch: 24 Global Step: 245165 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:20:49,567-Speed 129.36 samples/sec Loss 0.2648 Epoch: 24 Global Step: 245170 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:21:09,369-Speed 129.29 samples/sec Loss 0.2206 Epoch: 24 Global Step: 245175 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:21:29,248-Speed 128.78 samples/sec Loss 0.2966 Epoch: 24 Global Step: 245180 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:21:48,693-Speed 131.66 samples/sec Loss 0.2466 Epoch: 24 Global Step: 245185 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:22:08,575-Speed 128.77 samples/sec Loss 0.3078 Epoch: 24 Global Step: 245190 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:22:28,476-Speed 128.64 samples/sec Loss 0.3110 Epoch: 24 Global Step: 245195 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:22:48,011-Speed 131.05 samples/sec Loss 0.3194 Epoch: 24 Global Step: 245200 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:23:07,840-Speed 129.11 samples/sec Loss 0.3319 Epoch: 24 Global Step: 245205 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:23:27,386-Speed 130.98 samples/sec Loss 0.3202 Epoch: 24 Global Step: 245210 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:23:47,204-Speed 129.19 samples/sec Loss 0.3307 Epoch: 24 Global Step: 245215 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:24:06,961-Speed 129.58 samples/sec Loss 0.2692 Epoch: 24 Global Step: 245220 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:24:26,872-Speed 128.58 samples/sec Loss 0.3131 Epoch: 24 Global Step: 245225 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:24:46,312-Speed 131.69 samples/sec Loss 0.2636 Epoch: 24 Global Step: 245230 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:25:06,220-Speed 128.60 samples/sec Loss 0.3029 Epoch: 24 Global Step: 245235 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:25:26,113-Speed 128.69 samples/sec Loss 0.3037 Epoch: 24 Global Step: 245240 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:25:45,653-Speed 131.02 samples/sec Loss 0.2584 Epoch: 24 Global Step: 245245 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:26:05,481-Speed 129.12 samples/sec Loss 0.2690 Epoch: 24 Global Step: 245250 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:26:25,052-Speed 130.82 samples/sec Loss 0.2882 Epoch: 24 Global Step: 245255 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:26:45,018-Speed 128.22 samples/sec Loss 0.3261 Epoch: 24 Global Step: 245260 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:27:04,829-Speed 129.23 samples/sec Loss 0.3094 Epoch: 24 Global Step: 245265 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:27:24,648-Speed 129.18 samples/sec Loss 0.3133 Epoch: 24 Global Step: 245270 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:27:44,072-Speed 131.80 samples/sec Loss 0.3016 Epoch: 24 Global Step: 245275 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:28:04,134-Speed 127.61 samples/sec Loss 0.2690 Epoch: 24 Global Step: 245280 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:28:23,842-Speed 129.90 samples/sec Loss 0.2716 Epoch: 24 Global Step: 245285 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:28:43,349-Speed 131.24 samples/sec Loss 0.3181 Epoch: 24 Global Step: 245290 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:29:03,170-Speed 129.16 samples/sec Loss 0.3364 Epoch: 24 Global Step: 245295 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:29:22,695-Speed 131.12 samples/sec Loss 0.3048 Epoch: 24 Global Step: 245300 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:29:42,670-Speed 128.16 samples/sec Loss 0.3482 Epoch: 24 Global Step: 245305 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:30:02,431-Speed 129.55 samples/sec Loss 0.2534 Epoch: 24 Global Step: 245310 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:30:22,291-Speed 128.91 samples/sec Loss 0.3740 Epoch: 24 Global Step: 245315 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:30:41,748-Speed 131.58 samples/sec Loss 0.3380 Epoch: 24 Global Step: 245320 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:31:01,583-Speed 129.07 samples/sec Loss 0.2813 Epoch: 24 Global Step: 245325 Fp16 Grad Scale: 8192 Required: 5 hours Training: 2021-06-19 00:31:20,915-Speed 132.43 samples/sec Loss 0.3124 Epoch: 24 Global Step: 245330 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:31:40,816-Speed 128.64 samples/sec Loss 0.2726 Epoch: 24 Global Step: 245335 Fp16 Grad Scale: 4096 Required: 5 hours Training: 2021-06-19 00:32:00,326-Speed 131.22 samples/sec Loss 0.2748 Epoch: 24 Global Step: 245340 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:32:20,040-Speed 129.86 samples/sec Loss 0.3136 Epoch: 24 Global Step: 245345 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:32:39,925-Speed 128.74 samples/sec Loss 0.3107 Epoch: 24 Global Step: 245350 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:32:59,598-Speed 130.14 samples/sec Loss 0.3143 Epoch: 24 Global Step: 245355 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:33:19,476-Speed 128.80 samples/sec Loss 0.2901 Epoch: 24 Global Step: 245360 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:33:39,169-Speed 130.00 samples/sec Loss 0.2625 Epoch: 24 Global Step: 245365 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:33:59,372-Speed 126.72 samples/sec Loss 0.3080 Epoch: 24 Global Step: 245370 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:34:19,036-Speed 130.20 samples/sec Loss 0.1978 Epoch: 24 Global Step: 245375 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:34:39,068-Speed 127.80 samples/sec Loss 0.2711 Epoch: 24 Global Step: 245380 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:34:58,826-Speed 129.57 samples/sec Loss 0.3256 Epoch: 24 Global Step: 245385 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:35:18,802-Speed 128.16 samples/sec Loss 0.2693 Epoch: 24 Global Step: 245390 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:35:38,531-Speed 129.77 samples/sec Loss 0.3039 Epoch: 24 Global Step: 245395 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:35:58,224-Speed 130.00 samples/sec Loss 0.2960 Epoch: 24 Global Step: 245400 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:36:18,133-Speed 128.59 samples/sec Loss 0.3011 Epoch: 24 Global Step: 245405 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:36:37,866-Speed 129.74 samples/sec Loss 0.2503 Epoch: 24 Global Step: 245410 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:36:58,162-Speed 126.14 samples/sec Loss 0.3187 Epoch: 24 Global Step: 245415 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:37:17,904-Speed 129.68 samples/sec Loss 0.2929 Epoch: 24 Global Step: 245420 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:37:37,907-Speed 127.99 samples/sec Loss 0.2662 Epoch: 24 Global Step: 245425 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:37:57,658-Speed 129.61 samples/sec Loss 0.2920 Epoch: 24 Global Step: 245430 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:38:18,165-Speed 124.84 samples/sec Loss 0.2733 Epoch: 24 Global Step: 245435 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:38:37,870-Speed 129.92 samples/sec Loss 0.3081 Epoch: 24 Global Step: 245440 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:38:57,767-Speed 128.67 samples/sec Loss 0.2994 Epoch: 24 Global Step: 245445 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:39:17,490-Speed 129.81 samples/sec Loss 0.3523 Epoch: 24 Global Step: 245450 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:39:37,449-Speed 128.27 samples/sec Loss 0.3026 Epoch: 24 Global Step: 245455 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:39:56,809-Speed 132.24 samples/sec Loss 0.3091 Epoch: 24 Global Step: 245460 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:40:16,481-Speed 130.14 samples/sec Loss 0.3120 Epoch: 24 Global Step: 245465 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:40:36,882-Speed 125.49 samples/sec Loss 0.2804 Epoch: 24 Global Step: 245470 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:40:56,661-Speed 129.44 samples/sec Loss 0.2984 Epoch: 24 Global Step: 245475 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:41:16,221-Speed 130.89 samples/sec Loss 0.3123 Epoch: 24 Global Step: 245480 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:41:35,890-Speed 130.16 samples/sec Loss 0.3083 Epoch: 24 Global Step: 245485 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:41:55,551-Speed 130.21 samples/sec Loss 0.2831 Epoch: 24 Global Step: 245490 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:42:15,137-Speed 130.72 samples/sec Loss 0.3060 Epoch: 24 Global Step: 245495 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:42:35,013-Speed 128.80 samples/sec Loss 0.3179 Epoch: 24 Global Step: 245500 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:42:54,598-Speed 130.72 samples/sec Loss 0.2961 Epoch: 24 Global Step: 245505 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:43:14,333-Speed 129.73 samples/sec Loss 0.2440 Epoch: 24 Global Step: 245510 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:43:34,139-Speed 129.26 samples/sec Loss 0.2931 Epoch: 24 Global Step: 245515 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:43:54,066-Speed 128.47 samples/sec Loss 0.2707 Epoch: 24 Global Step: 245520 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:44:13,520-Speed 131.60 samples/sec Loss 0.3167 Epoch: 24 Global Step: 245525 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:44:33,286-Speed 129.52 samples/sec Loss 0.2987 Epoch: 24 Global Step: 245530 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:44:53,173-Speed 128.73 samples/sec Loss 0.2984 Epoch: 24 Global Step: 245535 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:45:12,780-Speed 130.57 samples/sec Loss 0.3206 Epoch: 24 Global Step: 245540 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:45:32,623-Speed 129.02 samples/sec Loss 0.3020 Epoch: 24 Global Step: 245545 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:45:52,186-Speed 130.87 samples/sec Loss 0.2857 Epoch: 24 Global Step: 245550 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:46:11,808-Speed 130.47 samples/sec Loss 0.2895 Epoch: 24 Global Step: 245555 Fp16 Grad Scale: 4096 Required: 4 hours Training: 2021-06-19 00:46:31,622-Speed 129.21 samples/sec Loss 0.2598 Epoch: 24 Global Step: 245560 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:46:51,632-Speed 127.94 samples/sec Loss 0.2311 Epoch: 24 Global Step: 245565 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:47:11,065-Speed 131.74 samples/sec Loss 0.2703 Epoch: 24 Global Step: 245570 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:47:30,839-Speed 129.47 samples/sec Loss 0.2937 Epoch: 24 Global Step: 245575 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:47:50,757-Speed 128.53 samples/sec Loss 0.3071 Epoch: 24 Global Step: 245580 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:48:10,339-Speed 130.74 samples/sec Loss 0.2724 Epoch: 24 Global Step: 245585 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:48:30,258-Speed 128.53 samples/sec Loss 0.3515 Epoch: 24 Global Step: 245590 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:48:49,787-Speed 131.09 samples/sec Loss 0.2700 Epoch: 24 Global Step: 245595 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:49:09,634-Speed 128.99 samples/sec Loss 0.2607 Epoch: 24 Global Step: 245600 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:49:29,484-Speed 128.97 samples/sec Loss 0.3474 Epoch: 24 Global Step: 245605 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:49:49,523-Speed 127.76 samples/sec Loss 0.2533 Epoch: 24 Global Step: 245610 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:50:08,967-Speed 131.67 samples/sec Loss 0.3113 Epoch: 24 Global Step: 245615 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:50:28,741-Speed 129.47 samples/sec Loss 0.2869 Epoch: 24 Global Step: 245620 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:50:48,651-Speed 128.58 samples/sec Loss 0.2462 Epoch: 24 Global Step: 245625 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:51:08,254-Speed 130.60 samples/sec Loss 0.3034 Epoch: 24 Global Step: 245630 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:51:28,119-Speed 128.88 samples/sec Loss 0.3309 Epoch: 24 Global Step: 245635 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:51:47,700-Speed 130.75 samples/sec Loss 0.2996 Epoch: 24 Global Step: 245640 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:52:07,475-Speed 129.47 samples/sec Loss 0.2836 Epoch: 24 Global Step: 245645 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:52:27,274-Speed 129.31 samples/sec Loss 0.2998 Epoch: 24 Global Step: 245650 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:52:47,276-Speed 127.99 samples/sec Loss 0.2641 Epoch: 24 Global Step: 245655 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:53:06,711-Speed 131.73 samples/sec Loss 0.2375 Epoch: 24 Global Step: 245660 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:53:26,723-Speed 127.93 samples/sec Loss 0.2887 Epoch: 24 Global Step: 245665 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:53:46,591-Speed 128.86 samples/sec Loss 0.2580 Epoch: 24 Global Step: 245670 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:54:06,122-Speed 131.08 samples/sec Loss 0.2521 Epoch: 24 Global Step: 245675 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:54:25,985-Speed 128.89 samples/sec Loss 0.3119 Epoch: 24 Global Step: 245680 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:54:45,514-Speed 131.10 samples/sec Loss 0.2997 Epoch: 24 Global Step: 245685 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:55:05,271-Speed 129.58 samples/sec Loss 0.3643 Epoch: 24 Global Step: 245690 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:55:25,033-Speed 129.54 samples/sec Loss 0.2488 Epoch: 24 Global Step: 245695 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:55:45,024-Speed 128.06 samples/sec Loss 0.3203 Epoch: 24 Global Step: 245700 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:56:04,469-Speed 131.66 samples/sec Loss 0.3137 Epoch: 24 Global Step: 245705 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:56:24,294-Speed 129.13 samples/sec Loss 0.2423 Epoch: 24 Global Step: 245710 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:56:44,280-Speed 128.10 samples/sec Loss 0.2930 Epoch: 24 Global Step: 245715 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:57:03,891-Speed 130.54 samples/sec Loss 0.2895 Epoch: 24 Global Step: 245720 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:57:23,802-Speed 128.58 samples/sec Loss 0.2918 Epoch: 24 Global Step: 245725 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:57:43,371-Speed 130.83 samples/sec Loss 0.3031 Epoch: 24 Global Step: 245730 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:58:03,135-Speed 129.54 samples/sec Loss 0.2639 Epoch: 24 Global Step: 245735 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:58:22,896-Speed 129.55 samples/sec Loss 0.2987 Epoch: 24 Global Step: 245740 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:58:42,922-Speed 127.84 samples/sec Loss 0.3230 Epoch: 24 Global Step: 245745 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:59:02,351-Speed 131.77 samples/sec Loss 0.2767 Epoch: 24 Global Step: 245750 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:59:22,117-Speed 129.52 samples/sec Loss 0.3101 Epoch: 24 Global Step: 245755 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 00:59:41,971-Speed 128.95 samples/sec Loss 0.2205 Epoch: 24 Global Step: 245760 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:00:01,774-Speed 129.28 samples/sec Loss 0.3003 Epoch: 24 Global Step: 245765 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:00:21,692-Speed 128.53 samples/sec Loss 0.2707 Epoch: 24 Global Step: 245770 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:00:41,170-Speed 131.44 samples/sec Loss 0.2599 Epoch: 24 Global Step: 245775 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:01:00,924-Speed 129.60 samples/sec Loss 0.2453 Epoch: 24 Global Step: 245780 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:01:20,701-Speed 129.45 samples/sec Loss 0.2547 Epoch: 24 Global Step: 245785 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:01:40,701-Speed 128.01 samples/sec Loss 0.2999 Epoch: 24 Global Step: 245790 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:02:00,130-Speed 131.77 samples/sec Loss 0.3104 Epoch: 24 Global Step: 245795 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:02:19,937-Speed 129.25 samples/sec Loss 0.2760 Epoch: 24 Global Step: 245800 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:02:39,790-Speed 128.95 samples/sec Loss 0.2378 Epoch: 24 Global Step: 245805 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:02:59,527-Speed 129.71 samples/sec Loss 0.2522 Epoch: 24 Global Step: 245810 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:03:19,409-Speed 128.77 samples/sec Loss 0.3060 Epoch: 24 Global Step: 245815 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:03:38,968-Speed 130.89 samples/sec Loss 0.3973 Epoch: 24 Global Step: 245820 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:03:58,657-Speed 130.03 samples/sec Loss 0.3242 Epoch: 24 Global Step: 245825 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:04:18,427-Speed 129.50 samples/sec Loss 0.2449 Epoch: 24 Global Step: 245830 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:04:38,447-Speed 127.88 samples/sec Loss 0.3150 Epoch: 24 Global Step: 245835 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:04:57,884-Speed 131.71 samples/sec Loss 0.3290 Epoch: 24 Global Step: 245840 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:05:17,795-Speed 128.58 samples/sec Loss 0.2529 Epoch: 24 Global Step: 245845 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:05:37,623-Speed 129.12 samples/sec Loss 0.2887 Epoch: 24 Global Step: 245850 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:05:57,151-Speed 131.10 samples/sec Loss 0.3108 Epoch: 24 Global Step: 245855 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:06:17,053-Speed 128.64 samples/sec Loss 0.2809 Epoch: 24 Global Step: 245860 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:06:36,599-Speed 130.98 samples/sec Loss 0.2844 Epoch: 24 Global Step: 245865 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:06:56,389-Speed 129.37 samples/sec Loss 0.2921 Epoch: 24 Global Step: 245870 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:07:16,183-Speed 129.34 samples/sec Loss 0.3006 Epoch: 24 Global Step: 245875 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:07:36,113-Speed 128.45 samples/sec Loss 0.2486 Epoch: 24 Global Step: 245880 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:07:55,538-Speed 131.79 samples/sec Loss 0.2737 Epoch: 24 Global Step: 245885 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:08:15,428-Speed 128.72 samples/sec Loss 0.2992 Epoch: 24 Global Step: 245890 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:08:35,216-Speed 129.38 samples/sec Loss 0.3033 Epoch: 24 Global Step: 245895 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:08:54,760-Speed 131.00 samples/sec Loss 0.3324 Epoch: 24 Global Step: 245900 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:09:14,629-Speed 128.85 samples/sec Loss 0.2679 Epoch: 24 Global Step: 245905 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:09:34,200-Speed 130.82 samples/sec Loss 0.3090 Epoch: 24 Global Step: 245910 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:09:54,013-Speed 129.21 samples/sec Loss 0.3313 Epoch: 24 Global Step: 245915 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:10:13,796-Speed 129.42 samples/sec Loss 0.2395 Epoch: 24 Global Step: 245920 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:10:33,671-Speed 128.81 samples/sec Loss 0.3078 Epoch: 24 Global Step: 245925 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:10:53,143-Speed 131.48 samples/sec Loss 0.3073 Epoch: 24 Global Step: 245930 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:11:13,114-Speed 128.20 samples/sec Loss 0.4033 Epoch: 24 Global Step: 245935 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:11:32,842-Speed 129.77 samples/sec Loss 0.3148 Epoch: 24 Global Step: 245940 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:11:52,378-Speed 131.05 samples/sec Loss 0.2902 Epoch: 24 Global Step: 245945 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:12:12,182-Speed 129.27 samples/sec Loss 0.3639 Epoch: 24 Global Step: 245950 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:12:31,643-Speed 131.55 samples/sec Loss 0.2671 Epoch: 24 Global Step: 245955 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:12:51,449-Speed 129.26 samples/sec Loss 0.3354 Epoch: 24 Global Step: 245960 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:13:11,238-Speed 129.38 samples/sec Loss 0.3239 Epoch: 24 Global Step: 245965 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:13:31,194-Speed 128.28 samples/sec Loss 0.3642 Epoch: 24 Global Step: 245970 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:13:50,618-Speed 131.81 samples/sec Loss 0.2643 Epoch: 24 Global Step: 245975 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:14:10,515-Speed 128.67 samples/sec Loss 0.3089 Epoch: 24 Global Step: 245980 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:14:30,253-Speed 129.71 samples/sec Loss 0.2979 Epoch: 24 Global Step: 245985 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:14:49,780-Speed 131.11 samples/sec Loss 0.2570 Epoch: 24 Global Step: 245990 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:15:09,642-Speed 128.90 samples/sec Loss 0.2755 Epoch: 24 Global Step: 245995 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:15:29,114-Speed 131.48 samples/sec Loss 0.2658 Epoch: 24 Global Step: 246000 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:15:49,021-Speed 128.60 samples/sec Loss 0.2879 Epoch: 24 Global Step: 246005 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:16:08,774-Speed 129.60 samples/sec Loss 0.3096 Epoch: 24 Global Step: 246010 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:16:28,823-Speed 127.69 samples/sec Loss 0.3147 Epoch: 24 Global Step: 246015 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:16:48,246-Speed 131.81 samples/sec Loss 0.3161 Epoch: 24 Global Step: 246020 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:17:08,061-Speed 129.20 samples/sec Loss 0.3283 Epoch: 24 Global Step: 246025 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:17:27,812-Speed 129.62 samples/sec Loss 0.2671 Epoch: 24 Global Step: 246030 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:17:47,412-Speed 130.62 samples/sec Loss 0.3451 Epoch: 24 Global Step: 246035 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:18:07,537-Speed 127.21 samples/sec Loss 0.3138 Epoch: 24 Global Step: 246040 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:18:26,993-Speed 131.59 samples/sec Loss 0.3404 Epoch: 24 Global Step: 246045 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:18:46,756-Speed 129.54 samples/sec Loss 0.2736 Epoch: 24 Global Step: 246050 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:19:06,559-Speed 129.29 samples/sec Loss 0.2266 Epoch: 24 Global Step: 246055 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:19:26,508-Speed 128.33 samples/sec Loss 0.2690 Epoch: 24 Global Step: 246060 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:19:45,928-Speed 131.83 samples/sec Loss 0.2787 Epoch: 24 Global Step: 246065 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:20:05,843-Speed 128.55 samples/sec Loss 0.2791 Epoch: 24 Global Step: 246070 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:20:25,543-Speed 129.95 samples/sec Loss 0.2667 Epoch: 24 Global Step: 246075 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:20:45,160-Speed 130.51 samples/sec Loss 0.3216 Epoch: 24 Global Step: 246080 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:21:05,013-Speed 128.96 samples/sec Loss 0.3839 Epoch: 24 Global Step: 246085 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:21:24,559-Speed 130.98 samples/sec Loss 0.2645 Epoch: 24 Global Step: 246090 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:21:44,278-Speed 129.83 samples/sec Loss 0.3048 Epoch: 24 Global Step: 246095 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:22:03,986-Speed 129.90 samples/sec Loss 0.3534 Epoch: 24 Global Step: 246100 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:22:23,994-Speed 127.95 samples/sec Loss 0.3159 Epoch: 24 Global Step: 246105 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:22:43,413-Speed 131.83 samples/sec Loss 0.2589 Epoch: 24 Global Step: 246110 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:23:03,259-Speed 129.00 samples/sec Loss 0.2348 Epoch: 24 Global Step: 246115 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:23:23,031-Speed 129.48 samples/sec Loss 0.3137 Epoch: 24 Global Step: 246120 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:23:42,649-Speed 130.50 samples/sec Loss 0.3236 Epoch: 24 Global Step: 246125 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:24:02,513-Speed 128.89 samples/sec Loss 0.3264 Epoch: 24 Global Step: 246130 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:24:22,058-Speed 130.98 samples/sec Loss 0.2982 Epoch: 24 Global Step: 246135 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:24:42,020-Speed 128.25 samples/sec Loss 0.2809 Epoch: 24 Global Step: 246140 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:25:01,766-Speed 129.65 samples/sec Loss 0.3052 Epoch: 24 Global Step: 246145 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:25:21,695-Speed 128.46 samples/sec Loss 0.2312 Epoch: 24 Global Step: 246150 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:25:41,112-Speed 131.85 samples/sec Loss 0.2798 Epoch: 24 Global Step: 246155 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:26:01,021-Speed 128.59 samples/sec Loss 0.3708 Epoch: 24 Global Step: 246160 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:26:20,738-Speed 129.85 samples/sec Loss 0.2500 Epoch: 24 Global Step: 246165 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:26:40,289-Speed 130.95 samples/sec Loss 0.2666 Epoch: 24 Global Step: 246170 Fp16 Grad Scale: 8192 Required: 4 hours Training: 2021-06-19 01:27:00,280-Speed 128.07 samples/sec Loss 0.2902 Epoch: 24 Global Step: 246175 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:27:19,805-Speed 131.12 samples/sec Loss 0.2845 Epoch: 24 Global Step: 246180 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:27:39,607-Speed 129.29 samples/sec Loss 0.2638 Epoch: 24 Global Step: 246185 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:27:59,305-Speed 129.97 samples/sec Loss 0.3305 Epoch: 24 Global Step: 246190 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:28:19,207-Speed 128.63 samples/sec Loss 0.3819 Epoch: 24 Global Step: 246195 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:28:38,632-Speed 131.80 samples/sec Loss 0.2707 Epoch: 24 Global Step: 246200 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:28:58,576-Speed 128.37 samples/sec Loss 0.2384 Epoch: 24 Global Step: 246205 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:29:18,286-Speed 129.89 samples/sec Loss 0.3207 Epoch: 24 Global Step: 246210 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:29:37,880-Speed 130.66 samples/sec Loss 0.3530 Epoch: 24 Global Step: 246215 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:29:57,681-Speed 129.30 samples/sec Loss 0.3034 Epoch: 24 Global Step: 246220 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:30:17,151-Speed 131.49 samples/sec Loss 0.2782 Epoch: 24 Global Step: 246225 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:30:36,984-Speed 129.09 samples/sec Loss 0.2522 Epoch: 24 Global Step: 246230 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:30:56,690-Speed 129.92 samples/sec Loss 0.2989 Epoch: 24 Global Step: 246235 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:31:16,631-Speed 128.38 samples/sec Loss 0.3585 Epoch: 24 Global Step: 246240 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:31:36,057-Speed 131.79 samples/sec Loss 0.3247 Epoch: 24 Global Step: 246245 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:31:55,526-Speed 131.50 samples/sec Loss 0.3428 Epoch: 24 Global Step: 246250 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:32:15,243-Speed 129.84 samples/sec Loss 0.2967 Epoch: 24 Global Step: 246255 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:32:34,797-Speed 130.93 samples/sec Loss 0.3048 Epoch: 24 Global Step: 246260 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:32:54,622-Speed 129.14 samples/sec Loss 0.2617 Epoch: 24 Global Step: 246265 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:33:14,190-Speed 130.83 samples/sec Loss 0.2756 Epoch: 24 Global Step: 246270 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:33:34,101-Speed 128.58 samples/sec Loss 0.2539 Epoch: 24 Global Step: 246275 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:33:53,816-Speed 129.86 samples/sec Loss 0.2903 Epoch: 24 Global Step: 246280 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:34:13,740-Speed 128.49 samples/sec Loss 0.3083 Epoch: 24 Global Step: 246285 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:34:33,165-Speed 131.79 samples/sec Loss 0.2884 Epoch: 24 Global Step: 246290 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:34:53,084-Speed 128.52 samples/sec Loss 0.3140 Epoch: 24 Global Step: 246295 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:35:12,801-Speed 129.85 samples/sec Loss 0.3148 Epoch: 24 Global Step: 246300 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:35:32,408-Speed 130.58 samples/sec Loss 0.2938 Epoch: 24 Global Step: 246305 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:35:52,302-Speed 128.69 samples/sec Loss 0.2546 Epoch: 24 Global Step: 246310 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:36:11,841-Speed 131.02 samples/sec Loss 0.2720 Epoch: 24 Global Step: 246315 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:36:31,752-Speed 128.58 samples/sec Loss 0.2713 Epoch: 24 Global Step: 246320 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:36:51,482-Speed 129.76 samples/sec Loss 0.2851 Epoch: 24 Global Step: 246325 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:37:11,406-Speed 128.49 samples/sec Loss 0.2772 Epoch: 24 Global Step: 246330 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:37:30,850-Speed 131.67 samples/sec Loss 0.3111 Epoch: 24 Global Step: 246335 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:37:50,798-Speed 128.34 samples/sec Loss 0.3326 Epoch: 24 Global Step: 246340 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:38:10,847-Speed 127.69 samples/sec Loss 0.3073 Epoch: 24 Global Step: 246345 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:38:30,379-Speed 131.08 samples/sec Loss 0.2374 Epoch: 24 Global Step: 246350 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:38:50,185-Speed 129.26 samples/sec Loss 0.3363 Epoch: 24 Global Step: 246355 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:39:09,729-Speed 130.99 samples/sec Loss 0.2865 Epoch: 24 Global Step: 246360 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:39:29,632-Speed 128.63 samples/sec Loss 0.2658 Epoch: 24 Global Step: 246365 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:39:49,293-Speed 130.21 samples/sec Loss 0.2534 Epoch: 24 Global Step: 246370 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:40:09,234-Speed 128.38 samples/sec Loss 0.3368 Epoch: 24 Global Step: 246375 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:40:28,668-Speed 131.73 samples/sec Loss 0.2949 Epoch: 24 Global Step: 246380 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:40:48,606-Speed 128.40 samples/sec Loss 0.2872 Epoch: 24 Global Step: 246385 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:41:08,745-Speed 127.13 samples/sec Loss 0.2348 Epoch: 24 Global Step: 246390 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:41:28,342-Speed 130.64 samples/sec Loss 0.3284 Epoch: 24 Global Step: 246395 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:41:48,060-Speed 129.83 samples/sec Loss 0.2782 Epoch: 24 Global Step: 246400 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:42:07,154-Speed 134.08 samples/sec Loss 0.2799 Epoch: 24 Global Step: 246405 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:42:26,820-Speed 130.18 samples/sec Loss 0.2658 Epoch: 24 Global Step: 246410 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:42:46,556-Speed 129.71 samples/sec Loss 0.2590 Epoch: 24 Global Step: 246415 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:43:06,168-Speed 130.54 samples/sec Loss 0.3920 Epoch: 24 Global Step: 246420 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:43:26,024-Speed 128.93 samples/sec Loss 0.2476 Epoch: 24 Global Step: 246425 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:43:46,044-Speed 127.88 samples/sec Loss 0.2908 Epoch: 24 Global Step: 246430 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:44:05,496-Speed 131.61 samples/sec Loss 0.2570 Epoch: 24 Global Step: 246435 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:44:25,475-Speed 128.14 samples/sec Loss 0.2359 Epoch: 24 Global Step: 246440 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:44:44,940-Speed 131.52 samples/sec Loss 0.2528 Epoch: 24 Global Step: 246445 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:45:04,805-Speed 128.88 samples/sec Loss 0.2658 Epoch: 24 Global Step: 246450 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:45:24,389-Speed 130.73 samples/sec Loss 0.2533 Epoch: 24 Global Step: 246455 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:45:44,118-Speed 129.77 samples/sec Loss 0.2593 Epoch: 24 Global Step: 246460 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:46:03,729-Speed 130.54 samples/sec Loss 0.3506 Epoch: 24 Global Step: 246465 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:46:23,680-Speed 128.32 samples/sec Loss 0.3326 Epoch: 24 Global Step: 246470 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:46:43,591-Speed 128.58 samples/sec Loss 0.3163 Epoch: 24 Global Step: 246475 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:47:03,010-Speed 131.84 samples/sec Loss 0.2795 Epoch: 24 Global Step: 246480 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:47:22,795-Speed 129.40 samples/sec Loss 0.3013 Epoch: 24 Global Step: 246485 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:47:42,402-Speed 130.57 samples/sec Loss 0.2686 Epoch: 24 Global Step: 246490 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:48:02,141-Speed 129.70 samples/sec Loss 0.2724 Epoch: 24 Global Step: 246495 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:48:21,718-Speed 130.77 samples/sec Loss 0.3057 Epoch: 24 Global Step: 246500 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:48:41,350-Speed 130.41 samples/sec Loss 0.3123 Epoch: 24 Global Step: 246505 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:49:00,973-Speed 130.46 samples/sec Loss 0.2916 Epoch: 24 Global Step: 246510 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:49:20,923-Speed 128.32 samples/sec Loss 0.3485 Epoch: 24 Global Step: 246515 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:49:40,875-Speed 128.32 samples/sec Loss 0.3715 Epoch: 24 Global Step: 246520 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:50:00,386-Speed 131.22 samples/sec Loss 0.3067 Epoch: 24 Global Step: 246525 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:50:20,311-Speed 128.49 samples/sec Loss 0.3627 Epoch: 24 Global Step: 246530 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:50:39,870-Speed 130.90 samples/sec Loss 0.2710 Epoch: 24 Global Step: 246535 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:50:59,630-Speed 129.56 samples/sec Loss 0.2436 Epoch: 24 Global Step: 246540 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:51:19,218-Speed 130.70 samples/sec Loss 0.2007 Epoch: 24 Global Step: 246545 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:51:38,962-Speed 129.67 samples/sec Loss 0.3157 Epoch: 24 Global Step: 246550 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:51:58,569-Speed 130.57 samples/sec Loss 0.2952 Epoch: 24 Global Step: 246555 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:52:18,652-Speed 127.48 samples/sec Loss 0.2655 Epoch: 24 Global Step: 246560 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:52:38,566-Speed 128.56 samples/sec Loss 0.2834 Epoch: 24 Global Step: 246565 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:52:58,111-Speed 130.99 samples/sec Loss 0.2668 Epoch: 24 Global Step: 246570 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:53:18,160-Speed 127.69 samples/sec Loss 0.3315 Epoch: 24 Global Step: 246575 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:53:37,607-Speed 131.65 samples/sec Loss 0.3039 Epoch: 24 Global Step: 246580 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:53:57,336-Speed 129.77 samples/sec Loss 0.2872 Epoch: 24 Global Step: 246585 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:54:16,953-Speed 130.50 samples/sec Loss 0.3402 Epoch: 24 Global Step: 246590 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:54:36,679-Speed 129.79 samples/sec Loss 0.3071 Epoch: 24 Global Step: 246595 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:54:56,217-Speed 131.04 samples/sec Loss 0.3001 Epoch: 24 Global Step: 246600 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:55:16,432-Speed 126.64 samples/sec Loss 0.3759 Epoch: 24 Global Step: 246605 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:55:36,419-Speed 128.09 samples/sec Loss 0.2500 Epoch: 24 Global Step: 246610 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:55:55,944-Speed 131.12 samples/sec Loss 0.3040 Epoch: 24 Global Step: 246615 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:56:15,666-Speed 129.81 samples/sec Loss 0.3343 Epoch: 24 Global Step: 246620 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:56:35,238-Speed 130.80 samples/sec Loss 0.3328 Epoch: 24 Global Step: 246625 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:56:54,975-Speed 129.71 samples/sec Loss 0.2544 Epoch: 24 Global Step: 246630 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 01:57:14,143-Speed 133.56 samples/sec Loss 0.2836 Epoch: 24 Global Step: 246635 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:57:33,705-Speed 130.88 samples/sec Loss 0.3043 Epoch: 24 Global Step: 246640 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:57:53,494-Speed 129.37 samples/sec Loss 0.2958 Epoch: 24 Global Step: 246645 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:58:13,243-Speed 129.63 samples/sec Loss 0.2870 Epoch: 24 Global Step: 246650 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:58:33,151-Speed 128.60 samples/sec Loss 0.3428 Epoch: 24 Global Step: 246655 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:58:52,758-Speed 130.57 samples/sec Loss 0.3305 Epoch: 24 Global Step: 246660 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:59:12,510-Speed 129.61 samples/sec Loss 0.2869 Epoch: 24 Global Step: 246665 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:59:32,413-Speed 128.63 samples/sec Loss 0.3515 Epoch: 24 Global Step: 246670 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 01:59:52,126-Speed 129.87 samples/sec Loss 0.2508 Epoch: 24 Global Step: 246675 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:00:12,109-Speed 128.11 samples/sec Loss 0.3209 Epoch: 24 Global Step: 246680 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:00:31,842-Speed 129.74 samples/sec Loss 0.2845 Epoch: 24 Global Step: 246685 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:00:51,688-Speed 129.00 samples/sec Loss 0.3304 Epoch: 24 Global Step: 246690 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:01:11,721-Speed 127.80 samples/sec Loss 0.3040 Epoch: 24 Global Step: 246695 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:01:31,610-Speed 128.72 samples/sec Loss 0.3445 Epoch: 24 Global Step: 246700 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:01:51,222-Speed 130.54 samples/sec Loss 0.3018 Epoch: 24 Global Step: 246705 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:02:10,994-Speed 129.49 samples/sec Loss 0.3010 Epoch: 24 Global Step: 246710 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:02:30,948-Speed 128.30 samples/sec Loss 0.2334 Epoch: 24 Global Step: 246715 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:02:50,656-Speed 129.90 samples/sec Loss 0.2600 Epoch: 24 Global Step: 246720 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:03:10,654-Speed 128.02 samples/sec Loss 0.3333 Epoch: 24 Global Step: 246725 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:03:30,376-Speed 129.81 samples/sec Loss 0.2953 Epoch: 24 Global Step: 246730 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:03:50,205-Speed 129.11 samples/sec Loss 0.3145 Epoch: 24 Global Step: 246735 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:04:10,257-Speed 127.67 samples/sec Loss 0.3123 Epoch: 24 Global Step: 246740 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:04:30,086-Speed 129.11 samples/sec Loss 0.3149 Epoch: 24 Global Step: 246745 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:04:49,749-Speed 130.20 samples/sec Loss 0.2926 Epoch: 24 Global Step: 246750 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:05:09,522-Speed 129.48 samples/sec Loss 0.2627 Epoch: 24 Global Step: 246755 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:05:29,469-Speed 128.35 samples/sec Loss 0.2895 Epoch: 24 Global Step: 246760 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:05:49,199-Speed 129.75 samples/sec Loss 0.3012 Epoch: 24 Global Step: 246765 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:06:09,173-Speed 128.17 samples/sec Loss 0.2818 Epoch: 24 Global Step: 246770 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:06:28,900-Speed 129.78 samples/sec Loss 0.3277 Epoch: 24 Global Step: 246775 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:06:48,719-Speed 129.18 samples/sec Loss 0.2928 Epoch: 24 Global Step: 246780 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:07:08,747-Speed 127.83 samples/sec Loss 0.2659 Epoch: 24 Global Step: 246785 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:07:28,535-Speed 129.37 samples/sec Loss 0.2552 Epoch: 24 Global Step: 246790 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:07:48,172-Speed 130.37 samples/sec Loss 0.3249 Epoch: 24 Global Step: 246795 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:08:07,942-Speed 129.50 samples/sec Loss 0.2825 Epoch: 24 Global Step: 246800 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:08:27,819-Speed 128.80 samples/sec Loss 0.2940 Epoch: 24 Global Step: 246805 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:08:47,523-Speed 129.93 samples/sec Loss 0.2810 Epoch: 24 Global Step: 246810 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:09:07,496-Speed 128.18 samples/sec Loss 0.2857 Epoch: 24 Global Step: 246815 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:09:27,240-Speed 129.67 samples/sec Loss 0.3059 Epoch: 24 Global Step: 246820 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:09:47,884-Speed 124.01 samples/sec Loss 0.3248 Epoch: 24 Global Step: 246825 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:10:07,883-Speed 128.01 samples/sec Loss 0.2512 Epoch: 24 Global Step: 246830 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:10:27,666-Speed 129.41 samples/sec Loss 0.2872 Epoch: 24 Global Step: 246835 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:10:47,258-Speed 130.67 samples/sec Loss 0.2491 Epoch: 24 Global Step: 246840 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:11:07,193-Speed 128.42 samples/sec Loss 0.2785 Epoch: 24 Global Step: 246845 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:11:26,924-Speed 129.75 samples/sec Loss 0.2774 Epoch: 24 Global Step: 246850 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:11:46,674-Speed 129.62 samples/sec Loss 0.2824 Epoch: 24 Global Step: 246855 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:12:06,624-Speed 128.33 samples/sec Loss 0.2458 Epoch: 24 Global Step: 246860 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:12:26,313-Speed 130.03 samples/sec Loss 0.3254 Epoch: 24 Global Step: 246865 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:12:46,253-Speed 128.39 samples/sec Loss 0.3180 Epoch: 24 Global Step: 246870 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:13:06,150-Speed 128.67 samples/sec Loss 0.2684 Epoch: 24 Global Step: 246875 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:13:25,941-Speed 129.36 samples/sec Loss 0.2710 Epoch: 24 Global Step: 246880 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:13:45,559-Speed 130.49 samples/sec Loss 0.2961 Epoch: 24 Global Step: 246885 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:14:05,461-Speed 128.64 samples/sec Loss 0.2889 Epoch: 24 Global Step: 246890 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:14:25,263-Speed 129.29 samples/sec Loss 0.3352 Epoch: 24 Global Step: 246895 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:14:44,907-Speed 130.32 samples/sec Loss 0.2911 Epoch: 24 Global Step: 246900 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:15:04,926-Speed 127.89 samples/sec Loss 0.2981 Epoch: 24 Global Step: 246905 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:15:24,626-Speed 129.96 samples/sec Loss 0.3571 Epoch: 24 Global Step: 246910 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:15:44,454-Speed 129.11 samples/sec Loss 0.3614 Epoch: 24 Global Step: 246915 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:16:04,429-Speed 128.17 samples/sec Loss 0.2369 Epoch: 24 Global Step: 246920 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:16:25,401-Speed 122.07 samples/sec Loss 0.3273 Epoch: 24 Global Step: 246925 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:16:45,149-Speed 129.64 samples/sec Loss 0.3580 Epoch: 24 Global Step: 246930 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:17:05,040-Speed 128.71 samples/sec Loss 0.3619 Epoch: 24 Global Step: 246935 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:17:24,921-Speed 128.77 samples/sec Loss 0.2854 Epoch: 24 Global Step: 246940 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:17:44,572-Speed 130.28 samples/sec Loss 0.3001 Epoch: 24 Global Step: 246945 Fp16 Grad Scale: 8192 Required: 3 hours Training: 2021-06-19 02:18:04,204-Speed 130.40 samples/sec Loss 0.3128 Epoch: 24 Global Step: 246950 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:18:23,913-Speed 129.90 samples/sec Loss 0.3049 Epoch: 24 Global Step: 246955 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:18:43,830-Speed 128.54 samples/sec Loss 0.2918 Epoch: 24 Global Step: 246960 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:19:03,719-Speed 128.73 samples/sec Loss 0.2548 Epoch: 24 Global Step: 246965 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:19:23,507-Speed 129.38 samples/sec Loss 0.2904 Epoch: 24 Global Step: 246970 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:19:43,200-Speed 130.00 samples/sec Loss 0.3188 Epoch: 24 Global Step: 246975 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:20:03,102-Speed 128.63 samples/sec Loss 0.2921 Epoch: 24 Global Step: 246980 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:20:22,890-Speed 129.38 samples/sec Loss 0.2910 Epoch: 24 Global Step: 246985 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:20:42,538-Speed 130.30 samples/sec Loss 0.3000 Epoch: 24 Global Step: 246990 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:21:02,546-Speed 127.95 samples/sec Loss 0.2639 Epoch: 24 Global Step: 246995 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:21:22,262-Speed 129.85 samples/sec Loss 0.2890 Epoch: 24 Global Step: 247000 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:21:42,250-Speed 128.09 samples/sec Loss 0.2522 Epoch: 24 Global Step: 247005 Fp16 Grad Scale: 4096 Required: 3 hours Training: 2021-06-19 02:22:02,228-Speed 128.15 samples/sec Loss 0.3287 Epoch: 24 Global Step: 247010 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:22:21,842-Speed 130.52 samples/sec Loss 0.3515 Epoch: 24 Global Step: 247015 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:22:41,532-Speed 130.02 samples/sec Loss 0.3088 Epoch: 24 Global Step: 247020 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:23:01,398-Speed 128.87 samples/sec Loss 0.2747 Epoch: 24 Global Step: 247025 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:23:21,192-Speed 129.34 samples/sec Loss 0.2638 Epoch: 24 Global Step: 247030 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:23:41,021-Speed 129.11 samples/sec Loss 0.3289 Epoch: 24 Global Step: 247035 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:24:00,959-Speed 128.40 samples/sec Loss 0.2758 Epoch: 24 Global Step: 247040 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:24:20,669-Speed 129.89 samples/sec Loss 0.3858 Epoch: 24 Global Step: 247045 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:24:40,628-Speed 128.27 samples/sec Loss 0.2637 Epoch: 24 Global Step: 247050 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:25:00,537-Speed 128.59 samples/sec Loss 0.3064 Epoch: 24 Global Step: 247055 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:25:20,274-Speed 129.72 samples/sec Loss 0.3036 Epoch: 24 Global Step: 247060 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:25:39,965-Speed 130.01 samples/sec Loss 0.3228 Epoch: 24 Global Step: 247065 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:25:59,863-Speed 128.66 samples/sec Loss 0.2971 Epoch: 24 Global Step: 247070 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:26:19,583-Speed 129.82 samples/sec Loss 0.3263 Epoch: 24 Global Step: 247075 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:26:39,495-Speed 128.57 samples/sec Loss 0.3101 Epoch: 24 Global Step: 247080 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:26:59,511-Speed 127.90 samples/sec Loss 0.3198 Epoch: 24 Global Step: 247085 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:27:19,262-Speed 129.62 samples/sec Loss 0.2725 Epoch: 24 Global Step: 247090 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:27:39,154-Speed 128.70 samples/sec Loss 0.2826 Epoch: 24 Global Step: 247095 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:27:59,210-Speed 127.65 samples/sec Loss 0.2576 Epoch: 24 Global Step: 247100 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:28:18,937-Speed 129.78 samples/sec Loss 0.3144 Epoch: 24 Global Step: 247105 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:28:38,671-Speed 129.73 samples/sec Loss 0.2850 Epoch: 24 Global Step: 247110 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:28:58,524-Speed 128.95 samples/sec Loss 0.2765 Epoch: 24 Global Step: 247115 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:29:18,241-Speed 129.84 samples/sec Loss 0.2932 Epoch: 24 Global Step: 247120 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:29:38,118-Speed 128.80 samples/sec Loss 0.3246 Epoch: 24 Global Step: 247125 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:29:58,040-Speed 128.51 samples/sec Loss 0.2722 Epoch: 24 Global Step: 247130 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:30:17,694-Speed 130.26 samples/sec Loss 0.3637 Epoch: 24 Global Step: 247135 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:30:37,642-Speed 128.35 samples/sec Loss 0.3026 Epoch: 24 Global Step: 247140 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:30:57,648-Speed 127.97 samples/sec Loss 0.2804 Epoch: 24 Global Step: 247145 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:31:17,266-Speed 130.50 samples/sec Loss 0.2746 Epoch: 24 Global Step: 247150 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:31:37,155-Speed 128.72 samples/sec Loss 0.2806 Epoch: 24 Global Step: 247155 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:31:56,927-Speed 129.49 samples/sec Loss 0.2517 Epoch: 24 Global Step: 247160 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:32:16,729-Speed 129.28 samples/sec Loss 0.3462 Epoch: 24 Global Step: 247165 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:32:36,522-Speed 129.34 samples/sec Loss 0.3458 Epoch: 24 Global Step: 247170 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:32:56,515-Speed 128.05 samples/sec Loss 0.3244 Epoch: 24 Global Step: 247175 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:33:16,199-Speed 130.06 samples/sec Loss 0.2491 Epoch: 24 Global Step: 247180 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:33:36,141-Speed 128.38 samples/sec Loss 0.2493 Epoch: 24 Global Step: 247185 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:33:56,129-Speed 128.08 samples/sec Loss 0.3392 Epoch: 24 Global Step: 247190 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:34:15,893-Speed 129.53 samples/sec Loss 0.3339 Epoch: 24 Global Step: 247195 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:34:35,650-Speed 129.58 samples/sec Loss 0.2933 Epoch: 24 Global Step: 247200 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:34:55,406-Speed 129.59 samples/sec Loss 0.2194 Epoch: 24 Global Step: 247205 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:35:15,209-Speed 129.28 samples/sec Loss 0.2768 Epoch: 24 Global Step: 247210 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:35:35,028-Speed 129.17 samples/sec Loss 0.2857 Epoch: 24 Global Step: 247215 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:35:55,041-Speed 127.92 samples/sec Loss 0.2580 Epoch: 24 Global Step: 247220 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:36:14,350-Speed 132.58 samples/sec Loss 0.2438 Epoch: 24 Global Step: 247225 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:36:34,162-Speed 129.23 samples/sec Loss 0.2543 Epoch: 24 Global Step: 247230 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:36:53,847-Speed 130.05 samples/sec Loss 0.3127 Epoch: 24 Global Step: 247235 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:37:13,693-Speed 129.00 samples/sec Loss 0.2997 Epoch: 24 Global Step: 247240 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:37:33,790-Speed 127.39 samples/sec Loss 0.3873 Epoch: 24 Global Step: 247245 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:37:53,414-Speed 130.45 samples/sec Loss 0.3409 Epoch: 24 Global Step: 247250 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:38:13,289-Speed 128.81 samples/sec Loss 0.2854 Epoch: 24 Global Step: 247255 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:38:33,046-Speed 129.58 samples/sec Loss 0.2331 Epoch: 24 Global Step: 247260 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:38:53,011-Speed 128.24 samples/sec Loss 0.3580 Epoch: 24 Global Step: 247265 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:39:12,804-Speed 129.34 samples/sec Loss 0.2027 Epoch: 24 Global Step: 247270 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:39:32,593-Speed 129.37 samples/sec Loss 0.3389 Epoch: 24 Global Step: 247275 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:39:52,529-Speed 128.42 samples/sec Loss 0.3175 Epoch: 24 Global Step: 247280 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:40:12,352-Speed 129.15 samples/sec Loss 0.2621 Epoch: 24 Global Step: 247285 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:40:32,391-Speed 127.75 samples/sec Loss 0.3036 Epoch: 24 Global Step: 247290 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:40:52,002-Speed 130.55 samples/sec Loss 0.3248 Epoch: 24 Global Step: 247295 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:41:11,823-Speed 129.16 samples/sec Loss 0.2599 Epoch: 24 Global Step: 247300 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:41:31,679-Speed 128.93 samples/sec Loss 0.3228 Epoch: 24 Global Step: 247305 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:41:51,591-Speed 128.57 samples/sec Loss 0.3129 Epoch: 24 Global Step: 247310 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:42:11,392-Speed 129.30 samples/sec Loss 0.3121 Epoch: 24 Global Step: 247315 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:42:31,172-Speed 129.43 samples/sec Loss 0.3180 Epoch: 24 Global Step: 247320 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 02:42:50,932-Speed 129.56 samples/sec Loss 0.2858 Epoch: 24 Global Step: 247325 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:43:10,800-Speed 128.86 samples/sec Loss 0.3043 Epoch: 24 Global Step: 247330 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:43:30,843-Speed 127.73 samples/sec Loss 0.2720 Epoch: 24 Global Step: 247335 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:43:50,519-Speed 130.12 samples/sec Loss 0.3434 Epoch: 24 Global Step: 247340 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:44:10,343-Speed 129.14 samples/sec Loss 0.2795 Epoch: 24 Global Step: 247345 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:44:30,209-Speed 128.87 samples/sec Loss 0.2823 Epoch: 24 Global Step: 247350 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:44:50,142-Speed 128.43 samples/sec Loss 0.2939 Epoch: 24 Global Step: 247355 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:45:09,936-Speed 129.34 samples/sec Loss 0.2560 Epoch: 24 Global Step: 247360 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:45:29,728-Speed 129.35 samples/sec Loss 0.3181 Epoch: 24 Global Step: 247365 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:45:49,475-Speed 129.65 samples/sec Loss 0.2992 Epoch: 24 Global Step: 247370 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:46:09,341-Speed 128.87 samples/sec Loss 0.2929 Epoch: 24 Global Step: 247375 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:46:29,349-Speed 127.96 samples/sec Loss 0.2683 Epoch: 24 Global Step: 247380 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:46:48,975-Speed 130.44 samples/sec Loss 0.2626 Epoch: 24 Global Step: 247385 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:47:08,814-Speed 129.05 samples/sec Loss 0.3324 Epoch: 24 Global Step: 247390 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:47:28,684-Speed 128.84 samples/sec Loss 0.3482 Epoch: 24 Global Step: 247395 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:47:48,591-Speed 128.61 samples/sec Loss 0.3199 Epoch: 24 Global Step: 247400 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:48:08,407-Speed 129.19 samples/sec Loss 0.2512 Epoch: 24 Global Step: 247405 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:48:28,198-Speed 129.36 samples/sec Loss 0.3059 Epoch: 24 Global Step: 247410 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:48:47,980-Speed 129.42 samples/sec Loss 0.3373 Epoch: 24 Global Step: 247415 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:49:08,020-Speed 127.75 samples/sec Loss 0.3521 Epoch: 24 Global Step: 247420 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:49:27,980-Speed 128.27 samples/sec Loss 0.2646 Epoch: 24 Global Step: 247425 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:49:47,599-Speed 130.49 samples/sec Loss 0.2960 Epoch: 24 Global Step: 247430 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:50:07,434-Speed 129.07 samples/sec Loss 0.2757 Epoch: 24 Global Step: 247435 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:50:27,299-Speed 128.88 samples/sec Loss 0.2932 Epoch: 24 Global Step: 247440 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:50:47,231-Speed 128.44 samples/sec Loss 0.2779 Epoch: 24 Global Step: 247445 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:51:07,387-Speed 127.02 samples/sec Loss 0.3107 Epoch: 24 Global Step: 247450 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:51:27,185-Speed 129.32 samples/sec Loss 0.3504 Epoch: 24 Global Step: 247455 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:51:47,091-Speed 128.61 samples/sec Loss 0.2945 Epoch: 24 Global Step: 247460 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:52:06,833-Speed 129.68 samples/sec Loss 0.2971 Epoch: 24 Global Step: 247465 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:52:26,813-Speed 128.14 samples/sec Loss 0.2823 Epoch: 24 Global Step: 247470 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:52:46,426-Speed 130.53 samples/sec Loss 0.3138 Epoch: 24 Global Step: 247475 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:53:06,422-Speed 128.03 samples/sec Loss 0.3499 Epoch: 24 Global Step: 247480 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:53:26,136-Speed 129.87 samples/sec Loss 0.3095 Epoch: 24 Global Step: 247485 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:53:46,095-Speed 128.27 samples/sec Loss 0.3210 Epoch: 24 Global Step: 247490 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:54:05,928-Speed 129.09 samples/sec Loss 0.3305 Epoch: 24 Global Step: 247495 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:54:25,877-Speed 128.33 samples/sec Loss 0.2960 Epoch: 24 Global Step: 247500 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:54:45,701-Speed 129.14 samples/sec Loss 0.3101 Epoch: 24 Global Step: 247505 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:55:05,443-Speed 129.68 samples/sec Loss 0.2206 Epoch: 24 Global Step: 247510 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:55:25,405-Speed 128.25 samples/sec Loss 0.3233 Epoch: 24 Global Step: 247515 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:55:45,011-Speed 130.58 samples/sec Loss 0.3294 Epoch: 24 Global Step: 247520 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:56:04,992-Speed 128.12 samples/sec Loss 0.2784 Epoch: 24 Global Step: 247525 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:56:24,695-Speed 129.94 samples/sec Loss 0.2870 Epoch: 24 Global Step: 247530 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:56:44,630-Speed 128.43 samples/sec Loss 0.2832 Epoch: 24 Global Step: 247535 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:57:04,361-Speed 129.75 samples/sec Loss 0.2848 Epoch: 24 Global Step: 247540 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:57:24,198-Speed 129.06 samples/sec Loss 0.3229 Epoch: 24 Global Step: 247545 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:57:44,057-Speed 128.92 samples/sec Loss 0.2752 Epoch: 24 Global Step: 247550 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:58:03,675-Speed 130.50 samples/sec Loss 0.2573 Epoch: 24 Global Step: 247555 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:58:23,735-Speed 127.62 samples/sec Loss 0.2796 Epoch: 24 Global Step: 247560 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:58:43,411-Speed 130.11 samples/sec Loss 0.2643 Epoch: 24 Global Step: 247565 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:59:03,344-Speed 128.44 samples/sec Loss 0.3133 Epoch: 24 Global Step: 247570 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:59:23,020-Speed 130.11 samples/sec Loss 0.3838 Epoch: 24 Global Step: 247575 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 02:59:43,067-Speed 127.71 samples/sec Loss 0.2941 Epoch: 24 Global Step: 247580 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:00:02,775-Speed 129.90 samples/sec Loss 0.3101 Epoch: 24 Global Step: 247585 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:00:22,660-Speed 128.75 samples/sec Loss 0.2684 Epoch: 24 Global Step: 247590 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:00:42,407-Speed 129.65 samples/sec Loss 0.2894 Epoch: 24 Global Step: 247595 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:01:02,016-Speed 130.56 samples/sec Loss 0.3857 Epoch: 24 Global Step: 247600 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:01:22,111-Speed 127.40 samples/sec Loss 0.2586 Epoch: 24 Global Step: 247605 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:01:41,919-Speed 129.25 samples/sec Loss 0.2849 Epoch: 24 Global Step: 247610 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:02:01,819-Speed 128.65 samples/sec Loss 0.2778 Epoch: 24 Global Step: 247615 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:02:21,487-Speed 130.17 samples/sec Loss 0.2879 Epoch: 24 Global Step: 247620 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:02:41,548-Speed 127.62 samples/sec Loss 0.3221 Epoch: 24 Global Step: 247625 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:03:01,263-Speed 129.86 samples/sec Loss 0.2714 Epoch: 24 Global Step: 247630 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:03:21,226-Speed 128.24 samples/sec Loss 0.2653 Epoch: 24 Global Step: 247635 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:03:40,833-Speed 130.57 samples/sec Loss 0.2963 Epoch: 24 Global Step: 247640 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:04:00,455-Speed 130.47 samples/sec Loss 0.2669 Epoch: 24 Global Step: 247645 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:04:20,554-Speed 127.38 samples/sec Loss 0.3684 Epoch: 24 Global Step: 247650 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:04:40,457-Speed 128.63 samples/sec Loss 0.2937 Epoch: 24 Global Step: 247655 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:05:00,249-Speed 129.35 samples/sec Loss 0.3370 Epoch: 24 Global Step: 247660 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:05:19,992-Speed 129.67 samples/sec Loss 0.2952 Epoch: 24 Global Step: 247665 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:05:40,042-Speed 127.69 samples/sec Loss 0.3770 Epoch: 24 Global Step: 247670 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:05:59,760-Speed 129.84 samples/sec Loss 0.2672 Epoch: 24 Global Step: 247675 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:06:19,591-Speed 129.10 samples/sec Loss 0.2812 Epoch: 24 Global Step: 247680 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:06:39,484-Speed 128.70 samples/sec Loss 0.2838 Epoch: 24 Global Step: 247685 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:06:59,139-Speed 130.25 samples/sec Loss 0.3212 Epoch: 24 Global Step: 247690 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:07:19,229-Speed 127.43 samples/sec Loss 0.2778 Epoch: 24 Global Step: 247695 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:07:39,058-Speed 129.11 samples/sec Loss 0.3139 Epoch: 24 Global Step: 247700 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:07:58,947-Speed 128.72 samples/sec Loss 0.3087 Epoch: 24 Global Step: 247705 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:08:18,667-Speed 129.82 samples/sec Loss 0.3139 Epoch: 24 Global Step: 247710 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:08:38,406-Speed 129.70 samples/sec Loss 0.2551 Epoch: 24 Global Step: 247715 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:08:57,827-Speed 131.82 samples/sec Loss 0.3521 Epoch: 24 Global Step: 247720 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:09:17,977-Speed 127.05 samples/sec Loss 0.2373 Epoch: 24 Global Step: 247725 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:09:37,811-Speed 129.08 samples/sec Loss 0.2518 Epoch: 24 Global Step: 247730 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:09:57,524-Speed 129.87 samples/sec Loss 0.3523 Epoch: 24 Global Step: 247735 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:10:17,601-Speed 127.52 samples/sec Loss 0.3509 Epoch: 24 Global Step: 247740 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:10:37,464-Speed 128.89 samples/sec Loss 0.2688 Epoch: 24 Global Step: 247745 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:10:57,194-Speed 129.76 samples/sec Loss 0.3045 Epoch: 24 Global Step: 247750 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:11:16,865-Speed 130.15 samples/sec Loss 0.2549 Epoch: 24 Global Step: 247755 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:11:36,725-Speed 128.91 samples/sec Loss 0.2988 Epoch: 24 Global Step: 247760 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:11:56,383-Speed 130.24 samples/sec Loss 0.2586 Epoch: 24 Global Step: 247765 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:12:16,336-Speed 128.31 samples/sec Loss 0.2824 Epoch: 24 Global Step: 247770 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:12:36,076-Speed 129.69 samples/sec Loss 0.2418 Epoch: 24 Global Step: 247775 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:12:55,697-Speed 130.48 samples/sec Loss 0.2841 Epoch: 24 Global Step: 247780 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:13:15,966-Speed 126.31 samples/sec Loss 0.3445 Epoch: 24 Global Step: 247785 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:13:35,952-Speed 128.09 samples/sec Loss 0.3249 Epoch: 24 Global Step: 247790 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:13:55,659-Speed 129.90 samples/sec Loss 0.3099 Epoch: 24 Global Step: 247795 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:14:15,310-Speed 130.28 samples/sec Loss 0.3022 Epoch: 24 Global Step: 247800 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:14:35,186-Speed 128.81 samples/sec Loss 0.3459 Epoch: 24 Global Step: 247805 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:14:54,910-Speed 129.80 samples/sec Loss 0.3384 Epoch: 24 Global Step: 247810 Fp16 Grad Scale: 4096 Required: 2 hours Training: 2021-06-19 03:15:14,808-Speed 128.66 samples/sec Loss 0.3045 Epoch: 24 Global Step: 247815 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:15:34,543-Speed 129.72 samples/sec Loss 0.2552 Epoch: 24 Global Step: 247820 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:15:54,209-Speed 130.18 samples/sec Loss 0.3131 Epoch: 24 Global Step: 247825 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:16:14,558-Speed 125.81 samples/sec Loss 0.2843 Epoch: 24 Global Step: 247830 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:16:34,707-Speed 127.06 samples/sec Loss 0.2875 Epoch: 24 Global Step: 247835 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:16:54,357-Speed 130.28 samples/sec Loss 0.3290 Epoch: 24 Global Step: 247840 Fp16 Grad Scale: 8192 Required: 2 hours Training: 2021-06-19 03:17:14,182-Speed 129.14 samples/sec Loss 0.2553 Epoch: 24 Global Step: 247845 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:17:34,196-Speed 127.92 samples/sec Loss 0.3414 Epoch: 24 Global Step: 247850 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:17:53,896-Speed 129.96 samples/sec Loss 0.2630 Epoch: 24 Global Step: 247855 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:18:13,847-Speed 128.32 samples/sec Loss 0.2690 Epoch: 24 Global Step: 247860 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:18:33,471-Speed 130.46 samples/sec Loss 0.2986 Epoch: 24 Global Step: 247865 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:18:53,203-Speed 129.74 samples/sec Loss 0.3188 Epoch: 24 Global Step: 247870 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:19:13,923-Speed 123.57 samples/sec Loss 0.2518 Epoch: 24 Global Step: 247875 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:19:33,934-Speed 127.94 samples/sec Loss 0.2311 Epoch: 24 Global Step: 247880 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:19:53,586-Speed 130.27 samples/sec Loss 0.3567 Epoch: 24 Global Step: 247885 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:20:13,374-Speed 129.38 samples/sec Loss 0.3260 Epoch: 24 Global Step: 247890 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:20:33,236-Speed 128.90 samples/sec Loss 0.3004 Epoch: 24 Global Step: 247895 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:20:53,416-Speed 126.86 samples/sec Loss 0.3613 Epoch: 24 Global Step: 247900 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:21:13,357-Speed 128.39 samples/sec Loss 0.2765 Epoch: 24 Global Step: 247905 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:21:32,991-Speed 130.39 samples/sec Loss 0.3412 Epoch: 24 Global Step: 247910 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:21:52,755-Speed 129.53 samples/sec Loss 0.2708 Epoch: 24 Global Step: 247915 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:22:13,094-Speed 125.87 samples/sec Loss 0.2386 Epoch: 24 Global Step: 247920 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:22:33,110-Speed 127.90 samples/sec Loss 0.2652 Epoch: 24 Global Step: 247925 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:22:52,989-Speed 128.79 samples/sec Loss 0.2682 Epoch: 24 Global Step: 247930 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:23:12,760-Speed 129.49 samples/sec Loss 0.2906 Epoch: 24 Global Step: 247935 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:23:32,595-Speed 129.07 samples/sec Loss 0.3202 Epoch: 24 Global Step: 247940 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:23:52,220-Speed 130.45 samples/sec Loss 0.2715 Epoch: 24 Global Step: 247945 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:24:12,117-Speed 128.67 samples/sec Loss 0.2836 Epoch: 24 Global Step: 247950 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:24:31,381-Speed 132.90 samples/sec Loss 0.3030 Epoch: 24 Global Step: 247955 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:24:51,087-Speed 129.92 samples/sec Loss 0.3403 Epoch: 24 Global Step: 247960 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:25:11,316-Speed 126.55 samples/sec Loss 0.3330 Epoch: 24 Global Step: 247965 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:25:31,284-Speed 128.21 samples/sec Loss 0.3240 Epoch: 24 Global Step: 247970 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:25:50,916-Speed 130.40 samples/sec Loss 0.3344 Epoch: 24 Global Step: 247975 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:26:11,032-Speed 127.27 samples/sec Loss 0.2977 Epoch: 24 Global Step: 247980 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:26:30,657-Speed 130.45 samples/sec Loss 0.2904 Epoch: 24 Global Step: 247985 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:26:50,374-Speed 129.85 samples/sec Loss 0.3300 Epoch: 24 Global Step: 247990 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:27:10,323-Speed 128.33 samples/sec Loss 0.3766 Epoch: 24 Global Step: 247995 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:27:30,093-Speed 129.50 samples/sec Loss 0.2451 Epoch: 24 Global Step: 248000 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:27:49,892-Speed 129.30 samples/sec Loss 0.2807 Epoch: 24 Global Step: 248005 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:28:10,088-Speed 126.77 samples/sec Loss 0.3187 Epoch: 24 Global Step: 248010 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:28:30,053-Speed 128.23 samples/sec Loss 0.3337 Epoch: 24 Global Step: 248015 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:28:49,654-Speed 130.61 samples/sec Loss 0.3063 Epoch: 24 Global Step: 248020 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:29:09,559-Speed 128.62 samples/sec Loss 0.2633 Epoch: 24 Global Step: 248025 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:29:29,179-Speed 130.48 samples/sec Loss 0.2962 Epoch: 24 Global Step: 248030 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:29:48,915-Speed 129.72 samples/sec Loss 0.3646 Epoch: 24 Global Step: 248035 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:30:08,811-Speed 128.68 samples/sec Loss 0.2923 Epoch: 24 Global Step: 248040 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:30:28,431-Speed 130.49 samples/sec Loss 0.3376 Epoch: 24 Global Step: 248045 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:30:48,341-Speed 128.58 samples/sec Loss 0.3389 Epoch: 24 Global Step: 248050 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:31:08,424-Speed 127.48 samples/sec Loss 0.3181 Epoch: 24 Global Step: 248055 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:31:28,389-Speed 128.23 samples/sec Loss 0.2937 Epoch: 24 Global Step: 248060 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:31:48,005-Speed 130.51 samples/sec Loss 0.2866 Epoch: 24 Global Step: 248065 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:32:07,778-Speed 129.48 samples/sec Loss 0.2841 Epoch: 24 Global Step: 248070 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:32:27,561-Speed 129.42 samples/sec Loss 0.3602 Epoch: 24 Global Step: 248075 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:32:46,785-Speed 133.17 samples/sec Loss 0.2697 Epoch: 24 Global Step: 248080 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:33:06,928-Speed 127.10 samples/sec Loss 0.3102 Epoch: 24 Global Step: 248085 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:33:26,533-Speed 130.59 samples/sec Loss 0.2675 Epoch: 24 Global Step: 248090 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:33:46,429-Speed 128.67 samples/sec Loss 0.2631 Epoch: 24 Global Step: 248095 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:34:06,564-Speed 127.15 samples/sec Loss 0.3161 Epoch: 24 Global Step: 248100 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:34:26,320-Speed 129.59 samples/sec Loss 0.2753 Epoch: 24 Global Step: 248105 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:34:46,295-Speed 128.17 samples/sec Loss 0.3277 Epoch: 24 Global Step: 248110 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:35:06,079-Speed 129.40 samples/sec Loss 0.2552 Epoch: 24 Global Step: 248115 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:35:25,899-Speed 129.17 samples/sec Loss 0.2616 Epoch: 24 Global Step: 248120 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:35:45,680-Speed 129.42 samples/sec Loss 0.2565 Epoch: 24 Global Step: 248125 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:36:05,496-Speed 129.20 samples/sec Loss 0.3821 Epoch: 24 Global Step: 248130 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:36:25,139-Speed 130.33 samples/sec Loss 0.2901 Epoch: 24 Global Step: 248135 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:36:44,823-Speed 130.06 samples/sec Loss 0.2636 Epoch: 24 Global Step: 248140 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:37:04,974-Speed 127.05 samples/sec Loss 0.3103 Epoch: 24 Global Step: 248145 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:37:24,777-Speed 129.28 samples/sec Loss 0.2906 Epoch: 24 Global Step: 248150 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:37:44,750-Speed 128.18 samples/sec Loss 0.3076 Epoch: 24 Global Step: 248155 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:38:04,614-Speed 128.88 samples/sec Loss 0.3942 Epoch: 24 Global Step: 248160 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:38:24,896-Speed 126.23 samples/sec Loss 0.3134 Epoch: 24 Global Step: 248165 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:38:44,582-Speed 130.05 samples/sec Loss 0.3048 Epoch: 24 Global Step: 248170 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:39:04,552-Speed 128.20 samples/sec Loss 0.3067 Epoch: 24 Global Step: 248175 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:39:24,269-Speed 129.84 samples/sec Loss 0.4178 Epoch: 24 Global Step: 248180 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:39:44,026-Speed 129.58 samples/sec Loss 0.3156 Epoch: 24 Global Step: 248185 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:40:04,161-Speed 127.15 samples/sec Loss 0.3615 Epoch: 24 Global Step: 248190 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:40:23,951-Speed 129.36 samples/sec Loss 0.3575 Epoch: 24 Global Step: 248195 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:40:44,208-Speed 126.39 samples/sec Loss 0.3093 Epoch: 24 Global Step: 248200 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:41:03,978-Speed 129.49 samples/sec Loss 0.2380 Epoch: 24 Global Step: 248205 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:41:23,868-Speed 128.71 samples/sec Loss 0.3525 Epoch: 24 Global Step: 248210 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:41:43,953-Speed 127.46 samples/sec Loss 0.2762 Epoch: 24 Global Step: 248215 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:42:03,940-Speed 128.09 samples/sec Loss 0.4326 Epoch: 24 Global Step: 248220 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:42:23,668-Speed 129.77 samples/sec Loss 0.2758 Epoch: 24 Global Step: 248225 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:42:43,421-Speed 129.60 samples/sec Loss 0.3455 Epoch: 24 Global Step: 248230 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:43:03,600-Speed 126.88 samples/sec Loss 0.3134 Epoch: 24 Global Step: 248235 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:43:23,390-Speed 129.36 samples/sec Loss 0.2479 Epoch: 24 Global Step: 248240 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:43:43,486-Speed 127.40 samples/sec Loss 0.3105 Epoch: 24 Global Step: 248245 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:44:03,304-Speed 129.18 samples/sec Loss 0.2659 Epoch: 24 Global Step: 248250 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:44:23,141-Speed 129.06 samples/sec Loss 0.2934 Epoch: 24 Global Step: 248255 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:44:42,852-Speed 129.89 samples/sec Loss 0.3195 Epoch: 24 Global Step: 248260 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:45:02,812-Speed 128.26 samples/sec Loss 0.3034 Epoch: 24 Global Step: 248265 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:45:22,533-Speed 129.82 samples/sec Loss 0.2924 Epoch: 24 Global Step: 248270 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:45:42,327-Speed 129.34 samples/sec Loss 0.2536 Epoch: 24 Global Step: 248275 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:46:02,500-Speed 126.91 samples/sec Loss 0.3630 Epoch: 24 Global Step: 248280 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:46:22,297-Speed 129.32 samples/sec Loss 0.3215 Epoch: 24 Global Step: 248285 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:46:41,653-Speed 132.26 samples/sec Loss 0.2781 Epoch: 24 Global Step: 248290 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:47:01,596-Speed 128.38 samples/sec Loss 0.3024 Epoch: 24 Global Step: 248295 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:47:21,773-Speed 126.88 samples/sec Loss 0.3371 Epoch: 24 Global Step: 248300 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:47:41,560-Speed 129.39 samples/sec Loss 0.3214 Epoch: 24 Global Step: 248305 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:48:01,169-Speed 130.56 samples/sec Loss 0.2512 Epoch: 24 Global Step: 248310 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:48:20,852-Speed 130.07 samples/sec Loss 0.3230 Epoch: 24 Global Step: 248315 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:48:40,540-Speed 130.04 samples/sec Loss 0.3527 Epoch: 24 Global Step: 248320 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:49:00,105-Speed 130.85 samples/sec Loss 0.2519 Epoch: 24 Global Step: 248325 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:49:19,901-Speed 129.32 samples/sec Loss 0.3361 Epoch: 24 Global Step: 248330 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:49:39,695-Speed 129.34 samples/sec Loss 0.2754 Epoch: 24 Global Step: 248335 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:49:59,513-Speed 129.18 samples/sec Loss 0.3409 Epoch: 24 Global Step: 248340 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:50:19,072-Speed 130.90 samples/sec Loss 0.2773 Epoch: 24 Global Step: 248345 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:50:38,960-Speed 128.72 samples/sec Loss 0.3100 Epoch: 24 Global Step: 248350 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:50:58,853-Speed 128.70 samples/sec Loss 0.3054 Epoch: 24 Global Step: 248355 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:51:18,334-Speed 131.41 samples/sec Loss 0.3289 Epoch: 24 Global Step: 248360 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:51:38,184-Speed 128.98 samples/sec Loss 0.2580 Epoch: 24 Global Step: 248365 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:51:57,772-Speed 130.70 samples/sec Loss 0.3371 Epoch: 24 Global Step: 248370 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:52:17,545-Speed 129.48 samples/sec Loss 0.3630 Epoch: 24 Global Step: 248375 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:52:37,346-Speed 129.29 samples/sec Loss 0.2750 Epoch: 24 Global Step: 248380 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:52:57,159-Speed 129.21 samples/sec Loss 0.3173 Epoch: 24 Global Step: 248385 Fp16 Grad Scale: 4096 Required: 1 hours Training: 2021-06-19 03:53:16,600-Speed 131.69 samples/sec Loss 0.4171 Epoch: 24 Global Step: 248390 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:53:36,430-Speed 129.10 samples/sec Loss 0.3266 Epoch: 24 Global Step: 248395 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:53:56,314-Speed 128.75 samples/sec Loss 0.3050 Epoch: 24 Global Step: 248400 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:54:15,741-Speed 131.79 samples/sec Loss 0.2828 Epoch: 24 Global Step: 248405 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:54:35,591-Speed 128.97 samples/sec Loss 0.3040 Epoch: 24 Global Step: 248410 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:54:55,197-Speed 130.58 samples/sec Loss 0.2967 Epoch: 24 Global Step: 248415 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:55:14,817-Speed 130.48 samples/sec Loss 0.2839 Epoch: 24 Global Step: 248420 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:55:34,632-Speed 129.21 samples/sec Loss 0.3135 Epoch: 24 Global Step: 248425 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:55:54,393-Speed 129.55 samples/sec Loss 0.3075 Epoch: 24 Global Step: 248430 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:56:14,026-Speed 130.40 samples/sec Loss 0.3051 Epoch: 24 Global Step: 248435 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:56:33,937-Speed 128.58 samples/sec Loss 0.3308 Epoch: 24 Global Step: 248440 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:56:53,796-Speed 128.91 samples/sec Loss 0.3307 Epoch: 24 Global Step: 248445 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:57:13,336-Speed 131.02 samples/sec Loss 0.3099 Epoch: 24 Global Step: 248450 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:57:33,097-Speed 129.55 samples/sec Loss 0.2950 Epoch: 24 Global Step: 248455 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:57:52,678-Speed 130.75 samples/sec Loss 0.2961 Epoch: 24 Global Step: 248460 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:58:12,413-Speed 129.72 samples/sec Loss 0.3237 Epoch: 24 Global Step: 248465 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:58:32,195-Speed 129.42 samples/sec Loss 0.2548 Epoch: 24 Global Step: 248470 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:58:51,968-Speed 129.47 samples/sec Loss 0.3268 Epoch: 24 Global Step: 248475 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:59:11,652-Speed 130.06 samples/sec Loss 0.2637 Epoch: 24 Global Step: 248480 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:59:31,399-Speed 129.65 samples/sec Loss 0.2619 Epoch: 24 Global Step: 248485 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 03:59:51,216-Speed 129.19 samples/sec Loss 0.3374 Epoch: 24 Global Step: 248490 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:00:10,700-Speed 131.40 samples/sec Loss 0.3206 Epoch: 24 Global Step: 248495 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:00:30,538-Speed 129.05 samples/sec Loss 0.2703 Epoch: 24 Global Step: 248500 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:00:50,116-Speed 130.77 samples/sec Loss 0.2868 Epoch: 24 Global Step: 248505 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:01:09,857-Speed 129.69 samples/sec Loss 0.2711 Epoch: 24 Global Step: 248510 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:01:29,649-Speed 129.35 samples/sec Loss 0.2914 Epoch: 24 Global Step: 248515 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:01:49,444-Speed 129.33 samples/sec Loss 0.3239 Epoch: 24 Global Step: 248520 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:02:09,100-Speed 130.25 samples/sec Loss 0.2915 Epoch: 24 Global Step: 248525 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:02:28,847-Speed 129.64 samples/sec Loss 0.2676 Epoch: 24 Global Step: 248530 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:02:48,710-Speed 128.89 samples/sec Loss 0.2814 Epoch: 24 Global Step: 248535 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:03:08,231-Speed 131.15 samples/sec Loss 0.2917 Epoch: 24 Global Step: 248540 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:03:28,102-Speed 128.84 samples/sec Loss 0.2847 Epoch: 24 Global Step: 248545 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:03:47,636-Speed 131.06 samples/sec Loss 0.2877 Epoch: 24 Global Step: 248550 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:04:07,357-Speed 129.82 samples/sec Loss 0.3160 Epoch: 24 Global Step: 248555 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:04:27,136-Speed 129.44 samples/sec Loss 0.3005 Epoch: 24 Global Step: 248560 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:04:47,068-Speed 128.44 samples/sec Loss 0.3085 Epoch: 24 Global Step: 248565 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:05:06,490-Speed 131.81 samples/sec Loss 0.3110 Epoch: 24 Global Step: 248570 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:05:26,273-Speed 129.41 samples/sec Loss 0.3253 Epoch: 24 Global Step: 248575 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:05:46,148-Speed 128.81 samples/sec Loss 0.3183 Epoch: 24 Global Step: 248580 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:06:05,587-Speed 131.71 samples/sec Loss 0.2701 Epoch: 24 Global Step: 248585 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:06:25,582-Speed 128.04 samples/sec Loss 0.3038 Epoch: 24 Global Step: 248590 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:06:45,035-Speed 131.60 samples/sec Loss 0.3767 Epoch: 24 Global Step: 248595 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:07:04,770-Speed 129.73 samples/sec Loss 0.2874 Epoch: 24 Global Step: 248600 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:07:24,562-Speed 129.36 samples/sec Loss 0.3133 Epoch: 24 Global Step: 248605 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:07:44,457-Speed 128.68 samples/sec Loss 0.3038 Epoch: 24 Global Step: 248610 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:08:03,892-Speed 131.73 samples/sec Loss 0.3837 Epoch: 24 Global Step: 248615 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:08:23,652-Speed 129.56 samples/sec Loss 0.2359 Epoch: 24 Global Step: 248620 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:08:43,526-Speed 128.82 samples/sec Loss 0.3341 Epoch: 24 Global Step: 248625 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:09:03,115-Speed 130.69 samples/sec Loss 0.3192 Epoch: 24 Global Step: 248630 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:09:23,032-Speed 128.54 samples/sec Loss 0.3410 Epoch: 24 Global Step: 248635 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:09:42,528-Speed 131.32 samples/sec Loss 0.3185 Epoch: 24 Global Step: 248640 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:10:02,301-Speed 129.48 samples/sec Loss 0.2884 Epoch: 24 Global Step: 248645 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:10:22,060-Speed 129.57 samples/sec Loss 0.2952 Epoch: 24 Global Step: 248650 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:10:42,048-Speed 128.08 samples/sec Loss 0.2675 Epoch: 24 Global Step: 248655 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:11:01,473-Speed 131.80 samples/sec Loss 0.2476 Epoch: 24 Global Step: 248660 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:11:21,335-Speed 128.89 samples/sec Loss 0.2755 Epoch: 24 Global Step: 248665 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:11:41,078-Speed 129.67 samples/sec Loss 0.2774 Epoch: 24 Global Step: 248670 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:12:00,604-Speed 131.12 samples/sec Loss 0.3150 Epoch: 24 Global Step: 248675 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:12:20,363-Speed 129.56 samples/sec Loss 0.2779 Epoch: 24 Global Step: 248680 Fp16 Grad Scale: 8192 Required: 1 hours Training: 2021-06-19 04:12:39,939-Speed 130.78 samples/sec Loss 0.2915 Epoch: 24 Global Step: 248685 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:12:59,671-Speed 129.74 samples/sec Loss 0.2633 Epoch: 24 Global Step: 248690 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:13:19,453-Speed 129.42 samples/sec Loss 0.2692 Epoch: 24 Global Step: 248695 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:13:39,294-Speed 129.03 samples/sec Loss 0.2963 Epoch: 24 Global Step: 248700 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:13:58,960-Speed 130.18 samples/sec Loss 0.2718 Epoch: 24 Global Step: 248705 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:14:18,751-Speed 129.36 samples/sec Loss 0.2929 Epoch: 24 Global Step: 248710 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:14:38,559-Speed 129.25 samples/sec Loss 0.2779 Epoch: 24 Global Step: 248715 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:14:58,150-Speed 130.68 samples/sec Loss 0.3375 Epoch: 24 Global Step: 248720 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:15:17,915-Speed 129.53 samples/sec Loss 0.2922 Epoch: 24 Global Step: 248725 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:15:37,534-Speed 130.49 samples/sec Loss 0.3246 Epoch: 24 Global Step: 248730 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:15:57,283-Speed 129.63 samples/sec Loss 0.2082 Epoch: 24 Global Step: 248735 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:16:17,077-Speed 129.34 samples/sec Loss 0.3188 Epoch: 24 Global Step: 248740 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:16:36,605-Speed 131.10 samples/sec Loss 0.2771 Epoch: 24 Global Step: 248745 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:16:56,047-Speed 131.68 samples/sec Loss 0.2895 Epoch: 24 Global Step: 248750 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:17:15,348-Speed 132.64 samples/sec Loss 0.3041 Epoch: 24 Global Step: 248755 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:17:34,955-Speed 130.57 samples/sec Loss 0.3055 Epoch: 24 Global Step: 248760 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:17:54,926-Speed 128.20 samples/sec Loss 0.1908 Epoch: 24 Global Step: 248765 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:18:14,538-Speed 130.54 samples/sec Loss 0.2863 Epoch: 24 Global Step: 248770 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:18:34,221-Speed 130.07 samples/sec Loss 0.2550 Epoch: 24 Global Step: 248775 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:18:53,972-Speed 129.62 samples/sec Loss 0.3330 Epoch: 24 Global Step: 248780 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:19:13,833-Speed 128.90 samples/sec Loss 0.2597 Epoch: 24 Global Step: 248785 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:19:33,495-Speed 130.21 samples/sec Loss 0.3063 Epoch: 24 Global Step: 248790 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:19:53,405-Speed 128.58 samples/sec Loss 0.3034 Epoch: 24 Global Step: 248795 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:20:13,757-Speed 125.79 samples/sec Loss 0.3192 Epoch: 24 Global Step: 248800 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:20:33,522-Speed 129.53 samples/sec Loss 0.2201 Epoch: 24 Global Step: 248805 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:20:53,540-Speed 127.89 samples/sec Loss 0.2713 Epoch: 24 Global Step: 248810 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:21:13,466-Speed 128.48 samples/sec Loss 0.2560 Epoch: 24 Global Step: 248815 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:21:33,191-Speed 129.79 samples/sec Loss 0.3324 Epoch: 24 Global Step: 248820 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:21:53,038-Speed 129.00 samples/sec Loss 0.2772 Epoch: 24 Global Step: 248825 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:22:12,760-Speed 129.81 samples/sec Loss 0.3525 Epoch: 24 Global Step: 248830 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:22:32,516-Speed 129.59 samples/sec Loss 0.3301 Epoch: 24 Global Step: 248835 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:22:52,423-Speed 128.61 samples/sec Loss 0.2697 Epoch: 24 Global Step: 248840 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:23:12,299-Speed 128.80 samples/sec Loss 0.3072 Epoch: 24 Global Step: 248845 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:23:32,254-Speed 128.30 samples/sec Loss 0.2846 Epoch: 24 Global Step: 248850 Fp16 Grad Scale: 2048 Required: 0 hours Training: 2021-06-19 04:23:52,118-Speed 128.89 samples/sec Loss 0.3056 Epoch: 24 Global Step: 248855 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:24:12,015-Speed 128.67 samples/sec Loss 0.3829 Epoch: 24 Global Step: 248860 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:24:31,736-Speed 129.81 samples/sec Loss 0.2864 Epoch: 24 Global Step: 248865 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:24:51,599-Speed 128.89 samples/sec Loss 0.2856 Epoch: 24 Global Step: 248870 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:25:11,400-Speed 129.29 samples/sec Loss 0.3659 Epoch: 24 Global Step: 248875 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:25:31,083-Speed 130.07 samples/sec Loss 0.2506 Epoch: 24 Global Step: 248880 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:25:50,970-Speed 128.74 samples/sec Loss 0.2917 Epoch: 24 Global Step: 248885 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:26:10,913-Speed 128.37 samples/sec Loss 0.2914 Epoch: 24 Global Step: 248890 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:26:30,877-Speed 128.24 samples/sec Loss 0.3350 Epoch: 24 Global Step: 248895 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:26:50,703-Speed 129.13 samples/sec Loss 0.3342 Epoch: 24 Global Step: 248900 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:27:10,583-Speed 128.78 samples/sec Loss 0.3029 Epoch: 24 Global Step: 248905 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:27:30,303-Speed 129.82 samples/sec Loss 0.3420 Epoch: 24 Global Step: 248910 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:27:50,118-Speed 129.20 samples/sec Loss 0.2419 Epoch: 24 Global Step: 248915 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:28:10,346-Speed 126.56 samples/sec Loss 0.3034 Epoch: 24 Global Step: 248920 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:28:30,219-Speed 128.83 samples/sec Loss 0.2630 Epoch: 24 Global Step: 248925 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:28:50,040-Speed 129.16 samples/sec Loss 0.2586 Epoch: 24 Global Step: 248930 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:29:09,996-Speed 128.29 samples/sec Loss 0.3306 Epoch: 24 Global Step: 248935 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:29:29,917-Speed 128.51 samples/sec Loss 0.3585 Epoch: 24 Global Step: 248940 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:29:49,882-Speed 128.23 samples/sec Loss 0.3004 Epoch: 24 Global Step: 248945 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:30:09,590-Speed 129.90 samples/sec Loss 0.2643 Epoch: 24 Global Step: 248950 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:30:29,342-Speed 129.61 samples/sec Loss 0.3366 Epoch: 24 Global Step: 248955 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:30:49,206-Speed 128.89 samples/sec Loss 0.3220 Epoch: 24 Global Step: 248960 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:31:08,991-Speed 129.39 samples/sec Loss 0.3435 Epoch: 24 Global Step: 248965 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:31:28,790-Speed 129.31 samples/sec Loss 0.3363 Epoch: 24 Global Step: 248970 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:31:48,496-Speed 129.91 samples/sec Loss 0.2966 Epoch: 24 Global Step: 248975 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:32:08,530-Speed 127.79 samples/sec Loss 0.2648 Epoch: 24 Global Step: 248980 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:32:28,509-Speed 128.15 samples/sec Loss 0.2616 Epoch: 24 Global Step: 248985 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:32:48,644-Speed 127.15 samples/sec Loss 0.2342 Epoch: 24 Global Step: 248990 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:33:08,298-Speed 130.26 samples/sec Loss 0.3185 Epoch: 24 Global Step: 248995 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:33:28,004-Speed 129.91 samples/sec Loss 0.3042 Epoch: 24 Global Step: 249000 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:33:47,966-Speed 128.25 samples/sec Loss 0.3019 Epoch: 24 Global Step: 249005 Fp16 Grad Scale: 8192 Required: 0 hours Training: 2021-06-19 04:34:07,272-Speed 132.61 samples/sec Loss 0.2583 Epoch: 24 Global Step: 249010 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:34:27,166-Speed 128.69 samples/sec Loss 0.3030 Epoch: 24 Global Step: 249015 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:34:46,815-Speed 130.29 samples/sec Loss 0.3273 Epoch: 24 Global Step: 249020 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:35:06,557-Speed 129.68 samples/sec Loss 0.2937 Epoch: 24 Global Step: 249025 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:35:26,339-Speed 129.41 samples/sec Loss 0.3037 Epoch: 24 Global Step: 249030 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:35:46,681-Speed 125.85 samples/sec Loss 0.2559 Epoch: 24 Global Step: 249035 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:36:06,490-Speed 129.24 samples/sec Loss 0.3139 Epoch: 24 Global Step: 249040 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:36:26,213-Speed 129.80 samples/sec Loss 0.2930 Epoch: 24 Global Step: 249045 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:36:46,039-Speed 129.14 samples/sec Loss 0.2955 Epoch: 24 Global Step: 249050 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:37:05,697-Speed 130.23 samples/sec Loss 0.3033 Epoch: 24 Global Step: 249055 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:37:25,603-Speed 128.61 samples/sec Loss 0.2821 Epoch: 24 Global Step: 249060 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:37:45,394-Speed 129.36 samples/sec Loss 0.3576 Epoch: 24 Global Step: 249065 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:38:05,170-Speed 129.45 samples/sec Loss 0.3861 Epoch: 24 Global Step: 249070 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:38:24,994-Speed 129.15 samples/sec Loss 0.2990 Epoch: 24 Global Step: 249075 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:38:45,245-Speed 126.41 samples/sec Loss 0.3241 Epoch: 24 Global Step: 249080 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:39:05,046-Speed 129.30 samples/sec Loss 0.2862 Epoch: 24 Global Step: 249085 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:39:24,770-Speed 129.79 samples/sec Loss 0.3218 Epoch: 24 Global Step: 249090 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:39:44,526-Speed 129.59 samples/sec Loss 0.4055 Epoch: 24 Global Step: 249095 Fp16 Grad Scale: 4096 Required: 0 hours Training: 2021-06-19 04:40:04,127-Speed 130.61 samples/sec Loss 0.3049 Epoch: 24 Global Step: 249100 Fp16 Grad Scale: 4096 Required: -0 hours Training: 2021-06-19 04:40:40,082-Pytorch Model Saved in 'ms1mv3_arcface_r1024/backbone.pth'